Skip to content

Open source CapCut alternative.

TypeScript

feat(site): the Cutout tab keys a clip's background away [rebuild]

David Le1a4e7da9857e
diff --git a/docs/guides/cut/README.md b/docs/guides/cut/README.md
index 086ed3da..a6a0090b 100644
--- a/docs/guides/cut/README.md
+++ b/docs/guides/cut/README.md
@@ -32,6 +32,8 @@ The client probes the engine's health endpoint (dedicated port first, dev server
**Framing is per clip.** Every video clip meets its box — the whole frame, or the region it was given — through one geometry: fitted with margins or covering with a crop, zoomed in further, panned to choose what the crop keeps, turned, faded, cornered, masked, and shadowed. One function computes where the picture lands, the preview draws that rectangle, and the export graph scales and crops to the same numbers, so a reframed clip looks the same on every surface, freeze frames and headless renders included. On the stage the selected clip wears two outlines: a blue box for its layout, with grips to move, resize, and rotate it, and a grey dashed one standing where the source really lands, so the part a crop discards stays visible — its corners zoom the picture and its interior pans it. The shadow is cast by the shape the clip actually shows, mask included, which is what makes a circle-masked camera bubble throw a circular shadow.
+**A cutout keys the clip to a matte.** Every video clip can carry a cutout — the Inspector's Cutout tab: an alpha matte decides which of its pixels stay, ink can trace the kept silhouette, and a color or image can fill in behind it, with lower tracks showing through wherever nothing fills. Every mode resolves to that one matte in the clip's own pixels. A chroma key computes it live, from reference math in the effects kit that the GPU shader, the CPU pass, and the export all share. The AI modes bake it into a small grayscale video stored as a project asset: auto mattes the person with the on-device segmenter in seconds and for free, then a hosted tracker (credits, like all AI work) re-bakes it at quality and swaps the asset in; a custom selection goes straight to the tracker, seeded by a few words naming the subject or by the strokes painted on the stage — the instant feedback while painting comes from a small on-device tap-to-select model, and the stored words and seeds make every bake reproducible. A fingerprint on the matte records what it was baked from — the source and the selection, trims left out — so a retrim inside the baked range keeps the matte and its bill, while a changed selection re-bakes; a background sweep settles bakes against the doc through undo, redo, and deletion, and holds them while a brush session is still painting. For the ffmpeg export each cutout clip ships as a client-rendered pair of videos — the keyed picture's color and its alpha — baked by the same compositor that draws the preview, so the engine and the worker composite exactly what the tab showed. Headless surfaces consume stored mattes; a turn that runs with no browser writes the cutout onto the clip and the next editor session's sweep starts the bake it owes.
+
**Words as the picture.** A lyric video, a kinetic-typography passage, a quote sequence: the assistant places a whole run of lines in one call, carrying a named look. A look is the design of the whole run rather than of one line — the frame color, the cards it cycles, and the ensemble the run walks across its length: the faces it rotates through, the entrances, where successive lines land in the frame, and what each line does while it holds. That is what stops a hundred lines coming out identical and dead center, which is the failure a text video actually has. The looks live in one registry the tools and the skills both read, so adding a look teaches the assistant a new style with no other change. Words ride one of two carriers — a caption track, which holds per-word timings, or elements, which give each line its own card, color and motion. Both carriers play a line word by word, and what a word does is data like everything else: three poses — before its moment, at it, and after — and the seconds between them. An emphasis travels along a line that is fully up, because the pose after a word's moment is the pose before it; a build assembles the line as it is spoken, because the pose after is the pose it arrived at. That one shape covers a karaoke highlight, a word swelling on the beat, a sentence appearing a word at a time, and a faint line going solid word by word, and it is what makes adding another an entry in the catalog. One evaluator resolves each word to a color, an opacity and a pose, and the preview, the burn-in and the frame samplers all read that, so a word effect looks the same wherever it plays. Text the user already has (lyrics, a script) is timed by aligning it to a transcript of the audio: the recognizer supplies the clock and the user's wording is kept exactly, which is what makes it survive singing.
**Motion is data.** Every animation in the editor — an entrance, an exit, a loop, and the moves an element makes while it holds — is one format: a pose keyframed over a normalized window with cubic-bezier easing, plus a range selector that says how the motion is handed from one character to the next. That is the After Effects text-animator model, the one Lottie serializes, and picking it means a preset and a hand-keyed track are the same object at two levels of convenience: the named entrances, loops, and moves are slots on the element, evaluated internally, and anything no name covers is keyed directly on the element's own pose track — the track stays whatever the user keys by hand. The library is a JSON asset in the effects kit rather than code, and the menus, tool enums, prompts, and the frame sampler's crop budget are all read from it, so adding an animation is adding an entry. One evaluator samples it, which is what keeps the preview, the export, and the headless worker showing the same motion. A slot may also carry a preset of its own rather than naming one from the library, in the same shape a library entry has — the evaluator cannot tell the two apart, so an animation composed inside a project plays wherever a shipped one does.
diff --git a/docs/guides/cut/local-compute.md b/docs/guides/cut/local-compute.md
index 139f478f..ca854cc1 100644
--- a/docs/guides/cut/local-compute.md
+++ b/docs/guides/cut/local-compute.md
@@ -79,6 +79,7 @@ In place is what makes the plain version of this work. Someone with a folder of
| Thumbnails, waveforms, media probing | the browser, always | it decodes the media itself |
| Converting media to MP4 | the machine holding the bytes: the Mac, the worker, or the tab — and the Mac for a browser project the tab can't decode | see Converting footage above |
| Image, video, and voice generation | hosted, always | no local counterpart |
+| Cutout mattes | the quick person matte in the tab, on-device and free; the quality and tracked bakes hosted (credits), driven by the tab | the tab decodes the clip and owns the session; the engine and the worker only consume the stored matte asset |
| The assistant's Gemini models | hosted, always | credits and the user's session |
| The assistant's Claude and Codex providers | the Mac, always | the user's own CLI logins |
diff --git a/site/.env.example b/site/.env.example
index 645da4a3..a11b8a11 100644
--- a/site/.env.example
+++ b/site/.env.example
@@ -15,6 +15,9 @@ OPENAI_API_KEY="replace-with-openai-api-key"
# Optional hosted inference providers.
ELEVENLABS_API_KEY="replace-with-elevenlabs-api-key"
+# Cut's background-removal matte segmenter.
+FAL_KEY="replace-with-fal-api-key"
+
# Required for Gemini / Google Cloud inference through Vertex AI. The adapter
# uses the global endpoint and derives the project from project_id.
GOOGLE_APPLICATION_CREDENTIALS_JSON='{"type":"service_account",...}'
diff --git a/site/packages/effects-kit/src/guidedFilter.test.ts b/site/packages/effects-kit/src/guidedFilter.test.ts
new file mode 100644
index 00000000..2c62eb0b
--- /dev/null
+++ b/site/packages/effects-kit/src/guidedFilter.test.ts
@@ -0,0 +1,62 @@
+import { describe, expect, test } from "bun:test";
+import { guidedFilterRefine, refineMatteAgainstFrame } from "./guidedFilter";
+
+const W = 32;
+const H = 16;
+
+/** A vertical step: dark left half, bright right half. */
+function stepGuide(): Float32Array {
+ const g = new Float32Array(W * H);
+ for (let y = 0; y < H; y++) for (let x = 0; x < W; x++) g[y * W + x] = x < W / 2 ? 0.1 : 0.9;
+ return g;
+}
+
+/** A matte ramping softly across the same edge. */
+function softMatte(): Float32Array {
+ const m = new Float32Array(W * H);
+ for (let y = 0; y < H; y++)
+ for (let x = 0; x < W; x++) m[y * W + x] = Math.min(1, Math.max(0, (x - W / 2 + 4) / 8));
+ return m;
+}
+
+describe("guided filter", () => {
+ test("a soft matte edge snaps toward the guide's step", () => {
+ const m = softMatte();
+ const before = { dark: m[8 * W + 13], bright: m[8 * W + 18] };
+ guidedFilterRefine(m, stepGuide(), W, H, 4, 1e-4);
+ // On the guide's dark side coverage drops; on the bright side it rises.
+ expect(m[8 * W + 13]).toBeLessThan(before.dark);
+ expect(m[8 * W + 18]).toBeGreaterThan(before.bright);
+ });
+
+ test("solid regions stay solid", () => {
+ const m = new Float32Array(W * H).fill(1);
+ guidedFilterRefine(m, stepGuide(), W, H, 4, 1e-3);
+ for (let i = 0; i < m.length; i++) expect(m[i]).toBeGreaterThan(0.95);
+ });
+
+ test("the RGBA wrapper writes gray coverage with full alpha", () => {
+ const matte = new Uint8ClampedArray(W * H * 4);
+ const framePx = new Uint8ClampedArray(W * H * 4);
+ const g = stepGuide();
+ const m = softMatte();
+ for (let i = 0; i < W * H; i++) {
+ const mv = Math.round(m[i] * 255);
+ matte[i * 4] = mv;
+ matte[i * 4 + 1] = mv;
+ matte[i * 4 + 2] = mv;
+ matte[i * 4 + 3] = 255;
+ const gv = Math.round(g[i] * 255);
+ framePx[i * 4] = gv;
+ framePx[i * 4 + 1] = gv;
+ framePx[i * 4 + 2] = gv;
+ framePx[i * 4 + 3] = 255;
+ }
+ refineMatteAgainstFrame(matte, framePx, W, H, 4, 1e-4);
+ for (let i = 0; i < W * H; i++) {
+ expect(matte[i * 4]).toBe(matte[i * 4 + 1]);
+ expect(matte[i * 4 + 1]).toBe(matte[i * 4 + 2]);
+ expect(matte[i * 4 + 3]).toBe(255);
+ }
+ });
+});
diff --git a/site/packages/effects-kit/src/guidedFilter.ts b/site/packages/effects-kit/src/guidedFilter.ts
new file mode 100644
index 00000000..6badc8bb
--- /dev/null
+++ b/site/packages/effects-kit/src/guidedFilter.ts
@@ -0,0 +1,118 @@
+/**
+ * Guided filter: sharpen a soft matte against the picture it belongs to.
+ *
+ * A tracked mask comes back blocky — model resolution, video compression —
+ * while the real edge lives in the source pixels. The guided filter slides
+ * the matte toward that edge: inside a window, the output is a local linear
+ * function of the guide (the source luma), so the matte snaps to picture
+ * edges and stays smooth across flat regions. Pure math on Float32 planes;
+ * every renderer that refines a matte runs this one implementation.
+ */
+
+/** Scratch planes, reused across frames: the refine runs per baked frame,
+ * synchronously, and reallocating a dozen frame-sized planes each time is
+ * pure GC pressure. Every plane is fully overwritten before it is read, so a
+ * reused one never leaks the previous frame. */
+function planeCache() {
+ let size = -1;
+ let list: Float32Array[] = [];
+ return (n: number, count: number): Float32Array[] => {
+ if (size !== n || list.length < count) {
+ size = n;
+ list = Array.from({ length: count }, () => new Float32Array(n));
+ }
+ return list;
+ };
+}
+const refinePlanes = planeCache();
+const framePlanes = planeCache();
+
+/** Box blur `src` into `out` (radius `r`, running sums per axis), through the
+ * caller's `tmp` plane. */
+function boxBlur(src: Float32Array, w: number, h: number, r: number, tmp: Float32Array, out: Float32Array): void {
+ // Horizontal pass.
+ for (let y = 0; y < h; y++) {
+ const row = y * w;
+ let sum = 0;
+ for (let x = -r; x <= r; x++) sum += src[row + Math.min(w - 1, Math.max(0, x))];
+ for (let x = 0; x < w; x++) {
+ tmp[row + x] = sum / (r * 2 + 1);
+ sum += src[row + Math.min(w - 1, x + r + 1)] - src[row + Math.max(0, x - r)];
+ }
+ }
+ // Vertical pass.
+ for (let x = 0; x < w; x++) {
+ let sum = 0;
+ for (let y = -r; y <= r; y++) sum += tmp[Math.min(h - 1, Math.max(0, y)) * w + x];
+ for (let y = 0; y < h; y++) {
+ out[y * w + x] = sum / (r * 2 + 1);
+ sum += tmp[Math.min(h - 1, y + r + 1) * w + x] - tmp[Math.max(0, y - r) * w + x];
+ }
+ }
+}
+
+/**
+ * Refine `matte` (0..1) against `guide` (0..1 luma, same dimensions), in
+ * place. `radius` is the window in pixels at the matte's own resolution;
+ * `eps` is the edge threshold — smaller hugs edges harder.
+ */
+export function guidedFilterRefine(
+ matte: Float32Array,
+ guide: Float32Array,
+ w: number,
+ h: number,
+ radius = 8,
+ eps = 1e-3
+): void {
+ const n = w * h;
+ const [tmp, meanI, meanP, cross, sq, meanIP, meanII, a, b, meanA, meanB] = refinePlanes(n, 11);
+ boxBlur(guide, w, h, radius, tmp, meanI);
+ boxBlur(matte, w, h, radius, tmp, meanP);
+ for (let i = 0; i < n; i++) {
+ cross[i] = guide[i] * matte[i];
+ sq[i] = guide[i] * guide[i];
+ }
+ boxBlur(cross, w, h, radius, tmp, meanIP);
+ boxBlur(sq, w, h, radius, tmp, meanII);
+ for (let i = 0; i < n; i++) {
+ const varI = meanII[i] - meanI[i] * meanI[i];
+ const covIP = meanIP[i] - meanI[i] * meanP[i];
+ a[i] = covIP / (varI + eps);
+ b[i] = meanP[i] - a[i] * meanI[i];
+ }
+ boxBlur(a, w, h, radius, tmp, meanA);
+ boxBlur(b, w, h, radius, tmp, meanB);
+ for (let i = 0; i < n; i++) {
+ matte[i] = Math.min(1, Math.max(0, meanA[i] * guide[i] + meanB[i]));
+ }
+}
+
+/**
+ * Refine a matte held as RGBA pixel buffers: the matte's red channel against
+ * the picture's BT.709 luma. Writes the refined coverage back into the
+ * matte's RGB (white × coverage) and alpha.
+ */
+export function refineMatteAgainstFrame(
+ mattePx: Uint8ClampedArray,
+ framePx: Uint8ClampedArray,
+ w: number,
+ h: number,
+ radius = 8,
+ eps = 1e-3
+): void {
+ const n = w * h;
+ const [matte, guide] = framePlanes(n, 2);
+ for (let i = 0; i < n; i++) {
+ matte[i] = mattePx[i * 4] / 255;
+ guide[i] =
+ (0.2126 * framePx[i * 4] + 0.7152 * framePx[i * 4 + 1] + 0.0722 * framePx[i * 4 + 2]) / 255;
+ }
+ guidedFilterRefine(matte, guide, w, h, radius, eps);
+ for (let i = 0; i < n; i++) {
+ const v = Math.round(matte[i] * 255);
+ mattePx[i * 4] = v;
+ mattePx[i * 4 + 1] = v;
+ mattePx[i * 4 + 2] = v;
+ mattePx[i * 4 + 3] = 255;
+ }
+}
diff --git a/site/packages/effects-kit/src/index.ts b/site/packages/effects-kit/src/index.ts
index 57b486b4..38911f51 100644
--- a/site/packages/effects-kit/src/index.ts
+++ b/site/packages/effects-kit/src/index.ts
@@ -9,6 +9,9 @@ export * from "./mask";
export * from "./render";
export * from "./registry";
export * from "./cutout";
+export * from "./guidedFilter";
+export * from "./removal";
+export * from "./strokeInk";
export * from "./lottie";
export * from "./looks";
export * from "./colorGrade";
diff --git a/site/packages/effects-kit/src/removal.test.ts b/site/packages/effects-kit/src/removal.test.ts
new file mode 100644
index 00000000..38726d89
--- /dev/null
+++ b/site/packages/effects-kit/src/removal.test.ts
@@ -0,0 +1,142 @@
+import { describe, expect, test } from "bun:test";
+import {
+ CHROMA_DEFAULT_INTENSITY,
+ CHROMA_DEFAULT_SOFTNESS,
+ CHROMA_DEFAULT_SPILL,
+ CHROMA_NEAR_MAX,
+ chromaAlphaInto,
+ chromaAlphaOf,
+ chromaParams,
+ removalActive,
+ removalFingerprint,
+ removalNeedsBake,
+ type ClipRemoval,
+} from "./removal";
+
+/** One RGBA pixel buffer of solid color. */
+function frame(px: [number, number, number][], alpha = 255): Uint8ClampedArray {
+ const data = new Uint8ClampedArray(px.length * 4);
+ px.forEach(([r, g, b], i) => {
+ data[i * 4] = r;
+ data[i * 4 + 1] = g;
+ data[i * 4 + 2] = b;
+ data[i * 4 + 3] = alpha;
+ });
+ return data;
+}
+
+describe("chroma params", () => {
+ test("defaults fill and the soft band sits past the tolerance", () => {
+ const p = chromaParams({ color: "#00ff00" });
+ expect(p.near).toBeCloseTo(CHROMA_DEFAULT_INTENSITY * CHROMA_NEAR_MAX, 5);
+ expect(p.far).toBeGreaterThan(p.near);
+ expect(p.spill).toBe(CHROMA_DEFAULT_SPILL);
+ expect(Math.hypot(p.dirCb, p.dirCr)).toBeCloseTo(1, 5);
+ });
+
+ test("intensity widens the removed core and softness widens the band", () => {
+ const tight = chromaParams({ color: "#00ff00", intensity: 0.2, softness: CHROMA_DEFAULT_SOFTNESS });
+ const wide = chromaParams({ color: "#00ff00", intensity: 0.9, softness: CHROMA_DEFAULT_SOFTNESS });
+ expect(wide.near).toBeGreaterThan(tight.near);
+ const hard = chromaParams({ color: "#00ff00", softness: 0.05 });
+ const soft = chromaParams({ color: "#00ff00", softness: 1 });
+ expect(soft.far - soft.near).toBeGreaterThan(hard.far - hard.near);
+ });
+
+ test("alpha ramps monotonically from removed to kept", () => {
+ const p = chromaParams({ color: "#00ff00" });
+ expect(chromaAlphaOf(0, p)).toBe(0);
+ expect(chromaAlphaOf(p.near, p)).toBe(0);
+ expect(chromaAlphaOf(p.far, p)).toBe(1);
+ expect(chromaAlphaOf(1, p)).toBe(1);
+ const mid = chromaAlphaOf((p.near + p.far) / 2, p);
+ expect(mid).toBeGreaterThan(0);
+ expect(mid).toBeLessThan(1);
+ });
+});
+
+describe("chroma keying", () => {
+ test("the key color goes transparent and distant colors stay", () => {
+ const px = frame([
+ [0, 255, 0], // the key itself
+ [255, 0, 0], // far from the key
+ [255, 255, 255], // neutral
+ ]);
+ chromaAlphaInto(px, { color: "#00ff00" });
+ expect(px[3]).toBe(0);
+ expect(px[7]).toBe(255);
+ expect(px[11]).toBe(255);
+ });
+
+ test("spill suppression pulls the key's tint off kept fringe", () => {
+ // A green-fringed pixel just past the key's soft band keeps its alpha
+ // while its green excess shrinks.
+ const px = frame([[120, 230, 90]]);
+ const before = px[1] - (px[0] + px[2]) / 2;
+ chromaAlphaInto(px, { color: "#00ff00", intensity: 0.6, spill: 1 });
+ const after = px[1] - (px[0] + px[2]) / 2;
+ expect(px[3]).toBeGreaterThan(200);
+ expect(after).toBeLessThan(before);
+ });
+
+ test("spill 0 leaves kept colors untouched", () => {
+ const px = frame([[150, 205, 130]]);
+ const want = [...px];
+ chromaAlphaInto(px, { color: "#00ff00", intensity: 0.1, softness: 0.1, spill: 0 });
+ expect(px[0]).toBe(want[0]);
+ expect(px[1]).toBe(want[1]);
+ expect(px[2]).toBe(want[2]);
+ });
+});
+
+describe("removal model", () => {
+ const seeds = { prompts: [{ t: 1, points: [{ x: 0.5, y: 0.5, label: 1 as const }] }] };
+
+ test("fingerprint tracks the source and the selection", () => {
+ expect(removalFingerprint("a", { mode: "auto" })).toBe(removalFingerprint("a", { mode: "auto" }));
+ expect(removalFingerprint("a", { mode: "auto" })).not.toBe(
+ removalFingerprint("b", { mode: "auto" })
+ );
+ expect(removalFingerprint("a", { mode: "auto" })).not.toBe(
+ removalFingerprint("a", { mode: "custom", seeds })
+ );
+ expect(removalFingerprint("a", { mode: "custom", seeds })).not.toBe(
+ removalFingerprint("a", { mode: "custom", seeds: { prompts: [] } })
+ );
+ });
+
+ test("auto mode's fingerprint ignores the seeds and the subject", () => {
+ expect(removalFingerprint("a", { mode: "auto", seeds })).toBe(
+ removalFingerprint("a", { mode: "auto" })
+ );
+ expect(removalFingerprint("a", { mode: "auto", subject: "the dog" })).toBe(
+ removalFingerprint("a", { mode: "auto" })
+ );
+ });
+
+ test("custom mode's fingerprint tracks the described subject", () => {
+ expect(removalFingerprint("a", { mode: "custom", subject: "the dog" })).not.toBe(
+ removalFingerprint("a", { mode: "custom", subject: "the cat" })
+ );
+ expect(removalFingerprint("a", { mode: "custom", subject: "the dog" })).toBe(
+ removalFingerprint("a", { mode: "custom", subject: "the dog" })
+ );
+ });
+
+ test("active and needs-bake read the mode", () => {
+ expect(removalActive(undefined)).toBe(false);
+ expect(removalActive({ mode: "auto" })).toBe(true);
+ expect(removalActive({ mode: "chroma" })).toBe(false);
+ expect(removalActive({ mode: "chroma", chroma: { color: "#00ff00" } })).toBe(true);
+ // A bake is owed only once it was explicitly started.
+ expect(removalNeedsBake({ mode: "auto" })).toBe(false);
+ expect(removalNeedsBake({ mode: "auto", requested: true })).toBe(true);
+ const baked: ClipRemoval = {
+ mode: "auto",
+ requested: true,
+ matte: { assetId: "m", fingerprint: "f", quality: "local", in: 0 },
+ };
+ expect(removalNeedsBake(baked)).toBe(false);
+ expect(removalNeedsBake({ mode: "chroma" })).toBe(false);
+ });
+});
diff --git a/site/packages/effects-kit/src/removal.ts b/site/packages/effects-kit/src/removal.ts
new file mode 100644
index 00000000..0805d854
--- /dev/null
+++ b/site/packages/effects-kit/src/removal.ts
@@ -0,0 +1,273 @@
+/**
+ * Background removal: the per-clip cutout model and the reference math every
+ * renderer shares.
+ *
+ * A removal turns a clip's picture into a keyed layer: an alpha matte decides
+ * what stays (a chroma key computed live, or an AI matte baked to a grayscale
+ * video the clip references), a stroke draws ink around the silhouette, and a
+ * backdrop fills in behind it. The model lives here so the doc, the panels,
+ * the chat tools, and all three renderers read one shape — and the chroma
+ * math lives beside it so the GPU shader, the CPU pass, and the export's
+ * matte videos resolve a key the same way to the pixel.
+ */
+
+export type RemovalMode = "auto" | "custom" | "chroma";
+
+/** A chroma key: the picked backdrop color plus how it reaches. */
+export interface ChromaKey {
+ /** Picked key color, "#rrggbb". */
+ color: string;
+ /** Tolerance 0..1 — how far from the key a color still keys out. */
+ intensity?: number;
+ /** Edge rolloff 0..1 — how wide the soft band past the tolerance runs. */
+ softness?: number;
+ /** Spill suppression 0..1 — how hard key-colored fringe on the kept
+ * pixels is pulled back toward neutral. */
+ spill?: number;
+}
+
+export const CHROMA_DEFAULT_INTENSITY = 0.4;
+export const CHROMA_DEFAULT_SOFTNESS = 0.25;
+export const CHROMA_DEFAULT_SPILL = 0.5;
+
+export const STROKE_STYLES = ["glow", "hand", "cut", "solid", "offset", "dotted"] as const;
+export type StrokeStyleId = (typeof STROKE_STYLES)[number];
+
+export const STROKE_STYLE_LABELS: Record<StrokeStyleId, string> = {
+ glow: "Glow",
+ hand: "Hand drawn",
+ cut: "Straight cut",
+ solid: "Solid",
+ offset: "Offset",
+ dotted: "Dotted",
+};
+
+/** Stroke lengths are design px at the 1080 short side, like masks. */
+export const STROKE_DEFAULT_WIDTH = 12;
+export const STROKE_WIDTH_MAX = 60;
+export const STROKE_OFFSET_MAX = 120;
+
+export interface RemovalStroke {
+ style: StrokeStyleId;
+ /** Ink color, "#rrggbb". */
+ color: string;
+ /** Ink thickness, design px; absent = STROKE_DEFAULT_WIDTH. */
+ width?: number;
+ /** Silhouette displacement for the "offset" style, design px per axis. */
+ offsetX?: number;
+ offsetY?: number;
+}
+
+export interface RemovalBackdrop {
+ kind: "none" | "color" | "image";
+ color?: string;
+ /** Image fill: a project asset's id. */
+ assetId?: string;
+}
+
+/** Custom-mode selection, persisted so a bake is reproducible and undoable. */
+export interface RemovalSeeds {
+ /** Source-time seconds → segmenter point prompts in frame fractions;
+ * label 1 keeps, 0 removes. */
+ prompts: { t: number; points: { x: number; y: number; label: 0 | 1 }[] }[];
+ /** Raw brush paint per edited source second: small grayscale PNG data URLs,
+ * white marks, `add` unions into the mask and `erase` subtracts. */
+ paint?: { t: number; add?: string; erase?: string }[];
+}
+
+export interface ClipRemoval {
+ mode: RemovalMode;
+ /** The key, for mode "chroma". */
+ chroma?: ChromaKey;
+ /** Custom mode by description: what to keep, in a few words ("the dog").
+ * The concept tracker mattes every match; painted seeds take over when the
+ * user brushes instead. */
+ subject?: string;
+ /** The baked matte, for modes "auto" and "custom": a grayscale video asset
+ * covering the clip's trimmed source range, white keeps the pixel. `in` is
+ * the source second the matte's own zero maps to, so a later retrim keeps
+ * reading the right frames; the fingerprint records what the matte was
+ * baked from, so edits re-bake. */
+ matte?: { assetId: string; fingerprint: string; quality: "local" | "hq"; in: number };
+ /** Custom-mode selection. */
+ seeds?: RemovalSeeds;
+ /** The bake was explicitly started — the panel's Start/Apply button or the
+ * chat tool. AI modes bake nothing until this is set, so picking a mode by
+ * itself never spends compute. */
+ requested?: boolean;
+ stroke?: RemovalStroke;
+ backdrop?: RemovalBackdrop;
+}
+
+/** Whether the removal changes the picture at all. */
+export function removalActive(r?: ClipRemoval): boolean {
+ if (!r) return false;
+ if (r.mode === "chroma") return !!r.chroma?.color;
+ return true;
+}
+
+/** An AI-mode removal whose started bake has no matte yet — still owed. */
+export function removalNeedsBake(r?: ClipRemoval): boolean {
+ return !!r && (r.mode === "auto" || r.mode === "custom") && !!r.requested && !r.matte;
+}
+
+/** What a baked matte was computed from: the source and the selection.
+ * Trims stay out of it — a matte covering a wider source range than the clip
+ * plays is still the right matte, so a retrim inside its coverage never
+ * re-bakes (and never re-bills the quality pass). A mismatch means the
+ * source or the seeds changed; the stale matte keeps drawing until the new
+ * one lands. */
+export function removalFingerprint(
+ assetId: string,
+ r: Pick<ClipRemoval, "mode" | "seeds" | "subject">
+): string {
+ const body = JSON.stringify([
+ assetId,
+ r.mode,
+ r.mode === "custom" ? (r.seeds ?? null) : null,
+ r.mode === "custom" ? (r.subject ?? null) : null,
+ ]);
+ // djb2 — short, stable, and identical everywhere the fingerprint is read.
+ let hash = 5381;
+ for (let i = 0; i < body.length; i++) hash = ((hash << 5) + hash + body.charCodeAt(i)) >>> 0;
+ return hash.toString(36);
+}
+
+// ---------------------------------------------------------------------------
+// Chroma reference math. The WebGL shader mirrors these formulas with the
+// same constants, so the GPU pass, this CPU pass, and the export mattes all
+// key the same frame to the same alpha.
+
+/** Chroma-plane distance that keys fully out at intensity 1. */
+export const CHROMA_NEAR_MAX = 0.38;
+/** Width of the soft band past the tolerance at softness 1. */
+export const CHROMA_SOFT_MAX = 0.3;
+/** How far past the soft band spill suppression keeps reaching. */
+export const CHROMA_SPILL_REACH = 2;
+
+export interface ChromaParams {
+ /** Key chroma in BT.709 Cb/Cr, each -0.5..0.5. */
+ keyCb: number;
+ keyCr: number;
+ /** Unit direction of the key's chroma, for the spill projection. */
+ dirCb: number;
+ dirCr: number;
+ /** Distances in chroma-plane units: fully removed inside `near`, fully
+ * kept past `far`. */
+ near: number;
+ far: number;
+ spill: number;
+}
+
+function hexChannel(hex: string, i: number): number {
+ const v = parseInt(hex.slice(1 + i * 2, 3 + i * 2), 16);
+ return Number.isFinite(v) ? v / 255 : 0;
+}
+
+/** BT.709 chroma of an r/g/b in 0..1. */
+function chromaOf(r: number, g: number, b: number): { cb: number; cr: number } {
+ const y = 0.2126 * r + 0.7152 * g + 0.0722 * b;
+ return { cb: (b - y) / 1.8556, cr: (r - y) / 1.5748 };
+}
+
+/** Resolve a key's sliders into the numbers the passes run on. */
+export function chromaParams(key: ChromaKey): ChromaParams {
+ const r = hexChannel(key.color, 0);
+ const g = hexChannel(key.color, 1);
+ const b = hexChannel(key.color, 2);
+ const { cb, cr } = chromaOf(r, g, b);
+ const len = Math.hypot(cb, cr);
+ const intensity = key.intensity ?? CHROMA_DEFAULT_INTENSITY;
+ const softness = key.softness ?? CHROMA_DEFAULT_SOFTNESS;
+ const near = Math.max(0.02, intensity * CHROMA_NEAR_MAX);
+ return {
+ keyCb: cb,
+ keyCr: cr,
+ dirCb: len > 0.001 ? cb / len : 0,
+ dirCr: len > 0.001 ? cr / len : 1,
+ near,
+ far: near + Math.max(0.01, softness * CHROMA_SOFT_MAX),
+ spill: key.spill ?? CHROMA_DEFAULT_SPILL,
+ };
+}
+
+const smooth = (t: number) => t * t * (3 - 2 * t);
+
+/** The alpha a pixel keeps under the key, 0..1, from its chroma distance. */
+export function chromaAlphaOf(d: number, p: ChromaParams): number {
+ const t = Math.min(1, Math.max(0, (d - p.near) / (p.far - p.near)));
+ return smooth(t);
+}
+
+/**
+ * Key an RGBA buffer in place: each pixel's alpha multiplies by how far its
+ * chroma sits from the key, and kept pixels near the key lose their fringe —
+ * the key's chroma direction is projected out of them, scaled by `spill` and
+ * by how close they sit. This is the reference pass; the shader repeats it.
+ */
+/** The baked-matte contract every producer and reader shares: encoded matte
+ * rate (coverage moves little in 1/15s, and every consumer's fps filter or
+ * nearest-frame read duplicates up to the output rate), the frame's short
+ * side (consumers scale it to the picture), and the encode rate. */
+export const MATTE_FPS = 15;
+export const MATTE_SHORT = 480;
+export const MATTE_BITRATE = 1_000_000;
+
+/** Turn a decoded matte frame's luma into alpha coverage, in place: alpha
+ * becomes the red channel's luma and the ink turns white, so the result
+ * draws as coverage (`destination-in`) or as a lit silhouette alike. Matte
+ * videos carry coverage as luma — H.264 has no alpha plane. */
+export function matteLumaToAlpha(px: Uint8ClampedArray): void {
+ for (let i = 0; i < px.length; i += 4) {
+ px[i + 3] = px[i];
+ px[i] = 255;
+ px[i + 1] = 255;
+ px[i + 2] = 255;
+ }
+}
+
+/** Flatten a tracker's mask frame to encodable gray, in place: coverage is
+ * whatever the tracker painted over black — some render white, some tint per
+ * object — so any lit channel reads as matte. */
+export function coverageToGray(px: Uint8ClampedArray): void {
+ for (let i = 0; i < px.length; i += 4) {
+ const v = Math.max(px[i], px[i + 1], px[i + 2]);
+ px[i] = v;
+ px[i + 1] = v;
+ px[i + 2] = v;
+ px[i + 3] = 255;
+ }
+}
+
+export function chromaAlphaInto(px: Uint8ClampedArray, key: ChromaKey): void {
+ const p = chromaParams(key);
+ const spillFar = p.near + (p.far - p.near) * CHROMA_SPILL_REACH;
+ for (let i = 0; i < px.length; i += 4) {
+ const r = px[i] / 255;
+ const g = px[i + 1] / 255;
+ const b = px[i + 2] / 255;
+ const y = 0.2126 * r + 0.7152 * g + 0.0722 * b;
+ const cb = (b - y) / 1.8556;
+ const cr = (r - y) / 1.5748;
+ const d = Math.hypot(cb - p.keyCb, cr - p.keyCr);
+ const a = chromaAlphaOf(d, p);
+ if (a < 1 || d < spillFar) {
+ px[i + 3] = Math.round(px[i + 3] * a);
+ if (a > 0 && p.spill > 0 && d < spillFar) {
+ const proj = cb * p.dirCb + cr * p.dirCr;
+ if (proj > 0) {
+ const reach = 1 - Math.min(1, Math.max(0, (d - p.near) / (spillFar - p.near)));
+ const cut = proj * p.spill * smooth(reach);
+ const ncb = cb - p.dirCb * cut;
+ const ncr = cr - p.dirCr * cut;
+ const nr = y + 1.5748 * ncr;
+ const nb = y + 1.8556 * ncb;
+ const ng = (y - 0.2126 * nr - 0.0722 * nb) / 0.7152;
+ px[i] = Math.max(0, Math.min(255, Math.round(nr * 255)));
+ px[i + 1] = Math.max(0, Math.min(255, Math.round(ng * 255)));
+ px[i + 2] = Math.max(0, Math.min(255, Math.round(nb * 255)));
+ }
+ }
+ }
+ }
+}
diff --git a/site/packages/effects-kit/src/strokeInk.test.ts b/site/packages/effects-kit/src/strokeInk.test.ts
new file mode 100644
index 00000000..89882537
--- /dev/null
+++ b/site/packages/effects-kit/src/strokeInk.test.ts
@@ -0,0 +1,45 @@
+import { describe, expect, test } from "bun:test";
+import { traceMatteContours } from "./strokeInk";
+
+/** An alpha mask as ImageData, solid inside the listed boxes. */
+function mask(w: number, h: number, boxes: { x0: number; y0: number; x1: number; y1: number }[]) {
+ const data = new Uint8ClampedArray(w * h * 4);
+ for (const b of boxes)
+ for (let y = b.y0; y < b.y1; y++)
+ for (let x = b.x0; x < b.x1; x++) data[(y * w + x) * 4 + 3] = 255;
+ return { width: w, height: h, data, colorSpace: "srgb" } as ImageData;
+}
+
+describe("matte contours", () => {
+ test("a solid block traces one ring along its boundary", () => {
+ const loops = traceMatteContours(mask(32, 32, [{ x0: 8, y0: 8, x1: 24, y1: 24 }]));
+ expect(loops.length).toBe(1);
+ const ring = loops[0];
+ // Every traced point sits on the block's boundary, none inside.
+ for (const p of ring) {
+ expect(p.x).toBeGreaterThanOrEqual(8);
+ expect(p.x).toBeLessThanOrEqual(23);
+ expect(p.y).toBeGreaterThanOrEqual(8);
+ expect(p.y).toBeLessThanOrEqual(23);
+ const onEdge = p.x === 8 || p.x === 23 || p.y === 8 || p.y === 23;
+ expect(onEdge).toBe(true);
+ }
+ // A 16×16 block's boundary ring is its perimeter minus the shared corners.
+ expect(ring.length).toBe(60);
+ });
+
+ test("separate blobs come back as separate loops, largest first", () => {
+ const loops = traceMatteContours(
+ mask(48, 24, [
+ { x0: 2, y0: 2, x1: 20, y1: 20 },
+ { x0: 30, y0: 8, x1: 40, y1: 16 },
+ ])
+ );
+ expect(loops.length).toBe(2);
+ expect(loops[0].length).toBeGreaterThan(loops[1].length);
+ });
+
+ test("specks are dropped", () => {
+ expect(traceMatteContours(mask(16, 16, [{ x0: 5, y0: 5, x1: 6, y1: 6 }]))).toEqual([]);
+ });
+});
diff --git a/site/packages/effects-kit/src/strokeInk.ts b/site/packages/effects-kit/src/strokeInk.ts
new file mode 100644
index 00000000..09a43559
--- /dev/null
+++ b/site/packages/effects-kit/src/strokeInk.ts
@@ -0,0 +1,306 @@
+/**
+ * Stroke ink: the outline a removal draws around its subject.
+ *
+ * One painter turns a silhouette (any image whose alpha marks the subject)
+ * into white ink on a transparent canvas; the host colors the ink and lays it
+ * behind the keyed layer. Styles come in two families: the dilate styles
+ * (glow, solid, offset) stamp or blur the silhouette itself, and the contour
+ * styles (dotted, straight cut, hand drawn) walk the silhouette's traced
+ * boundary. All of it is plain 2D canvas, so the preview, the in-tab export,
+ * and the headless renderers draw the same ink.
+ */
+
+import { kitCanvas } from "./surface";
+import type { RemovalStroke } from "./removal";
+import { STROKE_DEFAULT_WIDTH } from "./removal";
+
+type Canvas2D = CanvasRenderingContext2D | OffscreenCanvasRenderingContext2D;
+
+/** Contours trace on a copy about this tall — boundary shape at outline
+ * widths survives the downscale, and the per-frame scan stays cheap. */
+const TRACE_SHORT = 288;
+
+/** Round stamps that make a dilated edge read as smooth. */
+const DILATE_STAMPS = 24;
+
+export interface ContourPoint {
+ x: number;
+ y: number;
+}
+
+/**
+ * Boundary loops of the mask's solid pixels (alpha > 127), each an ordered
+ * ring of pixel coordinates, outermost first. Moore-neighbor tracing; specks
+ * under a few pixels are dropped.
+ */
+let planeSize = -1;
+let solidPlane = new Uint8Array(0);
+let tracedPlane = new Uint8Array(0);
+
+export function traceMatteContours(mask: ImageData): ContourPoint[][] {
+ const w = mask.width;
+ const h = mask.height;
+ // Reused across frames: `solid` is fully rewritten, `traced` cleared below.
+ if (planeSize !== w * h) {
+ planeSize = w * h;
+ solidPlane = new Uint8Array(planeSize);
+ tracedPlane = new Uint8Array(planeSize);
+ }
+ const solid = solidPlane;
+ for (let i = 0; i < w * h; i++) solid[i] = mask.data[i * 4 + 3] > 127 ? 1 : 0;
+ const at = (x: number, y: number) => (x >= 0 && x < w && y >= 0 && y < h ? solid[y * w + x] : 0);
+ // Clockwise Moore neighborhood, starting west.
+ const NB = [
+ [-1, 0],
+ [-1, -1],
+ [0, -1],
+ [1, -1],
+ [1, 0],
+ [1, 1],
+ [0, 1],
+ [-1, 1],
+ ];
+ const traced = tracedPlane;
+ traced.fill(0);
+ const loops: ContourPoint[][] = [];
+ for (let y = 0; y < h; y++) {
+ for (let x = 0; x < w; x++) {
+ const i = y * w + x;
+ // A boundary start: solid, unvisited, with background to its west (the
+ // scan reaches it from the west, which anchors the tracing direction).
+ if (!solid[i] || traced[i] || at(x - 1, y)) continue;
+ const loop: ContourPoint[] = [];
+ const sx = x;
+ const sy = y;
+ let cx = x;
+ let cy = y;
+ // Entered from the west.
+ let from = 0;
+ const cap = w * h * 4;
+ for (let step = 0; step < cap; step++) {
+ traced[cy * w + cx] = 1;
+ loop.push({ x: cx, y: cy });
+ // Search the neighborhood clockwise from just past where we came in.
+ let found = -1;
+ for (let k = 1; k <= 8; k++) {
+ const dir = (from + k) % 8;
+ if (at(cx + NB[dir][0], cy + NB[dir][1])) {
+ found = dir;
+ break;
+ }
+ }
+ if (found < 0) break; // an isolated pixel
+ cx += NB[found][0];
+ cy += NB[found][1];
+ // Next search starts from the direction pointing back at the pixel we
+ // just left, so the trace hugs the boundary.
+ from = (found + 5) % 8;
+ if (cx === sx && cy === sy) break;
+ }
+ if (loop.length >= 8) loops.push(loop);
+ }
+ }
+ loops.sort((a, b) => b.length - a.length);
+ return loops;
+}
+
+/** Ramer–Douglas–Peucker simplification of a closed ring. */
+function simplifyRing(points: ContourPoint[], epsilon: number): ContourPoint[] {
+ if (points.length <= 4) return points;
+ const keep = new Uint8Array(points.length);
+ keep[0] = 1;
+ keep[points.length - 1] = 1;
+ const stack: [number, number][] = [[0, points.length - 1]];
+ while (stack.length) {
+ const [a, b] = stack.pop()!;
+ const pa = points[a];
+ const pb = points[b];
+ const dx = pb.x - pa.x;
+ const dy = pb.y - pa.y;
+ const len = Math.hypot(dx, dy) || 1;
+ let worst = 0;
+ let worstI = -1;
+ for (let i = a + 1; i < b; i++) {
+ const p = points[i];
+ const d = Math.abs((p.x - pa.x) * dy - (p.y - pa.y) * dx) / len;
+ if (d > worst) {
+ worst = d;
+ worstI = i;
+ }
+ }
+ if (worstI > 0 && worst > epsilon) {
+ keep[worstI] = 1;
+ stack.push([a, worstI], [worstI, b]);
+ }
+ }
+ const out: ContourPoint[] = [];
+ for (let i = 0; i < points.length; i++) if (keep[i]) out.push(points[i]);
+ return out;
+}
+
+/** Resample a ring to points evenly spaced `gap` apart along its length. */
+function resampleRing(points: ContourPoint[], gap: number): ContourPoint[] {
+ const out: ContourPoint[] = [];
+ let carry = 0;
+ for (let i = 0; i < points.length; i++) {
+ const a = points[i];
+ const b = points[(i + 1) % points.length];
+ const seg = Math.hypot(b.x - a.x, b.y - a.y);
+ if (seg <= 0) continue;
+ let d = carry;
+ while (d < seg) {
+ const p = d / seg;
+ out.push({ x: a.x + (b.x - a.x) * p, y: a.y + (b.y - a.y) * p });
+ d += gap;
+ }
+ carry = d - seg;
+ }
+ return out;
+}
+
+/** Scratch canvases reused across frames — the ink paints every preview
+ * frame, and each scratch is cleared and fully redrawn before use. One of
+ * each is enough: a paint call stages one silhouette (or one contour read)
+ * at a time. */
+let silCanvas: HTMLCanvasElement | null = null;
+let contourCanvas: HTMLCanvasElement | null = null;
+
+function sizedScratch(held: HTMLCanvasElement | null, w: number, h: number): HTMLCanvasElement {
+ if (held && held.width === w && held.height === h) return held;
+ return kitCanvas(w, h);
+}
+
+/** The silhouette as white fill, for stamping. */
+function whiteSilhouette(silhouette: CanvasImageSource, w: number, h: number): HTMLCanvasElement {
+ const sil = (silCanvas = sizedScratch(silCanvas, w, h));
+ const ctx = sil.getContext("2d")!;
+ ctx.globalCompositeOperation = "source-over";
+ ctx.clearRect(0, 0, w, h);
+ ctx.drawImage(silhouette, 0, 0, w, h);
+ ctx.globalCompositeOperation = "source-in";
+ ctx.fillStyle = "#ffffff";
+ ctx.fillRect(0, 0, w, h);
+ ctx.globalCompositeOperation = "source-over";
+ return sil;
+}
+
+/** Stamp `sil` at `count` points around a circle of radius `r`. */
+function stampAround(ctx: Canvas2D, sil: CanvasImageSource, r: number, count = DILATE_STAMPS) {
+ for (let k = 0; k < count; k++) {
+ const a = (k / count) * Math.PI * 2;
+ ctx.drawImage(sil, Math.cos(a) * r, Math.sin(a) * r);
+ }
+}
+
+/** The traced rings of a silhouette, downscaled for the scan and mapped back
+ * to canvas coordinates, largest first. */
+function contoursOf(silhouette: CanvasImageSource, w: number, h: number): ContourPoint[][] {
+ const scale = Math.min(1, TRACE_SHORT / Math.max(1, Math.min(w, h)));
+ const tw = Math.max(2, Math.round(w * scale));
+ const th = Math.max(2, Math.round(h * scale));
+ const small = (contourCanvas = sizedScratch(contourCanvas, tw, th));
+ const sctx = small.getContext("2d", { willReadFrequently: true })!;
+ sctx.clearRect(0, 0, tw, th);
+ sctx.drawImage(silhouette, 0, 0, tw, th);
+ const rings = traceMatteContours(sctx.getImageData(0, 0, tw, th));
+ const up = 1 / scale;
+ return rings.map((ring) => ring.map((p) => ({ x: p.x * up, y: p.y * up })));
+}
+
+/**
+ * Paint the stroke's ink — white on transparent — into `ctx`, sized to the
+ * `w`×`h` frame the silhouette was drawn at. `scale` converts the stroke's
+ * design px (1080 short side); `t` is the layer's local seconds, driving the
+ * hand style's boil.
+ */
+export function paintStrokeInk(
+ ctx: Canvas2D,
+ silhouette: CanvasImageSource,
+ w: number,
+ h: number,
+ stroke: RemovalStroke,
+ t: number,
+ scale: number
+): void {
+ const width = Math.max(1, (stroke.width ?? STROKE_DEFAULT_WIDTH) * scale);
+ ctx.save();
+ ctx.setTransform(1, 0, 0, 1, 0, 0);
+ ctx.globalCompositeOperation = "source-over";
+ ctx.fillStyle = "#ffffff";
+ ctx.strokeStyle = "#ffffff";
+ if (stroke.style === "solid") {
+ stampAround(ctx, whiteSilhouette(silhouette, w, h), width);
+ } else if (stroke.style === "offset") {
+ const sil = whiteSilhouette(silhouette, w, h);
+ ctx.drawImage(sil, (stroke.offsetX ?? 0) * scale, (stroke.offsetY ?? 0) * scale);
+ } else if (stroke.style === "glow") {
+ const sil = whiteSilhouette(silhouette, w, h);
+ if ("filter" in ctx) {
+ ctx.filter = `blur(${Math.max(1, width).toFixed(1)}px)`;
+ // Twice, so the halo reads against bright footage too.
+ ctx.drawImage(sil, 0, 0);
+ ctx.drawImage(sil, 0, 0);
+ ctx.filter = "none";
+ } else {
+ // No blur available (a bare context): fade the halo by stamping rings
+ // of falling alpha.
+ const bare = ctx as Canvas2D;
+ for (let ring = 1; ring <= 3; ring++) {
+ bare.globalAlpha = 0.5 / ring;
+ stampAround(bare, sil, (width * ring) / 2);
+ }
+ bare.globalAlpha = 1;
+ }
+ } else {
+ // Contour styles.
+ const rings = contoursOf(silhouette, w, h);
+ if (stroke.style === "dotted") {
+ const gap = Math.max(width * 2.6, 6);
+ for (const ring of rings) {
+ for (const p of resampleRing(ring, gap)) {
+ ctx.beginPath();
+ ctx.arc(p.x, p.y, width / 2, 0, Math.PI * 2);
+ ctx.fill();
+ }
+ }
+ } else if (stroke.style === "cut") {
+ // Coarse polygon: the sticker-scissors look.
+ ctx.lineWidth = width;
+ ctx.lineJoin = "miter";
+ for (const ring of rings) {
+ const poly = simplifyRing(ring, Math.max(6, Math.min(w, h) * 0.02));
+ if (poly.length < 3) continue;
+ ctx.beginPath();
+ ctx.moveTo(poly[0].x, poly[0].y);
+ for (let i = 1; i < poly.length; i++) ctx.lineTo(poly[i].x, poly[i].y);
+ ctx.closePath();
+ ctx.stroke();
+ }
+ } else {
+ // Hand drawn: the boundary resampled and jittered by two sinusoids that
+ // drift with time, so the line boils the way a drawn-on-frames outline
+ // does.
+ ctx.lineWidth = Math.max(1.5, width * 0.6);
+ ctx.lineJoin = "round";
+ ctx.lineCap = "round";
+ const amp = Math.max(1.5, width * 0.5);
+ const phase = t * 2.1;
+ for (const ring of rings) {
+ const pts = resampleRing(ring, Math.max(6, amp * 2.2));
+ if (pts.length < 3) continue;
+ ctx.beginPath();
+ for (let i = 0; i <= pts.length; i++) {
+ const p = pts[i % pts.length];
+ const wob = Math.sin(i * 0.9 + phase) * amp + Math.sin(i * 0.37 - phase * 0.7) * amp * 0.5;
+ const x = p.x + Math.cos(i * 0.53 + phase) * wob * 0.4;
+ const y = p.y + Math.sin(i * 0.71 - phase) * wob * 0.4 + wob * 0.3;
+ if (i === 0) ctx.moveTo(x, y);
+ else ctx.lineTo(x, y);
+ }
+ ctx.closePath();
+ ctx.stroke();
+ }
+ }
+ }
+ ctx.restore();
+}
diff --git a/site/public/mediapipe/magic_touch.tflite b/site/public/mediapipe/magic_touch.tflite
new file mode 100644
index 00000000..6abfeeea
Binary files /dev/null and b/site/public/mediapipe/magic_touch.tflite differ
diff --git a/site/scripts/copy-mediapipe-assets.mjs b/site/scripts/copy-mediapipe-assets.mjs
index 6174d0b4..3ad48178 100644
--- a/site/scripts/copy-mediapipe-assets.mjs
+++ b/site/scripts/copy-mediapipe-assets.mjs
@@ -1,7 +1,8 @@
// Stage the MediaPipe vision runtime into public/ so segmentation runs
// self-hosted (no runtime Google fetch). The wasm pair is ~12MB, so it is
-// copied from node_modules on install rather than committed; the tiny
-// person-segmentation model is committed beside it. Runs from postinstall.
+// copied from node_modules on install rather than committed; the
+// person-segmentation and tap-to-select models are committed beside it.
+// Runs from postinstall.
import { copyFileSync, existsSync, mkdirSync } from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
diff --git a/site/src/app/api/inference/assets/route.ts b/site/src/app/api/inference/assets/route.ts
index 6e555949..3cd21a47 100644
--- a/site/src/app/api/inference/assets/route.ts
+++ b/site/src/app/api/inference/assets/route.ts
@@ -41,16 +41,54 @@ export const POST = withDonkeyAuth(async (request) => {
return validationErrorResponse(parsed.error);
}
- // Select the provider and settle the model it will bill before the preflight, so credit limits
- // and pricing are scoped to what actually runs. Callers routinely omit the model and let the
- // adapter pick one (music picks by requested length), so asking the provider is the only way the
- // preflight sees the id that gets charged.
+ // Select the provider, settle the model it will bill, and count the units the request will
+ // charge, all before the preflight, so credit limits and pricing are scoped to what actually
+ // runs. Callers routinely omit the model and let the adapter pick one (music picks by requested
+ // length), so asking the provider is the only way the preflight sees the id that gets charged.
+ // Reference pictures and seed frames ride storage, not the request body — they become inline
+ // data on the server-to-server leg. That resolution reads whole objects out of storage, so it
+ // waits for the balance to clear; only a provider whose charge scales with the input (the matte
+ // segmenter's frame chunks) resolves ahead of the preflight, because its count is read from the
+ // resolved media.
+ const resolveRequest = async () => {
+ const inputs: JsonObject | undefined = parsed.data.inputs
+ ? toJsonObject(
+ await resolveInferenceBlobs(
+ toJsonValue(parsed.data.inputs),
+ request.donkey.userId,
+ ),
+ )
+ : undefined;
+ return inputs ? { ...parsed.data, inputs } : parsed.data;
+ };
let provider;
let model;
+ let generationRequest;
+ let generationCount: number | undefined;
try {
provider = createProviderRegistry().assetProvider(parsed.data);
model = provider.assetModelFor?.(parsed.data) ?? parsed.data.model;
+ generationRequest = parsed.data;
+ if (provider.assetGenerationCountFor) {
+ generationRequest = await resolveRequest();
+ generationCount = await provider.assetGenerationCountFor(generationRequest);
+ }
} catch (error) {
+ // A rejected request still leaves its diagnostic trail, same as a
+ // failure after the preflight.
+ await recordFailedInferenceUsage({
+ clientId: client.clientId,
+ conversationId: request.donkey.conversationId,
+ errorCode: inferenceErrorCode(error),
+ metadata: {
+ assetKind: parsed.data.kind,
+ },
+ model: model ?? parsed.data.model ?? "default",
+ provider: provider?.id ?? parsed.data.provider ?? "unresolved",
+ requestKind: "asset_generation",
+ route: inferenceUsageRoutes.assets,
+ userId: request.donkey.userId,
+ });
if (error instanceof InferenceProviderError) {
return inferenceProviderErrorResponse(error);
}
@@ -63,9 +101,10 @@ export const POST = withDonkeyAuth(async (request) => {
route: inferenceUsageRoutes.assets,
userId: request.donkey.userId,
// The billed model is known here, so the preflight holds the generation to it: an unpriced
- // model never runs and bills upstream, and a flat-priced one never starts on a balance that
- // can't cover it.
+ // model never runs and bills upstream, and a priced one never starts on a balance that
+ // can't cover the full unit count.
enforceModelPrice: true,
+ generationCount,
});
if (!credits.ok) {
return credits.response;
@@ -74,24 +113,16 @@ export const POST = withDonkeyAuth(async (request) => {
const failedUsageProvider = provider.id;
try {
+ if (!provider.assetGenerationCountFor) generationRequest = await resolveRequest();
const generationId = generationIDForRequest(parsed.data);
const generation = {
id: generationId,
kind: parsed.data.kind,
};
- // Reference pictures and seed frames ride storage, not the request body —
- // they become inline data here, on the server-to-server leg.
- const inputs: JsonObject | undefined = parsed.data.inputs
- ? toJsonObject(
- await resolveInferenceBlobs(
- toJsonValue(parsed.data.inputs),
- request.donkey.userId,
- ),
- )
- : undefined;
const result = await provider.generateAsset?.({
generationId,
- request: inputs ? { ...parsed.data, inputs } : parsed.data,
+ request: generationRequest,
+ generationCount,
});
if (!result) {
diff --git a/site/src/cut/components/Inspector.tsx b/site/src/cut/components/Inspector.tsx
index 99ab6b7c..532b8b3a 100644
--- a/site/src/cut/components/Inspector.tsx
+++ b/site/src/cut/components/Inspector.tsx
@@ -1,7 +1,7 @@
"use client";
import { createContext, useCallback, useContext, useEffect, useMemo, useRef, useState, useSyncExternalStore } from "react";
-import { AlignCenter, AlignHorizontalSpaceAround, AlignLeft, AlignRight, AlignVerticalSpaceAround, Bold, ChevronLeft, ChevronRight, Diamond, Frame, Italic, Loader2, Palette, Smile, Trash2, Type, Volume2 } from "lucide-react";
+import { AlignCenter, AlignHorizontalSpaceAround, AlignLeft, AlignRight, AlignVerticalSpaceAround, Bold, ChevronLeft, ChevronRight, Diamond, Frame, Italic, Loader2, Palette, Scissors, Smile, Trash2, Type, Volume2 } from "lucide-react";
import { Button } from "@/components/ui/button";
import { EmojiPicker } from "@/cut/components/EmojiPicker";
import { FontPicker } from "@/cut/components/FontPicker";
@@ -49,6 +49,7 @@ import {
type WordEffectId,
} from "@donkeycut/effects-kit";
import { clipWindow, useEditor, type EditorState } from "@/cut/lib/store";
+import { usePanelView } from "@/cut/lib/panelViews";
import { usePreviewTime } from "@/cut/lib/playhead";
import { CLIP_MAX_ZOOM, clipKeyed, clipPoseAt, clipZoom, contentRect } from "@/cut/lib/types";
import { AnimationCard, AnimationTiles } from "@/cut/components/AnimationTiles";
@@ -116,6 +117,7 @@ import { cn } from "@/lib/utils";
import { ScrollArea } from "@/components/ui/scroll-area";
import { InfoTip, ResetButton, Row, useSliderCheckpoint, Value } from "@/cut/components/panelBits";
import { ColorPanel } from "@/cut/components/ColorPanel";
+import { RemovalPanel } from "@/cut/components/RemovalPanel";
/** Where the animation panel is, for the panel that offers the way in. It is
* held by the Inspector rather than by each panel because the animation panel
@@ -164,13 +166,28 @@ function InspectorColumn({
audio?: AudioClip;
overlay?: Overlay;
}) {
- const [animView, setAnimView] = useState<"closed" | "open" | "returned">("closed");
+ // Whether the Animations view is open holds for the session per overlay,
+ // so an overlay reselected after a look elsewhere reopens on it. The
+ // transitional "returned" state stays local — it exists for scroll restore
+ // on the way back, and only "open"/"closed" are worth remembering.
+ const [heldAnim, setHeldAnim] = usePanelView<"open" | "closed">(
+ `overlay-anim:${overlay?.id ?? "none"}`,
+ "closed"
+ );
+ const [animView, setAnimView] = useState<"closed" | "open" | "returned">(heldAnim);
const nav = useMemo<AnimNav>(
() => ({
view: animView,
- open: () => setAnimView("open"),
- back: () => setAnimView("returned"),
+ open: () => {
+ setHeldAnim("open");
+ setAnimView("open");
+ },
+ back: () => {
+ setHeldAnim("closed");
+ setAnimView("returned");
+ },
}),
+ // eslint-disable-next-line react-hooks/exhaustive-deps
[animView]
);
@@ -296,7 +313,7 @@ function Section({
}) {
const gated = !!onEnabledChange;
return (
- <section className="mt-1.5 border-t border-border pt-1.5">
+ <section className="mt-1.5 border-t border-border pt-1.5 first:mt-0 first:border-t-0">
<div className="flex min-h-9 items-center justify-between gap-2">
<span className="flex min-w-0 items-center gap-1 text-[13px] font-medium">
<span className="truncate">{title}</span>
@@ -395,6 +412,7 @@ function LayoutButtons({
* to the main settings. */
const CLIP_TABS = [
{ id: "color", label: "Color", Icon: Palette },
+ { id: "cutout", label: "Cutout", Icon: Scissors },
{ id: "frame", label: "Frame", Icon: Frame },
{ id: "audio", label: "Audio", Icon: Volume2 },
] as const;
@@ -404,16 +422,19 @@ type ClipTab = (typeof CLIP_TABS)[number]["id"] | "main";
* The video clip's column: the main settings with the Color, Frame, and Audio
* views behind a toolbar pinned under the scroller, so it stays put whichever
* view is open. Color lays out its own bands around its own scroller; the
- * others ride the column's. The column is keyed on the selection, so picking
- * another clip lands back on the main view.
+ * others ride the column's. The open view holds for the session per clip, so
+ * deselecting to look at something and coming back lands this clip on its
+ * own view; a clip whose panel was never opened starts at the main one.
*/
function ClipColumn({ clip }: { clip: VideoClip }) {
- const [tab, setTab] = useState<ClipTab>("main");
+ const [tab, setTab] = usePanelView<ClipTab>(`clip-tab:${clip.id}`, "main");
const back = () => setTab("main");
return (
<div className="flex min-h-0 flex-1 flex-col">
{tab === "color" ? (
<ColorPanel clip={clip} onBack={back} />
+ ) : tab === "cutout" ? (
+ <RemovalPanel clip={clip} onBack={back} />
) : (
<ScrollArea className="min-h-0 flex-1">
{tab === "frame" ? (
@@ -1926,7 +1947,8 @@ type AnimSlot = "in" | "out" | "loop" | "move" | "words";
* which a stack of grids would not be. */
function AnimationPanel({ overlay: o, onBack }: { overlay: Overlay; onBack: () => void }) {
const anim = o.anim ?? {};
- const [picked, setSlot] = useState<AnimSlot>("in");
+ // The active slot holds for the session, like every settings view's tab.
+ const [picked, setSlot] = usePanelView<AnimSlot>(`anim-slot:${o.id}`, "in");
// Word emphasis is a title's slot; a shape or a sticker has no words, so
// selecting one hands the picker back to the entrance.
const tabs: AnimSlot[] = isTextOverlay(o)
diff --git a/site/src/cut/components/Preview.tsx b/site/src/cut/components/Preview.tsx
index 60528b38..fa61a2f2 100644
--- a/site/src/cut/components/Preview.tsx
+++ b/site/src/cut/components/Preview.tsx
@@ -4,6 +4,7 @@ import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState, typ
import { Check, Copy, Loader2, ZoomIn, ZoomOut } from "lucide-react";
import { usePlayback } from "@/cut/hooks/usePlayback";
import { startDrag } from "@/cut/lib/drag";
+import { useBrushUi } from "@/cut/lib/removal/brushUi";
import {
playheadAt,
previewAt,
@@ -23,6 +24,7 @@ import { CLIP_MAX_ZOOM, clipCovers, clipKeyed, clipPoseAt, clipZoom, contentRect
import { hasMaskKeys, type MaskKey } from "@donkeycut/effects-kit";
import { cn } from "@/lib/utils";
import { MaskGizmoCore, OverlayChromeHost, OverlayLayer, StagePress } from "./OverlayLayer";
+import { RemovalBrush } from "./RemovalBrush";
import { CORNER_HANDLES, HANDLE_AXIS, TransformHandles, type ResizeHandle } from "./TransformHandles";
import {
StageEffectPaint,
@@ -150,6 +152,24 @@ function pictureInPane(
return visX >= Math.min(ONSCREEN_MIN_PX, stage.w) && visY >= Math.min(ONSCREEN_MIN_PX, stage.h);
}
+/** While the Cutout panel's Original toggle is on, the stage says so — the
+ * cutout looks gone otherwise. Clicking the badge brings the cutout back. */
+function OriginalPeekBadge() {
+ const peeking = useEditor((st) => !!st.removalPeek);
+ if (!peeking) return null;
+ return (
+ <div className="absolute inset-x-0 top-2 flex justify-center">
+ <button
+ type="button"
+ className="clip-peek-badge rounded-full bg-black/55 px-3 py-1 text-[11.5px] font-medium text-white shadow-lg transition-colors hover:bg-black/70"
+ onClick={() => useEditor.setState({ removalPeek: null })}
+ >
+ Showing the original — click to see the cutout
+ </button>
+ </div>
+ );
+}
+
/** A play held at a standstill, waiting on the picture. The playhead is not
* moving, so this says what the frozen frame is doing. */
function BufferingBadge() {
@@ -674,6 +694,7 @@ export function Preview() {
/>
</StagePictureFx>
<BufferingBadge />
+ <OriginalPeekBadge />
<StagePress.Provider value={stagePress}>
{slices.map((slice) =>
slice.kind === "elements" ? (
@@ -710,6 +731,7 @@ export function Preview() {
<div className="pointer-events-none absolute inset-x-0 top-1/2 z-20 h-px bg-neutral-400/80" />
)}
<ClipMaskGizmo stage={stage} />
+ <RemovalBrush stage={stage} />
<ClipTransformGizmo stage={stage} />
</div>
</div>
@@ -928,6 +950,7 @@ function ClipTransformGizmo({ stage }: { stage: Stage }) {
// that answer instead of to the clock: one render when the clip comes and
// goes, rather than one per frame while it stays.
const selectedClip = selection?.kind === "clip" ? clips.find((c) => c.id === selection.id) : null;
+ const brushClipId = useBrushUi((s) => s.clipId);
const live = usePreviewSelector((t) => {
if (!selectedClip) return false;
const speed = selectedClip.speed && selectedClip.speed > 0 ? selectedClip.speed : 1;
@@ -939,6 +962,9 @@ function ClipTransformGizmo({ stage }: { stage: Stage }) {
if (skimTime !== null) return null;
const clip = selectedClip;
if (!clip || clip.hidden || !live) return null;
+ // An open brush session owns the stage for its clip: presses paint the
+ // removal, so the move box and its grips step aside until the session ends.
+ if (brushClipId === clip.id && clip.removal?.mode === "custom") return null;
const st = () => useEditor.getState();
const patch = (p: Partial<VideoClip>) => st().updateClipTransient(clip.id, p);
diff --git a/site/src/cut/components/RemovalBrush.tsx b/site/src/cut/components/RemovalBrush.tsx
new file mode 100644
index 00000000..20e041b6
--- /dev/null
+++ b/site/src/cut/components/RemovalBrush.tsx
@@ -0,0 +1,395 @@
+"use client";
+
+import { useEffect, useRef, useState } from "react";
+import { createPortal } from "react-dom";
+import { useBrushUi } from "@/cut/lib/removal/brushUi";
+import { QuickSelectSession } from "@/cut/lib/removal/interactive";
+import { usePreviewSelector, useSkim } from "@/cut/lib/playhead";
+import { decodeRasterImageUrl } from "@/cut/lib/raster";
+import { useEditor } from "@/cut/lib/store";
+import {
+ clipCovers,
+ clipKeyed,
+ clipPoseAt,
+ clipZoom,
+ contentRect,
+ rectOf,
+ type MediaAsset,
+ type RemovalSeeds,
+ type VideoClip,
+} from "@/cut/lib/types";
+
+/**
+ * The custom-removal brush on the stage: a pointer surface laid over the
+ * selected clip's picture while its brush session is open. Quick strokes run
+ * the on-device tap-to-select segmenter and preview the pick live under the
+ * pointer; manual strokes paint the mask directly. The selection shows as a
+ * red overlay, a loupe magnifies the pixels being worked, and every finished
+ * stroke lands in the doc as a seed record — the prompts and paint the hosted
+ * tracker replays across the whole clip. A size ring rides the pointer the
+ * whole session — over the picture, the stage margins, and the settings
+ * panel — so the brush's reach is visible while the slider moves.
+ */
+
+const LOUPE_PX = 128;
+const LOUPE_ZOOM = 3;
+/** Pointer samples closer than this (frame fractions) fold into one point. */
+const MIN_STEP = 0.008;
+/** The most points one stroke stores as a tracker prompt. */
+const PROMPT_POINTS_MAX = 24;
+
+type Frac = { x: number; y: number };
+
+const decodeSeed = async (url: string) => {
+ const img = await decodeRasterImageUrl(url).catch(() => null);
+ return img ? img.source : null;
+};
+
+export function RemovalBrush({ stage }: { stage: { w: number; h: number } }) {
+ const brushClipId = useBrushUi((s) => s.clipId);
+ const selection = useEditor((s) => s.selection);
+ const clips = useEditor((s) => s.clips);
+ const assets = useEditor((s) => s.assets);
+ const skimTime = useSkim();
+ const clip =
+ brushClipId && selection?.kind === "clip" && selection.id === brushClipId
+ ? clips.find((c) => c.id === brushClipId) ?? null
+ : null;
+ const armed = !!clip && clip.removal?.mode === "custom" && !clip.hidden;
+ const tLocal = usePreviewSelector((t) => (armed && clip ? t - clip.start : -1));
+ if (!armed || !clip || skimTime !== null) return null;
+ const speed = clip.speed && clip.speed > 0 ? clip.speed : 1;
+ const len = Math.max(0.1, (clip.out - clip.in) / speed);
+ if (tLocal < 0 || tLocal >= len) return null;
+ const asset = assets.find((a) => a.id === clip.assetId);
+ if (!asset?.width || !asset?.height) return null;
+ return <BrushSurface clip={clip} asset={asset} stage={stage} tLocal={tLocal} />;
+}
+
+function BrushSurface({
+ clip,
+ asset,
+ stage,
+ tLocal,
+}: {
+ clip: VideoClip;
+ asset: MediaAsset;
+ stage: { w: number; h: number };
+ tLocal: number;
+}) {
+ const tool = useBrushUi((s) => s.tool);
+ const size = useBrushUi((s) => s.size);
+ const sessionRef = useRef<QuickSelectSession | null>(null);
+ const [sessionEpoch, setSessionEpoch] = useState(0);
+ const overlayRef = useRef<HTMLCanvasElement | null>(null);
+ const loupeRef = useRef<HTMLCanvasElement | null>(null);
+ const ringRef = useRef<HTMLDivElement | null>(null);
+ const ringGeom = useRef({ w: 0, h: 0, px: 0 });
+ const [loupeRight, setLoupeRight] = useState(false);
+ const [painting, setPainting] = useState(false);
+
+ /** Hover tracking rides refs and direct style writes — a pointer move must
+ * never re-render the stage. The loupe's side flip is the one piece of
+ * state, and React bails when the boolean holds. */
+ const updatePointer = (p: Frac | null) => {
+ const ring = ringRef.current;
+ if (ring) {
+ const g = ringGeom.current;
+ ring.style.display = p ? "block" : "none";
+ if (p) {
+ ring.style.left = `${p.x * g.w - g.px / 2}px`;
+ ring.style.top = `${p.y * g.h - g.px / 2}px`;
+ }
+ }
+ setLoupeRight(!!p && p.x < 0.4 && p.y < 0.4);
+ };
+ /** The seeds object this surface last wrote, so the replay effect only
+ * rebuilds the mask for outside writes — undo, redo, the panel's Reset. */
+ const lastWritten = useRef<RemovalSeeds | undefined | null>(null);
+ const gesture = useRef<{
+ points: Frac[];
+ erase: boolean;
+ quick: boolean;
+ lastSeg: number;
+ } | null>(null);
+
+ useEffect(() => {
+ let alive = true;
+ void QuickSelectSession.open().then((s) => {
+ if (!alive) return;
+ sessionRef.current = s;
+ setSessionEpoch((e) => e + 1);
+ });
+ return () => {
+ alive = false;
+ sessionRef.current = null;
+ gesture.current = null;
+ };
+ }, [clip.id]);
+
+ const paintOverlay = () => {
+ const s = sessionRef.current;
+ const c = overlayRef.current;
+ if (!s || !c) return;
+ if (c.width !== s.width || c.height !== s.height) {
+ c.width = s.width;
+ c.height = s.height;
+ }
+ const ctx = c.getContext("2d");
+ if (ctx) s.paintOverlay(ctx, c.width, c.height);
+ };
+
+ const srcT = () => clip.in + Math.max(0, tLocal) * (clip.speed && clip.speed > 0 ? clip.speed : 1);
+
+ // Outside writes to the seeds rebuild the working mask from them.
+ const seeds = clip.removal?.seeds;
+ useEffect(() => {
+ const s = sessionRef.current;
+ if (!s || lastWritten.current === seeds) return;
+ lastWritten.current = seeds;
+ if (!s.refreshFrame(clip.id)) return;
+ void s.replaySeeds(seeds, decodeSeed, srcT()).then(paintOverlay);
+ // srcT reads the frame the rebuild runs against; a playhead move alone
+ // must not re-replay, and the lastWritten guard above holds that line.
+ }, [seeds, sessionEpoch, clip.id]); // eslint-disable-line react-hooks/exhaustive-deps
+
+ const paintLoupe = (p: Frac) => {
+ const s = sessionRef.current;
+ const c = loupeRef.current;
+ if (!s || !c) return;
+ const side = LOUPE_PX / LOUPE_ZOOM;
+ const sx = p.x * s.width - side / 2;
+ const sy = p.y * s.height - side / 2;
+ const ctx = c.getContext("2d");
+ if (!ctx) return;
+ ctx.fillStyle = "#000000";
+ ctx.fillRect(0, 0, LOUPE_PX, LOUPE_PX);
+ ctx.imageSmoothingEnabled = true;
+ ctx.drawImage(s.frame(), sx, sy, side, side, 0, 0, LOUPE_PX, LOUPE_PX);
+ const ov = overlayRef.current;
+ if (ov && ov.width > 2) ctx.drawImage(ov, sx, sy, side, side, 0, 0, LOUPE_PX, LOUPE_PX);
+ ctx.strokeStyle = "rgba(255,255,255,0.9)";
+ ctx.lineWidth = 1.5;
+ ctx.beginPath();
+ ctx.arc(
+ LOUPE_PX / 2,
+ LOUPE_PX / 2,
+ (size / 2) * Math.min(s.width, s.height) * LOUPE_ZOOM,
+ 0,
+ Math.PI * 2
+ );
+ ctx.stroke();
+ };
+
+ /** Merge one stroke into the clip's stored seeds under one undo step. */
+ const writeSeeds = (mutate: (next: RemovalSeeds) => void) => {
+ const st = useEditor.getState();
+ const cur = st.clips.find((c) => c.id === clip.id)?.removal;
+ if (!cur || cur.mode !== "custom") return;
+ const next: RemovalSeeds = {
+ prompts: [...(cur.seeds?.prompts ?? [])],
+ ...(cur.seeds?.paint ? { paint: [...cur.seeds.paint] } : {}),
+ };
+ mutate(next);
+ st.pushHistory();
+ lastWritten.current = next;
+ // A new stroke reopens the selection: the bake waits for the next Apply.
+ st.updateClipTransient(clip.id, { removal: { ...cur, seeds: next, requested: undefined } });
+ };
+
+ const finishStroke = (g: NonNullable<typeof gesture.current>) => {
+ const s = sessionRef.current;
+ if (!s) return;
+ const t = srcT();
+ if (g.quick) {
+ s.commitTentative();
+ paintOverlay();
+ if (!s.quickAvailable) return;
+ const label: 0 | 1 = g.erase ? 0 : 1;
+ const kept =
+ g.points.length <= PROMPT_POINTS_MAX
+ ? g.points
+ : g.points.filter(
+ (_, i) => i % Math.ceil(g.points.length / PROMPT_POINTS_MAX) === 0
+ );
+ writeSeeds((next) => {
+ const at = next.prompts.findIndex((p) => Math.abs(p.t - t) < 0.02);
+ const points = kept.map((p) => ({ x: p.x, y: p.y, label }));
+ if (at >= 0)
+ next.prompts[at] = { ...next.prompts[at], points: [...next.prompts[at].points, ...points] };
+ else next.prompts.push({ t, points });
+ });
+ return;
+ }
+ paintOverlay();
+ void s.paintSeed(g.points, size / 2).then((url) => {
+ writeSeeds((next) => {
+ const field = g.erase ? "erase" : "add";
+ const at = (next.paint ?? []).findIndex((p) => Math.abs(p.t - t) < 0.02 && !p[field]);
+ const paint = [...(next.paint ?? [])];
+ if (at >= 0) paint[at] = { ...paint[at], [field]: url };
+ else paint.push({ t, [field]: url });
+ next.paint = paint;
+ });
+ });
+ };
+
+ const localPoint = (e: React.PointerEvent<HTMLDivElement>): Frac => {
+ const el = e.currentTarget;
+ return {
+ x: Math.min(1, Math.max(0, e.nativeEvent.offsetX / Math.max(1, el.clientWidth))),
+ y: Math.min(1, Math.max(0, e.nativeEvent.offsetY / Math.max(1, el.clientHeight))),
+ };
+ };
+
+ const onPointerDown = (e: React.PointerEvent<HTMLDivElement>) => {
+ if (e.button !== 0) return;
+ e.stopPropagation();
+ const s = sessionRef.current;
+ if (!s) return;
+ e.currentTarget.setPointerCapture(e.pointerId);
+ s.refreshFrame(clip.id);
+ const p = localPoint(e);
+ const erase = tool === "erase" || tool === "quickErase";
+ const quick = tool === "quick" || tool === "quickErase";
+ gesture.current = { points: [p], erase, quick, lastSeg: 0 };
+ if (quick) {
+ if (s.quickPreview([p], erase)) paintOverlay();
+ } else {
+ s.paintStroke([p], size / 2, erase);
+ paintOverlay();
+ }
+ updatePointer(p);
+ setPainting(true);
+ paintLoupe(p);
+ };
+
+ const onPointerMove = (e: React.PointerEvent<HTMLDivElement>) => {
+ const p = localPoint(e);
+ updatePointer(p);
+ const g = gesture.current;
+ const s = sessionRef.current;
+ if (!g || !s) return;
+ const last = g.points[g.points.length - 1];
+ if (Math.hypot(p.x - last.x, p.y - last.y) < MIN_STEP) return;
+ g.points.push(p);
+ if (g.quick) {
+ const now = performance.now();
+ if (now - g.lastSeg > 60) {
+ g.lastSeg = now;
+ if (s.quickPreview(g.points, g.erase)) paintOverlay();
+ }
+ } else {
+ s.paintStroke([last, p], size / 2, g.erase);
+ paintOverlay();
+ }
+ paintLoupe(p);
+ };
+
+ const endGesture = (commit: boolean) => {
+ const g = gesture.current;
+ gesture.current = null;
+ setPainting(false);
+ if (!g) return;
+ if (commit) finishStroke(g);
+ else sessionRef.current?.dropTentative();
+ };
+
+ // The same geometry the compositor draws: the picture's rect inside the
+ // clip's box in stage pixels, hung on the pose anchor so a keyed clip's
+ // surface travels with its picture.
+ const r = rectOf(clip);
+ const boxPx = { x: r.x * stage.w, y: r.y * stage.h, w: r.w * stage.w, h: r.h * stage.h };
+ const pic = contentRect(
+ boxPx,
+ asset.width!,
+ asset.height!,
+ clipCovers(clip),
+ clipZoom(clip),
+ clip.panX ?? 0,
+ clip.panY ?? 0
+ );
+ const pose = clipKeyed(clip) ? clipPoseAt(clip, tLocal) : null;
+ const ax = (pose ? pose.x : r.x + r.w / 2) * stage.w;
+ const ay = (pose ? pose.y : r.y + r.h / 2) * stage.h;
+ const rot = pose ? pose.rotation : clip.rotation ?? 0;
+ const scl = pose ? pose.scale : 1;
+ const brushPx = size * Math.min(pic.w, pic.h);
+ // The cursor ring is positioned by direct style writes on pointer moves;
+ // this keeps the geometry those writes use in step with the layout.
+ useEffect(() => {
+ ringGeom.current = { w: pic.w, h: pic.h, px: brushPx };
+ });
+ // Off the picture the ring keeps following the pointer — over the stage
+ // margins and the settings panel — positioned in viewport pixels from a
+ // window listener; over the picture the surface's own ring takes over.
+ const freeRingRef = useRef<HTMLDivElement | null>(null);
+ useEffect(() => {
+ const onMove = (e: PointerEvent) => {
+ const el = freeRingRef.current;
+ if (!el) return;
+ const onSurface = e.target instanceof Element && !!e.target.closest(".clip-removal-brush");
+ el.style.display = onSurface ? "none" : "block";
+ el.style.left = `${e.clientX - brushPx / 2}px`;
+ el.style.top = `${e.clientY - brushPx / 2}px`;
+ };
+ window.addEventListener("pointermove", onMove);
+ return () => window.removeEventListener("pointermove", onMove);
+ }, [brushPx]);
+
+ return (
+ <>
+ <div
+ className="pointer-events-none absolute"
+ style={{
+ left: ax,
+ top: ay,
+ width: 0,
+ height: 0,
+ transform: rot || scl !== 1 ? `rotate(${rot}deg) scale(${scl})` : undefined,
+ }}
+ >
+ <div
+ className="clip-removal-brush pointer-events-auto absolute cursor-crosshair touch-none"
+ style={{
+ left: pic.x - ax,
+ top: pic.y - ay,
+ width: pic.w,
+ height: pic.h,
+ }}
+ onPointerDown={onPointerDown}
+ onPointerMove={onPointerMove}
+ onPointerUp={() => endGesture(true)}
+ onPointerCancel={() => endGesture(false)}
+ onPointerLeave={() => {
+ if (!gesture.current) updatePointer(null);
+ }}
+ >
+ <canvas ref={overlayRef} className="pointer-events-none absolute inset-0 size-full" />
+ <div
+ ref={ringRef}
+ className="pointer-events-none absolute rounded-full border border-white/90 shadow-[0_0_0_1px_rgba(0,0,0,0.4)]"
+ style={{ display: "none", width: brushPx, height: brushPx }}
+ />
+ </div>
+ </div>
+ {createPortal(
+ <div
+ ref={freeRingRef}
+ className="pointer-events-none fixed z-50 rounded-full border border-white/90 shadow-[0_0_0_1px_rgba(0,0,0,0.4)]"
+ style={{ display: "none", width: brushPx, height: brushPx }}
+ />,
+ document.body
+ )}
+ {painting && (
+ <canvas
+ ref={loupeRef}
+ width={LOUPE_PX}
+ height={LOUPE_PX}
+ className="clip-removal-loupe pointer-events-none absolute top-2 z-20 rounded-lg border border-white/40 shadow-lg"
+ style={loupeRight ? { right: 8 } : { left: 8 }}
+ />
+ )}
+ </>
+ );
+}
diff --git a/site/src/cut/components/RemovalPanel.tools.ts b/site/src/cut/components/RemovalPanel.tools.ts
new file mode 100644
index 00000000..2e5e3dad
--- /dev/null
+++ b/site/src/cut/components/RemovalPanel.tools.ts
@@ -0,0 +1,97 @@
+/**
+ * The assistant's cutout tools — the background-removal suite the Inspector's
+ * Cutout tab exposes on a video clip: the removal mode, the silhouette
+ * stroke, and the fill behind the subject. Kept beside the panel; the catalog
+ * spreads this list into the model's toolset and `aiTools.ts` keys its
+ * handlers on `RemovalToolName`. Enums and ranges come from the same kit
+ * constants the panel renders from.
+ */
+
+import {
+ CHROMA_DEFAULT_INTENSITY,
+ CHROMA_DEFAULT_SOFTNESS,
+ CHROMA_DEFAULT_SPILL,
+ STROKE_DEFAULT_WIDTH,
+ STROKE_OFFSET_MAX,
+ STROKE_STYLE_LABELS,
+ STROKE_STYLES,
+ STROKE_WIDTH_MAX,
+} from "@donkeycut/effects-kit";
+import { num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const REMOVAL_TOOLS = [
+ {
+ name: "set_removal",
+ description:
+ 'Cut a video clip\'s background away — the Inspector\'s Cutout tab, on any track. Modes: "auto" mattes the person in the shot with AI; "custom" mattes whatever `subject` describes in a few words ("the dog") and tracks every match through the clip — picking one exact instance by hand is the panel\'s brush flow, so send the user there for that; "chroma" keys out a backdrop color (green screen) with intensity/softness/spill; "off" removes the whole cutout, stroke and background fill included. The AI mattes bake in the background — wait_for_renders covers the bake. Where the background was, lower tracks or the project background show through; set_removal_background fills that area within the clip itself. Read the background-removal skill for the full recipes.',
+ inputSchema: obj(
+ {
+ clipId: str("Video clip id"),
+ mode: {
+ type: "string",
+ enum: ["off", "auto", "custom", "chroma"],
+ description: "Cutout mode",
+ },
+ subject: str(
+ 'Custom mode: what to keep, in a few words ("the dog", "the red car") — every match is matted and tracked'
+ ),
+ color: str(
+ 'Chroma key color "#rrggbb" (omitted: sampled from the footage borders, where a backdrop lives)'
+ ),
+ intensity: num(
+ `Chroma tolerance 0..1 — how far from the key a color still keys out (default ${CHROMA_DEFAULT_INTENSITY})`
+ ),
+ softness: num(`Chroma edge rolloff 0..1 (default ${CHROMA_DEFAULT_SOFTNESS})`),
+ spill: num(
+ `Chroma spill suppression 0..1 — pulls key-colored fringe off the kept pixels (default ${CHROMA_DEFAULT_SPILL})`
+ ),
+ },
+ ["clipId", "mode"]
+ ),
+ },
+ {
+ name: "set_removal_stroke",
+ description: `Draw ink around a cutout clip's silhouette (the clip needs an active set_removal cutout). Styles: ${STROKE_STYLES.map((s) => `${s} (${STROKE_STYLE_LABELS[s].toLowerCase()})`).join(", ")} — glow haloes the edge, hand looks marker-drawn, cut is a coarse straight-edged sticker cut, solid is a clean even line, offset throws a filled silhouette copy behind the subject (offset_x/offset_y place it), dotted walks dots along the edge. Pass style "none" to remove the stroke.`,
+ inputSchema: obj(
+ {
+ clipId: str("Video clip id"),
+ style: {
+ type: "string",
+ enum: [...STROKE_STYLES, "none"],
+ description: 'Stroke style, or "none" to remove it',
+ },
+ color: str('Ink color "#rrggbb" (default white)'),
+ width: num(
+ `Ink thickness, design px at the 1080 short side, 1..${STROKE_WIDTH_MAX} (default ${STROKE_DEFAULT_WIDTH})`
+ ),
+ offset_x: num(
+ `Offset style: silhouette displacement right, design px, -${STROKE_OFFSET_MAX}..${STROKE_OFFSET_MAX}`
+ ),
+ offset_y: num(
+ `Offset style: silhouette displacement down, design px, -${STROKE_OFFSET_MAX}..${STROKE_OFFSET_MAX}`
+ ),
+ },
+ ["clipId", "style"]
+ ),
+ },
+ {
+ name: "set_removal_background",
+ description:
+ 'Fill in behind a cutout clip\'s subject (the clip needs an active set_removal cutout): a flat color or a project image asset drawn where the removed background was, inside the clip\'s own picture. Kind "none" clears the fill and lets lower tracks show through again.',
+ inputSchema: obj(
+ {
+ clipId: str("Video clip id"),
+ kind: {
+ type: "string",
+ enum: ["none", "color", "image"],
+ description: "Backdrop fill behind the subject",
+ },
+ color: str('Fill color "#rrggbb" (kind "color")'),
+ asset_id: str('Project image asset id (kind "image")'),
+ },
+ ["clipId", "kind"]
+ ),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type RemovalToolName = (typeof REMOVAL_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/RemovalPanel.tsx b/site/src/cut/components/RemovalPanel.tsx
new file mode 100644
index 00000000..9d091646
--- /dev/null
+++ b/site/src/cut/components/RemovalPanel.tsx
@@ -0,0 +1,739 @@
+"use client";
+
+import { useEffect, useRef, useState } from "react";
+import { Ban, ChevronLeft, Loader2 } from "lucide-react";
+import {
+ Select,
+ SelectContent,
+ SelectItem,
+ SelectTrigger,
+ SelectValue,
+} from "@/components/ui/select";
+import {
+ BRUSH_SIZE_MAX,
+ BRUSH_SIZE_MIN,
+ useBrushUi,
+ type BrushTool,
+} from "@/cut/lib/removal/brushUi";
+import {
+ confirmMatteBake,
+ ensureMatteBake,
+ retryMatteBake,
+ useMatteBakes,
+ type MatteBakeJob,
+} from "@/cut/lib/removal/bakeJobs";
+import { clipKeyColor, suggestKeyColor } from "@/cut/lib/removal/keyColor";
+import { HostedErrorText } from "@/cut/components/hostedError";
+import { useEditor } from "@/cut/lib/store";
+import { usePanelView } from "@/cut/lib/panelViews";
+import {
+ CHROMA_DEFAULT_INTENSITY,
+ CHROMA_DEFAULT_SOFTNESS,
+ CHROMA_DEFAULT_SPILL,
+ paintStrokeInk,
+ STROKE_DEFAULT_WIDTH,
+ STROKE_OFFSET_MAX,
+ STROKE_WIDTH_MAX,
+} from "@donkeycut/effects-kit";
+import {
+ removalActive,
+ removalFingerprint,
+ STROKE_STYLE_LABELS,
+ STROKE_STYLES,
+ type ClipRemoval,
+ type RemovalMode,
+ type StrokeStyleId,
+ type VideoClip,
+} from "@/cut/lib/types";
+import { ColorField } from "@/cut/components/ColorField";
+import { Row, useSliderCheckpoint } from "@/cut/components/panelBits";
+import { parseNumberInput } from "@/cut/components/ScrubValue";
+import { ValueSlider } from "@/cut/components/ValueSlider";
+import { cn } from "@/lib/utils";
+import { ScrollArea } from "@/components/ui/scroll-area";
+
+/**
+ * The clip panel's Cutout view: labeled inspector rows. Layer picks what the
+ * rows below edit — Removal (the mode and its controls), Stroke (ink around
+ * the kept silhouette), Background (the fill behind it). Removal leads with
+ * Auto — most people want the background gone in one pick — and brushing sits
+ * underneath as the refinement. Every write lands on the one `clip.removal`
+ * field; drafts stream through the transient updater under one checkpoint per
+ * gesture, the way the color panel writes its grade.
+ */
+
+const LAYERS = [
+ { id: "removal", label: "Removal" },
+ { id: "stroke", label: "Stroke" },
+ { id: "background", label: "Background" },
+] as const;
+type Layer = (typeof LAYERS)[number]["id"];
+
+const MODES: { id: RemovalMode | "off"; label: string }[] = [
+ { id: "auto", label: "Auto" },
+ { id: "custom", label: "Custom" },
+ { id: "chroma", label: "Chroma" },
+ { id: "off", label: "Off" },
+];
+
+const BRUSH_TOOLS: { id: BrushTool; label: string }[] = [
+ { id: "quick", label: "Quick brush" },
+ { id: "brush", label: "Brush" },
+ { id: "quickErase", label: "Quick erase" },
+ { id: "erase", label: "Erase" },
+];
+
+/** Shared write path: transiently while a gesture is live, committed at its
+ * end, one checkpoint per gesture. */
+function useRemovalWriter(clip: VideoClip) {
+ const ck = useSliderCheckpoint();
+ const draft = (r: ClipRemoval | undefined) => {
+ ck.begin();
+ useEditor.getState().updateClipTransient(clip.id, { removal: r });
+ };
+ const commit = (r: ClipRemoval | undefined) => {
+ draft(r);
+ ck.end();
+ };
+ return { draft, commit };
+}
+
+/** One inspector row: a label and a compact dropdown, the shape every row in
+ * this panel shares. */
+function RowSelect<T extends string>({
+ label,
+ value,
+ options,
+ onChange,
+ hook,
+}: {
+ label: string;
+ value: T;
+ options: readonly { id: T; label: string }[];
+ onChange: (next: T) => void;
+ hook: string;
+}) {
+ return (
+ <Row label={label}>
+ <Select
+ value={value}
+ items={Object.fromEntries(options.map((o) => [o.id, o.label]))}
+ onValueChange={(v) => onChange(v as T)}
+ >
+ <SelectTrigger className={`${hook} h-8 w-36 text-[12px]`}>
+ <SelectValue />
+ </SelectTrigger>
+ <SelectContent>
+ {options.map((o) => (
+ <SelectItem key={o.id} value={o.id} className={`${hook}-${o.id} text-[12px]`}>
+ {o.label}
+ </SelectItem>
+ ))}
+ </SelectContent>
+ </Select>
+ </Row>
+ );
+}
+
+export function RemovalPanel({ clip, onBack }: { clip: VideoClip; onBack: () => void }) {
+ // The open layer holds for the session per clip, so leaving the panel and
+ // coming back lands on the same rows.
+ const [layer, setLayer] = usePanelView<Layer>(`cutout-layer:${clip.id}`, "removal");
+ const removal = clip.removal;
+ const peeking = useEditor((s) => s.removalPeek === clip.id);
+
+ // Peek only while held; a switched selection or a closed panel lets go too.
+ useEffect(
+ () => () => {
+ if (useEditor.getState().removalPeek) useEditor.setState({ removalPeek: null });
+ },
+ [clip.id]
+ );
+
+ // The bake the clip's AI matte owes, restarted whenever trims or seeds move
+ // it. The doc sweep covers the closed-panel cases; this keeps the open
+ // panel immediate.
+ const fp =
+ removal && (removal.mode === "auto" || removal.mode === "custom")
+ ? removalFingerprint(clip.assetId, removal)
+ : null;
+ useEffect(() => {
+ if (fp) ensureMatteBake(clip.id);
+ }, [clip.id, fp, clip.in, clip.out]);
+
+ // The brush session follows custom mode while its panel is open.
+ useEffect(() => {
+ const brush = useBrushUi.getState();
+ if (removal?.mode === "custom") brush.open(clip.id);
+ else if (brush.clipId === clip.id) brush.close();
+ return () => {
+ const b = useBrushUi.getState();
+ if (b.clipId === clip.id) b.close();
+ };
+ }, [clip.id, removal?.mode]);
+
+ return (
+ <div className="flex min-h-0 flex-1 flex-col">
+ <div className="shrink-0 bg-card pb-2">
+ <div className="flex h-10 shrink-0 items-center gap-1 px-2.5 text-sm font-semibold tracking-tight">
+ <button
+ type="button"
+ aria-label="Back"
+ className="clip-cutout-back grid size-6 place-items-center rounded text-muted-foreground transition-colors hover:text-foreground"
+ onClick={onBack}
+ >
+ <ChevronLeft className="size-4" />
+ </button>
+ Cutout
+ </div>
+ </div>
+ <ScrollArea
+ key={layer}
+ className="min-h-0 flex-1"
+ viewportClassName="overscroll-contain"
+ contentClassName="flex flex-col gap-1 px-3.5 pt-1 pb-4"
+ >
+ <RowSelect
+ label="Layer"
+ value={layer}
+ options={LAYERS}
+ onChange={setLayer}
+ hook="clip-cutout-layer"
+ />
+ {layer === "removal" && <RemovalView clip={clip} />}
+ {layer === "stroke" && <StrokeView clip={clip} />}
+ {layer === "background" && <BackgroundView clip={clip} />}
+ {removalActive(removal) && (
+ <Row label="Original">
+ <button
+ type="button"
+ aria-pressed={peeking}
+ className={cn(
+ "clip-cutout-peek rounded-md border px-2 py-0.5 text-[11.5px] font-medium transition-colors",
+ peeking
+ ? "border-neutral-900 bg-neutral-900 text-white"
+ : "border-input text-muted-foreground hover:text-foreground"
+ )}
+ onClick={() => useEditor.setState({ removalPeek: peeking ? null : clip.id })}
+ >
+ {peeking ? "Hide" : "Show"}
+ </button>
+ </Row>
+ )}
+ {layer === "removal" && <RemovalActions clip={clip} />}
+ </ScrollArea>
+ </div>
+ );
+}
+
+/* ------------------------------------------------------------------ */
+/* Removal */
+/* ------------------------------------------------------------------ */
+
+/** The suggested key: the live frame when a decoder shows the clip, its own
+ * decoded in-point frame otherwise, and green as the visible last resort the
+ * user corrects with the eyedropper. */
+async function keyColorOf(clip: VideoClip): Promise<string> {
+ const live = suggestKeyColor(clip.id);
+ if (live) return live;
+ const asset = useEditor.getState().assets.find((a) => a.id === clip.assetId);
+ return (asset && (await clipKeyColor(asset, clip))) || "#00ff00";
+}
+
+/** The Mode row: every layer's entry into the cutout. Stroke and Background
+ * render it too while no mode is picked, so the pick happens in place. */
+function ModeRow({ clip }: { clip: VideoClip }) {
+ const { commit } = useRemovalWriter(clip);
+ const removal = clip.removal;
+ const mode: RemovalMode | "off" = removal?.mode ?? "off";
+
+ const pickMode = (next: RemovalMode | "off") => {
+ if (next === mode) return;
+ if (next === "off") {
+ commit(undefined);
+ return;
+ }
+ const base: ClipRemoval = { ...removal, mode: next };
+ if (next === "chroma" && !base.chroma) {
+ // The key defaults to the footage's border color. A clip away from the
+ // playhead has no live frame, so its own in-point frame decodes first.
+ void keyColorOf(clip).then((color) => commit({ ...base, chroma: { color } }));
+ return;
+ }
+ if (next === "custom" && !base.seeds) base.seeds = { prompts: [] };
+ commit(base);
+ };
+
+ return (
+ <RowSelect
+ label="Mode"
+ value={mode}
+ options={MODES}
+ onChange={pickMode}
+ hook="clip-removal-mode"
+ />
+ );
+}
+
+function RemovalView({ clip }: { clip: VideoClip }) {
+ const { draft, commit } = useRemovalWriter(clip);
+ const removal = clip.removal;
+ const mode: RemovalMode | "off" = removal?.mode ?? "off";
+
+ return (
+ <>
+ <ModeRow clip={clip} />
+ {mode === "custom" && <CustomControls />}
+ {mode === "chroma" && removal?.chroma && (
+ <ChromaControls removal={removal} draft={draft} commit={commit} />
+ )}
+ </>
+ );
+}
+
+function BakeStatus({ clipId, job }: { clipId: string; job: MatteBakeJob | undefined }) {
+ if (!job) return null;
+ if (job.status === "error")
+ return (
+ <div className="flex items-center justify-between gap-2 pt-1 text-[11.5px] text-muted-foreground">
+ <span className="min-w-0 truncate" title={job.error}>
+ <HostedErrorText error={job.error ?? "The cutout could not be prepared."} />
+ </span>
+ <button
+ type="button"
+ className="clip-removal-retry shrink-0 rounded-md border border-input px-2 py-0.5 font-medium transition-colors hover:text-foreground"
+ onClick={() => retryMatteBake(clipId)}
+ >
+ Retry
+ </button>
+ </div>
+ );
+ return (
+ <div className="flex items-center gap-2 pt-1 text-[11.5px] text-muted-foreground">
+ <Loader2 className="size-3 shrink-0 animate-spin" />
+ <span>{job.quality === "hq" ? "Refining the cutout…" : "Preparing the cutout…"}</span>
+ <span className="ml-auto font-mono tabular-nums">
+ {Math.round(job.progress * 100)}%
+ {job.secondsLeft !== undefined && job.etaAt !== undefined && (
+ <TickingEta secondsLeft={job.secondsLeft} etaAt={job.etaAt} />
+ )}
+ </span>
+ </div>
+ );
+}
+
+/** The countdown between progress steps: each estimate ticks down in real
+ * time from the moment it was stamped until the next one replaces it. */
+function TickingEta({ secondsLeft, etaAt }: { secondsLeft: number; etaAt: number }) {
+ const [now, setNow] = useState(etaAt);
+ useEffect(() => {
+ const id = setInterval(() => setNow(Date.now()), 1000);
+ return () => clearInterval(id);
+ }, []);
+ const left = Math.max(0, secondsLeft - Math.max(0, Math.round((now - etaAt) / 1000)));
+ return `, ${Math.floor(left / 60)}:${String(left % 60).padStart(2, "0")}`;
+}
+
+function CustomControls() {
+ const tool = useBrushUi((s) => s.tool);
+ const size = useBrushUi((s) => s.size);
+ return (
+ <>
+ <RowSelect
+ label="Tool"
+ value={tool}
+ options={BRUSH_TOOLS}
+ onChange={(t) => useBrushUi.getState().setTool(t)}
+ hook="clip-brush-tool"
+ />
+ <Row label="Size">
+ <ValueSlider
+ label="Size"
+ sliderClassName="clip-brush-size data-horizontal:w-24"
+ valueClassName="w-12 text-muted-foreground"
+ value={Math.round(size * 1080)}
+ min={Math.round(BRUSH_SIZE_MIN * 1080)}
+ max={Math.round(BRUSH_SIZE_MAX * 1080)}
+ step={1}
+ format={(v) => `${Math.round(v)}px`}
+ parse={parseNumberInput}
+ onDraft={(v) => useBrushUi.getState().setSize(v / 1080)}
+ onCommit={(v) => useBrushUi.getState().setSize(v / 1080)}
+ />
+ </Row>
+ <p className="pt-1 text-[12px] text-muted-foreground">
+ Paint on the preview to pick what stays.
+ </p>
+ </>
+ );
+}
+
+/** The panel floor: the bake's status line and its actions. The bake runs
+ * only from here — Start (auto) or Apply (custom) marks the removal
+ * requested, so picking a mode by itself never spends compute. */
+function RemovalActions({ clip }: { clip: VideoClip }) {
+ const job = useMatteBakes((s) => s.jobs[clip.id]);
+ const { commit } = useRemovalWriter(clip);
+ const r = clip.removal;
+ if (!r || (r.mode !== "auto" && r.mode !== "custom")) return null;
+ const custom = r.mode === "custom";
+ const hasSelection =
+ custom && !!(r.seeds?.prompts.length || r.seeds?.paint?.length || r.subject?.trim());
+ const running = job?.status === "running";
+ const start = () => {
+ commit({ ...r, requested: true });
+ if (custom) confirmMatteBake(clip.id);
+ else ensureMatteBake(clip.id);
+ };
+ const showStart = !custom && !r.requested && !running;
+ const showCustom = custom && hasSelection && !running;
+ return (
+ <>
+ <BakeStatus clipId={clip.id} job={job} />
+ {(showStart || showCustom) && (
+ <div className="flex items-center justify-end gap-1.5 pt-1">
+ {showCustom && (
+ <button
+ type="button"
+ className="clip-brush-reset shrink-0 rounded-md border border-input px-2 py-0.5 text-[11.5px] font-medium text-muted-foreground transition-colors hover:text-foreground"
+ onClick={() =>
+ commit({
+ ...r,
+ seeds: { prompts: [] },
+ matte: undefined,
+ requested: undefined,
+ })
+ }
+ >
+ Reset
+ </button>
+ )}
+ <button
+ type="button"
+ className={cn(
+ "shrink-0 rounded-md bg-neutral-900 px-2.5 py-0.5 text-[11.5px] font-medium text-white transition-colors hover:bg-neutral-700",
+ custom ? "clip-brush-apply" : "clip-removal-start"
+ )}
+ onClick={start}
+ >
+ {custom ? "Apply" : "Start"}
+ </button>
+ </div>
+ )}
+ </>
+ );
+}
+
+function ChromaControls({
+ removal,
+ draft,
+ commit,
+}: {
+ removal: ClipRemoval;
+ draft: (r: ClipRemoval) => void;
+ commit: (r: ClipRemoval) => void;
+}) {
+ const key = removal.chroma!;
+ const setKey = (patch: Partial<typeof key>, live?: boolean) =>
+ (live ? draft : commit)({ ...removal, chroma: { ...key, ...patch } });
+ const pct = (v: number | undefined, fallback: number) => Math.round((v ?? fallback) * 100);
+ const slider = (
+ label: string,
+ field: "intensity" | "softness" | "spill",
+ fallback: number
+ ) => (
+ <Row label={label}>
+ <ValueSlider
+ label={label}
+ sliderClassName={`clip-chroma-${field} data-horizontal:w-24`}
+ valueClassName="w-9 text-muted-foreground"
+ value={pct(key[field], fallback)}
+ min={0}
+ max={100}
+ step={1}
+ format={(v) => `${Math.round(v)}%`}
+ parse={parseNumberInput}
+ onDraft={(v) => setKey({ [field]: v / 100 }, true)}
+ onCommit={(v) => setKey({ [field]: v / 100 })}
+ />
+ </Row>
+ );
+ return (
+ <>
+ <Row label="Key color">
+ <ColorField
+ value={key.color}
+ label="Key color"
+ onBegin={() => {}}
+ onLive={(hex) => setKey({ color: hex }, true)}
+ onCommit={(hex) => setKey({ color: hex })}
+ />
+ </Row>
+ {slider("Intensity", "intensity", CHROMA_DEFAULT_INTENSITY)}
+ {slider("Softness", "softness", CHROMA_DEFAULT_SOFTNESS)}
+ {slider("Spill", "spill", CHROMA_DEFAULT_SPILL)}
+ </>
+ );
+}
+
+/* ------------------------------------------------------------------ */
+/* Stroke */
+/* ------------------------------------------------------------------ */
+
+/** A blob silhouette drawn into `ctx`, the stand-in subject the style tiles
+ * ink around. */
+function paintBlob(ctx: CanvasRenderingContext2D, w: number, h: number, fill: string) {
+ ctx.fillStyle = fill;
+ ctx.beginPath();
+ ctx.ellipse(w * 0.5, h * 0.62, w * 0.22, h * 0.3, 0, 0, Math.PI * 2);
+ ctx.ellipse(w * 0.5, h * 0.3, w * 0.13, h * 0.17, 0, 0, Math.PI * 2);
+ ctx.fill();
+}
+
+function StrokeTile({
+ style,
+ color,
+ selected,
+ onPick,
+}: {
+ style: StrokeStyleId;
+ color: string;
+ selected: boolean;
+ onPick: () => void;
+}) {
+ const ref = useRef<HTMLCanvasElement | null>(null);
+ useEffect(() => {
+ const canvas = ref.current;
+ const ctx = canvas?.getContext("2d");
+ if (!canvas || !ctx) return;
+ const w = canvas.width;
+ const h = canvas.height;
+ ctx.clearRect(0, 0, w, h);
+ const sil = document.createElement("canvas");
+ sil.width = w;
+ sil.height = h;
+ const sctx = sil.getContext("2d")!;
+ paintBlob(sctx, w, h, "#ffffff");
+ const ink = document.createElement("canvas");
+ ink.width = w;
+ ink.height = h;
+ const ictx = ink.getContext("2d")!;
+ paintStrokeInk(ictx, sil, w, h, { style, color, width: 6 }, 0.6, 1);
+ ictx.globalCompositeOperation = "source-in";
+ ictx.fillStyle = color;
+ ictx.fillRect(0, 0, w, h);
+ paintBlob(ctx, w, h, "#52525b");
+ ctx.globalCompositeOperation = "destination-over";
+ ctx.drawImage(ink, 0, 0);
+ ctx.globalCompositeOperation = "source-over";
+ }, [style, color]);
+ return (
+ <button
+ type="button"
+ aria-pressed={selected}
+ className={cn(
+ `clip-stroke-${style} flex flex-col items-center gap-1 rounded-lg border p-1.5 pb-1 text-[10.5px] font-medium transition-colors`,
+ selected
+ ? "border-neutral-900 text-foreground"
+ : "border-input text-muted-foreground hover:text-foreground"
+ )}
+ onClick={onPick}
+ >
+ <canvas ref={ref} width={96} height={64} className="w-full rounded bg-neutral-950/90" />
+ {STROKE_STYLE_LABELS[style]}
+ </button>
+ );
+}
+
+function StrokeView({ clip }: { clip: VideoClip }) {
+ const { draft, commit } = useRemovalWriter(clip);
+ const removal = clip.removal;
+ if (!removal)
+ return (
+ <>
+ <ModeRow clip={clip} />
+ <p className="pt-1 text-[12px] text-muted-foreground">
+ The stroke draws around what the cutout keeps.
+ </p>
+ </>
+ );
+ const stroke = removal.stroke;
+ const color = stroke?.color ?? "#ffffff";
+ const setStroke = (patch: Partial<NonNullable<typeof stroke>>, live?: boolean) =>
+ (live ? draft : commit)({
+ ...removal,
+ stroke: { style: stroke?.style ?? "solid", color, ...stroke, ...patch },
+ });
+ return (
+ <>
+ <div className="grid grid-cols-3 gap-1.5 pt-1">
+ <button
+ type="button"
+ aria-pressed={!stroke}
+ className={cn(
+ "clip-stroke-none flex flex-col items-center justify-center gap-1 rounded-lg border p-1.5 pb-1 text-[10.5px] font-medium transition-colors",
+ !stroke
+ ? "border-neutral-900 text-foreground"
+ : "border-input text-muted-foreground hover:text-foreground"
+ )}
+ onClick={() => commit({ ...removal, stroke: undefined })}
+ >
+ <span className="grid aspect-[3/2] w-full place-items-center rounded bg-neutral-950/90 text-neutral-500">
+ <Ban className="size-4" />
+ </span>
+ None
+ </button>
+ {STROKE_STYLES.map((s) => (
+ <StrokeTile
+ key={s}
+ style={s}
+ color={color}
+ selected={stroke?.style === s}
+ onPick={() => setStroke({ style: s })}
+ />
+ ))}
+ </div>
+ {stroke && (
+ <>
+ <Row label="Color">
+ <ColorField
+ value={color}
+ onBegin={() => {}}
+ onLive={(hex) => setStroke({ color: hex }, true)}
+ onCommit={(hex) => setStroke({ color: hex })}
+ />
+ </Row>
+ <Row label="Width">
+ <ValueSlider
+ label="Width"
+ sliderClassName="clip-stroke-width data-horizontal:w-24"
+ valueClassName="w-9 text-muted-foreground"
+ value={stroke.width ?? STROKE_DEFAULT_WIDTH}
+ min={1}
+ max={STROKE_WIDTH_MAX}
+ step={1}
+ format={(v) => `${Math.round(v)}`}
+ parse={parseNumberInput}
+ onDraft={(v) => setStroke({ width: v }, true)}
+ onCommit={(v) => setStroke({ width: v })}
+ />
+ </Row>
+ {stroke.style === "offset" && (
+ <>
+ <Row label="Horizontal">
+ <ValueSlider
+ label="Horizontal"
+ sliderClassName="clip-stroke-offset-x data-horizontal:w-24"
+ valueClassName="w-9 text-muted-foreground"
+ value={stroke.offsetX ?? 0}
+ min={-STROKE_OFFSET_MAX}
+ max={STROKE_OFFSET_MAX}
+ step={1}
+ snap={[0]}
+ format={(v) => `${Math.round(v)}`}
+ parse={parseNumberInput}
+ onDraft={(v) => setStroke({ offsetX: v }, true)}
+ onCommit={(v) => setStroke({ offsetX: v || undefined })}
+ />
+ </Row>
+ <Row label="Vertical">
+ <ValueSlider
+ label="Vertical"
+ sliderClassName="clip-stroke-offset-y data-horizontal:w-24"
+ valueClassName="w-9 text-muted-foreground"
+ value={stroke.offsetY ?? 0}
+ min={-STROKE_OFFSET_MAX}
+ max={STROKE_OFFSET_MAX}
+ step={1}
+ snap={[0]}
+ format={(v) => `${Math.round(v)}`}
+ parse={parseNumberInput}
+ onDraft={(v) => setStroke({ offsetY: v }, true)}
+ onCommit={(v) => setStroke({ offsetY: v || undefined })}
+ />
+ </Row>
+ </>
+ )}
+ </>
+ )}
+ </>
+ );
+}
+
+/* ------------------------------------------------------------------ */
+/* Background */
+/* ------------------------------------------------------------------ */
+
+function BackgroundView({ clip }: { clip: VideoClip }) {
+ const { draft, commit } = useRemovalWriter(clip);
+ const removal = clip.removal;
+ const assets = useEditor((s) => s.assets);
+ const images = assets.filter((a) => a.type === "image" && !a.origin);
+ if (!removal)
+ return (
+ <>
+ <ModeRow clip={clip} />
+ <p className="pt-1 text-[12px] text-muted-foreground">
+ The background fills the area the cutout removes.
+ </p>
+ </>
+ );
+ const backdrop = removal.backdrop;
+ const kind = backdrop?.kind ?? "none";
+ const setBackdrop = (b: ClipRemoval["backdrop"], live?: boolean) =>
+ (live ? draft : commit)({ ...removal, backdrop: b });
+ const pill = (id: "none" | "color" | "image", label: string) => (
+ <button
+ key={id}
+ type="button"
+ className={cn(
+ `clip-backdrop-${id} relative flex-1 rounded-md px-1.5 py-1 transition-colors`,
+ kind === id ? "bg-neutral-900 text-white" : "text-muted-foreground hover:text-foreground"
+ )}
+ onClick={() =>
+ setBackdrop(
+ id === "none"
+ ? undefined
+ : id === "color"
+ ? { kind: "color", color: backdrop?.color ?? "#101014" }
+ : { kind: "image", color: backdrop?.color, assetId: backdrop?.assetId ?? images[0]?.id }
+ )
+ }
+ >
+ {label}
+ </button>
+ );
+ return (
+ <>
+ <div className="flex shrink-0 rounded-lg bg-muted p-0.5 text-[11.5px] font-medium">
+ {pill("none", "None")}
+ {pill("color", "Color")}
+ {pill("image", "Image")}
+ </div>
+ {kind === "color" && (
+ <Row label="Color">
+ <ColorField
+ value={backdrop?.color ?? "#101014"}
+ onBegin={() => {}}
+ onLive={(hex) => setBackdrop({ kind: "color", color: hex }, true)}
+ onCommit={(hex) => setBackdrop({ kind: "color", color: hex })}
+ />
+ </Row>
+ )}
+ {kind === "image" &&
+ (images.length === 0 ? (
+ <p className="pt-2 text-[12px] text-muted-foreground">
+ Import an image into Media to use it as the background.
+ </p>
+ ) : (
+ <RowSelect
+ label="Image"
+ value={backdrop?.assetId ?? images[0].id}
+ options={images.map((a) => ({ id: a.id, label: a.name }))}
+ onChange={(id) => setBackdrop({ kind: "image", assetId: id })}
+ hook="clip-backdrop-image"
+ />
+ ))}
+ </>
+ );
+}
diff --git a/site/src/cut/hooks/usePlayback.ts b/site/src/cut/hooks/usePlayback.ts
index 672dbab8..77cf5ebd 100644
--- a/site/src/cut/hooks/usePlayback.ts
+++ b/site/src/cut/hooks/usePlayback.ts
@@ -8,6 +8,7 @@ import {
projectDuration,
useEditor,
} from "@/cut/lib/store";
+import { matteLumaToAlpha } from "@donkeycut/effects-kit";
import { playheadAt, previewAt, setPlayhead, subscribePlayhead } from "@/cut/lib/playhead";
import { assetIsSilent, clipCovers, projectFadeSeconds, rectOf } from "@/cut/lib/types";
import type { ClipSpan, MediaAsset, VideoClip } from "@/cut/lib/types";
@@ -31,6 +32,8 @@ import {
tracing,
} from "@/cut/lib/perfTrace";
import { registerSourceSampler } from "@/cut/lib/previewCanvas";
+import { backdropStill } from "@/cut/lib/backdropStills";
+import { createRasterCanvas, type RasterSurface } from "@/cut/lib/raster";
/**
* The preview engine.
@@ -172,6 +175,16 @@ class Engine {
private playSince = 0;
private wasPlaying = false;
private lastWalkCost = 0;
+ /** Whether the tick being drawn is a playing one — read by the matte
+ * provider, which the compositor calls mid-frame. */
+ private renderPlaying = false;
+ /** Per removal clip: the last decoded matte luma frame converted to alpha,
+ * so the conversion runs once per new frame rather than once per draw. Keyed
+ * by clip — two clips reading the same matte at different times each keep
+ * their own frame — and bounded LRU, so a long session's re-bakes and
+ * deleted clips never pile up canvases. */
+ private matteAlpha = new Map<string, { ts: number; canvas: RasterSurface }>();
+ private static readonly MATTE_ALPHA_MAX = 8;
private unsubscribe: () => void;
private unwatch: () => void;
@@ -179,6 +192,11 @@ class Engine {
constructor(private canvas: HTMLCanvasElement) {
this.comp = new FrameCompositor(canvas);
+ this.comp.removalMatteProvider = (clip, at) => this.matteFor(clip, at);
+ // A backdrop still decodes out of band; the frame that finds it missing
+ // draws without it and the landing repaints.
+ this.comp.backdropImageProvider = (assetId) =>
+ backdropStill(useEditor.getState().assets, assetId, () => this.wake());
this.tick = this.tick.bind(this);
// A moved playhead and an edited document both change the picture. Either
// one wakes the loop; nothing else does, so an idle editor costs nothing.
@@ -310,6 +328,57 @@ class Engine {
return src.frameAt(sourceTimeOf(clip, previewAt()), clip.in, clip.out)?.image ?? null;
}
+ /**
+ * The alpha matte for a removal clip at timeline time `t`: the baked matte
+ * video read like any other source, its luma turned into alpha. Null while
+ * nothing has decoded yet — the picture draws plain until the matte lands.
+ */
+ private matteFor(clip: VideoClip, at: number): CanvasImageSource | null {
+ const m = clip.removal?.matte;
+ if (!m) return null;
+ const s = useEditor.getState();
+ const asset = s.assets.find((a) => a.id === m.assetId);
+ if (!asset) return null;
+ const dur = Math.max(0.001, asset.duration);
+ const mt = Math.max(0, Math.min(sourceTimeOf(clip, at) - m.in, dur - 0.001));
+ // Coverage tolerates far less resolution than the picture, and the luma →
+ // alpha read below is a per-frame pixel pass; capping the matte's decode
+ // keeps that pass off the frame budget.
+ const src = this.pool.get(
+ mappingKey(asset.id, clipSpeed(clip), m.in, clip.start),
+ asset,
+ Math.min(this.decodeHeight(), 480)
+ );
+ this.used.add(src);
+ src.want(mt, this.renderPlaying);
+ const frame = src.frameAt(mt, 0, dur);
+ if (!frame) return null;
+ const held = this.matteAlpha.get(clip.id);
+ // Re-insert on every touch so the cap below drops the clip drawn longest
+ // ago.
+ this.matteAlpha.delete(clip.id);
+ if (held && held.ts === frame.timestamp) {
+ this.matteAlpha.set(clip.id, held);
+ return held.canvas as CanvasImageSource;
+ }
+ const canvas = held?.canvas ?? createRasterCanvas(frame.width, frame.height);
+ if (canvas.width !== frame.width) canvas.width = frame.width;
+ if (canvas.height !== frame.height) canvas.height = frame.height;
+ const ctx = canvas.getContext("2d") as CanvasRenderingContext2D | null;
+ if (!ctx) return null;
+ ctx.clearRect(0, 0, canvas.width, canvas.height);
+ ctx.drawImage(frame.image, 0, 0, canvas.width, canvas.height);
+ const img = ctx.getImageData(0, 0, canvas.width, canvas.height);
+ matteLumaToAlpha(img.data);
+ ctx.putImageData(img, 0, 0);
+ this.matteAlpha.set(clip.id, { ts: frame.timestamp, canvas });
+ if (this.matteAlpha.size > Engine.MATTE_ALPHA_MAX) {
+ const oldest = this.matteAlpha.keys().next().value;
+ if (oldest !== undefined) this.matteAlpha.delete(oldest);
+ }
+ return canvas as CanvasImageSource;
+ }
+
/**
* The picture a clip shows at timeline time `t`.
*
@@ -591,6 +660,8 @@ class Engine {
// and every letterbox below paints it, so a cut with no footage at all
// still plays a picture.
this.comp.background = s.background;
+ this.comp.removalBypass = s.removalPeek;
+ this.renderPlaying = playing;
this.pool.beginFrame();
this.used.clear();
diff --git a/site/src/cut/lib/aiContext.ts b/site/src/cut/lib/aiContext.ts
index 181bd6f4..ceaae594 100644
--- a/site/src/cut/lib/aiContext.ts
+++ b/site/src/cut/lib/aiContext.ts
@@ -3,6 +3,7 @@
import { hasOverlayAnim } from "@donkeycut/effects-kit";
import { chatOwner } from "./chatAssets";
import { useGenerate } from "./generate";
+import { useMatteBakes } from "./removal/bakeJobs";
import { getClipSpans, overlayLayers, resolveTransitions, totalDuration, useEditor } from "./store";
import { playheadAt, skimAt } from "./playhead";
import { cueWordCount } from "./cueChunk";
@@ -66,6 +67,46 @@ function clipEffects(clip: VideoClip) {
: {}),
...color,
...(clip.boxStyle ? { boxStyle: clip.boxStyle } : {}),
+ ...describeRemoval(clip),
+ };
+}
+
+/** The clip's cutout state: mode, whether its AI matte is ready (and at which
+ * tier), a live bake's progress, and the stroke/background riding on it —
+ * what set_removal claims must be grounded in. */
+function describeRemoval(clip: VideoClip) {
+ const rm = clip.removal;
+ if (!rm) return {};
+ const job = useMatteBakes.getState().jobs[clip.id];
+ return {
+ removal: {
+ mode: rm.mode,
+ ...(rm.subject ? { subject: rm.subject } : {}),
+ ...(rm.mode === "chroma" && rm.chroma ? { chroma: rm.chroma } : {}),
+ ...(rm.mode === "auto" || rm.mode === "custom"
+ ? {
+ matte: rm.matte ? `ready (${rm.matte.quality === "hq" ? "quality" : "quick"})` : "none",
+ ...(job
+ ? {
+ bake:
+ job.status === "running"
+ ? `baking ${job.quality === "hq" ? "quality" : "quick"} (${Math.round(job.progress * 100)}%)`
+ : `error: ${job.error ?? "failed"}`,
+ }
+ : {}),
+ }
+ : {}),
+ ...(rm.stroke
+ ? {
+ stroke: {
+ style: rm.stroke.style,
+ color: rm.stroke.color,
+ ...(rm.stroke.width !== undefined ? { width: rm.stroke.width } : {}),
+ },
+ }
+ : {}),
+ ...(rm.backdrop && rm.backdrop.kind !== "none" ? { background: rm.backdrop } : {}),
+ },
};
}
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index 09436fd0..28d984ec 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -29,6 +29,11 @@ import {
type OverlayAnim,
type OverlayAnimStyle,
type OverlayLoopStyle,
+ STROKE_OFFSET_MAX,
+ STROKE_STYLES,
+ STROKE_WIDTH_MAX,
+ type RemovalStroke,
+ type StrokeStyleId,
} from "@donkeycut/effects-kit";
import type { AiPanelToolName } from "@/cut/components/AiPanel.tools";
import type { OverlayAnimationToolName } from "@/cut/components/AnimationTiles.tools";
@@ -41,6 +46,7 @@ import type { ImageGenToolName } from "@/cut/components/ImageGenPanel.tools";
import type { InspectorToolName } from "@/cut/components/Inspector.tools";
import type { LibraryToolName } from "@/cut/components/LibraryView.tools";
import type { PreviewToolName } from "@/cut/components/Preview.tools";
+import type { RemovalToolName } from "@/cut/components/RemovalPanel.tools";
import type { SceneToolName } from "@/cut/components/SceneCard.tools";
import type { SidePanelToolName } from "@/cut/components/SidePanel.tools";
import type { StockToolName } from "@/cut/components/StockPanels.tools";
@@ -88,6 +94,13 @@ import { convertAssetToMp4 } from "./mediaConvert";
import { isLottieAsset } from "./lottieAssets";
import { AIR, MIN_AIR, placeEdge, REACH, settleJoint, type Pace, type PlacedEdge } from "./cutRefine";
import { requestSidePanel } from "./panelRequest";
+import {
+ confirmMatteBake,
+ ensureMatteBake,
+ matteBakesAvailable,
+ useMatteBakes,
+} from "./removal/bakeJobs";
+import { clipKeyColor, suggestKeyColor } from "./removal/keyColor";
import { blobToInlineAudio, refToInlineAudio, visualRefs, type InlineImage } from "./refMedia";
import { characterPrompt, stockAspectDims, stockTitle } from "./stock";
import { STOCK_IMAGES } from "./stockManifest";
@@ -341,6 +354,7 @@ type BrowserToolName = Exclude<
| LibraryToolName
| OverlayAnimationToolName
| PreviewToolName
+ | RemovalToolName
| SceneToolName
| SidePanelToolName
| StockToolName
@@ -2045,6 +2059,146 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
return { id: clip.id, boxStyle: boxStyle ?? null };
},
+ set_removal: async (s, input) => {
+ const clip = requireItem(s.clips, input.clipId, "video clip");
+ const mode = String(input.mode ?? "");
+ if (mode === "off") {
+ s.updateClip(clip.id, { removal: undefined });
+ return { id: clip.id, removal: null };
+ }
+ const bakeNote = matteBakesAvailable
+ ? "The cutout matte is baking in the background — wait_for_renders reports when it lands."
+ : "The cutout is set; its matte bakes the next time the project is open in the editor.";
+ if (mode === "auto") {
+ // The tool call is the explicit start, so the bake is requested here
+ // the way the panel's Start button requests it.
+ s.updateClip(clip.id, { removal: { ...clip.removal, mode: "auto", requested: true } });
+ ensureMatteBake(clip.id);
+ return { id: clip.id, mode, note: bakeNote };
+ }
+ if (mode === "custom") {
+ const subject = typeof input.subject === "string" ? input.subject.trim() : "";
+ if (!subject || subject.length > 100)
+ throw new ToolError('mode "custom" needs `subject` — what to keep, in a few words.');
+ // A named subject replaces any painted selection: the seeds point at
+ // whatever the user brushed before, and the words are the new ask.
+ s.updateClip(clip.id, {
+ removal: {
+ ...clip.removal,
+ mode: "custom",
+ subject,
+ seeds: { prompts: [] },
+ requested: true,
+ },
+ });
+ confirmMatteBake(clip.id);
+ return { id: clip.id, mode, subject, note: bakeNote };
+ }
+ if (mode !== "chroma")
+ throw new ToolError('mode must be "off", "auto", "custom", or "chroma".');
+ const prev = clip.removal?.chroma;
+ const explicit =
+ typeof input.color === "string" && /^#[0-9a-f]{6}$/i.test(input.color)
+ ? input.color.toLowerCase()
+ : null;
+ // The default key comes from the clip's own footage: the live frame
+ // when a decoder shows it, its decoded in-point frame when the clip is
+ // away from the playhead or the turn runs headless.
+ let color = explicit ?? prev?.color ?? suggestKeyColor(clip.id);
+ if (!color) {
+ const asset = s.assets.find((a) => a.id === clip.assetId);
+ color = asset ? await clipKeyColor(asset, clip) : null;
+ }
+ if (!color)
+ throw new ToolError(
+ "The clip's picture could not be read for a key suggestion — pass `color` as #rrggbb."
+ );
+ const chroma = {
+ color,
+ ...(isNum(input.intensity)
+ ? { intensity: clamp(input.intensity, 0, 1) }
+ : prev?.intensity !== undefined && { intensity: prev.intensity }),
+ ...(isNum(input.softness)
+ ? { softness: clamp(input.softness, 0, 1) }
+ : prev?.softness !== undefined && { softness: prev.softness }),
+ ...(isNum(input.spill)
+ ? { spill: clamp(input.spill, 0, 1) }
+ : prev?.spill !== undefined && { spill: prev.spill }),
+ };
+ s.updateClip(clip.id, { removal: { ...clip.removal, mode: "chroma", chroma } });
+ return { id: clip.id, mode, chroma };
+ },
+
+ set_removal_stroke: (s, input) => {
+ const clip = requireItem(s.clips, input.clipId, "video clip");
+ const removal = clip.removal;
+ if (!removal) throw new ToolError("The clip has no cutout yet — call set_removal first.");
+ const style = String(input.style ?? "");
+ if (style === "none") {
+ s.updateClip(clip.id, { removal: { ...removal, stroke: undefined } });
+ return { id: clip.id, stroke: null };
+ }
+ if (!(STROKE_STYLES as readonly string[]).includes(style))
+ throw new ToolError(`style must be one of ${[...STROKE_STYLES, "none"].join(", ")}.`);
+ const prev = removal.stroke;
+ // Canvas silently ignores an invalid fillStyle, so a bad color would
+ // store and draw as whatever came before it.
+ if (typeof input.color === "string" && input.color && !/^#[0-9a-f]{6}$/i.test(input.color))
+ throw new ToolError("color must be #rrggbb.");
+ const stroke: RemovalStroke = {
+ style: style as StrokeStyleId,
+ color:
+ typeof input.color === "string" && input.color
+ ? input.color.toLowerCase()
+ : prev?.color ?? "#ffffff",
+ ...(isNum(input.width)
+ ? { width: clamp(input.width, 1, STROKE_WIDTH_MAX) }
+ : prev?.width !== undefined && { width: prev.width }),
+ ...(isNum(input.offset_x)
+ ? input.offset_x !== 0 && {
+ offsetX: clamp(input.offset_x, -STROKE_OFFSET_MAX, STROKE_OFFSET_MAX),
+ }
+ : prev?.offsetX !== undefined && { offsetX: prev.offsetX }),
+ ...(isNum(input.offset_y)
+ ? input.offset_y !== 0 && {
+ offsetY: clamp(input.offset_y, -STROKE_OFFSET_MAX, STROKE_OFFSET_MAX),
+ }
+ : prev?.offsetY !== undefined && { offsetY: prev.offsetY }),
+ };
+ s.updateClip(clip.id, { removal: { ...removal, stroke } });
+ return { id: clip.id, stroke };
+ },
+
+ set_removal_background: (s, input) => {
+ const clip = requireItem(s.clips, input.clipId, "video clip");
+ const removal = clip.removal;
+ if (!removal) throw new ToolError("The clip has no cutout yet — call set_removal first.");
+ const kind = String(input.kind ?? "");
+ if (kind === "none") {
+ s.updateClip(clip.id, { removal: { ...removal, backdrop: undefined } });
+ return { id: clip.id, background: null };
+ }
+ if (kind === "color") {
+ if (typeof input.color === "string" && input.color && !/^#[0-9a-f]{6}$/i.test(input.color))
+ throw new ToolError("color must be #rrggbb.");
+ const color =
+ typeof input.color === "string" && input.color
+ ? input.color.toLowerCase()
+ : removal.backdrop?.color ?? "#101014";
+ s.updateClip(clip.id, { removal: { ...removal, backdrop: { kind: "color", color } } });
+ return { id: clip.id, background: { kind, color } };
+ }
+ if (kind !== "image") throw new ToolError('kind must be "none", "color", or "image".');
+ const assetId = typeof input.asset_id === "string" ? input.asset_id : "";
+ const bgAsset = s.assets.find((a) => a.id === assetId);
+ if (bgAsset?.type !== "image")
+ throw new ToolError(
+ assetId ? `No image asset with id ${assetId}.` : "asset_id is required for kind \"image\"."
+ );
+ s.updateClip(clip.id, { removal: { ...removal, backdrop: { kind: "image", assetId } } });
+ return { id: clip.id, background: { kind, assetId } };
+ },
+
freeze_frame: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
@@ -2298,8 +2452,13 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
(j) => j.kind === "video" && j.status === "running" && j.projectId === projectId
)
.map((j) => j.id);
- if (watched.length === 0) {
- return { renders: [], note: "No renders in flight — check `renders` in the state for recent outcomes." };
+ // Cutout matte bakes settle here too, so "the background is removed"
+ // waits for the matte the claim rests on.
+ const watchedBakes = Object.entries(useMatteBakes.getState().jobs)
+ .filter(([, j]) => j.status === "running")
+ .map(([clipId]) => clipId);
+ if (watched.length === 0 && watchedBakes.length === 0) {
+ return { renders: [], note: "No renders or cutout bakes in flight — check `renders` in the state for recent outcomes." };
}
// Poll the job store rather than the settlement promises: it also tracks
// renders another tab owns. Stay under the tool bridge's 2-minute cap;
@@ -2307,7 +2466,11 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
const deadline = Date.now() + 100_000;
while (Date.now() < deadline) {
const jobs = useGenerate.getState().jobs;
- if (!watched.some((id) => jobs.find((j) => j.id === id)?.status === "running")) break;
+ const bakes = useMatteBakes.getState().jobs;
+ const busy =
+ watched.some((id) => jobs.find((j) => j.id === id)?.status === "running") ||
+ watchedBakes.some((id) => bakes[id]?.status === "running");
+ if (!busy) break;
await new Promise((r) => setTimeout(r, 1000));
}
const jobs = useGenerate.getState().jobs;
@@ -2323,9 +2486,26 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
...(j.status === "error" && j.error ? { error: j.error } : {}),
};
});
- const stillRunning = renders.filter((x) => x.status === "running").length;
+ const bakes = useMatteBakes.getState().jobs;
+ const cutouts = watchedBakes.map((clipId) => {
+ const j = bakes[clipId];
+ if (!j) {
+ const m = cur.clips.find((c) => c.id === clipId)?.removal?.matte;
+ return { clipId, status: "done" as const, ...(m ? { quality: m.quality } : {}) };
+ }
+ return {
+ clipId,
+ status: j.status,
+ quality: j.quality,
+ ...(j.error ? { error: j.error } : {}),
+ };
+ });
+ const stillRunning =
+ renders.filter((x) => x.status === "running").length +
+ cutouts.filter((c) => c.status === "running").length;
return {
renders,
+ ...(cutouts.length > 0 ? { cutouts } : {}),
...(stillRunning > 0
? { note: `${stillRunning} still rendering — call wait_for_renders again to keep waiting.` }
: {}),
diff --git a/site/src/cut/lib/assetRef.ts b/site/src/cut/lib/assetRef.ts
index 384fa57b..af77837d 100644
--- a/site/src/cut/lib/assetRef.ts
+++ b/site/src/cut/lib/assetRef.ts
@@ -447,11 +447,13 @@ const EMPTY_LIBRARY: LibraryData = { assets: [], folders: [], templates: [] };
* keyframes, takes — visible nowhere the user works, so they don't become
* candidates; a take placed on the timeline is reachable as its clip (@c2).
* Uploaded fonts are project assets too, but there is nothing to say about a
- * typeface file in a prompt, so they stay out of the handle numbering. */
+ * typeface file in a prompt, so they stay out of the handle numbering. Baked
+ * mattes (origin "matte") are a cutout's internals the same way — numbering
+ * them would shift every later @v handle when a bake lands. */
export function projectRefs(assets: MediaAsset[]): AssetRef[] {
const counters = { v: 0, i: 0, a: 0 };
return assets
- .filter((a) => a.origin !== "chat" && a.type !== "font")
+ .filter((a) => a.origin !== "chat" && a.origin !== "matte" && a.type !== "font")
.map((a) => {
const prefix = a.type === "image" ? "i" : a.type === "video" ? "v" : "a";
counters[prefix] += 1;
diff --git a/site/src/cut/lib/backdropStills.ts b/site/src/cut/lib/backdropStills.ts
new file mode 100644
index 00000000..7971e7ad
--- /dev/null
+++ b/site/src/cut/lib/backdropStills.ts
@@ -0,0 +1,77 @@
+"use client";
+
+/**
+ * Decoded stills for removal backdrops. A backdrop fill names an image asset;
+ * the compositor asks for its pixels synchronously, mid-frame, so the decode
+ * happens once out of band and the answer is a cache read. Both hosts share
+ * it: the preview asks and repaints when the decode lands, the export awaits
+ * the load up front and then reads the same cache.
+ */
+
+import { decodeRasterImageUrl, type RasterImage } from "./raster";
+import type { MediaAsset } from "./types";
+
+/** Decoded backdrops kept at once; a stale one re-decodes on demand. */
+const CACHE_MAX = 12;
+
+/** How long a failed decode rests before the next attempt. A failure is
+ * usually transient — an expired signed URL, an asset mid-upload — so it
+ * never caches: it just holds off long enough to keep a repainting preview
+ * from hammering the fetch. */
+const RETRY_MS = 5_000;
+
+const cache = new Map<string, RasterImage>();
+const failedAt = new Map<string, number>();
+const loading = new Map<string, Promise<RasterImage | null>>();
+
+function remember(assetId: string, img: RasterImage) {
+ cache.delete(assetId);
+ cache.set(assetId, img);
+ if (cache.size > CACHE_MAX) {
+ const oldest = cache.keys().next().value;
+ if (oldest !== undefined) cache.delete(oldest);
+ }
+}
+
+/** Start (or join) the decode of one backdrop asset. */
+export function loadBackdropStill(asset: MediaAsset): Promise<RasterImage | null> {
+ const hit = cache.get(asset.id);
+ if (hit) return Promise.resolve(hit);
+ const running = loading.get(asset.id);
+ if (running) return running;
+ const job = decodeRasterImageUrl(asset.url)
+ .then((img) => {
+ if (img) {
+ remember(asset.id, img);
+ failedAt.delete(asset.id);
+ } else {
+ failedAt.set(asset.id, Date.now());
+ }
+ return img;
+ })
+ .finally(() => loading.delete(asset.id));
+ loading.set(asset.id, job);
+ return job;
+}
+
+/**
+ * The decoded backdrop, or null while it is still decoding (the decode is
+ * kicked off and `onLand` fires when it arrives, so the caller can repaint).
+ * A failed decode rests briefly and then tries again on the next ask.
+ */
+export function backdropStill(
+ assets: MediaAsset[],
+ assetId: string,
+ onLand?: () => void
+): CanvasImageSource | null {
+ const hit = cache.get(assetId);
+ if (hit) return hit.source;
+ const failed = failedAt.get(assetId);
+ if (failed !== undefined && Date.now() - failed < RETRY_MS) return null;
+ const asset = assets.find((a) => a.id === assetId);
+ if (!asset) return null;
+ void loadBackdropStill(asset).then((img) => {
+ if (img) onLand?.();
+ });
+ return null;
+}
diff --git a/site/src/cut/lib/canvasVideo.ts b/site/src/cut/lib/canvasVideo.ts
new file mode 100644
index 00000000..27ce230c
--- /dev/null
+++ b/site/src/cut/lib/canvasVideo.ts
@@ -0,0 +1,91 @@
+"use client";
+
+/**
+ * One canvas-fed H.264 encode — the ceremony behind every baked video the
+ * removal suite writes: mattes, tracker segments, keyed layers. The caller
+ * draws each frame onto `canvas` and calls `add`; `finish` closes the stream
+ * and hands back the file. Throws when no encoder or drawing surface is
+ * available.
+ */
+
+import {
+ BufferTarget,
+ CanvasSource,
+ getFirstEncodableVideoCodec,
+ Mp4OutputFormat,
+ Output,
+ Quality,
+} from "mediabunny";
+import { VIDEO_CODECS } from "./exportRender";
+import { createRasterCanvas, type RasterSurface } from "./raster";
+
+export interface CanvasVideoEncoder {
+ canvas: RasterSurface;
+ ctx: CanvasRenderingContext2D;
+ /** Stage the canvas as the frame at `t` seconds, held for `dur` seconds. */
+ add(t: number, dur: number): Promise<void>;
+ /** Close the stream and hand back the file. */
+ finish(): Promise<Blob>;
+ /** Abandon the encode (a cancelled bake). */
+ cancel(): Promise<void>;
+}
+
+export async function openCanvasVideo(opts: {
+ width: number;
+ height: number;
+ fps: number;
+ frames: number;
+ bitrate: number;
+ /** The frame loop reads pixels back off the canvas. */
+ readback?: boolean;
+}): Promise<CanvasVideoEncoder> {
+ const codec = await getFirstEncodableVideoCodec(VIDEO_CODECS, {
+ width: opts.width,
+ height: opts.height,
+ });
+ if (!codec) throw new Error("No video encoder is available.");
+ const canvas = createRasterCanvas(opts.width, opts.height);
+ const ctx = canvas.getContext(
+ "2d",
+ opts.readback ? { willReadFrequently: true } : undefined
+ ) as CanvasRenderingContext2D | null;
+ if (!ctx) throw new Error("No drawing surface for the encode.");
+ const target = new BufferTarget();
+ const output = new Output({ format: new Mp4OutputFormat(), target });
+ const source = new CanvasSource(canvas as HTMLCanvasElement, {
+ codec,
+ quality: new Quality({ bitrate: opts.bitrate }),
+ });
+ output.addVideoTrack(source, {
+ frameRate: opts.fps,
+ maximumPacketCount: opts.frames + 8,
+ });
+ await output.start();
+ return {
+ canvas,
+ ctx,
+ add: (t, dur) => source.add(t, dur),
+ finish: async () => {
+ await output.finalize();
+ const buffer = target.buffer;
+ if (!buffer) throw new Error("The encode produced nothing.");
+ return new Blob([buffer], { type: "video/mp4" });
+ },
+ cancel: () => output.cancel().catch(() => {}),
+ };
+}
+
+/** Even-dimensioned frame size for a source scaled down to `maxShort` on its
+ * short side — even because the encoder's 4:2:0 subsampling needs it, capped
+ * at 1 so a small source keeps its size. */
+export function scaledEvenSize(
+ srcW: number,
+ srcH: number,
+ maxShort: number
+): { w: number; h: number } {
+ const scale = Math.min(1, maxShort / Math.max(1, Math.min(srcW, srcH)));
+ return {
+ w: Math.max(2, Math.round((srcW * scale) / 2) * 2),
+ h: Math.max(2, Math.round((srcH * scale) / 2) * 2),
+ };
+}
diff --git a/site/src/cut/lib/chromaGpu.ts b/site/src/cut/lib/chromaGpu.ts
new file mode 100644
index 00000000..d347b353
--- /dev/null
+++ b/site/src/cut/lib/chromaGpu.ts
@@ -0,0 +1,89 @@
+/**
+ * GPU chroma key for the preview and in-tab export: the clip's frame renders
+ * through a WebGL2 pass that repeats the kit's reference math —
+ * `chromaAlphaInto` in removal.ts — formula for formula, so the shader and
+ * the CPU fallback key a frame to the same alpha. The canvas comes from the
+ * raster seam; under a headless factory WebGL2 is absent and callers fall
+ * back to the CPU pass.
+ */
+
+import { CHROMA_SPILL_REACH, chromaParams, type ChromaKey } from "@donkeycut/effects-kit";
+import { gpuPassSingleton, uploadPassSource } from "./gpuPass";
+import type { RasterSurface } from "./raster";
+
+// BT.709 luma and the kit's Cb/Cr normalization; smoothstep matches the CPU
+// pass's cubic.
+const FRAG = `#version 300 es
+precision highp float;
+uniform sampler2D uSrc;
+uniform vec2 uKey; // key cb, cr
+uniform vec2 uDir; // unit key chroma direction
+uniform vec3 uRange; // near, far, spillFar
+uniform float uSpill;
+in vec2 vUv;
+out vec4 outColor;
+
+void main() {
+ vec4 s = texture(uSrc, vUv);
+ float y = dot(s.rgb, vec3(0.2126, 0.7152, 0.0722));
+ float cb = (s.b - y) / 1.8556;
+ float cr = (s.r - y) / 1.5748;
+ float d = distance(vec2(cb, cr), uKey);
+ float a = smoothstep(uRange.x, uRange.y, d);
+ vec3 rgb = s.rgb;
+ if (uSpill > 0.0 && d < uRange.z) {
+ float proj = cb * uDir.x + cr * uDir.y;
+ if (proj > 0.0) {
+ float reach = 1.0 - clamp((d - uRange.x) / (uRange.z - uRange.x), 0.0, 1.0);
+ float cut = proj * uSpill * (reach * reach * (3.0 - 2.0 * reach));
+ float ncb = cb - uDir.x * cut;
+ float ncr = cr - uDir.y * cut;
+ float nr = y + 1.5748 * ncr;
+ float nb = y + 1.8556 * ncb;
+ float ng = (y - 0.2126 * nr - 0.0722 * nb) / 0.7152;
+ rgb = clamp(vec3(nr, ng, nb), 0.0, 1.0);
+ }
+ }
+ outColor = vec4(rgb, s.a * a);
+}`;
+
+const acquire = gpuPassSingleton(FRAG, ({ gl, program }) => {
+ const keyLoc = gl.getUniformLocation(program, "uKey");
+ const dirLoc = gl.getUniformLocation(program, "uDir");
+ const rangeLoc = gl.getUniformLocation(program, "uRange");
+ const spillLoc = gl.getUniformLocation(program, "uSpill");
+ if (!keyLoc || !dirLoc || !rangeLoc || !spillLoc) return null;
+ return { keyLoc, dirLoc, rangeLoc, spillLoc };
+});
+
+/**
+ * Render `source` through the key and return a canvas holding the keyed
+ * frame, or null when WebGL2 is unavailable (headless) — callers then use
+ * the kit's CPU pass. The returned canvas is shared and valid until the
+ * next call.
+ */
+export function chromaKeyGpu(
+ source: CanvasImageSource,
+ w: number,
+ h: number,
+ key: ChromaKey
+): RasterSurface | null {
+ const held = acquire(w, h);
+ if (!held) return null;
+ const { pass, ext } = held;
+ const { gl } = pass;
+ try {
+ const p = chromaParams(key);
+ uploadPassSource(pass, source);
+ gl.uniform2f(ext.keyLoc, p.keyCb, p.keyCr);
+ gl.uniform2f(ext.dirLoc, p.dirCb, p.dirCr);
+ gl.uniform3f(ext.rangeLoc, p.near, p.far, p.near + (p.far - p.near) * CHROMA_SPILL_REACH);
+ gl.uniform1f(ext.spillLoc, p.spill);
+ gl.clearColor(0, 0, 0, 0);
+ gl.clear(gl.COLOR_BUFFER_BIT);
+ gl.drawArrays(gl.TRIANGLES, 0, 3);
+ return pass.canvas;
+ } catch {
+ return null;
+ }
+}
diff --git a/site/src/cut/lib/composite.ts b/site/src/cut/lib/composite.ts
index 7e1ddffe..68e42fff 100644
--- a/site/src/cut/lib/composite.ts
+++ b/site/src/cut/lib/composite.ts
@@ -19,7 +19,8 @@
* it was reached by playing there or by rendering the 135th frame.
*/
-import { applyLutToImageData, applyMaskToCanvas, buildGradeLut, gradeKey, gradeNeedsLut, gradeTint, gradeToCssFilter, grainTile, isNeutralGrade, lookCssFilter, lookPost, maskComposite, type GradeLut } from "@donkeycut/effects-kit";
+import { applyLutToImageData, applyMaskToCanvas, buildGradeLut, chromaAlphaInto, gradeKey, gradeNeedsLut, gradeTint, gradeToCssFilter, grainTile, isNeutralGrade, lookCssFilter, lookPost, maskComposite, paintStrokeInk, removalActive, type GradeLut } from "@donkeycut/effects-kit";
+import { chromaKeyGpu } from "./chromaGpu";
import { applyLutGpu } from "./gradeGpu";
import { createRasterCanvas } from "./raster";
import { clipCovers, clipPosed, clipPoseAt, clipZoom, contentRect, DEFAULT_BACKGROUND, isFullRect, rectOf, shadowInk } from "./types";
@@ -133,6 +134,18 @@ export class FrameCompositor {
private poseScratch: Surface | null = null;
/** Where a shadow is cast before the picture goes down over it. */
private shadowScratch: Surface | null = null;
+ /** The removal pass: the keyed picture builds in `removalScratch`, its
+ * silhouette in `removalSil`, the stroke's ink in `removalInk`, and an AI
+ * matte lands in `removalMatte` on its way to the alpha multiply. */
+ private removalScratch: Surface | null = null;
+ private removalMatte: Surface | null = null;
+ private removalSil: Surface | null = null;
+ private removalInk: Surface | null = null;
+ /** The piece renderer's look bake: the post passes run over a copy in
+ * `removalLookA`, then land back inside the layer's own coverage through
+ * `removalLookB`. */
+ private removalLookA: Surface | null = null;
+ private removalLookB: Surface | null = null;
/** Where a subject-masked clip's person matte comes from: the host hands a
* reader over the canvas as it stands (the layers beneath the clip), so
@@ -144,6 +157,19 @@ export class FrameCompositor {
| ((at: number) => { alpha: CanvasImageSource | null } | null)
| null = null;
+ /** Where a removal clip's baked AI matte comes from: an alpha image
+ * (subject opaque, backdrop transparent — the host converts a decoded
+ * luma frame) covering the clip's source frame at `at` timeline seconds.
+ * Null = no matte yet; the picture draws plain until the bake lands. */
+ removalMatteProvider: ((clip: VideoClip, at: number) => CanvasImageSource | null) | null = null;
+
+ /** Resolves a backdrop fill's image asset to a drawable picture. */
+ backdropImageProvider: ((assetId: string) => CanvasImageSource | null) | null = null;
+
+ /** A clip id whose removal is bypassed this frame — the panel's eye toggle
+ * showing the original picture. */
+ removalBypass: string | null = null;
+
/**
* The frame's own color, behind everything drawn into it. The project owns
* it, so the caller writes it before each frame; black keeps a compositor
@@ -182,7 +208,13 @@ export class FrameCompositor {
| "layerScratch"
| "maskScratch"
| "poseScratch"
- | "shadowScratch",
+ | "shadowScratch"
+ | "removalScratch"
+ | "removalMatte"
+ | "removalSil"
+ | "removalInk"
+ | "removalLookA"
+ | "removalLookB",
w: number,
h: number
): { surface: Surface; resized: boolean } {
@@ -305,6 +337,155 @@ export class FrameCompositor {
return scratch;
}
+ /**
+ * The clip's graded picture with its background removal applied: the key
+ * multiplies an alpha into the frame (a chroma key computed here, or the
+ * baked AI matte the host provides), the stroke's ink goes down behind the
+ * silhouette, and the backdrop fills in behind everything. Runs in the
+ * clip's source pixel space, before the fit/fill draw, so the keyed layer
+ * rides zoom, pan, masks, poses and transitions like any other picture.
+ */
+ /** The keyed layer alone, in source pixel space — what the ffmpeg export's
+ * piece renderer encodes so the engine composites the very pixels the
+ * preview draws. `bakeLookPost` folds the look's post passes (grain,
+ * vignette, glow, washes) into the layer for the overlay pieces, where the
+ * engine applies no look to an alpha layer: the passes run over a copy and
+ * land back inside the layer's own coverage, so the keyed-out hole stays
+ * transparent. */
+ removedLayer(
+ frame: Extract<Frame, { kind: "ready" }>,
+ clip: VideoClip | undefined,
+ at: number,
+ opts: { bakeLookPost?: boolean } = {}
+ ): CanvasImageSource {
+ const out = this.removedSource(frame, clip, at);
+ if (!opts.bakeLookPost || !lookPost(clip?.look, clip?.lookAmount)) return out;
+ const w = frame.width;
+ const h = frame.height;
+ const { surface: processed } = this.scratch("removalLookA", w, h);
+ const pctx = processed.getContext("2d") as Ctx | null;
+ const { surface: baked } = this.scratch("removalLookB", w, h);
+ const bctx = baked.getContext("2d") as Ctx | null;
+ if (!pctx || !bctx) return out;
+ for (const c of [pctx, bctx]) {
+ c.setTransform(1, 0, 0, 1, 0, 0);
+ c.globalAlpha = 1;
+ c.globalCompositeOperation = "source-over";
+ c.clearRect(0, 0, w, h);
+ c.drawImage(out, 0, 0, w, h);
+ }
+ this.applyLookPost(clip, 0, 0, w, h, 1, at, { canvas: processed, ctx: pctx });
+ bctx.globalCompositeOperation = "source-atop";
+ bctx.drawImage(processed, 0, 0, w, h);
+ bctx.globalCompositeOperation = "source-over";
+ return baked as CanvasImageSource;
+ }
+
+ private removedSource(
+ frame: Extract<Frame, { kind: "ready" }>,
+ clip: VideoClip | undefined,
+ at: number
+ ): CanvasImageSource {
+ const base = this.gradedSource(frame, clip);
+ const r = clip?.removal;
+ if (!r || !removalActive(r) || (clip && this.removalBypass === clip.id)) return base;
+ const w = frame.width;
+ const h = frame.height;
+ // The matte answers before any scratch is staged: while the bake is still
+ // running the picture stays plain — the stroke and backdrop wait with it —
+ // and a full-frame clear and composite per tick would be pure waste.
+ const matte =
+ r.mode === "chroma" ? null : clip ? (this.removalMatteProvider?.(clip, at) ?? null) : null;
+ if (r.mode !== "chroma" && !matte) return base;
+ const { surface } = this.scratch("removalScratch", w, h);
+ const ctx = surface.getContext("2d") as Ctx | null;
+ if (!ctx) return base;
+ ctx.setTransform(1, 0, 0, 1, 0, 0);
+ ctx.globalAlpha = 1;
+ ctx.globalCompositeOperation = "source-over";
+ ctx.clearRect(0, 0, w, h);
+
+ // 1. The key: alpha into the picture.
+ if (r.mode === "chroma" && r.chroma) {
+ const gpu = chromaKeyGpu(base, w, h, r.chroma);
+ if (gpu) {
+ ctx.drawImage(gpu as CanvasImageSource, 0, 0, w, h);
+ } else {
+ ctx.drawImage(base, 0, 0, w, h);
+ const img = ctx.getImageData(0, 0, w, h);
+ chromaAlphaInto(img.data, r.chroma);
+ ctx.putImageData(img, 0, 0);
+ }
+ } else {
+ ctx.drawImage(base, 0, 0, w, h);
+ const { surface: cover } = this.scratch("removalMatte", w, h);
+ const cctx = cover.getContext("2d") as Ctx | null;
+ if (cctx) {
+ cctx.setTransform(1, 0, 0, 1, 0, 0);
+ cctx.clearRect(0, 0, w, h);
+ cctx.imageSmoothingEnabled = true;
+ cctx.drawImage(matte!, 0, 0, w, h);
+ maskComposite(ctx, cover as CanvasImageSource, false);
+ }
+ }
+
+ const tLocal = Math.max(0, at - (clip?.start ?? 0));
+ // Design px at the source frame's own short side: ink painted here is
+ // scaled with the picture into its box, so the stroke holds its design
+ // size wherever the clip lands.
+ const ds = Math.min(w, h) / 1080;
+
+ // 2. Stroke ink around the keyed silhouette, behind the picture.
+ if (r.stroke) {
+ const { surface: sil } = this.scratch("removalSil", w, h);
+ const sctx = sil.getContext("2d") as Ctx | null;
+ const { surface: ink } = this.scratch("removalInk", w, h);
+ const ictx = ink.getContext("2d") as Ctx | null;
+ if (sctx && ictx) {
+ sctx.setTransform(1, 0, 0, 1, 0, 0);
+ sctx.globalCompositeOperation = "source-over";
+ sctx.clearRect(0, 0, w, h);
+ sctx.drawImage(surface, 0, 0);
+ sctx.globalCompositeOperation = "source-in";
+ sctx.fillStyle = "#ffffff";
+ sctx.fillRect(0, 0, w, h);
+ sctx.globalCompositeOperation = "source-over";
+ ictx.setTransform(1, 0, 0, 1, 0, 0);
+ ictx.globalCompositeOperation = "source-over";
+ ictx.clearRect(0, 0, w, h);
+ paintStrokeInk(ictx, sil as CanvasImageSource, w, h, r.stroke, tLocal, ds);
+ // Color the white ink, then lay it under the picture.
+ ictx.globalCompositeOperation = "source-in";
+ ictx.fillStyle = r.stroke.color;
+ ictx.fillRect(0, 0, w, h);
+ ictx.globalCompositeOperation = "source-over";
+ ctx.globalCompositeOperation = "destination-over";
+ ctx.drawImage(ink, 0, 0);
+ ctx.globalCompositeOperation = "source-over";
+ }
+ }
+
+ // 3. Backdrop behind it all, covering the source frame.
+ const bd = r.backdrop;
+ if (bd && bd.kind !== "none") {
+ ctx.globalCompositeOperation = "destination-over";
+ if (bd.kind === "color" && bd.color) {
+ ctx.fillStyle = bd.color;
+ ctx.fillRect(0, 0, w, h);
+ } else if (bd.kind === "image" && bd.assetId) {
+ const img = this.backdropImageProvider?.(bd.assetId) ?? null;
+ const iw = img ? Number((img as { width?: number }).width ?? 0) : 0;
+ const ih = img ? Number((img as { height?: number }).height ?? 0) : 0;
+ if (img && iw > 0 && ih > 0) {
+ const s = Math.max(w / iw, h / ih);
+ ctx.drawImage(img, (w - iw * s) / 2, (h - ih * s) / 2, iw * s, ih * s);
+ }
+ }
+ ctx.globalCompositeOperation = "source-over";
+ }
+ return surface;
+ }
+
/**
* Draw a look's post passes over one composited layer's footprint (canvas
* pixels): vignette, animated grain, self-copy glow, color washes, chroma
@@ -320,14 +501,18 @@ export class FrameCompositor {
rw: number,
rh: number,
alpha: number,
- at: number
+ at: number,
+ // A different surface to paint over — the piece renderer bakes the post
+ // passes into the keyed layer itself. Default: the frame canvas.
+ target?: { canvas: Surface; ctx: Ctx }
) {
const post = lookPost(clip?.look, clip?.lookAmount);
if (!post || alpha <= 0 || rw <= 0 || rh <= 0) return;
- const ctx = this.ctx();
+ const canvas = target?.canvas ?? this.canvas;
+ const ctx = target?.ctx ?? this.ctx();
if (!ctx || !("filter" in ctx)) return;
- const W = this.canvas.width;
- const H = this.canvas.height;
+ const W = canvas.width;
+ const H = canvas.height;
ctx.save();
ctx.beginPath();
ctx.rect(rx, ry, rw, rh);
@@ -343,7 +528,7 @@ export class FrameCompositor {
sctx.filter = post.glow.bright
? `contrast(2.5) brightness(0.55) saturate(1.4) sepia(0.35) blur(${blurPx}px)`
: `blur(${blurPx}px)`;
- sctx.drawImage(this.canvas, rx, ry, rw, rh, rx, ry, rw, rh);
+ sctx.drawImage(canvas, rx, ry, rw, rh, rx, ry, rw, rh);
sctx.filter = "none";
ctx.globalAlpha = post.glow.alpha * alpha;
ctx.globalCompositeOperation = post.glow.mode;
@@ -358,14 +543,14 @@ export class FrameCompositor {
const shift = Math.max(1, post.ghost.shiftFrac * W);
sctx.clearRect(rx, ry, rw, rh);
sctx.filter = "sepia(1) saturate(4) hue-rotate(-40deg) brightness(0.55)";
- sctx.drawImage(this.canvas, rx, ry, rw, rh, rx, ry, rw, rh);
+ sctx.drawImage(canvas, rx, ry, rw, rh, rx, ry, rw, rh);
sctx.filter = "none";
ctx.globalAlpha = post.ghost.alpha * alpha;
ctx.globalCompositeOperation = "lighter";
ctx.drawImage(scratch, rx, ry, rw, rh, rx + shift, ry, rw, rh);
sctx.clearRect(rx, ry, rw, rh);
sctx.filter = "sepia(1) saturate(4) hue-rotate(140deg) brightness(0.55)";
- sctx.drawImage(this.canvas, rx, ry, rw, rh, rx, ry, rw, rh);
+ sctx.drawImage(canvas, rx, ry, rw, rh, rx, ry, rw, rh);
sctx.filter = "none";
ctx.drawImage(scratch, rx, ry, rw, rh, rx - shift, ry, rw, rh);
}
@@ -615,7 +800,7 @@ export class FrameCompositor {
clip?.panX ?? 0,
clip?.panY ?? 0
);
- const src = this.gradedSource(frame, clip);
+ const src = this.removedSource(frame, clip, at);
const bs = clip?.boxStyle;
// Box style lengths are design px at the 1080 short side, like masks.
const ds = Math.min(W, H) / 1080;
@@ -725,7 +910,7 @@ export class FrameCompositor {
);
const prevAlpha = ctx.globalAlpha;
ctx.globalAlpha = Math.max(0, Math.min(1, alpha));
- ctx.drawImage(this.gradedSource(frame, clip), dx, dy, dw, dh);
+ ctx.drawImage(this.removedSource(frame, clip, at), dx, dy, dw, dh);
ctx.globalAlpha = prevAlpha;
this.applyLookPost(clip, 0, 0, W, H, alpha, at);
if (hasFx) ctx.restore();
diff --git a/site/src/cut/lib/cutout.ts b/site/src/cut/lib/cutout.ts
index 625520e6..a002b5cc 100644
--- a/site/src/cut/lib/cutout.ts
+++ b/site/src/cut/lib/cutout.ts
@@ -146,6 +146,84 @@ export function segmentSubjectAlpha(
}
}
+const TOUCH_MODEL = "/mediapipe/magic_touch.tflite";
+type TouchSegmenter = import("@mediapipe/tasks-vision").InteractiveSegmenterLegacy;
+
+let touchOnce: Promise<TouchSegmenter | null> | null = null;
+
+/** The shared interactive (tap-to-select) segmenter, created on first use.
+ * Null when the runtime assets are missing. */
+export function interactiveSegmenter(): Promise<TouchSegmenter | null> {
+ touchOnce ??= (async () => {
+ try {
+ const { FilesetResolver, InteractiveSegmenterLegacy } = await import(
+ "@mediapipe/tasks-vision"
+ );
+ const fileset = await FilesetResolver.forVisionTasks(WASM_BASE);
+ return await withQuietWasmLogs(() =>
+ InteractiveSegmenterLegacy.createFromOptions(fileset, {
+ baseOptions: { modelAssetPath: TOUCH_MODEL },
+ outputCategoryMask: false,
+ outputConfidenceMasks: true,
+ })
+ );
+ } catch {
+ touchOnce = null;
+ return null;
+ }
+ })();
+ return touchOnce;
+}
+
+/**
+ * Segment the object under a tap (or a scribbled path) and return its alpha
+ * canvas at the mask's own resolution, white where the object is. Points are
+ * frame fractions. Null when nothing registers. The `source` should be small
+ * (≤ ~512px) — the model resamples anyway, and quick-brush strokes segment
+ * per pointer move.
+ */
+export function segmentTouchAlpha(
+ segmenter: TouchSegmenter,
+ source: HTMLCanvasElement,
+ points: { x: number; y: number }[]
+): HTMLCanvasElement | null {
+ if (points.length === 0) return null;
+ const roi =
+ points.length === 1
+ ? { keypoint: { x: points[0].x, y: points[0].y } }
+ : { scribble: points.map((p) => ({ x: p.x, y: p.y })) };
+ const result = segmenter.segment(source, roi);
+ try {
+ const masks = result.confidenceMasks;
+ if (!masks?.length) return null;
+ // Two masks: background first, the touched object second (mirroring the
+ // selfie model's layout). A single mask is the object itself.
+ const conf = masks.length > 1 ? masks[1].getAsFloat32Array() : masks[0].getAsFloat32Array();
+ const mw = masks[0].width;
+ const mh = masks[0].height;
+ const [out, ctx] = canvasOf(mw, mh);
+ const img = ctx.createImageData(mw, mh);
+ let kept = 0;
+ for (let i = 0; i < mw * mh; i++) {
+ const subject = conf[i];
+ const a =
+ subject > PERSON_THRESHOLD
+ ? Math.min(255, Math.round((subject - PERSON_THRESHOLD) * 4 * 255))
+ : 0;
+ img.data[i * 4] = 255;
+ img.data[i * 4 + 1] = 255;
+ img.data[i * 4 + 2] = 255;
+ img.data[i * 4 + 3] = a;
+ if (a > 128) kept++;
+ }
+ if (kept < mw * mh * 0.001) return null;
+ ctx.putImageData(img, 0, 0);
+ return out;
+ } finally {
+ result.close();
+ }
+}
+
/** Person matting, fully on-device. Null when no person registers (callers
* fall through to the hosted path). */
export async function personCutout(blob: Blob): Promise<HTMLCanvasElement | null> {
diff --git a/site/src/cut/lib/exportClient.ts b/site/src/cut/lib/exportClient.ts
index d58b95cc..6a682777 100644
--- a/site/src/cut/lib/exportClient.ts
+++ b/site/src/cut/lib/exportClient.ts
@@ -10,14 +10,15 @@ import { exportsDir, readFileAt, saveExport } from "./backend/browser/opfs";
import { holdRegistered, registerBlobFile, releaseRegistered } from "./backend/browser/registry";
import { quotaErrorMessage } from "./backend/cloud";
import { downloadFromUrl } from "./download";
-import { renderProjectToMp4 } from "./exportRender";
+import { ClipReader, renderProjectToMp4 } from "./exportRender";
import { putSigned } from "./media";
+import { renderRemovalPieces } from "./removalVideo";
import { createRasterCanvas, rasterCanvasToPng } from "./raster";
import { clipSpeed, getClipSpans, overlayLayers, projectDuration, spanSequence, useEditor } from "./store";
import { captionStyle, cueOverlay, cueWordFrames, laneCues, laneHidden, subtitleLaneCount, trackPos } from "./subtitles";
-import { isMaskAnimated, isOverlayAnimated, normalizeGrade, paintMaskLuma } from "@donkeycut/effects-kit";
+import { chromaAlphaInto, isMaskAnimated, isOverlayAnimated, matteLumaToAlpha, normalizeGrade, paintMaskLuma, paintStrokeInk } from "@donkeycut/effects-kit";
import { renderElementFrames, renderElementPng } from "./textRender";
-import { clipCovers, clipKeyed, clipPosed, clipPoseAt, clipZoom, contentRect, frameOf, isStickerOverlay, isTextOverlay, laneOf, overlayAnimStyle, projectBackground, rectOf, regionPx, shadowInk, subjectMasked } from "./types";
+import { clipCovers, clipKeyed, clipPosed, clipPoseAt, clipZoom, contentRect, frameOf, isStickerOverlay, isTextOverlay, laneOf, overlayAnimStyle, projectBackground, rectOf, regionPx, removalActive, shadowInk, subjectMasked } from "./types";
import type {
Aspect,
AudioClip,
@@ -283,14 +284,113 @@ async function renderClipMaskPictures(
return { frames };
}
+/** The keyed silhouette a removal clip's shadow falls from: white coverage
+ * over transparency in the clip's source frame, sampled at a shadow frame's
+ * local second. Null means the shadow keeps the picture's rectangle — no
+ * active removal, a backdrop fill restoring the full picture, or an AI
+ * removal whose matte hasn't baked (the export shows the plain picture then,
+ * and a rectangular shadow matches it). */
+function removalShadowSilhouette(
+ clip: VideoClip,
+ assets: MediaAsset[]
+): { at: (tLocal: number) => Promise<CanvasImageSource | null>; dispose: () => void } | null {
+ const r = clip.removal;
+ if (!removalActive(r)) return null;
+ if (r!.backdrop && r!.backdrop.kind !== "none") return null;
+ const asset = assets.find((a) => a.id === clip.assetId);
+ if (!asset) return null;
+ const speed = clipSpeed(clip);
+ const still = asset.type === "image";
+ const canvas = createRasterCanvas(2, 2);
+ const ctx = canvas.getContext("2d", {
+ willReadFrequently: true,
+ }) as CanvasRenderingContext2D | null;
+ if (!ctx) return null;
+ const stage = (w: number, h: number) => {
+ if (canvas.width !== w || canvas.height !== h) {
+ canvas.width = w;
+ canvas.height = h;
+ }
+ ctx.clearRect(0, 0, w, h);
+ };
+ // The preview's shadow falls from the whole layer — keyed picture plus the
+ // stroke ink laid behind it — so the ink joins the silhouette here too;
+ // a thick or offset stroke moves the shadow's edge with it.
+ const ink = createRasterCanvas(2, 2);
+ const inkCtx = ink.getContext("2d") as CanvasRenderingContext2D | null;
+ const withStroke = (w: number, h: number, tLocal: number): CanvasImageSource => {
+ const stroke = r!.stroke;
+ if (!stroke || !inkCtx) return canvas as CanvasImageSource;
+ if (ink.width !== w || ink.height !== h) {
+ ink.width = w;
+ ink.height = h;
+ }
+ inkCtx.setTransform(1, 0, 0, 1, 0, 0);
+ inkCtx.globalCompositeOperation = "source-over";
+ inkCtx.clearRect(0, 0, w, h);
+ paintStrokeInk(inkCtx, canvas as CanvasImageSource, w, h, stroke, tLocal, Math.min(w, h) / 1080);
+ ctx.globalCompositeOperation = "destination-over";
+ ctx.drawImage(ink, 0, 0);
+ ctx.globalCompositeOperation = "source-over";
+ return canvas as CanvasImageSource;
+ };
+ if (r!.mode === "chroma") {
+ const key = r!.chroma;
+ if (!key) return null;
+ const reader = new ClipReader(asset, () => asset.url);
+ return {
+ at: async (tLocal) => {
+ const frame = await reader.frameAt(still ? 0 : clip.in + tLocal * speed);
+ if (frame.kind !== "ready") return null;
+ // Sampled small — the silhouette blurs into a shadow anyway.
+ const s = Math.min(1, 480 / Math.max(1, Math.min(frame.width, frame.height)));
+ const w = Math.max(2, Math.round(frame.width * s));
+ const h = Math.max(2, Math.round(frame.height * s));
+ stage(w, h);
+ ctx.drawImage(frame.image, 0, 0, w, h);
+ const px = ctx.getImageData(0, 0, w, h);
+ chromaAlphaInto(px.data, key);
+ for (let i = 0; i < px.data.length; i += 4) {
+ px.data[i] = 255;
+ px.data[i + 1] = 255;
+ px.data[i + 2] = 255;
+ }
+ ctx.putImageData(px, 0, 0);
+ return withStroke(w, h, tLocal);
+ },
+ dispose: () => reader.dispose(),
+ };
+ }
+ const matte = r!.matte ? assets.find((a) => a.id === r!.matte!.assetId) : undefined;
+ if (!matte) return null;
+ const reader = new ClipReader(matte, () => matte.url);
+ const mdur = Math.max(0.1, matte.duration || 0.1);
+ return {
+ at: async (tLocal) => {
+ const srcT = still ? 0 : clip.in + tLocal * speed;
+ const frame = await reader.frameAt(
+ Math.min(Math.max(0, srcT - r!.matte!.in), mdur - 0.001)
+ );
+ if (frame.kind !== "ready") return null;
+ stage(frame.width, frame.height);
+ ctx.drawImage(frame.image, 0, 0);
+ const px = ctx.getImageData(0, 0, frame.width, frame.height);
+ matteLumaToAlpha(px.data);
+ ctx.putImageData(px, 0, 0);
+ return withStroke(frame.width, frame.height, tLocal);
+ },
+ dispose: () => reader.dispose(),
+ };
+}
+
/**
* Paint the shadow a clip casts, frame-sized and transparent everywhere else.
* The silhouette is the picture the clip really shows — its box, its rounded
- * corners, its mask, its pose — thrown behind itself and then punched back
- * out, so the graph can lay the result over the frame and get what sitting
- * behind the clip would have looked like. A moving clip samples the shadow
- * per frame, the way a keyframed mask samples its coverage. Null without a
- * shadow.
+ * corners, its mask, its pose, its keyed cutout — thrown behind itself and
+ * then punched back out, so the graph can lay the result over the frame and
+ * get what sitting behind the clip would have looked like. A moving clip
+ * samples the shadow per frame, the way a keyframed mask samples its
+ * coverage. Null without a shadow.
*/
async function renderClipShadowPictures(
clip: VideoClip,
@@ -299,10 +399,14 @@ async function renderClipShadowPictures(
H: number,
dur: number,
tag: string,
- pngs: ExportPayload["pngs"]
+ pngs: ExportPayload["pngs"],
+ assets: MediaAsset[]
): Promise<SpecMask | undefined> {
const sh = clip.boxStyle?.shadow;
if (!sh) return undefined;
+ // A cutout's shadow falls from the keyed subject, sampled per frame from
+ // the matte (or the chroma key), the way the preview casts it.
+ const silhouette = removalShadowSilhouette(clip, assets);
const scale = Math.min(W, H) / 1080;
const frame = { width: W, height: H, scale };
const rect = rectOf(clip);
@@ -319,7 +423,8 @@ async function renderClipShadowPictures(
const cover = createRasterCanvas(W, H);
const canvas = createRasterCanvas(W, H);
const anchor = { x: rect.x + rect.w / 2, y: rect.y + rect.h / 2 };
- const blobAt = (tLocal: number) => {
+ const blobAt = async (tLocal: number) => {
+ const sil = silhouette ? await silhouette.at(tLocal) : null;
const shapeCtx = shape.getContext("2d") as CanvasRenderingContext2D;
shapeCtx.setTransform(1, 0, 0, 1, 0, 0);
shapeCtx.clearRect(0, 0, W, H);
@@ -330,13 +435,18 @@ async function renderClipShadowPictures(
shapeCtx.scale(pose.scale, pose.scale);
shapeCtx.translate(-anchor.x * W, -anchor.y * H);
}
- // The lit shape: the picture, trimmed at the box's rounded corners.
+ // The lit shape: the picture — a cutout's keyed silhouette in its place —
+ // trimmed at the box's rounded corners.
shapeCtx.save();
shapeCtx.beginPath();
shapeCtx.roundRect(box.x, box.y, box.w, box.h, radius);
shapeCtx.clip();
- shapeCtx.fillStyle = "#ffffff";
- shapeCtx.fillRect(pic.x, pic.y, pic.w, pic.h);
+ if (sil) {
+ shapeCtx.drawImage(sil, pic.x, pic.y, pic.w, pic.h);
+ } else {
+ shapeCtx.fillStyle = "#ffffff";
+ shapeCtx.fillRect(pic.x, pic.y, pic.w, pic.h);
+ }
shapeCtx.restore();
if (mask) {
// The mask decides the rest of the shape; its luma becomes coverage.
@@ -362,21 +472,27 @@ async function renderClipShadowPictures(
ctx.globalCompositeOperation = "source-over";
return rasterCanvasToPng(canvas);
};
- const moves = (mask && isMaskAnimated(mask)) || clipKeyed(clip);
- if (!moves) {
- const name = `${tag}.png`;
- pngs.push({ name, blob: await blobAt(0) });
- return { file: name };
- }
- const step = 1 / MASK_SAMPLE_FPS;
- const n = Math.max(1, Math.round(dur * MASK_SAMPLE_FPS));
- const frames: { file: string; duration: number }[] = [];
- for (let i = 0; i < n; i++) {
- const name = `${tag}_f${i}.png`;
- pngs.push({ name, blob: await blobAt(i * step) });
- frames.push({ file: name, duration: i === n - 1 ? Math.max(step, dur - (n - 1) * step) : step });
+ try {
+ // A keyed subject moves every frame, so its shadow samples like an
+ // animated mask does.
+ const moves = (mask && isMaskAnimated(mask)) || clipKeyed(clip) || !!silhouette;
+ if (!moves) {
+ const name = `${tag}.png`;
+ pngs.push({ name, blob: await blobAt(0) });
+ return { file: name };
+ }
+ const step = 1 / MASK_SAMPLE_FPS;
+ const n = Math.max(1, Math.round(dur * MASK_SAMPLE_FPS));
+ const frames: { file: string; duration: number }[] = [];
+ for (let i = 0; i < n; i++) {
+ const name = `${tag}_f${i}.png`;
+ pngs.push({ name, blob: await blobAt(i * step) });
+ frames.push({ file: name, duration: i === n - 1 ? Math.max(step, dur - (n - 1) * step) : step });
+ }
+ return { frames };
+ } finally {
+ silhouette?.dispose();
}
- return { frames };
}
/** Paint the clip's border ring — a stroked rounded rect along its box edge,
@@ -514,6 +630,7 @@ export async function buildExportPayload(
shadow: undefined as SpecMask | undefined,
kf: posed(sp.clip).kf,
border: undefined as string | undefined,
+ removal: undefined as { rgb: string; alpha: string } | undefined,
}));
// Track-0 segments render at the full output frame (regioned clips pad out
// to it), so their masks paint full-frame. A subject mask rides the shared
@@ -558,8 +675,33 @@ export async function buildExportPayload(
settings.height,
dur,
`shadow_c${i}`,
- pngs
+ pngs,
+ doc.assets
);
+ // A removal clip's picture travels as its keyed layer, client-rendered:
+ // a color/alpha pair the engine merges and frames in place of the source
+ // decode. Grade and key are baked in (the entry's grade clears so the
+ // engine never grades twice); the look stays on the entry — the engine
+ // runs it over the flattened segment. A null is the declared degrade (no
+ // matte yet — the export shows the plain picture, like the preview); a
+ // broken render throws and fails the export instead of silently dropping
+ // the cutout.
+ if (removalActive(c.removal) && !c.hidden) {
+ const pieces = await renderRemovalPieces(spans[i].asset, spans[i].clip, doc.assets, {
+ fps: settings.fps,
+ maxShort: Math.min(
+ 2160,
+ Math.round(Math.min(settings.width, settings.height) * clipZoom(c))
+ ),
+ bakeLook: false,
+ });
+ if (pieces) {
+ pngs.push({ name: `removal_c${i}_rgb.mp4`, blob: pieces.rgb });
+ pngs.push({ name: `removal_c${i}_a.mp4`, blob: pieces.alpha });
+ clipEntries[i].removal = { rgb: `removal_c${i}_rgb.mp4`, alpha: `removal_c${i}_a.mp4` };
+ clipEntries[i].grade = undefined;
+ }
+ }
}
// The server's video graph is a sequential fold, so gaps between the
@@ -674,6 +816,7 @@ export async function buildExportPayload(
shadow: undefined as SpecMask | undefined,
kf: posed(c).kf,
border: undefined as string | undefined,
+ removal: undefined as { rgb: string; alpha: string } | undefined,
...ramp,
};
overlayClipOf.set(entry, posed(c));
@@ -725,8 +868,30 @@ export async function buildExportPayload(
settings.height,
olen,
`shadow_ov${i}`,
- pngs
+ pngs,
+ doc.assets
);
+ // The keyed layer keeps its alpha over the tracks beneath, so the engine
+ // applies no grade or look to it — both bake into the pieces here, the
+ // way image overlays already carry their pixels ready-made.
+ const oAsset = assetById.get(c.assetId);
+ if (removalActive(c.removal) && oAsset) {
+ const pieces = await renderRemovalPieces(oAsset, c, doc.assets, {
+ fps: settings.fps,
+ maxShort: Math.min(
+ 2160,
+ Math.round(Math.min(settings.width, settings.height) * clipZoom(c))
+ ),
+ bakeLook: true,
+ });
+ if (pieces) {
+ pngs.push({ name: `removal_ov${i}_rgb.mp4`, blob: pieces.rgb });
+ pngs.push({ name: `removal_ov${i}_a.mp4`, blob: pieces.alpha });
+ entry.removal = { rgb: `removal_ov${i}_rgb.mp4`, alpha: `removal_ov${i}_a.mp4` };
+ entry.grade = undefined;
+ entry.look = undefined;
+ }
+ }
}
const audio = doc.audioClips
diff --git a/site/src/cut/lib/exportRender.ts b/site/src/cut/lib/exportRender.ts
index a83e8d8d..e6cff685 100644
--- a/site/src/cut/lib/exportRender.ts
+++ b/site/src/cut/lib/exportRender.ts
@@ -34,7 +34,8 @@ import { overlayPlan, trackZeroPlan } from "./framePlan";
import { frameSink, openMedia, videoTrackOf } from "./mediaRead";
import { getClipSpans, overlayLayers, projectDuration, spanSequence } from "./store";
import { captionStyle, cueOverlay, cueWordFrames, laneCues, laneHidden, subtitleLaneCount, trackPos } from "./subtitles";
-import { applyEffectToCanvas, evalOverlayFrame, grainTile, isAudioEffect, isMaskAnimated, isOverlayAnimated, maskFrameAt, planAnimatedLayers, type LottieHandle, type OverlayAnim, type PaintPhase } from "@donkeycut/effects-kit";
+import { applyEffectToCanvas, evalOverlayFrame, grainTile, isAudioEffect, isMaskAnimated, isOverlayAnimated, maskFrameAt, MATTE_FPS, matteLumaToAlpha, planAnimatedLayers, type LottieHandle, type OverlayAnim, type PaintPhase } from "@donkeycut/effects-kit";
+import { backdropStill, loadBackdropStill } from "./backdropStills";
import { hasSubjectOverlays, SubjectMaskCompositor } from "./behindPass";
import { createRasterCanvas, type RasterSurface } from "./raster";
import { renderElementPng } from "./textRender";
@@ -807,6 +808,15 @@ export class FramePainter {
private readers = new Map<string, ClipReader>();
private behind: SubjectMaskCompositor | null = null;
private fxScratch: RasterSurface | null = null;
+ /** This frame's removal mattes, one alpha image per keyed clip — fetched
+ * with the layer frames (readers are async) and read synchronously by the
+ * compositor's provider mid-draw. */
+ private matteFrames = new Map<string, CanvasImageSource>();
+ /** Per-clip conversion scratch for those mattes (luma frame → alpha). */
+ private matteScratch = new Map<string, RasterSurface>();
+ /** Which matte frame each scratch currently holds, so the luma → alpha
+ * pixel pass runs once per matte frame. */
+ private matteStamp = new Map<string, string>();
/** Text layers, deepest lane first — so a walk of them is a walk up the stack. */
private stacked: StampedLayer[] = [];
private spans: ClipSpan[] = [];
@@ -849,6 +859,23 @@ export class FramePainter {
this.comp.subjectMatteProvider = (at) =>
behind.clipMatteOf(this.canvas, at, { minMaskInterval: 0 });
}
+ // Removal: baked mattes are prefetched per frame into `matteFrames`; the
+ // provider is a synchronous read of that map. Backdrop stills decode once
+ // here, so mid-render reads are cache hits.
+ this.comp.removalMatteProvider = (clip) => this.matteFrames.get(clip.id) ?? null;
+ this.comp.backdropImageProvider = (assetId) => backdropStill(this.doc.assets, assetId);
+ const backdropIds = new Set(
+ this.doc.clips
+ .map((c) => c.removal?.backdrop)
+ .filter((b) => b?.kind === "image" && b.assetId)
+ .map((b) => b!.assetId!)
+ );
+ await Promise.all(
+ [...backdropIds].map((id) => {
+ const asset = this.doc.assets.find((a) => a.id === id);
+ return asset ? loadBackdropStill({ ...asset, url: this.resolve(asset) }) : null;
+ })
+ );
// Span geometry is a property of the document, which does not change while
// rendering — so it is computed once rather than per overlay per frame.
this.spans = getClipSpans(this.doc.clips, this.doc.assets, 0);
@@ -866,6 +893,43 @@ export class FramePainter {
return r;
}
+ /** Pull a removal clip's baked matte frame for timeline time `t` and stage
+ * it, luma turned to alpha, where the compositor's provider reads. A clip
+ * with no matte (chroma, or a bake still owed) stages nothing. */
+ private async fetchRemovalMatte(span: ClipSpan, t: number): Promise<void> {
+ const clip = span.clip;
+ const m = clip.removal?.matte;
+ this.matteFrames.delete(clip.id);
+ if (!m) return;
+ const asset = this.doc.assets.find((a) => a.id === m.assetId);
+ if (!asset) return;
+ const dur = Math.max(0.001, asset.duration);
+ const mt = Math.max(0, Math.min(sourceTimeAt(span, t) - m.in, dur - 0.001));
+ const frame = await this.readerFor(asset).frameAt(mt);
+ if (frame.kind !== "ready") return;
+ let scratch = this.matteScratch.get(clip.id);
+ // The matte advances at its own baked rate below the export's, so a
+ // converted frame is reused until the read crosses into the next one —
+ // the pixel pass is per matte frame, never per output frame.
+ const stamp = `${m.assetId}:${Math.floor(mt * MATTE_FPS)}`;
+ if (scratch && this.matteStamp.get(clip.id) === stamp) {
+ this.matteFrames.set(clip.id, scratch as CanvasImageSource);
+ return;
+ }
+ if (!scratch) this.matteScratch.set(clip.id, (scratch = createRasterCanvas(frame.width, frame.height)));
+ if (scratch.width !== frame.width) scratch.width = frame.width;
+ if (scratch.height !== frame.height) scratch.height = frame.height;
+ const ctx = scratch.getContext("2d") as CanvasRenderingContext2D | null;
+ if (!ctx) return;
+ ctx.clearRect(0, 0, scratch.width, scratch.height);
+ ctx.drawImage(frame.image, 0, 0, scratch.width, scratch.height);
+ const img = ctx.getImageData(0, 0, scratch.width, scratch.height);
+ matteLumaToAlpha(img.data);
+ ctx.putImageData(img, 0, 0);
+ this.matteStamp.set(clip.id, stamp);
+ this.matteFrames.set(clip.id, scratch as CanvasImageSource);
+ }
+
/** Draw the whole cut at timeline time `t` onto the canvas. */
async drawAt(t: number): Promise<void> {
const { canvas, comp, doc, stamps } = this;
@@ -876,7 +940,10 @@ export class FramePainter {
const master = this.spans.find((sp) => t >= sp.start && t < sp.start + sp.len);
if (master) {
const plan = trackZeroPlan(master, this.spans, t);
+ await this.fetchRemovalMatte(master, t);
+ if (plan.incoming) await this.fetchRemovalMatte(plan.incoming, t);
if (plan.backdrop) {
+ await this.fetchRemovalMatte(plan.backdrop.span, t);
const frame = await this.readerFor(plan.backdrop.span.asset).frameAt(plan.backdrop.at);
comp.drawLayer(frame, plan.backdrop.span.clip, false, 1, t);
}
@@ -912,6 +979,7 @@ export class FramePainter {
for (const layer of overlayPlan(this.overlayTracks, spansOf, t)) {
const span = this.spanOfClip.get(layer.clip.id);
if (!span) continue;
+ await this.fetchRemovalMatte(span, t);
const frame = await this.readerFor(layer.asset).frameAt(sourceTimeAt(span, t));
comp.drawIntoRect(
frame,
diff --git a/site/src/cut/lib/gpuPass.ts b/site/src/cut/lib/gpuPass.ts
new file mode 100644
index 00000000..b5be478f
--- /dev/null
+++ b/site/src/cut/lib/gpuPass.ts
@@ -0,0 +1,115 @@
+/**
+ * The lazy WebGL2 pass the GPU pixel effects share: one raster canvas, a
+ * fullscreen triangle, a source texture on unit 0, and the lifecycle around
+ * them. A pass supplies its fragment shader and one-time uniform setup; the
+ * helper owns creation, the resize of the shared canvas, and context loss —
+ * a lost context turns every GL call into a silent no-op and the shared
+ * canvas would hand back stale pixels forever, so the pass drops itself and
+ * the next call rebuilds it, or settles on null when WebGL2 stays gone
+ * (headless) and callers keep their CPU fallback.
+ */
+
+import { createRasterCanvas, type RasterSurface } from "./raster";
+
+// The V axis is inverted here rather than at upload: `UNPACK_FLIP_Y_WEBGL` is
+// ignored for an ImageBitmap source (a still's frame), so a pass that leaned on
+// it processed stills upside down while video frames came out upright. Sampling
+// the texture top-down puts every source kind — bitmap, canvas, video frame —
+// the same way up.
+const VERT = `#version 300 es
+in vec2 aPos;
+out vec2 vUv;
+void main() {
+ vUv = vec2(aPos.x * 0.5 + 0.5, 0.5 - aPos.y * 0.5);
+ gl_Position = vec4(aPos, 0.0, 1.0);
+}`;
+
+export interface GpuPass {
+ canvas: RasterSurface;
+ gl: WebGL2RenderingContext;
+ program: WebGLProgram;
+ srcTex: WebGLTexture;
+}
+
+function initPass(frag: string): GpuPass | false {
+ const canvas = createRasterCanvas(2, 2);
+ const gl = (canvas.getContext as (id: string, opts?: unknown) => unknown)("webgl2", {
+ premultipliedAlpha: false,
+ preserveDrawingBuffer: true,
+ }) as WebGL2RenderingContext | null;
+ if (!gl) return false;
+ const compile = (type: number, src: string) => {
+ const sh = gl.createShader(type)!;
+ gl.shaderSource(sh, src);
+ gl.compileShader(sh);
+ if (!gl.getShaderParameter(sh, gl.COMPILE_STATUS)) return null;
+ return sh;
+ };
+ const vs = compile(gl.VERTEX_SHADER, VERT);
+ const fs = compile(gl.FRAGMENT_SHADER, frag);
+ if (!vs || !fs) return false;
+ const program = gl.createProgram()!;
+ gl.attachShader(program, vs);
+ gl.attachShader(program, fs);
+ gl.linkProgram(program);
+ if (!gl.getProgramParameter(program, gl.LINK_STATUS)) return false;
+ gl.useProgram(program);
+ const buf = gl.createBuffer();
+ gl.bindBuffer(gl.ARRAY_BUFFER, buf);
+ gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1, -1, 3, -1, -1, 3]), gl.STATIC_DRAW);
+ const posLoc = gl.getAttribLocation(program, "aPos");
+ gl.enableVertexAttribArray(posLoc);
+ gl.vertexAttribPointer(posLoc, 2, gl.FLOAT, false, 0, 0);
+ const srcTex = gl.createTexture()!;
+ gl.activeTexture(gl.TEXTURE0);
+ gl.bindTexture(gl.TEXTURE_2D, srcTex);
+ gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
+ gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
+ gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
+ gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
+ gl.uniform1i(gl.getUniformLocation(program, "uSrc"), 0);
+ return { canvas, gl, program, srcTex };
+}
+
+/**
+ * A singleton pass over `frag`. `setup` runs once per (re)build — uniform
+ * lookups, extra texture units — and its null means the shader is missing
+ * something, which settles the pass off for good. The returned acquire
+ * function sizes the canvas and viewport and hands the pass back, or null
+ * while WebGL2 is out; each call re-checks context loss.
+ */
+export function gpuPassSingleton<T>(
+ frag: string,
+ setup: (pass: GpuPass) => T | null
+): (w: number, h: number) => { pass: GpuPass; ext: T } | null {
+ let state: { pass: GpuPass; ext: T } | null | false = null;
+ return (w, h) => {
+ if (state && state.pass.gl.isContextLost()) state = null;
+ if (state === false) return null;
+ if (!state) {
+ try {
+ const pass = initPass(frag);
+ const ext = pass ? setup(pass) : null;
+ state = pass && ext !== null ? { pass, ext } : false;
+ } catch {
+ state = false;
+ }
+ if (!state) return null;
+ }
+ const { pass } = state;
+ if (pass.canvas.width !== w || pass.canvas.height !== h) {
+ pass.canvas.width = w;
+ pass.canvas.height = h;
+ }
+ pass.gl.viewport(0, 0, w, h);
+ return state;
+ };
+}
+
+/** Upload `source` as the pass's source texture on unit 0. */
+export function uploadPassSource(pass: GpuPass, source: CanvasImageSource): void {
+ const { gl } = pass;
+ gl.activeTexture(gl.TEXTURE0);
+ gl.bindTexture(gl.TEXTURE_2D, pass.srcTex);
+ gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, source as TexImageSource);
+}
diff --git a/site/src/cut/lib/gradeGpu.ts b/site/src/cut/lib/gradeGpu.ts
index a4438b2a..4b3892e4 100644
--- a/site/src/cut/lib/gradeGpu.ts
+++ b/site/src/cut/lib/gradeGpu.ts
@@ -7,20 +7,8 @@
*/
import type { GradeLut } from "@donkeycut/effects-kit";
-import { createRasterCanvas, type RasterSurface } from "./raster";
-
-// The V axis is inverted here rather than at upload: `UNPACK_FLIP_Y_WEBGL` is
-// ignored for an ImageBitmap source (a still's frame), so a pass that leaned on
-// it graded stills upside down while video frames came out upright. Sampling
-// the texture top-down puts every source kind — bitmap, canvas, video frame —
-// the same way up.
-const VERT = `#version 300 es
-in vec2 aPos;
-out vec2 vUv;
-void main() {
- vUv = vec2(aPos.x * 0.5 + 0.5, 0.5 - aPos.y * 0.5);
- gl_Position = vec4(aPos, 0.0, 1.0);
-}`;
+import { gpuPassSingleton, uploadPassSource, type GpuPass } from "./gpuPass";
+import type { RasterSurface } from "./raster";
const FRAG = `#version 300 es
precision highp float;
@@ -58,11 +46,7 @@ void main() {
outColor = vec4(rgb, s.a);
}`;
-interface GpuState {
- canvas: RasterSurface;
- gl: WebGL2RenderingContext;
- program: WebGLProgram;
- srcTex: WebGLTexture;
+interface GradeExt {
sizeLoc: WebGLUniformLocation;
/** One uploaded texture per grade, so two graded clips in the same frame
* take turns binding instead of re-uploading 33³ floats twice a frame. */
@@ -72,61 +56,23 @@ interface GpuState {
/** Uploaded LUT textures kept at once; a 33³ RGBA32F texture is ~0.6MB. */
const LUT_TEXTURE_MAX = 6;
-let state: GpuState | null | false = null;
-
-function init(): GpuState | false {
- const canvas = createRasterCanvas(2, 2);
- const gl = (canvas.getContext as (id: string, opts?: unknown) => unknown)("webgl2", {
- premultipliedAlpha: false,
- preserveDrawingBuffer: true,
- }) as WebGL2RenderingContext | null;
- if (!gl) return false;
- const compile = (type: number, src: string) => {
- const sh = gl.createShader(type)!;
- gl.shaderSource(sh, src);
- gl.compileShader(sh);
- if (!gl.getShaderParameter(sh, gl.COMPILE_STATUS)) return null;
- return sh;
- };
- const vs = compile(gl.VERTEX_SHADER, VERT);
- const fs = compile(gl.FRAGMENT_SHADER, FRAG);
- if (!vs || !fs) return false;
- const program = gl.createProgram()!;
- gl.attachShader(program, vs);
- gl.attachShader(program, fs);
- gl.linkProgram(program);
- if (!gl.getProgramParameter(program, gl.LINK_STATUS)) return false;
- gl.useProgram(program);
- const buf = gl.createBuffer();
- gl.bindBuffer(gl.ARRAY_BUFFER, buf);
- gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1, -1, 3, -1, -1, 3]), gl.STATIC_DRAW);
- const posLoc = gl.getAttribLocation(program, "aPos");
- gl.enableVertexAttribArray(posLoc);
- gl.vertexAttribPointer(posLoc, 2, gl.FLOAT, false, 0, 0);
- const srcTex = gl.createTexture()!;
- gl.activeTexture(gl.TEXTURE0);
- gl.bindTexture(gl.TEXTURE_2D, srcTex);
- gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
- gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
- gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
- gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
- gl.uniform1i(gl.getUniformLocation(program, "uSrc"), 0);
+const acquire = gpuPassSingleton(FRAG, ({ gl, program }): GradeExt | null => {
gl.uniform1i(gl.getUniformLocation(program, "uLut"), 1);
const sizeLoc = gl.getUniformLocation(program, "uSize");
- if (!sizeLoc) return false;
- return { canvas, gl, program, srcTex, sizeLoc, luts: new Map() };
-}
+ if (!sizeLoc) return null;
+ return { sizeLoc, luts: new Map() };
+});
/** Bind the LUT's texture, uploading it the first time this grade is seen. */
-function bindLut(s: GpuState, lut: GradeLut, key: string): void {
- const { gl } = s;
+function bindLut(s: { pass: GpuPass; ext: GradeExt }, lut: GradeLut, key: string): void {
+ const { gl } = s.pass;
gl.activeTexture(gl.TEXTURE1);
- const held = s.luts.get(key);
+ const held = s.ext.luts.get(key);
if (held) {
gl.bindTexture(gl.TEXTURE_3D, held);
// Re-insert so the cap drops the grade least recently drawn.
- s.luts.delete(key);
- s.luts.set(key, held);
+ s.ext.luts.delete(key);
+ s.ext.luts.set(key, held);
return;
}
const tex = gl.createTexture()!;
@@ -145,13 +91,13 @@ function bindLut(s: GpuState, lut: GradeLut, key: string): void {
rgba[i * 4 + 3] = 1;
}
gl.texImage3D(gl.TEXTURE_3D, 0, gl.RGBA32F, lut.size, lut.size, lut.size, 0, gl.RGBA, gl.FLOAT, rgba);
- s.luts.set(key, tex);
- if (s.luts.size > LUT_TEXTURE_MAX) {
- const oldest = s.luts.keys().next().value;
+ s.ext.luts.set(key, tex);
+ if (s.ext.luts.size > LUT_TEXTURE_MAX) {
+ const oldest = s.ext.luts.keys().next().value;
if (oldest !== undefined && oldest !== key) {
- const stale = s.luts.get(oldest);
+ const stale = s.ext.luts.get(oldest);
if (stale) gl.deleteTexture(stale);
- s.luts.delete(oldest);
+ s.ext.luts.delete(oldest);
}
}
}
@@ -168,30 +114,16 @@ export function applyLutGpu(
lut: GradeLut,
key: string
): RasterSurface | null {
- if (state === false) return null;
- if (!state) {
- try {
- state = init();
- } catch {
- state = false;
- }
- if (!state) return null;
- }
- const s = state;
- const { gl } = s;
+ const held = acquire(w, h);
+ if (!held) return null;
+ const { pass, ext } = held;
+ const { gl } = pass;
try {
- if (s.canvas.width !== w || s.canvas.height !== h) {
- s.canvas.width = w;
- s.canvas.height = h;
- }
- bindLut(s, lut, key);
- gl.viewport(0, 0, w, h);
- gl.activeTexture(gl.TEXTURE0);
- gl.bindTexture(gl.TEXTURE_2D, s.srcTex);
- gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, source as TexImageSource);
- gl.uniform1f(s.sizeLoc, lut.size);
+ bindLut(held, lut, key);
+ uploadPassSource(pass, source);
+ gl.uniform1f(ext.sizeLoc, lut.size);
gl.drawArrays(gl.TRIANGLES, 0, 3);
- return s.canvas;
+ return pass.canvas;
} catch {
return null;
}
diff --git a/site/src/cut/lib/hostedBlobs.ts b/site/src/cut/lib/hostedBlobs.ts
index 3e569e31..f874c1e9 100644
--- a/site/src/cut/lib/hostedBlobs.ts
+++ b/site/src/cut/lib/hostedBlobs.ts
@@ -127,6 +127,21 @@ async function put(url: string, bytes: Uint8Array, mimeType: string): Promise<bo
}
}
+/**
+ * Store one blob in the inference scratch space and return its object key,
+ * for a request that references media by key (e.g. `blobUrl` placeholders)
+ * instead of carrying payload fields. Null when storage is unavailable.
+ */
+export async function storeHostedBlob(blob: Blob, clientId: string): Promise<string | null> {
+ const mimeType = blob.type || "application/octet-stream";
+ const bytes = new Uint8Array(await blob.arrayBuffer());
+ const sha256 = await sha256Hex(bytes);
+ const [claim] = await claimKeys([{ sha256, mimeType, bytes: bytes.byteLength }], clientId);
+ if (!claim?.blobRef) return null;
+ if (claim.putUrl && !(await put(claim.putUrl, bytes, mimeType))) return null;
+ return claim.blobRef;
+}
+
/**
* A copy of `body` with its media payloads moved to storage and replaced by
* references. Returns `body` itself when there is nothing worth moving.
diff --git a/site/src/cut/lib/removal/bakeJobs.ts b/site/src/cut/lib/removal/bakeJobs.ts
new file mode 100644
index 00000000..2bf8630f
--- /dev/null
+++ b/site/src/cut/lib/removal/bakeJobs.ts
@@ -0,0 +1,434 @@
+"use client";
+
+/**
+ * Matte bake jobs: one running bake per clip, keyed by clip id.
+ *
+ * `ensureMatteBake` is idempotent — the panel and the doc sweep call it
+ * whenever an AI-mode removal is on, and the store compares the clip's
+ * fingerprint against the stored matte and the bake in flight, so it only
+ * ever starts work the doc actually owes. The ladder: auto mode bakes the
+ * free on-device matte first, then the hosted quality pass replaces it when
+ * the account has credits; custom mode goes straight to the hosted tracker
+ * (only it can follow a brushed object). A finished bake lands as a project
+ * asset (origin "matte") and repoints `clip.removal.matte` through a
+ * transient write: background landings stay off the undo stack, and the clip
+ * change still marks the doc for saving.
+ */
+
+import { create } from "zustand";
+import { importFileToProject } from "../media";
+import { useEditor } from "../store";
+import { removalFingerprint, type ClipRemoval, type VideoClip } from "../types";
+import { useBrushUi } from "./brushUi";
+import { hostedBakeMatte, MatteTrackFailed, type HostedBakeTicket } from "./hostedBake";
+import { localBakeMatte, MatteBakeCancelled } from "./localBake";
+
+type MatteQuality = "local" | "hq";
+
+export interface MatteBakeJob {
+ quality: MatteQuality;
+ status: "running" | "error";
+ /** 0..1 while running. */
+ progress: number;
+ /** Estimated seconds to done, from the run's own pace — stamped with each
+ * progress step, absent until the estimate has anything to stand on. */
+ secondsLeft?: number;
+ /** When secondsLeft was stamped, so the panel counts it down between
+ * progress steps. */
+ etaAt?: number;
+ error?: string;
+}
+
+interface MatteBakes {
+ jobs: Record<string, MatteBakeJob>;
+}
+
+export const useMatteBakes = create<MatteBakes>(() => ({ jobs: {} }));
+
+/** Whether this surface can run bakes at all. Bakes decode and encode
+ * through browser media machinery, so a headless turn writes the removal and
+ * leaves the bake to the next editor session — the doc sweep starts it on
+ * load, since the clip's fingerprint still owes a matte. */
+export const matteBakesAvailable = typeof window !== "undefined";
+
+const setJob = (clipId: string, job: MatteBakeJob | null) =>
+ useMatteBakes.setState((s) => {
+ const jobs = { ...s.jobs };
+ if (job) jobs[clipId] = job;
+ else delete jobs[clipId];
+ return { jobs };
+ });
+
+/** The in-flight bake per clip: what it's baking, and its kill switch. */
+const running = new Map<string, { fp: string; quality: MatteQuality; ctl: AbortController }>();
+
+// The hosted bake bills at submit, so its poll ticket persists per browser: a
+// reload re-attaches to the paid track through it and pays nothing new. One
+// record per clip. A ticket outlives cancellation — clearing a cutout and
+// undoing it comes back to the same fingerprint, and the undo resumes the
+// paid track. Landing or terminal failure drops it; the sweep retires the
+// rest: a changed selection, a clip gone from the doc, or plain age.
+const TICKETS_KEY = "cut-matte-bake-tickets";
+const TICKET_TTL_MS = 48 * 60 * 60 * 1000;
+type PersistedTicket = {
+ projectId: string;
+ clipId: string;
+ fp: string;
+ gen: HostedBakeTicket;
+ /** When the submit happened; absent on records from before the stamp. */
+ at?: number;
+};
+
+function readTickets(): PersistedTicket[] {
+ try {
+ const v = JSON.parse(localStorage.getItem(TICKETS_KEY) ?? "[]") as unknown;
+ return Array.isArray(v) ? (v as PersistedTicket[]) : [];
+ } catch {
+ return [];
+ }
+}
+
+function writeTickets(tickets: PersistedTicket[]): void {
+ try {
+ localStorage.setItem(TICKETS_KEY, JSON.stringify(tickets));
+ } catch {
+ // Storage full/blocked — the resume convenience just won't persist.
+ }
+}
+
+function saveTicket(ticket: PersistedTicket): void {
+ writeTickets([
+ ...readTickets().filter((t) => t.clipId !== ticket.clipId),
+ { ...ticket, at: Date.now() },
+ ]);
+}
+
+function dropTicket(clipId: string): void {
+ const held = readTickets();
+ const rest = held.filter((t) => t.clipId !== clipId);
+ if (rest.length !== held.length) writeTickets(rest);
+}
+
+/** The persisted in-flight track for this exact bake, if a previous session
+ * paid for one that never landed. */
+function heldTicket(projectId: string, clipId: string, fp: string): HostedBakeTicket | undefined {
+ return readTickets().find(
+ (t) => t.projectId === projectId && t.clipId === clipId && t.fp === fp
+ )?.gen;
+}
+
+/** Bakes that failed, per quality, so a doc sweep leaves the error standing
+ * until the clip changes or the user retries. */
+const failed = new Map<string, { fp: string; quality: MatteQuality }>();
+
+const clipOf = (clipId: string): VideoClip | undefined =>
+ useEditor.getState().clips.find((c) => c.id === clipId);
+
+/** The fingerprint an AI-mode clip's matte must carry, or null when the clip
+ * owes no matte at all. */
+function owedFingerprint(clip: VideoClip | undefined): string | null {
+ const r = clip?.removal;
+ if (!clip || !r || (r.mode !== "auto" && r.mode !== "custom")) return null;
+ // Bakes are explicitly started — the panel button or the chat tool sets
+ // `requested`. Picking a mode alone owes nothing.
+ if (!r.requested) return null;
+ // A custom removal with nothing selected yet owes nothing — the tracker
+ // needs painted seeds or a described subject to follow.
+ if (
+ r.mode === "custom" &&
+ !r.seeds?.prompts.length &&
+ !r.seeds?.paint?.length &&
+ !r.subject?.trim()
+ )
+ return null;
+ return removalFingerprint(clip.assetId, r);
+}
+
+/** Whether the stored matte's baked range still spans the clip's trims —
+ * a matte wider than the clip plays is fine; one the trim reached past
+ * wants a re-bake. Stills always cover. */
+function matteCovers(matte: NonNullable<ClipRemoval["matte"]>, clip: VideoClip): boolean {
+ const s = useEditor.getState();
+ const matteAsset = s.assets.find((a) => a.id === matte.assetId);
+ const source = s.assets.find((a) => a.id === clip.assetId);
+ if (!matteAsset) return false;
+ if (source?.type === "image") return true;
+ const dur = matteAsset.duration ?? 0;
+ return clip.in >= matte.in - 0.05 && clip.out <= matte.in + dur + 0.2;
+}
+
+/** A matte some other clip already carries for this exact selection, whose
+ * baked range spans this clip's trims. hq wins when both tiers exist. */
+function twinMatte(
+ clipId: string,
+ fp: string,
+ clip: VideoClip
+): NonNullable<ClipRemoval["matte"]> | null {
+ const s = useEditor.getState();
+ let found: NonNullable<ClipRemoval["matte"]> | null = null;
+ for (const c of s.clips) {
+ if (c.id === clipId) continue;
+ const m = c.removal?.matte;
+ if (!m || m.fingerprint !== fp || !s.assets.some((a) => a.id === m.assetId)) continue;
+ if (!matteCovers(m, clip)) continue;
+ if (m.quality === "hq") return m;
+ found ??= m;
+ }
+ return found;
+}
+
+export function cancelMatteBake(clipId: string): void {
+ running.get(clipId)?.ctl.abort();
+ running.delete(clipId);
+ failed.delete(clipId);
+ // The ticket stays: the paid track keeps running upstream, and an undo
+ // that brings the same selection back resumes it free. The sweep retires
+ // tickets that can never land.
+ setJob(clipId, null);
+}
+
+/** Clear a failed bake and run the ladder again — the panel's Retry, and the
+ * upgrade affordance after a credit top-up. */
+export function retryMatteBake(clipId: string): void {
+ failed.delete(clipId);
+ setJob(clipId, null);
+ ensureMatteBake(clipId, { whileBrushing: true });
+}
+
+/** The brush panel's Apply: start the tracked bake for what's painted so
+ * far, with the session still open for touch-ups. */
+export function confirmMatteBake(clipId: string): void {
+ ensureMatteBake(clipId, { whileBrushing: true });
+}
+
+/**
+ * Make sure the clip's AI matte exists and matches its current trims and
+ * seeds; start (or restart) whatever rung of the ladder is owed. Safe to
+ * call on every panel render.
+ */
+export function ensureMatteBake(clipId: string, opts: { whileBrushing?: boolean } = {}): void {
+ if (!matteBakesAvailable) return;
+ const clip = clipOf(clipId);
+ const fp = owedFingerprint(clip);
+ if (!clip || !fp) {
+ cancelMatteBake(clipId);
+ return;
+ }
+ // An open brush session re-fingerprints on every stroke; the bake (and the
+ // hosted tier's bill) waits until it closes or the panel's Apply confirms.
+ if (!opts.whileBrushing && useBrushUi.getState().clipId === clipId) return;
+ const r = clip.removal!;
+ // Settled: the stored matte matches the clip and its asset is still around
+ // (an undo can revive a matte pointer whose asset was collected).
+ const matte = r.matte;
+ const stored = matte?.fingerprint === fp && matteCovers(matte, clip) ? matte : null;
+ // A split or a paste leaves two clips owing the same selection. A twin
+ // matte another clip already carries is adopted whole — same asset, one
+ // bake, one bill — as long as its baked range spans this clip's trims. A
+ // stored local matte adopts a twin only for the hq upgrade it still owes.
+ const twin = twinMatte(clipId, fp, clip);
+ const adopt = !stored ? twin : stored.quality === "local" && twin?.quality === "hq" ? twin : null;
+ if (adopt) {
+ useEditor.getState().updateClipTransient(clipId, { removal: { ...r, matte: adopt } });
+ ensureMatteBake(clipId, opts);
+ return;
+ }
+ // What the ladder owes next. Custom tracks an arbitrary object, so only the
+ // hosted tracker can bake it; auto gets the free local matte immediately
+ // and the hosted pass upgrades it.
+ const want: MatteQuality | null =
+ r.mode === "custom"
+ ? stored
+ ? null
+ : "hq"
+ : stored
+ ? stored.quality === "local"
+ ? "hq"
+ : null
+ : "local";
+ if (!want) {
+ cancelMatteBake(clipId);
+ return;
+ }
+ const stuck = failed.get(clipId);
+ if (stuck && stuck.fp === fp && stuck.quality === want) return;
+ const inFlight = running.get(clipId);
+ if (inFlight && inFlight.fp === fp && inFlight.quality === want) return;
+ // Another clip is already baking this exact selection — one submit, one
+ // bill. When it lands, the sweep runs this clip's ensure again and the
+ // twin adoption above picks the result up (or bakes for real when the
+ // twin's range turns out not to cover this clip's trims).
+ for (const [otherId, run] of running) {
+ if (otherId !== clipId && run.fp === fp && (run.quality === want || run.quality === "hq"))
+ return;
+ }
+ inFlight?.ctl.abort();
+
+ const ctl = new AbortController();
+ running.set(clipId, { fp, quality: want, ctl });
+ failed.delete(clipId);
+ setJob(clipId, { quality: want, status: "running", progress: 0 });
+ void runBake(clipId, fp, want, ctl).catch((e: unknown) => {
+ if (running.get(clipId)?.ctl !== ctl) return;
+ running.delete(clipId);
+ if (e instanceof MatteBakeCancelled) {
+ setJob(clipId, null);
+ return;
+ }
+ // A dead track's ticket has nothing to resume — drop it so Retry submits
+ // fresh. A transient failure keeps it, and the retry re-attaches.
+ if (e instanceof MatteTrackFailed) dropTicket(clipId);
+ failed.set(clipId, { fp, quality: want });
+ setJob(clipId, {
+ quality: want,
+ status: "error",
+ progress: 0,
+ error: e instanceof Error ? e.message : "The matte could not be baked.",
+ });
+ });
+}
+
+// Bakes run one at a time: each opens its own video decoders on top of the
+// preview's, and the browser's decoder pool is finite — two clips refining at
+// once is how a healthy mask dies with a decoder error.
+let bakeTurn: Promise<void> = Promise.resolve();
+
+async function runBake(
+ clipId: string,
+ fp: string,
+ quality: MatteQuality,
+ ctl: AbortController
+): Promise<void> {
+ const prev = bakeTurn;
+ let release!: () => void;
+ bakeTurn = new Promise((r) => (release = r));
+ try {
+ await prev;
+ await runBakeNow(clipId, fp, quality, ctl);
+ } finally {
+ release();
+ }
+}
+
+async function runBakeNow(
+ clipId: string,
+ fp: string,
+ quality: MatteQuality,
+ ctl: AbortController
+): Promise<void> {
+ if (ctl.signal.aborted) throw new MatteBakeCancelled();
+ const state = useEditor.getState();
+ const clip = clipOf(clipId);
+ const asset = clip && state.assets.find((a) => a.id === clip.assetId);
+ if (!clip || !asset || !state.projectId) throw new MatteBakeCancelled();
+
+ let shown = 0;
+ const startedAt = Date.now();
+ const onProgress = (f: number) => {
+ if (f - shown < 0.03 && f < 1) return;
+ shown = f;
+ if (running.get(clipId)?.ctl !== ctl) return;
+ const elapsed = (Date.now() - startedAt) / 1000;
+ const secondsLeft =
+ f >= 0.05 && elapsed >= 3
+ ? Math.max(0, Math.round((elapsed * (1 - f)) / f))
+ : undefined;
+ setJob(clipId, {
+ quality,
+ status: "running",
+ progress: f,
+ secondsLeft,
+ etaAt: secondsLeft !== undefined ? Date.now() : undefined,
+ });
+ };
+ const projectId = state.projectId;
+ const baked =
+ quality === "hq"
+ ? await hostedBakeMatte(asset, clip, {
+ signal: ctl.signal,
+ onProgress,
+ resume: heldTicket(projectId, clipId, fp),
+ onSubmitted: (gen) => saveTicket({ projectId, clipId, fp, gen }),
+ })
+ : await localBakeMatte(asset, clip, { signal: ctl.signal, onProgress });
+ if (ctl.signal.aborted) throw new MatteBakeCancelled();
+
+ const file = new File([baked.blob], `matte-${quality}-${fp}.mp4`, { type: "video/mp4" });
+ const matteAsset = await importFileToProject(state.projectId, file);
+ if (!matteAsset) throw new Error("The matte could not be saved into the project.");
+ matteAsset.name = `${asset.name} matte`;
+ matteAsset.origin = "matte";
+
+ // The doc may have moved while the bake ran — land only onto the removal
+ // the matte was baked for.
+ const editor = useEditor.getState();
+ const latest = clipOf(clipId);
+ const owed = owedFingerprint(latest);
+ editor.addAsset(matteAsset);
+ dropTicket(clipId);
+ if (ctl.signal.aborted || !latest || owed !== fp) {
+ editor.removeAsset(matteAsset.id);
+ running.delete(clipId);
+ setJob(clipId, null);
+ // Trims or seeds changed under the bake: the clip still owes a matte.
+ if (owed) ensureMatteBake(clipId);
+ return;
+ }
+
+ const removal: ClipRemoval = {
+ ...latest.removal!,
+ matte: { assetId: matteAsset.id, fingerprint: fp, quality, in: baked.in },
+ };
+ editor.updateClipTransient(clipId, { removal });
+ // The matte this replaced may still be another clip's (a split, a paste);
+ // the sweep collects it once nothing points at it.
+ running.delete(clipId);
+ setJob(clipId, null);
+ // Auto mode's local matte just landed: the quality pass is owed next.
+ ensureMatteBake(clipId);
+}
+
+/** Drop a cleared removal's matte asset (and its file) from the project. */
+export function dropMatteAsset(assetId: string | undefined): void {
+ if (!assetId) return;
+ const editor = useEditor.getState();
+ if (editor.assets.some((a) => a.id === assetId && a.origin === "matte"))
+ editor.removeAsset(assetId);
+}
+
+/** Settle every clip's matte against the doc: start owed bakes, stop bakes
+ * whose clip is gone or left AI mode, collect matte assets nothing points at.
+ * Undo, redo, trims and clip deletion all land here. */
+function sweepMattes(): void {
+ const s = useEditor.getState();
+ if (!s.loaded || s.readOnly || !s.projectId) return;
+ for (const id of [...running.keys()]) if (!owedFingerprint(clipOf(id))) cancelMatteBake(id);
+ for (const c of s.clips) if (owedFingerprint(c)) ensureMatteBake(c.id);
+ // Tickets retire by age alone. A cutout that is cleared, a clip that is
+ // deleted, a selection that moved on — each keeps its ticket, because an
+ // undo brings the old fingerprint back and resumes the paid track for
+ // free; a landing or a dead track already dropped its own record, and age
+ // retires whatever never comes back.
+ for (const t of readTickets()) {
+ if (t.projectId !== s.projectId) continue;
+ if (Date.now() - (t.at ?? 0) > TICKET_TTL_MS) dropTicket(t.clipId);
+ }
+ const pointed = new Set<string>();
+ for (const c of s.clips) if (c.removal?.matte) pointed.add(c.removal.matte.assetId);
+ for (const a of s.assets) if (a.origin === "matte" && !pointed.has(a.id)) s.removeAsset(a.id);
+}
+
+// Debounced: a trim drag patches the doc per pointer move, and a landing bake
+// registers its asset one set() before the clip points at it — the pause lets
+// each settle before the sweep reads it.
+let sweepTimer: ReturnType<typeof setTimeout> | null = null;
+useEditor.subscribe((s, prev) => {
+ if (s.clips === prev.clips && s.assets === prev.assets) return;
+ if (sweepTimer) clearTimeout(sweepTimer);
+ sweepTimer = setTimeout(sweepMattes, 600);
+});
+
+// A closed brush session releases the held bake for its clip.
+useBrushUi.subscribe((s, prev) => {
+ if (prev.clipId && prev.clipId !== s.clipId) ensureMatteBake(prev.clipId);
+});
diff --git a/site/src/cut/lib/removal/brushUi.ts b/site/src/cut/lib/removal/brushUi.ts
new file mode 100644
index 00000000..d90f9bcc
--- /dev/null
+++ b/site/src/cut/lib/removal/brushUi.ts
@@ -0,0 +1,38 @@
+"use client";
+
+/**
+ * The custom-removal brush session: which clip is being painted on, the tool
+ * in hand, and the brush size. View state shared by the removal panel (the
+ * tool buttons) and the stage gizmo (the pointer) — never saved; the strokes
+ * themselves land in the doc as `removal.seeds`.
+ */
+
+import { create } from "zustand";
+
+export type BrushTool = "quick" | "brush" | "quickErase" | "erase";
+
+/** Brush diameter bounds, as a fraction of the frame's short side. */
+export const BRUSH_SIZE_MIN = 0.02;
+export const BRUSH_SIZE_MAX = 0.25;
+export const BRUSH_SIZE_DEFAULT = 0.07;
+
+interface BrushUi {
+ /** The clip being painted, or null when no brush session is open. */
+ clipId: string | null;
+ tool: BrushTool;
+ size: number;
+ open: (clipId: string) => void;
+ close: () => void;
+ setTool: (tool: BrushTool) => void;
+ setSize: (size: number) => void;
+}
+
+export const useBrushUi = create<BrushUi>((set) => ({
+ clipId: null,
+ tool: "quick",
+ size: BRUSH_SIZE_DEFAULT,
+ open: (clipId) => set({ clipId, tool: "quick" }),
+ close: () => set({ clipId: null }),
+ setTool: (tool) => set({ tool }),
+ setSize: (size) => set({ size: Math.min(BRUSH_SIZE_MAX, Math.max(BRUSH_SIZE_MIN, size)) }),
+}));
diff --git a/site/src/cut/lib/removal/hostedBake.ts b/site/src/cut/lib/removal/hostedBake.ts
new file mode 100644
index 00000000..784eb9d8
--- /dev/null
+++ b/site/src/cut/lib/removal/hostedBake.ts
@@ -0,0 +1,391 @@
+"use client";
+
+/**
+ * The quality matte bake: the clip's trimmed segment goes to the hosted
+ * tracker (credits), which propagates the subject across every frame; the
+ * returned mask video is refined here against the source pixels with the
+ * kit's guided filter and encoded as the final matte asset. Auto mode seeds
+ * the tracker from the on-device person matte; custom mode seeds it from the
+ * brush's stored prompts and paint. Driven by the page or the cloud worker —
+ * the local engine only ever consumes the stored matte.
+ */
+
+import {
+ coverageToGray,
+ MATTE_BITRATE,
+ MATTE_FPS,
+ MATTE_SHORT,
+ refineMatteAgainstFrame,
+} from "@donkeycut/effects-kit";
+import { openCanvasVideo, scaledEvenSize } from "../canvasVideo";
+import { NO_CREDITS_MESSAGE } from "../credits";
+import { ClipReader } from "../exportRender";
+import { hostedPost } from "../hosted";
+import { storeHostedBlob } from "../hostedBlobs";
+import { createRasterCanvas, decodeRasterImageUrl } from "../raster";
+import type { MediaAsset, RemovalSeeds, VideoClip } from "../types";
+import { localBakeMatte, MatteBakeCancelled } from "./localBake";
+
+/** Segment sent to the tracker: capped resolution and rate — the mask comes
+ * back at the same shape, and tracking cost scales with both. The bitrate
+ * also shrinks with length so the whole segment stays under the upload
+ * route's 32MB blob cap: a long clip trades bitrate for admission. */
+const SEGMENT_SHORT = 720;
+const SEGMENT_BITRATE = 2_500_000;
+const SEGMENT_BYTE_BUDGET = 30 * 1024 * 1024;
+const POLL_MS = 5_000;
+const DEADLINE_MS = 10 * 60_000;
+
+/** The hosted tier answered 402: credits unlock it, the local matte stands.
+ * Carries the app-wide balance message so every surface renders it the same
+ * way, with the credits link. */
+export class MatteNeedsCredits extends Error {
+ constructor() {
+ super(NO_CREDITS_MESSAGE);
+ }
+}
+
+/** The tracker settled this generation as failed — the charge is spent and
+ * the track is dead, so the caller drops its persisted ticket and a retry
+ * submits fresh. Transient errors (a poll that couldn't reach the route)
+ * throw plain errors instead, and the ticket stands for a resume. */
+export class MatteTrackFailed extends Error {}
+
+type Point = { x: number; y: number; frame: number; label: 0 | 1; object: string };
+
+interface Generation {
+ id: string;
+ status: string;
+ provider: string;
+ model: string;
+ providerJobId?: string | null;
+ providerGenerationId?: string | null;
+ providerPollingUrl?: string | null;
+ metadata?: Record<string, unknown>;
+ outputs: { url?: string; dataBase64?: string; contentType?: string }[];
+ error?: unknown;
+}
+
+function checkAborted(signal: AbortSignal | undefined) {
+ if (signal?.aborted) throw new MatteBakeCancelled();
+}
+
+/** Encode the clip's trimmed source range as a compact video-only segment. */
+async function renderSegment(
+ asset: MediaAsset,
+ clip: VideoClip,
+ signal: AbortSignal | undefined
+): Promise<{ blob: Blob; from: number; w: number; h: number; frames: number }> {
+ const still = asset.type === "image";
+ const from = still ? 0 : Math.max(0, clip.in);
+ const to = still ? from + 1 / MATTE_FPS : Math.min(clip.out, asset.duration || clip.out);
+ const frames = still ? 1 : Math.max(1, Math.ceil(Math.max(1 / MATTE_FPS, to - from) * MATTE_FPS));
+ const reader = new ClipReader(asset, () => asset.url);
+ try {
+ const first = await reader.frameAt(from);
+ if (first.kind !== "ready") throw new Error("The clip's picture could not be read.");
+ const { w, h } = scaledEvenSize(first.width, first.height, SEGMENT_SHORT);
+ const seconds = frames / MATTE_FPS;
+ const enc = await openCanvasVideo({
+ width: w,
+ height: h,
+ fps: MATTE_FPS,
+ frames,
+ bitrate: Math.min(SEGMENT_BITRATE, Math.floor((SEGMENT_BYTE_BUDGET * 8) / seconds)),
+ });
+ for (let i = 0; i < frames; i++) {
+ if (signal?.aborted) {
+ await enc.cancel();
+ throw new MatteBakeCancelled();
+ }
+ const at = Math.min(to - 1 / (MATTE_FPS * 2), from + i / MATTE_FPS);
+ const frame = await reader.frameAt(at);
+ enc.ctx.fillStyle = "#000000";
+ enc.ctx.fillRect(0, 0, w, h);
+ if (frame.kind === "ready") enc.ctx.drawImage(frame.image, 0, 0, w, h);
+ await enc.add(i / MATTE_FPS, 1 / MATTE_FPS);
+ }
+ return { blob: await enc.finish(), from, w, h, frames };
+ } finally {
+ reader.dispose();
+ }
+}
+
+/** Centroid of the white pixels in an alpha or luma image, as fractions;
+ * null when nothing is marked. */
+function centroidOf(px: Uint8ClampedArray, w: number, h: number, channel: 0 | 3): { x: number; y: number } | null {
+ let sx = 0;
+ let sy = 0;
+ let count = 0;
+ for (let y = 0; y < h; y++) {
+ for (let x = 0; x < w; x++) {
+ if (px[(y * w + x) * 4 + channel] > 160) {
+ sx += x;
+ sy += y;
+ count++;
+ }
+ }
+ }
+ // A handful of lit pixels is a real mark — the smallest brush dab covers
+ // ~20px² at the paint bitmap's 256px short side.
+ if (count < 4) return null;
+ return { x: sx / count / w, y: sy / count / h };
+}
+
+/** Custom mode's tracker seeds, from the brush's stored prompts and paint.
+ * A seed the trims moved past points at a frame the segment doesn't hold —
+ * clamped it would prompt on a different moment of the shot, so it is
+ * dropped instead. */
+async function seedPoints(
+ seeds: RemovalSeeds,
+ seg: { w: number; h: number; frames: number },
+ from: number
+): Promise<Point[]> {
+ const frameOf = (t: number): number | null => {
+ const frame = Math.round((t - from) * MATTE_FPS);
+ return frame >= 0 && frame < seg.frames ? frame : null;
+ };
+ const out: Point[] = [];
+ seeds.prompts.forEach((p, i) => {
+ const frame = frameOf(p.t);
+ if (frame === null) return;
+ const keeps = p.points.some((pt) => pt.label === 1);
+ for (const pt of p.points) {
+ out.push({
+ x: pt.x * seg.w,
+ y: pt.y * seg.h,
+ frame,
+ label: pt.label,
+ // A refine-only stroke (all background points) attaches to the first
+ // tracked object; a keep stroke tracks its own object.
+ object: keeps ? `obj${i}` : "obj0",
+ });
+ }
+ });
+ for (const p of seeds.paint ?? []) {
+ const frame = frameOf(p.t);
+ if (frame === null) continue;
+ for (const [url, label] of [
+ [p.add, 1],
+ [p.erase, 0],
+ ] as const) {
+ if (!url) continue;
+ const img = await decodeRasterImageUrl(url).catch(() => null);
+ if (!img) continue;
+ const c = createRasterCanvas(img.width, img.height);
+ const ctx = c.getContext("2d", { willReadFrequently: true }) as CanvasRenderingContext2D;
+ ctx.drawImage(img.source, 0, 0);
+ const px = ctx.getImageData(0, 0, img.width, img.height).data;
+ const point = centroidOf(px, img.width, img.height, 0);
+ if (point) {
+ out.push({ x: point.x * seg.w, y: point.y * seg.h, frame, label, object: "paint" });
+ }
+ }
+ }
+ return out;
+}
+
+/** What a reload needs to re-attach to a paid in-flight track: the
+ * generation's poll payload, as the submit returned it. */
+export type HostedBakeTicket = Generation;
+
+/**
+ * Run the hosted quality bake for a clip and return the refined matte, ready
+ * to store as the project's matte asset. Throws `MatteNeedsCredits` on a 402,
+ * `MatteTrackFailed` when the tracker settled the paid run as dead, and
+ * `MatteBakeCancelled` on abort. The submit bills, so the caller persists the
+ * ticket handed to `onSubmitted` and passes it back as `resume` after a
+ * reload — the bake then re-attaches to the running track and pays nothing
+ * new.
+ */
+export async function hostedBakeMatte(
+ asset: MediaAsset,
+ clip: VideoClip,
+ opts: {
+ onProgress?: (fraction: number) => void;
+ signal?: AbortSignal;
+ resume?: HostedBakeTicket;
+ onSubmitted?: (ticket: HostedBakeTicket) => void;
+ } = {}
+): Promise<{ blob: Blob; in: number }> {
+ const r = clip.removal;
+ if (!r) throw new Error("The clip has no removal to bake.");
+ const still = asset.type === "image";
+ const from = still ? 0 : Math.max(0, clip.in);
+ let gen: Generation;
+ if (opts.resume) {
+ gen = opts.resume;
+ opts.onProgress?.(0.2);
+ } else {
+ opts.onProgress?.(0.02);
+ const segment = await renderSegment(asset, clip, opts.signal);
+ opts.onProgress?.(0.15);
+ // The words and the strokes ride together: auto mode and a described
+ // subject send a text prompt, painted seeds send point prompts, and a
+ // custom removal with both sends both — the segmenter's points refine what
+ // the words detected. The route picks the configured segmenter.
+ const painted =
+ r.mode === "custom" && !!(r.seeds?.prompts.length || r.seeds?.paint?.length);
+ const concept = r.mode === "custom" ? (r.subject ?? "").trim() : "person";
+ const points = painted ? await seedPoints(r.seeds!, segment, segment.from) : [];
+ if (!concept && points.length === 0) throw new Error("No selection to track.");
+ const parameters = {
+ ...(concept ? { prompt: concept } : {}),
+ ...(points.length > 0 ? { points } : {}),
+ };
+ checkAborted(opts.signal);
+
+ const blobRef = await storeHostedBlob(segment.blob, "donkey-cut");
+ if (!blobRef) throw new Error("The segment could not reach storage.");
+ opts.onProgress?.(0.2);
+
+ const submit = await hostedPost("/api/inference/assets", {
+ kind: "matte",
+ prompt: "Track the selected subject through the clip.",
+ inputs: { video: { blobRef, blobField: "url", blobUrl: true, mimeType: "video/mp4" } },
+ parameters,
+ });
+ if (submit.status === 402) throw new MatteNeedsCredits();
+ if (!submit.ok) throw new Error("The quality cutout could not start.");
+ gen = (await submit.json()) as Generation;
+ // The submit is the billable moment: from here the ticket outlives the
+ // page, so a reload resumes this track instead of paying for another.
+ opts.onSubmitted?.(gen);
+ }
+
+ const deadline = Date.now() + DEADLINE_MS;
+ while (gen.status === "in_progress" || gen.status === "pending") {
+ checkAborted(opts.signal);
+ if (Date.now() > deadline) throw new Error("The quality cutout is taking too long.");
+ await new Promise((r2) => setTimeout(r2, POLL_MS));
+ checkAborted(opts.signal);
+ const poll = await hostedPost("/api/inference/assets/refresh", {
+ id: gen.id,
+ kind: "matte",
+ provider: gen.provider,
+ model: gen.model,
+ providerJobId: gen.providerJobId,
+ providerGenerationId: gen.providerGenerationId,
+ providerPollingUrl: gen.providerPollingUrl,
+ metadata: gen.metadata ?? {},
+ });
+ if (!poll.ok) throw new Error("The quality cutout failed.");
+ gen = (await poll.json()) as Generation;
+ opts.onProgress?.(Math.min(0.7, 0.2 + (1 - (deadline - Date.now()) / DEADLINE_MS) * 2));
+ }
+ if (gen.status !== "completed") {
+ const message =
+ typeof (gen.error as { message?: string } | undefined)?.message === "string"
+ ? (gen.error as { message: string }).message
+ : "The quality cutout failed.";
+ throw new MatteTrackFailed(message);
+ }
+ const maskUrl = gen.outputs.find((o) => o.url)?.url;
+ if (!maskUrl) throw new MatteTrackFailed("The tracker returned no mask.");
+ checkAborted(opts.signal);
+ opts.onProgress?.(0.72);
+
+ return refineAndEncode(asset, clip, maskUrl, from, opts);
+}
+
+/** Refine the tracker's mask against the source pixels and encode the final
+ * grayscale matte. */
+async function refineAndEncode(
+ asset: MediaAsset,
+ clip: VideoClip,
+ maskUrl: string,
+ from: number,
+ opts: { onProgress?: (fraction: number) => void; signal?: AbortSignal }
+): Promise<{ blob: Blob; in: number }> {
+ const still = asset.type === "image";
+ const to = still ? from + 1 / MATTE_FPS : Math.min(clip.out, asset.duration || clip.out);
+ const frames = still ? 1 : Math.max(1, Math.ceil(Math.max(1 / MATTE_FPS, to - from) * MATTE_FPS));
+ // The browser reads the mask from a local blob: the whole file downloads
+ // once, and the decoder never streams the provider's CDN. Headless runs
+ // have no object URLs and read the URL directly.
+ let localUrl: string | null = null;
+ if (typeof URL !== "undefined" && typeof URL.createObjectURL === "function") {
+ const got = await fetch(maskUrl).catch(() => null);
+ if (!got?.ok) throw new Error("The tracked mask could not be downloaded.");
+ localUrl = URL.createObjectURL(await got.blob());
+ }
+ const readUrl = localUrl ?? maskUrl;
+ const maskAsset: MediaAsset = {
+ id: "hosted-mask",
+ type: "video",
+ url: readUrl,
+ fileName: "hosted-mask.mp4",
+ name: "hosted-mask",
+ } as MediaAsset;
+ const maskReader = new ClipReader(maskAsset, () => readUrl);
+ const srcReader = new ClipReader(asset, () => asset.url);
+ try {
+ const firstMask = await maskReader.frameAt(0);
+ if (firstMask.kind !== "ready") throw new Error("The tracked mask could not be read.");
+ const { w, h } = scaledEvenSize(firstMask.width, firstMask.height, MATTE_SHORT);
+ const enc = await openCanvasVideo({
+ width: w,
+ height: h,
+ fps: MATTE_FPS,
+ frames,
+ bitrate: MATTE_BITRATE,
+ readback: true,
+ });
+ const ctx = enc.ctx;
+ const guide = createRasterCanvas(w, h);
+ const gctx = guide.getContext("2d", { willReadFrequently: true }) as CanvasRenderingContext2D;
+ // A track that detected nothing comes back as an all-black mask video; an
+ // empty matte would make the clip vanish, so it fails the bake instead.
+ let lit = false;
+
+ for (let i = 0; i < frames; i++) {
+ if (opts.signal?.aborted) {
+ await enc.cancel();
+ throw new MatteBakeCancelled();
+ }
+ const t = i / MATTE_FPS;
+ const mask = await maskReader.frameAt(Math.min(t, (frames - 0.5) / MATTE_FPS));
+ const src = await srcReader.frameAt(still ? 0 : Math.min(to - 1 / (MATTE_FPS * 2), from + t));
+ ctx.fillStyle = "#000000";
+ ctx.fillRect(0, 0, w, h);
+ if (mask.kind === "ready") {
+ ctx.imageSmoothingEnabled = true;
+ ctx.drawImage(mask.image, 0, 0, w, h);
+ const mattePx = ctx.getImageData(0, 0, w, h);
+ coverageToGray(mattePx.data);
+ if (!lit) {
+ const px = mattePx.data;
+ for (let p = 0; p < px.length; p += 4) {
+ if (px[p] > 32) {
+ lit = true;
+ break;
+ }
+ }
+ }
+ if (src.kind === "ready") {
+ gctx.drawImage(src.image, 0, 0, w, h);
+ const framePx = gctx.getImageData(0, 0, w, h);
+ refineMatteAgainstFrame(mattePx.data, framePx.data, w, h, 6);
+ }
+ ctx.putImageData(mattePx, 0, 0);
+ }
+ await enc.add(t, 1 / MATTE_FPS);
+ opts.onProgress?.(0.72 + ((i + 1) / frames) * 0.27);
+ }
+ if (!lit) {
+ await enc.cancel();
+ throw new MatteTrackFailed(
+ "The tracker found nothing to keep — describe the subject or paint it."
+ );
+ }
+ return { blob: await enc.finish(), in: from };
+ } finally {
+ maskReader.dispose();
+ srcReader.dispose();
+ if (localUrl) URL.revokeObjectURL(localUrl);
+ }
+}
+
+// The local quick bake rides beside the hosted one so callers pick a rung
+// without importing two modules.
+export { localBakeMatte, MatteBakeCancelled };
diff --git a/site/src/cut/lib/removal/interactive.ts b/site/src/cut/lib/removal/interactive.ts
new file mode 100644
index 00000000..601a5fcb
--- /dev/null
+++ b/site/src/cut/lib/removal/interactive.ts
@@ -0,0 +1,305 @@
+"use client";
+
+/**
+ * The custom-removal selection session: what the brush has picked on the
+ * current frame, held as a working mask the stage draws as a red overlay.
+ * Quick strokes run the on-device tap-to-select segmenter for instant
+ * feedback; manual strokes paint the mask directly. Every finished stroke
+ * also yields its seed record — the point prompts and paint bitmaps the doc
+ * stores, which the hosted tracker replays across the whole clip.
+ */
+
+import { matteLumaToAlpha } from "@donkeycut/effects-kit";
+import { interactiveSegmenter, segmentTouchAlpha } from "../cutout";
+import { sampleClipSource } from "../previewCanvas";
+import { createRasterCanvas, rasterCanvasToDataUrl, type RasterSurface } from "../raster";
+import type { RemovalSeeds } from "../types";
+
+/** Working-mask short side: enough for a crisp overlay, cheap to re-segment
+ * on every pointer move. */
+const WORK_SHORT = 512;
+/** Stored paint bitmaps stay small — they seed a tracker, never a render. */
+const PAINT_SHORT = 256;
+
+type Point = { x: number; y: number };
+
+export class QuickSelectSession {
+ /** The current frame, drawn at working size. */
+ private work: RasterSurface;
+ /** White where the selection is. */
+ private mask: RasterSurface;
+ /** The stroke in flight, previewed over (or out of) the mask until the
+ * pointer lifts. */
+ private tentative: RasterSurface;
+ /** Scratch for compositing mask + tentative into the red overlay tint. */
+ private view: RasterSurface;
+ private tentativeErase = false;
+ private hasTentative = false;
+ private w = 2;
+ private h = 2;
+ private seg: Awaited<ReturnType<typeof interactiveSegmenter>> = null;
+
+ private constructor() {
+ this.work = createRasterCanvas(2, 2);
+ this.mask = createRasterCanvas(2, 2);
+ this.tentative = createRasterCanvas(2, 2);
+ this.view = createRasterCanvas(2, 2);
+ }
+
+ /** Null when the tap-to-select model is unavailable — the manual brush and
+ * erase still work through the returned session in that case. */
+ static async open(): Promise<QuickSelectSession> {
+ const s = new QuickSelectSession();
+ s.seg = await interactiveSegmenter();
+ return s;
+ }
+
+ get quickAvailable(): boolean {
+ return !!this.seg;
+ }
+
+ /** Pull the clip's current decoder frame into the working canvas. False
+ * when no frame is ready yet. */
+ refreshFrame(clipId: string): boolean {
+ const src = sampleClipSource(clipId);
+ if (!src) return false;
+ const sw = Number((src as { width?: number }).width ?? 0) || 16;
+ const sh = Number((src as { height?: number }).height ?? 0) || 9;
+ const scale = WORK_SHORT / Math.max(1, Math.min(sw, sh));
+ const w = Math.max(2, Math.round(sw * scale));
+ const h = Math.max(2, Math.round(sh * scale));
+ if (this.w !== w || this.h !== h) {
+ this.w = w;
+ this.h = h;
+ this.work.width = w;
+ this.work.height = h;
+ const m = this.mask.getContext("2d") as CanvasRenderingContext2D | null;
+ const keep = m && (this.mask.width > 2 ? this.mask : null);
+ const prev = keep ? createRasterCanvas(this.mask.width, this.mask.height) : null;
+ if (prev && keep) {
+ (prev.getContext("2d") as CanvasRenderingContext2D).drawImage(keep as CanvasImageSource, 0, 0);
+ }
+ this.mask.width = w;
+ this.mask.height = h;
+ if (prev && m) m.drawImage(prev as CanvasImageSource, 0, 0, w, h);
+ this.tentative.width = w;
+ this.tentative.height = h;
+ this.hasTentative = false;
+ }
+ const ctx = this.work.getContext("2d") as CanvasRenderingContext2D | null;
+ if (!ctx) return false;
+ ctx.drawImage(src, 0, 0, w, h);
+ return true;
+ }
+
+ /** Run tap-to-select over the stroke and fold the result into the mask.
+ * `erase` subtracts the picked object. True when something registered. */
+ quickStroke(points: Point[], erase: boolean): boolean {
+ if (!this.seg || points.length === 0) return false;
+ const alpha = segmentTouchAlpha(this.seg, this.work as HTMLCanvasElement, points);
+ if (!alpha) return false;
+ const ctx = this.mask.getContext("2d") as CanvasRenderingContext2D | null;
+ if (!ctx) return false;
+ ctx.globalCompositeOperation = erase ? "destination-out" : "source-over";
+ ctx.imageSmoothingEnabled = true;
+ ctx.drawImage(alpha, 0, 0, this.w, this.h);
+ ctx.globalCompositeOperation = "source-over";
+ return true;
+ }
+
+ /** Preview a quick stroke while the pointer is still down: the picked
+ * object shows over (or carves out of) the mask, folded in only on
+ * `commitTentative`. */
+ quickPreview(points: Point[], erase: boolean): boolean {
+ if (!this.seg || points.length === 0) return false;
+ const alpha = segmentTouchAlpha(this.seg, this.work as HTMLCanvasElement, points);
+ const ctx = this.tentative.getContext("2d") as CanvasRenderingContext2D | null;
+ if (!ctx) return false;
+ ctx.clearRect(0, 0, this.w, this.h);
+ if (!alpha) {
+ this.hasTentative = false;
+ return false;
+ }
+ ctx.imageSmoothingEnabled = true;
+ ctx.drawImage(alpha, 0, 0, this.w, this.h);
+ this.tentativeErase = erase;
+ this.hasTentative = true;
+ return true;
+ }
+
+ /** Fold the previewed stroke into the mask. */
+ commitTentative(): void {
+ if (!this.hasTentative) return;
+ const ctx = this.mask.getContext("2d") as CanvasRenderingContext2D | null;
+ if (ctx) {
+ ctx.globalCompositeOperation = this.tentativeErase ? "destination-out" : "source-over";
+ ctx.drawImage(this.tentative as CanvasImageSource, 0, 0);
+ ctx.globalCompositeOperation = "source-over";
+ }
+ this.dropTentative();
+ }
+
+ dropTentative(): void {
+ this.hasTentative = false;
+ const ctx = this.tentative.getContext("2d") as CanvasRenderingContext2D | null;
+ ctx?.clearRect(0, 0, this.w, this.h);
+ }
+
+ /** Paint one manual stroke into the mask. `radius` is a fraction of the
+ * frame's short side. */
+ paintStroke(points: Point[], radius: number, erase: boolean): void {
+ const ctx = this.mask.getContext("2d") as CanvasRenderingContext2D | null;
+ if (!ctx || points.length === 0) return;
+ const r = Math.max(1, radius * Math.min(this.w, this.h));
+ ctx.globalCompositeOperation = erase ? "destination-out" : "source-over";
+ ctx.strokeStyle = "#ffffff";
+ ctx.fillStyle = "#ffffff";
+ ctx.lineWidth = r * 2;
+ ctx.lineCap = "round";
+ ctx.lineJoin = "round";
+ ctx.beginPath();
+ ctx.moveTo(points[0].x * this.w, points[0].y * this.h);
+ for (let i = 1; i < points.length; i++) ctx.lineTo(points[i].x * this.w, points[i].y * this.h);
+ if (points.length === 1) {
+ ctx.arc(points[0].x * this.w, points[0].y * this.h, r, 0, Math.PI * 2);
+ ctx.fill();
+ } else {
+ ctx.stroke();
+ }
+ ctx.globalCompositeOperation = "source-over";
+ }
+
+ /** The stroke's stored paint bitmap: the stroke alone, small grayscale PNG
+ * (white marks) for the seed record. */
+ async paintSeed(points: Point[], radius: number): Promise<string> {
+ const scale = PAINT_SHORT / Math.max(1, Math.min(this.w, this.h));
+ const w = Math.max(2, Math.round(this.w * scale));
+ const h = Math.max(2, Math.round(this.h * scale));
+ const c = createRasterCanvas(w, h);
+ const ctx = c.getContext("2d") as CanvasRenderingContext2D;
+ ctx.fillStyle = "#000000";
+ ctx.fillRect(0, 0, w, h);
+ const r = Math.max(1, radius * Math.min(w, h));
+ ctx.strokeStyle = "#ffffff";
+ ctx.fillStyle = "#ffffff";
+ ctx.lineWidth = r * 2;
+ ctx.lineCap = "round";
+ ctx.lineJoin = "round";
+ ctx.beginPath();
+ ctx.moveTo(points[0].x * w, points[0].y * h);
+ for (let i = 1; i < points.length; i++) ctx.lineTo(points[i].x * w, points[i].y * h);
+ if (points.length === 1) {
+ ctx.arc(points[0].x * w, points[0].y * h, r, 0, Math.PI * 2);
+ ctx.fill();
+ } else {
+ ctx.stroke();
+ }
+ return rasterCanvasToDataUrl(c, "image/png");
+ }
+
+ /** Rebuild the mask from the seeds recorded on the current frame: prompts
+ * re-segment, paint bitmaps replay over them. `at` is the frame's source
+ * second; a stroke recorded on another frame stays off this overlay — the
+ * segmenter would pick whatever sits under its points now, a different
+ * object than the stroke selected. */
+ replaySeeds(
+ seeds: RemovalSeeds | undefined,
+ decode: (url: string) => Promise<CanvasImageSource | null>,
+ at: number
+ ): Promise<void> {
+ this.reset();
+ if (!seeds) return Promise.resolve();
+ const here = (t: number) => Math.abs(t - at) < 0.05;
+ for (const p of seeds.prompts) {
+ if (!here(p.t)) continue;
+ const keeps = p.points.filter((pt) => pt.label === 1);
+ const drops = p.points.filter((pt) => pt.label === 0);
+ if (keeps.length) this.quickStroke(keeps, false);
+ if (drops.length) this.quickStroke(drops, true);
+ }
+ const paints = (seeds.paint ?? []).filter((p) => here(p.t));
+ return (async () => {
+ const ctx = this.mask.getContext("2d") as CanvasRenderingContext2D | null;
+ if (!ctx) return;
+ for (const p of paints) {
+ for (const [url, erase] of [
+ [p.add, false],
+ [p.erase, true],
+ ] as const) {
+ if (!url) continue;
+ const img = await decode(url);
+ if (!img) continue;
+ ctx.globalCompositeOperation = erase ? "destination-out" : "source-over";
+ // The paint is white on black; its luma is the stroke, so cut the
+ // black away first via a lightened draw onto a scratch.
+ const s = createRasterCanvas(this.w, this.h);
+ const sctx = s.getContext("2d") as CanvasRenderingContext2D;
+ sctx.drawImage(img, 0, 0, this.w, this.h);
+ const px = sctx.getImageData(0, 0, this.w, this.h);
+ matteLumaToAlpha(px.data);
+ sctx.putImageData(px, 0, 0);
+ ctx.drawImage(s as CanvasImageSource, 0, 0);
+ }
+ }
+ ctx.globalCompositeOperation = "source-over";
+ })();
+ }
+
+ reset(): void {
+ const ctx = this.mask.getContext("2d") as CanvasRenderingContext2D | null;
+ ctx?.clearRect(0, 0, this.mask.width, this.mask.height);
+ }
+
+ /** Whether anything is selected right now. */
+ hasSelection(): boolean {
+ const ctx = this.mask.getContext("2d") as CanvasRenderingContext2D | null;
+ if (!ctx) return false;
+ const px = ctx.getImageData(0, 0, this.mask.width, this.mask.height).data;
+ for (let i = 3; i < px.length; i += 64) if (px[i] > 32) return true;
+ return false;
+ }
+
+ /** Paint the selection (committed mask plus the stroke in flight) into a
+ * display context as the red overlay. */
+ paintOverlay(ctx: CanvasRenderingContext2D, w: number, h: number): void {
+ if (this.view.width !== this.w || this.view.height !== this.h) {
+ this.view.width = this.w;
+ this.view.height = this.h;
+ }
+ const view = this.view;
+ const vctx = view.getContext("2d") as CanvasRenderingContext2D;
+ vctx.globalCompositeOperation = "source-over";
+ vctx.clearRect(0, 0, this.w, this.h);
+ vctx.drawImage(this.mask as CanvasImageSource, 0, 0);
+ if (this.hasTentative) {
+ vctx.globalCompositeOperation = this.tentativeErase ? "destination-out" : "source-over";
+ vctx.drawImage(this.tentative as CanvasImageSource, 0, 0);
+ vctx.globalCompositeOperation = "source-over";
+ }
+ vctx.globalCompositeOperation = "source-in";
+ vctx.fillStyle = "#ff2244";
+ vctx.fillRect(0, 0, this.w, this.h);
+ ctx.clearRect(0, 0, w, h);
+ ctx.globalAlpha = 0.5;
+ ctx.imageSmoothingEnabled = true;
+ ctx.drawImage(view as CanvasImageSource, 0, 0, w, h);
+ ctx.globalAlpha = 1;
+ }
+
+ /** The selection mask (white, alpha-carrying) for the stage's red overlay. */
+ overlay(): CanvasImageSource {
+ return this.mask as CanvasImageSource;
+ }
+
+ /** The current frame at working size, for the loupe. */
+ frame(): CanvasImageSource {
+ return this.work as CanvasImageSource;
+ }
+
+ get width() {
+ return this.w;
+ }
+ get height() {
+ return this.h;
+ }
+}
diff --git a/site/src/cut/lib/removal/keyColor.ts b/site/src/cut/lib/removal/keyColor.ts
new file mode 100644
index 00000000..eb2edf3b
--- /dev/null
+++ b/site/src/cut/lib/removal/keyColor.ts
@@ -0,0 +1,64 @@
+"use client";
+
+import { ClipReader } from "../exportRender";
+import { sampleClipFrameData } from "../previewCanvas";
+import { createRasterCanvas } from "../raster";
+import type { MediaAsset, VideoClip } from "../types";
+
+const SAMPLE_W = 96;
+const SAMPLE_H = 54;
+
+/** The median of a small frame readout's border pixels, where a backdrop
+ * lives. */
+function medianBorder(data: Uint8ClampedArray, w: number): string {
+ const h = Math.round(data.length / 4 / w);
+ const chans: number[][] = [[], [], []];
+ const take = (x: number, y: number) => {
+ const i = (y * w + x) * 4;
+ for (let c = 0; c < 3; c++) chans[c].push(data[i + c]);
+ };
+ for (let x = 0; x < w; x += 2) {
+ take(x, 0);
+ take(x, h - 1);
+ }
+ for (let y = 1; y < h - 1; y += 2) {
+ take(0, y);
+ take(w - 1, y);
+ }
+ const hex = chans
+ .map((v) => {
+ v.sort((a, b) => a - b);
+ return v[Math.floor(v.length / 2)].toString(16).padStart(2, "0");
+ })
+ .join("");
+ return `#${hex}`;
+}
+
+/** The key color the clip's live preview frame suggests; null when no
+ * decoder currently shows the clip (off the playhead, headless). */
+export function suggestKeyColor(clipId: string): string | null {
+ const data = sampleClipFrameData(clipId);
+ return data ? medianBorder(data, SAMPLE_W) : null;
+}
+
+/** The key color read from the clip's own footage, decoded at its in-point —
+ * for callers with no live decoder on the clip (the chat tool, a clip away
+ * from the playhead). Null when the picture can't be read. */
+export async function clipKeyColor(asset: MediaAsset, clip: VideoClip): Promise<string | null> {
+ const reader = new ClipReader(asset, () => asset.url);
+ try {
+ const frame = await reader.frameAt(asset.type === "image" ? 0 : Math.max(0, clip.in));
+ if (frame.kind !== "ready") return null;
+ const canvas = createRasterCanvas(SAMPLE_W, SAMPLE_H);
+ const ctx = canvas.getContext("2d", {
+ willReadFrequently: true,
+ }) as CanvasRenderingContext2D | null;
+ if (!ctx) return null;
+ ctx.drawImage(frame.image, 0, 0, SAMPLE_W, SAMPLE_H);
+ return medianBorder(ctx.getImageData(0, 0, SAMPLE_W, SAMPLE_H).data, SAMPLE_W);
+ } catch {
+ return null;
+ } finally {
+ reader.dispose();
+ }
+}
diff --git a/site/src/cut/lib/removal/localBake.ts b/site/src/cut/lib/removal/localBake.ts
new file mode 100644
index 00000000..79309fb4
--- /dev/null
+++ b/site/src/cut/lib/removal/localBake.ts
@@ -0,0 +1,85 @@
+"use client";
+
+/**
+ * The free matte bake: MediaPipe person segmentation walked over the clip's
+ * trimmed source range and encoded as a grayscale H.264 clip (white = person
+ * — luma, never alpha, which browsers can't encode reliably). It runs in
+ * seconds, spends nothing, and gives every surface — headless renders
+ * included — a stored matte the moment auto removal is switched on; the
+ * hosted quality bake replaces it when it lands.
+ */
+
+import { MATTE_BITRATE, MATTE_FPS, MATTE_SHORT } from "@donkeycut/effects-kit";
+import { openCanvasVideo, scaledEvenSize } from "../canvasVideo";
+import { personSegmenter, segmentSubjectAlpha } from "../cutout";
+import { ClipReader } from "../exportRender";
+import { createRasterCanvas } from "../raster";
+import type { MediaAsset, VideoClip } from "../types";
+
+export class MatteBakeCancelled extends Error {
+ constructor() {
+ super("The bake was cancelled.");
+ }
+}
+
+/**
+ * Segment the clip's people over `[clip.in, clip.out]` and encode the matte.
+ * A still image bakes as a single frame. Throws when no segmenter is
+ * available (missing assets) or the encode has no codec; a cancellation
+ * surfaces as `MatteBakeCancelled`. `in` is the source second the matte's
+ * zero maps to.
+ */
+export async function localBakeMatte(
+ asset: MediaAsset,
+ clip: VideoClip,
+ opts: { onProgress?: (fraction: number) => void; signal?: AbortSignal } = {}
+): Promise<{ blob: Blob; in: number }> {
+ const segmenter = await personSegmenter();
+ if (!segmenter) throw new Error("On-device segmentation is unavailable in this browser.");
+ const still = asset.type === "image";
+ const from = still ? 0 : Math.max(0, clip.in);
+ const to = still ? from + 1 / MATTE_FPS : Math.min(clip.out, asset.duration || clip.out);
+ const seconds = Math.max(1 / MATTE_FPS, to - from);
+ const frames = still ? 1 : Math.max(1, Math.ceil(seconds * MATTE_FPS));
+
+ const reader = new ClipReader(asset, () => asset.url);
+ try {
+ const first = await reader.frameAt(from);
+ if (first.kind !== "ready") throw new Error("The clip's picture could not be read.");
+ const { w, h } = scaledEvenSize(first.width, first.height, MATTE_SHORT);
+ const enc = await openCanvasVideo({
+ width: w,
+ height: h,
+ fps: MATTE_FPS,
+ frames,
+ bitrate: MATTE_BITRATE,
+ });
+ const small = createRasterCanvas(w, h);
+ const sctx = small.getContext("2d") as CanvasRenderingContext2D | null;
+
+ for (let i = 0; i < frames; i++) {
+ if (opts.signal?.aborted) {
+ await enc.cancel();
+ throw new MatteBakeCancelled();
+ }
+ const at = Math.min(to - 1 / (MATTE_FPS * 2), from + i / MATTE_FPS);
+ const frame = await reader.frameAt(at);
+ enc.ctx.fillStyle = "#000000";
+ enc.ctx.fillRect(0, 0, w, h);
+ if (frame.kind === "ready" && sctx) {
+ sctx.clearRect(0, 0, w, h);
+ sctx.drawImage(frame.image, 0, 0, w, h);
+ const alpha = segmentSubjectAlpha(segmenter, small as HTMLCanvasElement);
+ if (alpha) {
+ enc.ctx.imageSmoothingEnabled = true;
+ enc.ctx.drawImage(alpha, 0, 0, w, h);
+ }
+ }
+ await enc.add(i / MATTE_FPS, 1 / MATTE_FPS);
+ opts.onProgress?.((i + 1) / frames);
+ }
+ return { blob: await enc.finish(), in: from };
+ } finally {
+ reader.dispose();
+ }
+}
diff --git a/site/src/cut/lib/removalVideo.ts b/site/src/cut/lib/removalVideo.ts
new file mode 100644
index 00000000..ec06a723
--- /dev/null
+++ b/site/src/cut/lib/removalVideo.ts
@@ -0,0 +1,174 @@
+"use client";
+
+/**
+ * Removal pieces for the ffmpeg export: each removal clip ships its keyed
+ * layer as two client-rendered, segment-aligned videos — the color (grade and
+ * key and stroke ink and backdrop baked, straight alpha colors) and the alpha
+ * as luma. The frames come out of the compositor's own removal stage, so the
+ * engine composites the very pixels the preview draws; the graph just merges
+ * the pair and frames it like any other picture. H.264 carries no alpha
+ * channel, which is why the layer travels as a pair.
+ */
+
+import { matteLumaToAlpha } from "@donkeycut/effects-kit";
+import { openCanvasVideo, scaledEvenSize } from "./canvasVideo";
+import { FrameCompositor, type Frame } from "./composite";
+import { ClipReader } from "./exportRender";
+import { createRasterCanvas, decodeRasterImageUrl, type RasterSurface } from "./raster";
+import { clipSpeed } from "./store";
+import type { MediaAsset, VideoClip } from "./types";
+
+export interface RemovalPieces {
+ rgb: Blob;
+ alpha: Blob;
+}
+
+/** Straight-alpha colors survive a getImageData round trip; premultiplied
+ * draws would darken every soft edge once the engine re-applies the alpha. */
+function splitFrame(
+ layer: CanvasImageSource,
+ lw: number,
+ lh: number,
+ read: { surface: RasterSurface; ctx: CanvasRenderingContext2D },
+ a: ImageData,
+ rgb: CanvasRenderingContext2D,
+ alpha: CanvasRenderingContext2D,
+ w: number,
+ h: number
+) {
+ read.ctx.clearRect(0, 0, w, h);
+ read.ctx.drawImage(layer, 0, 0, lw, lh, 0, 0, w, h);
+ const px = read.ctx.getImageData(0, 0, w, h);
+ for (let i = 0; i < px.data.length; i += 4) {
+ const v = px.data[i + 3];
+ a.data[i] = v;
+ a.data[i + 1] = v;
+ a.data[i + 2] = v;
+ a.data[i + 3] = 255;
+ px.data[i + 3] = 255;
+ }
+ rgb.putImageData(px, 0, 0);
+ alpha.putImageData(a, 0, 0);
+}
+
+/**
+ * Render one removal clip's keyed layer over its segment: `fps` frames per
+ * second for the segment's exact play length, sized to `maxShort` on the
+ * short side. AI modes read the baked matte asset; without one the export
+ * shows the plain picture, the preview's own degrade, so the two never
+ * disagree about a clip whose bake hasn't run. `bakeLook` folds the clip's
+ * look color pass into the pixels (overlay tracks, where the engine applies
+ * no look to an alpha layer); track 0 leaves it out and the engine grades the
+ * flattened segment. Null when the removal cannot draw here.
+ */
+export async function renderRemovalPieces(
+ asset: MediaAsset,
+ clip: VideoClip,
+ assets: MediaAsset[],
+ opts: { fps: number; maxShort: number; bakeLook: boolean }
+): Promise<RemovalPieces | null> {
+ const r = clip.removal;
+ if (!r) return null;
+ const matteAsset = r.matte ? assets.find((a) => a.id === r.matte!.assetId) : undefined;
+ const ai = r.mode === "auto" || r.mode === "custom";
+ if (ai && !matteAsset) return null;
+
+ const speed = clipSpeed(clip);
+ const still = asset.type === "image";
+ const dur = Math.max(0.1, (clip.out - clip.in) / speed);
+ const frames = Math.max(1, Math.ceil(dur * opts.fps));
+ const baked = clip.removal!.matte;
+
+ const reader = new ClipReader(asset, () => asset.url);
+ const matteReader = matteAsset ? new ClipReader(matteAsset, () => matteAsset.url) : null;
+ try {
+ const first = await reader.frameAt(still ? 0 : clip.in);
+ if (first.kind !== "ready") throw new Error("The removal clip's picture could not be read.");
+ const { w, h } = scaledEvenSize(first.width, first.height, opts.maxShort);
+ const px = w * h;
+ const rgbOut = await openCanvasVideo({
+ width: w,
+ height: h,
+ fps: opts.fps,
+ frames,
+ bitrate: Math.min(20_000_000, Math.max(2_000_000, Math.round(px * opts.fps * 0.12))),
+ });
+ const alphaOut = await openCanvasVideo({
+ width: w,
+ height: h,
+ fps: opts.fps,
+ frames,
+ bitrate: Math.min(4_000_000, Math.max(600_000, Math.round(px * opts.fps * 0.02))),
+ });
+ const rgbCtx = rgbOut.ctx;
+ const alphaCtx = alphaOut.ctx;
+ const readSurface = createRasterCanvas(w, h);
+ const readCtx = readSurface.getContext("2d", {
+ willReadFrequently: true,
+ }) as CanvasRenderingContext2D | null;
+ if (!readCtx) throw new Error("No drawing surface for the removal layer.");
+
+ // The compositor renders the layer with the preview's own code; its
+ // drawing canvas never draws, the scratches do the work.
+ const comp = new FrameCompositor(createRasterCanvas(2, 2));
+ const bakeClip = opts.bakeLook ? clip : { ...clip, look: undefined };
+ const matteFrame = createRasterCanvas(2, 2);
+ // The alpha plane's staging buffer, rewritten whole every frame.
+ const alphaPlane = new ImageData(w, h);
+ const backdrops = new Map<string, CanvasImageSource | null>();
+ if (r.backdrop?.kind === "image" && r.backdrop.assetId) {
+ const bd = assets.find((a) => a.id === r.backdrop!.assetId);
+ const img = bd ? await decodeRasterImageUrl(bd.url).catch(() => null) : null;
+ backdrops.set(r.backdrop.assetId, img ? img.source : null);
+ }
+ comp.backdropImageProvider = (assetId) => backdrops.get(assetId) ?? null;
+ let matteReady: Frame | null = null;
+ comp.removalMatteProvider = () => {
+ if (matteReady?.kind !== "ready") return null;
+ // Luma to alpha, once per staged frame.
+ const mw = matteReady.width;
+ const mh = matteReady.height;
+ if (matteFrame.width !== mw || matteFrame.height !== mh) {
+ matteFrame.width = mw;
+ matteFrame.height = mh;
+ }
+ const mctx = matteFrame.getContext("2d", {
+ willReadFrequently: true,
+ }) as CanvasRenderingContext2D;
+ mctx.drawImage(matteReady.image, 0, 0);
+ const mpx = mctx.getImageData(0, 0, mw, mh);
+ matteLumaToAlpha(mpx.data);
+ mctx.putImageData(mpx, 0, 0);
+ return matteFrame as CanvasImageSource;
+ };
+
+ for (let i = 0; i < frames; i++) {
+ const s = Math.min(dur - 1 / (opts.fps * 2), i / opts.fps);
+ const srcT = still ? 0 : clip.in + s * speed;
+ const frame = await reader.frameAt(srcT);
+ if (frame.kind !== "ready") {
+ rgbCtx.fillStyle = "#000000";
+ rgbCtx.fillRect(0, 0, w, h);
+ alphaCtx.fillStyle = "#000000";
+ alphaCtx.fillRect(0, 0, w, h);
+ } else {
+ matteReady = null;
+ if (matteReader && baked) {
+ const mdur = Math.max(0.1, matteAsset!.duration || 0.1);
+ const mt = Math.min(Math.max(0, srcT - baked.in), mdur - 0.001);
+ matteReady = await matteReader.frameAt(mt);
+ }
+ const layer = comp.removedLayer(frame, bakeClip, clip.start + s, {
+ bakeLookPost: opts.bakeLook,
+ });
+ splitFrame(layer, frame.width, frame.height, { surface: readSurface, ctx: readCtx }, alphaPlane, rgbCtx, alphaCtx, w, h);
+ }
+ await rgbOut.add(i / opts.fps, 1 / opts.fps);
+ await alphaOut.add(i / opts.fps, 1 / opts.fps);
+ }
+ return { rgb: await rgbOut.finish(), alpha: await alphaOut.finish() };
+ } finally {
+ reader.dispose();
+ matteReader?.dispose();
+ }
+}
diff --git a/site/src/cut/lib/store.ts b/site/src/cut/lib/store.ts
index 01288049..992adad5 100644
--- a/site/src/cut/lib/store.ts
+++ b/site/src/cut/lib/store.ts
@@ -300,6 +300,10 @@ export interface EditorState {
/** While playing a scoped effect preview, the time playback auto-pauses at;
* null otherwise. Manual seek/play/pause clears it. */
previewStopAt: number | null;
+ /** A clip whose background removal is temporarily bypassed — the removal
+ * panel's eye toggle showing the original picture; null otherwise. View
+ * state, never saved. */
+ removalPeek: string | null;
pxPerSec: number;
/** Timeline panel height in px (drag the panel's top border to change). */
timelineH: number;
@@ -1508,6 +1512,7 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {
playing: false,
buffering: false,
previewStopAt: null,
+ removalPeek: null,
subtitles: emptySubtitles(),
subtitleLane: 0,
subtitleStatus: "idle",
@@ -1557,6 +1562,7 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {
playing: false,
buffering: false,
previewStopAt: null,
+ removalPeek: null,
pxPerSec: 60,
timelineH: TIMELINE_H_DEFAULT,
publish: { caption: "", tags: "", soundTitle: "", handle: "" },
diff --git a/site/src/cut/lib/types.ts b/site/src/cut/lib/types.ts
index 418bd2e7..5b9d03ca 100644
--- a/site/src/cut/lib/types.ts
+++ b/site/src/cut/lib/types.ts
@@ -4,6 +4,7 @@ import {
poseAt,
stampOverlayKinds,
stripDefaultOverlayKinds,
+ type ClipRemoval,
type ColorGrade,
type EffectOverlay,
type LookStyle,
@@ -192,7 +193,7 @@ export interface StoredAsset {
* drop, or upload), so it belongs in the Media panel. Any value marks media
* Cut created or fetched — it lives where it was made (the timeline, a
* generation panel, or an AI chat card) and is kept out of the Media panel. */
- origin?: "voiceover" | "generated" | "recording" | "stock" | "freeze" | "chat" | "sticker";
+ origin?: "voiceover" | "generated" | "recording" | "stock" | "freeze" | "chat" | "sticker" | "matte";
/** BCP-47 of the audio's spoken language, when known (stamped on voiceovers
* at synthesis) — what transcription should run its recognizer in. */
language?: string;
@@ -554,6 +555,11 @@ export interface VideoClip {
kf?: OverlayKey[];
/** Rounded corners and a border stroke on the clip's box; absent = plain. */
boxStyle?: BoxStyle;
+ /** Background removal: keys the clip's picture to an alpha matte (chroma
+ * key, or an AI matte baked to a grayscale video asset), with an optional
+ * stroke around the silhouette and a backdrop filled in behind it (see the
+ * kit's removal.ts). Absent = the whole picture shows. */
+ removal?: ClipRemoval;
}
// Color grading (the dual-renderer math) lives in the effects kit; the model
@@ -561,6 +567,25 @@ export interface VideoClip {
export { GRADE_BASIC_FIELDS, GRADE_HUE_MAX, GRADE_MAX, HSL_BANDS } from "@donkeycut/effects-kit";
export type { ColorGrade, GradePresetRef, HslBand } from "@donkeycut/effects-kit";
+// Background removal follows the same split: math and model in the kit, the
+// doc-model types re-exported here.
+export {
+ removalActive,
+ removalFingerprint,
+ removalNeedsBake,
+ STROKE_STYLE_LABELS,
+ STROKE_STYLES,
+} from "@donkeycut/effects-kit";
+export type {
+ ChromaKey,
+ ClipRemoval,
+ RemovalBackdrop,
+ RemovalMode,
+ RemovalSeeds,
+ RemovalStroke,
+ StrokeStyleId,
+} from "@donkeycut/effects-kit";
+
/** Speed slider range. Typed entry and tools may go beyond it; SPEED_FLOOR is
* the only hard bound, keeping rates positive so length math stays finite. */
export const SPEED_MIN = 0.25;
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index 12b8a7e3..c1d7866f 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -30,6 +30,7 @@ import { IMAGE_GEN_TOOLS } from "@/cut/components/ImageGenPanel.tools";
import { INSPECTOR_TOOLS } from "@/cut/components/Inspector.tools";
import { LIBRARY_TOOLS } from "@/cut/components/LibraryView.tools";
import { PREVIEW_TOOLS } from "@/cut/components/Preview.tools";
+import { REMOVAL_TOOLS } from "@/cut/components/RemovalPanel.tools";
import { SCENE_TOOLS } from "@/cut/components/SceneCard.tools";
import { SIDE_PANEL_TOOLS } from "@/cut/components/SidePanel.tools";
import { STOCK_TOOLS } from "@/cut/components/StockPanels.tools";
@@ -54,6 +55,7 @@ export const AI_TOOLS: AiToolDef[] = [
...PREVIEW_TOOLS,
...TIMELINE_TOOLS,
...INSPECTOR_TOOLS,
+ ...REMOVAL_TOOLS,
...ELEMENTS_TOOLS,
...EFFECTS_TOOLS,
...OVERLAY_ANIMATION_TOOLS,
@@ -93,6 +95,7 @@ Times are in seconds on the shared timeline. The playhead is currentTime; a skim
- Resizes ripple the same way deletes do: while track 0 is the only video track, a track-0 trim or speed change slides everything past the clip's tail — clips, titles, captions, soundtrack — with the moved edge, in both directions, every existing gap keeping its width. With overlay video tracks present, growth pushes the track-0 run and a shrink leaves a gap.
- set_speed sets a clip's playback rate; it changes the clip's timeline length and carries the timeline behind it like a trim.
- set_transition joins a clip into the next one (0–2s, ${TRANSITION_STYLE_IDS.length} styles); set_animation animates one clip's own entrance/exit — read the transitions-and-fades skill before styling cuts. Splitting or deleting clears the affected transition. A transition is a bar on its own row at an absolute time, so an edit can leave one lining up with nothing: every timeline mutation reports those as \`parkedTransitions\`, and clearing them (remove_transition) is part of finishing the edit. Grading a clip's picture is the color tools (set_color_preset, set_color_grade, set_color_curves, set_color_wheels, set_color_hsl — read the color-grading skill); a stylized post treatment (grain, VHS, halation…) is add_effect, dropped over the stretch of timeline it covers — as is a treatment on the sound (echo, reverb, telephone…).
+- Cutting a clip's background away — the AI person matte, the green-screen chroma key, ink around the silhouette, a fill behind the subject — is the cutout suite (set_removal, set_removal_stroke, set_removal_background); read the background-removal skill.
- split_at cuts the track-0 clip under that time into two clips at the exact frame. With a soundtrack or overlay clip selected it splits that instead.
- The Media and Library panels take the same gestures: ⌘/⇧-click a tile, or sweep a rubber-band over the grid, and dragging any picked tile onto the timeline lays the whole set down one after another from the drop point.
- The user can multi-select (⌘/⇧-click, or sweep a rubber-band over empty track space) across every kind at once — clips, audio, titles, cues. Deleting or copying acts on the whole selection, and dragging any member moves the set as one: every item shifts by the same time, relative spacing kept, while unselected items slide out of the way and flow back. A selection of one kind on the multi-row tracks (audio, titles) also retracks together on a vertical drag, one row past the end opening a new track. A hover chip on each video clip toggles its own audio.
@@ -124,6 +127,16 @@ Grade like a colorist: read the image, correct it, then style it — in that ord
**Craft.** Skin lives around orange at modest saturation — check faces after strong saturation, hue, or wheel moves, and reach for protect_skin or a gentle orange-band correction rather than abandoning the look. Keep adjacent clips consistent: one preset + per-clip correction reads as one film; different presets per clip read as a slideshow. Strong looks sit better with contrast established first. Grades are per-clip — "grade the whole video" means every clip, same recipe.`,
+ "background-removal": `# Background removal (Cutout)
+Every video clip — any track, stills too — can carry a cutout, the Inspector's Cutout tab: the clip's picture keys to an alpha matte, the subject stays, and where the background was, lower tracks or the project background show through. Preview, timeline thumbnails, and export render the same matte.
+- set_removal mode "auto": the AI person matte. A free on-device pass lands in seconds, and a hosted quality pass (spends credits like generation) upgrades hair and edge detail in the background. The tool call starts the bake itself; in the panel, picking a mode only previews the live matte, and the user's Start (auto) or Apply (custom) press starts the bake — hand-instructions include that step. Each cutout clip reports under \`removal\` in editor_state videoTrack/overlayVideo — mode, matte readiness ("ready (quick)" / "ready (quality)" / "none"), and a live bake's progress — and wait_for_renders settles on bakes, so claims about a finished cutout come from there. Short of credits, the quick matte stands and the panel offers the upgrade.
+- set_removal mode "custom" with \`subject\`: an arbitrary object by description — "the dog", "the red car" — matted and tracked through the clip (spends credits, less than a brushed track). Every match of the words is kept, so "cut out the dog" is one call. Picking one exact instance among lookalikes is hands-on: the user picks Custom in the Cutout tab, paints it on the preview with the quick and manual brushes, and presses Apply — the hosted tracker follows those exact strokes; with a subject set too, the strokes refine which instance the words picked.
+- set_removal mode "chroma": the green-screen key. The key color plus intensity (tolerance), softness (edge rolloff), and spill (pulls key-colored fringe off the kept pixels); omitting color samples the footage's borders. Footage shot or generated over a solid backdrop keys cleanly and costs nothing.
+- set_removal_stroke draws ink around the kept silhouette — glow, hand-drawn, straight cut, solid, offset (a thrown silhouette copy placed with offset_x/offset_y), dotted — with color and width. The die-cut sticker look is a white solid or cut stroke.
+- set_removal_background fills in behind the subject inside the clip's own picture: a flat color or a project image. For new scenery behind a person, an image fill here or scenery on a lower track both read; the lower track keeps the scenery independently editable.
+Panel geography, for hand-instructions: the Cutout tab is labeled rows, and the Layer dropdown at the top switches the rows between Removal (the mode and its controls), Stroke, and Background (None / Color / Image; the Image grid lists pictures imported into Media). Say "set Layer to Background", never "under the Background settings".
+Recipes: "remove the background" on a person = auto. "Cut out the dog" = custom with subject "the dog". "Put me on a beach" = auto + the beach behind (background image or lower track). "Green screen" = chroma. Sticker/die-cut = auto + white cut stroke. A bold silhouette pop = offset stroke in a loud color. Turning the whole thing off = set_removal mode "off", which drops the stroke and background fill with it.`,
+
"watching-and-cutting": `# Watching footage & cutting by content
When a request depends on what the footage actually contains — "cut the dead air", "clip the best moment", "remove the boring part", "split where the scene changes" — watch it first. Never guess at content you haven't seen.
diff --git a/site/src/cut/server/cloud/r2.ts b/site/src/cut/server/cloud/r2.ts
index 320d67ff..f2fb3d18 100644
--- a/site/src/cut/server/cloud/r2.ts
+++ b/site/src/cut/server/cloud/r2.ts
@@ -83,6 +83,14 @@ export function presignPut(key: string, mime: string): Promise<string> {
);
}
+/** A time-limited read URL, for handing an object to an inference provider
+ * that fetches its input itself. */
+export function presignGet(key: string): Promise<string> {
+ return getSignedUrl(r2(), new GetObjectCommand({ Bucket: R2_BUCKET, Key: key }), {
+ expiresIn: PUT_EXPIRY_SECONDS,
+ });
+}
+
/** Object size/type, or null when the object does not exist. */
export async function head(
diff --git a/site/src/cut/server/exportPipeline.ts b/site/src/cut/server/exportPipeline.ts
index a849bdec..f41d057e 100644
--- a/site/src/cut/server/exportPipeline.ts
+++ b/site/src/cut/server/exportPipeline.ts
@@ -139,6 +139,12 @@ export interface ExportSpec {
file?: string;
frames?: { file: string; duration: number }[];
};
+ /** Background removal: the clip's keyed layer, client-rendered as a
+ * segment-aligned color/alpha video pair (grade, key, stroke ink and
+ * backdrop baked in, straight alpha colors). The pair replaces the source
+ * decode for the picture; audio still reads the original file, and the
+ * look applies to the flattened segment. */
+ removal?: { rgb: string; alpha: string };
}[];
/** Video tracks composited over the track-0 `clips`, lowest track first. */
overlayVideos?: {
@@ -204,6 +210,9 @@ export interface ExportSpec {
file?: string;
frames?: { file: string; duration: number }[];
};
+ /** Background removal, as on `clips`; this layer keeps its alpha over the
+ * tracks beneath, so grade and look are both baked into the pair. */
+ removal?: { rgb: string; alpha: string };
}[];
audio: {
file: string;
@@ -678,6 +687,25 @@ export async function runExport(
overlayBorderInput.set(oc, borderStill(oc.border, olen));
}
+ // Removal pieces: each removal clip's keyed layer arrives as a color/alpha
+ // video pair uploaded with the stills; the pair replaces the source decode.
+ const removalInputs = (rm: { rgb: string; alpha: string }) => {
+ const rgb = nInputs++;
+ inputs.push("-i", path.join(job.tmpDir, path.basename(rm.rgb)));
+ const a = nInputs++;
+ inputs.push("-i", path.join(job.tmpDir, path.basename(rm.alpha)));
+ return { rgb, a };
+ };
+ const clipRemovalInput = new Map<number, { rgb: number; a: number }>();
+ for (let j = 0; j < spec.clips.length; j++) {
+ const c = spec.clips[j];
+ if (c.removal) clipRemovalInput.set(j, removalInputs(c.removal));
+ }
+ const overlayRemovalInput = new Map<(typeof overlayVideos)[number], { rgb: number; a: number }>();
+ for (const oc of overlayVideos) {
+ if (oc.removal) overlayRemovalInput.set(oc, removalInputs(oc.removal));
+ }
+
// One concat-demuxer input per subtitle track: within a track cues never
// overlap, so each plays as a slideshow with transparent filler
// ("sub_blank.png", uploaded with the stills) covering the gaps. Tracks
@@ -1093,13 +1121,29 @@ export async function runExport(
if (!tailFx && tf <= 0.01 && czTail > 0.01) {
tailFx = { kind: "zoom", secs: Math.min(czTail, dur - (headFx?.secs ?? 0)) };
}
+ const rmIn = clipRemovalInput.get(j);
// A still's looped input is already the right length at `fps`; it has no
- // source span to trim. Footage trims `in..out` and re-times by speed.
- const timebase = c.image
- ? `[${idx}:v]setpts=PTS-STARTPTS`
- : `[${idx}:v]trim=${num(c.in)}:${num(c.out)},setpts=(PTS-STARTPTS)/${num(speed)}`;
- if ((c.image || videoPresence.get(c.file)) && !c.hidden) {
+ // source span to trim. Footage trims `in..out` and re-times by speed. A
+ // removal clip's picture is its uploaded keyed pair instead: the color
+ // stream merges its alpha luma, clone-padded so a piece a frame short
+ // never shortens the join, and rides the same framing chain in rgba.
+ let timebase: string;
+ if (rmIn) {
+ filters.push(`[${rmIn.a}:v]setpts=PTS-STARTPTS,format=gray[rva${j}]`);
+ filters.push(`[${rmIn.rgb}:v]setpts=PTS-STARTPTS,format=rgba[rvc${j}]`);
+ filters.push(`[rvc${j}][rva${j}]alphamerge,tpad=stop_mode=clone:stop_duration=1[rvs${j}]`);
+ timebase = `[rvs${j}]null`;
+ } else {
+ timebase = c.image
+ ? `[${idx}:v]setpts=PTS-STARTPTS`
+ : `[${idx}:v]trim=${num(c.in)}:${num(c.out)},setpts=(PTS-STARTPTS)/${num(speed)}`;
+ }
+ if ((c.image || rmIn || videoPresence.get(c.file)) && !c.hidden) {
const region = regionPx(c.frame, W, H);
+ // The keyed layer's letterbox stays transparent (the flatten below lays
+ // the frame color behind it), and its alpha survives the chain.
+ const segPad = rmIn ? "[email protected]" : padColor;
+ const segFmt = rmIn ? "rgba" : clipFmt;
let frame: string;
if (region) {
// A regioned track-0 clip (split-screen half) scales into its rect,
@@ -1117,22 +1161,24 @@ export async function runExport(
// the rect exactly, a fitted one keeps its margins.
frame =
`${boxFraming(rw, rh, c.fit === "fill", c.zoom, c.panX, c.panY)},` +
- `pad=${bw}:${bh}:${rx - bx}+(${rw}-iw)/2:${ry - by}+(${rh}-ih)/2:color=${padColor}${win}`;
+ `pad=${bw}:${bh}:${rx - bx}+(${rw}-iw)/2:${ry - by}+(${rh}-ih)/2:color=${segPad}${win}`;
} else {
const cover = c.fit === "fill";
frame =
boxFraming(W, H, cover, c.zoom, c.panX, c.panY) +
// A covering picture already spans the frame; a fitted one letterboxes.
- (cover ? "" : `,pad=${W}:${H}:(ow-iw)/2:(oh-ih)/2:color=${padColor}`);
+ (cover ? "" : `,pad=${W}:${H}:(ow-iw)/2:(oh-ih)/2:color=${segPad}`);
}
// setpts/speed rescales the clip's duration on the timeline (footage);
// a still just replays its looped input.
// The grade sits after the color conversion (so it acts on the same
// BT.709 values the preview shows) and before the terminal format.
- let core = `${timebase},fps=${fps},${frame},setsar=1,${colorFix.get(c.file) ?? ""}${gradeChain(c.grade)}format=${clipFmt}`;
+ let core = `${timebase},fps=${fps},${frame},setsar=1,${rmIn ? "" : colorFix.get(c.file) ?? ""}${gradeChain(rmIn ? undefined : c.grade)}format=${segFmt}`;
// The look bakes in after grade + framing, before the edge effects, so
- // animations move already-graded pixels (matching the preview order).
- if (c.look) {
+ // animations move already-graded pixels (matching the preview order). A
+ // removal clip's look runs after the flatten below instead — the chain's
+ // internal blends would drop the keyed layer's alpha.
+ if (c.look && !rmIn) {
const lines = lookFilterLines(`lki${j}`, `lko${j}`, c.look, c.lookAmount, H, clipFmt, `c${j}`);
if (lines) {
filters.push(`${core}[lki${j}]`);
@@ -1145,12 +1191,16 @@ export async function runExport(
const brIdx = clipBorderInput.get(j);
if (brIdx !== undefined) {
filters.push(`${core}[cbi${j}]`);
- filters.push(`[cbi${j}][${brIdx}:v]overlay=0:0:eof_action=pass,format=${clipFmt}[cbo${j}]`);
+ filters.push(`[cbi${j}][${brIdx}:v]overlay=0:0:eof_action=pass,format=${segFmt}[cbo${j}]`);
core = `[cbo${j}]null`;
}
+ // A removal segment carries alpha, so its fades must ramp the alpha
+ // plane too — a color-only fade darkens the still-opaque silhouette to
+ // black where the preview dissolves it.
+ const fadeAlpha = rmIn ? ":alpha=1" : "";
const fades =
- (hf > 0.01 ? `,fade=t=in:st=0:d=${num(hf)}` : "") +
- (tf > 0.01 ? `,fade=t=out:st=${num(Math.max(0, dur - tf))}:d=${num(tf)}` : "");
+ (hf > 0.01 ? `,fade=t=in:st=0:d=${num(hf)}${fadeAlpha}` : "") +
+ (tf > 0.01 ? `,fade=t=out:st=${num(Math.max(0, dur - tf))}:d=${num(tf)}${fadeAlpha}` : "");
// A neighbor's backdrop animation freezes a frame of this segment —
// split the copies it needs off the pre-backdrop picture.
const nFz = (needFirstFreeze[j] ? 1 : 0) + (needLastFreeze[j] ? 1 : 0);
@@ -1164,7 +1214,7 @@ export async function runExport(
const mkIdx = clipMaskInput.get(j);
const subjMask = subjectActive && matteConsumers > 0 ? c.mask?.subject : undefined;
const keyed = !!c.kf?.length;
- if (mkIdx !== undefined || subjMask || keyed) {
+ if (mkIdx !== undefined || subjMask || keyed || rmIn) {
// Masked or keyframed track-0 clip, composed in the preview's order:
// the painted coverage multiplies first (it rides the clip), the
// pose transforms and positions the result over a transparent base,
@@ -1172,7 +1222,7 @@ export async function runExport(
// restores the constant-size opaque frame the join expects. The
// multiply chains (alphaextract → blend → alphamerge) compose with
// any alpha the segment carries.
- pushEdgeFx(core, dur, headFx, tailFx, W, H, clipFmt, fades, `vmr${j}`, `c${j}`);
+ pushEdgeFx(core, dur, headFx, tailFx, W, H, rmIn ? "yuva420p" : clipFmt, fades, `vmr${j}`, `c${j}`);
let cur = `vmr${j}`;
if (mkIdx !== undefined) {
filters.push(`[${mkIdx}:v]fps=${fps},scale=${W}:${H},setsar=1,format=gray[cmk${j}]`);
@@ -1215,10 +1265,22 @@ export async function runExport(
filters.push(`[cs0_${j}][csm${j}]alphamerge[csc${j}]`);
cur = `csc${j}`;
}
- filters.push(`color=c=black:s=${W}x${H}:r=${fps}:d=${num(dur)}[cmb${j}]`);
+ // A keyed layer flattens onto the frame color — where the removal
+ // left the picture transparent, the export shows what the preview
+ // shows behind track 0.
+ filters.push(
+ `color=c=${rmIn ? padColor : "black"}:s=${W}x${H}:r=${fps}:d=${num(dur)}[cmb${j}]`
+ );
+ const flatOut = rmIn && c.look ? `vrl${j}` : segCore;
filters.push(
- `[cmb${j}][${cur}]overlay=0:0:shortest=1,format=${clipFmt},fps=${fps}[${segCore}]`
+ `[cmb${j}][${cur}]overlay=0:0:shortest=1,format=${clipFmt},fps=${fps}[${flatOut}]`
);
+ if (flatOut !== segCore) {
+ // The removal clip's look, over the flattened opaque segment.
+ const lines = lookFilterLines(`vrl${j}`, segCore, c.look!, c.lookAmount, H, clipFmt, `cr${j}`);
+ if (lines) filters.push(...lines);
+ else filters.push(`[vrl${j}]null[${segCore}]`);
+ }
} else {
pushEdgeFx(core, dur, headFx, tailFx, W, H, clipFmt, fades, segCore, `c${j}`);
}
@@ -1442,7 +1504,8 @@ export async function runExport(
// Overlay one track clip onto `onto`, returning the new label; also queues
// its audio.
const addOverlay = (oc: (typeof overlayVideos)[number], onto: string): string => {
- if (!oc.image && !videoPresence.get(oc.file)) return onto;
+ const orIn = overlayRemovalInput.get(oc);
+ if (!oc.image && !orIn && !videoPresence.get(oc.file)) return onto;
const idx = oc.image ? imageOverlayInput.get(oc)! : inputIndex.get(oc.file)!;
const ospeed = oc.speed && oc.speed > 0 ? oc.speed : 1;
const olen = Math.max(0.1, (oc.out - oc.in) / ospeed);
@@ -1491,7 +1554,7 @@ export async function runExport(
? `,format=yuva420p,pad=${boxW}:${boxH}:(ow-iw)/2:(oh-ih)/2:[email protected]`
: "";
if (boxed) pos = region ? `${region.rx}:${region.ry}` : "0:0";
- const fmt = hf > 0.01 || tf > 0.01 || boxed ? "yuva420p" : "yuv420p";
+ const fmt = hf > 0.01 || tf > 0.01 || boxed || orIn ? "yuva420p" : "yuv420p";
// The look chain reads the pre-pad picture, which is opaque.
const lookFmt = boxed ? "yuv420p" : fmt;
const fades =
@@ -1500,15 +1563,25 @@ export async function runExport(
const k = ovk++;
const seg = `ovv${k}`;
// A still replays its looped input; footage trims its source span and
- // re-times by speed. tpad then delays the clip to its timeline start.
- const timebase = oc.image
- ? `[${idx}:v]setpts=PTS-STARTPTS`
- : `[${idx}:v]trim=${num(oc.in)}:${num(oc.out)},setpts=(PTS-STARTPTS)/${num(ospeed)}`;
- let core = `${timebase},fps=${fps},${framing},setsar=1,${colorFix.get(oc.file) ?? ""}${gradeChain(oc.grade)}format=${lookFmt}`;
+ // re-times by speed. tpad then delays the clip to its timeline start. A
+ // removal overlay's picture is its uploaded keyed pair, alpha merged in
+ // and clone-padded, grade and look already baked into the pixels.
+ let timebase: string;
+ if (orIn) {
+ filters.push(`[${orIn.a}:v]setpts=PTS-STARTPTS,format=gray[ova${k}]`);
+ filters.push(`[${orIn.rgb}:v]setpts=PTS-STARTPTS,format=rgba[ovc${k}]`);
+ filters.push(`[ovc${k}][ova${k}]alphamerge,tpad=stop_mode=clone:stop_duration=1[ovr${k}]`);
+ timebase = `[ovr${k}]null`;
+ } else {
+ timebase = oc.image
+ ? `[${idx}:v]setpts=PTS-STARTPTS`
+ : `[${idx}:v]trim=${num(oc.in)}:${num(oc.out)},setpts=(PTS-STARTPTS)/${num(ospeed)}`;
+ }
+ let core = `${timebase},fps=${fps},${framing},setsar=1,${orIn ? "" : colorFix.get(oc.file) ?? ""}${gradeChain(orIn ? undefined : oc.grade)}format=${orIn ? "yuva420p" : lookFmt}`;
// Looks bake into footage overlays only: an image may carry alpha, which
// the look chain's internal filters would flatten onto black over the
// tracks beneath. The alpha fades stay safe: they apply after the look.
- if (oc.look && !oc.image) {
+ if (oc.look && !oc.image && !orIn) {
const lines = lookFilterLines(`olki${k}`, `olko${k}`, oc.look, oc.lookAmount, H, lookFmt, `o${k}`);
if (lines) {
filters.push(`${core}[olki${k}]`);
diff --git a/site/src/lib/credits/inference.ts b/site/src/lib/credits/inference.ts
index 3df04010..001e34d0 100644
--- a/site/src/lib/credits/inference.ts
+++ b/site/src/lib/credits/inference.ts
@@ -85,6 +85,10 @@ type CreditPreflightInput = {
// billed model differs from the request (a provider that pins its own model) must leave this off
// and rely on the post-generation price resolution instead.
enforceModelPrice?: boolean;
+ // The generation units the request will bill (the provider's own count, e.g. the matte
+ // segmenter's frame chunks). The affordability check multiplies the per-unit price by it, so
+ // a charge that scales with the input is held to the balance in full. Unset means one unit.
+ generationCount?: number;
};
export type RecordedInferenceUsage = {
@@ -320,8 +324,12 @@ async function assertModelIsAffordable(
const rate = await resolveCreditRate(prisma, input.route, provider, model);
// A DB rate override replaces the code pricing wholesale and carries no flat per-generation
// price, so an overridden model falls back to the positive-balance gate above.
+ const generationCount = BigInt(
+ Math.max(1, Math.floor(input.generationCount ?? 1)),
+ );
const generationCostMicros =
- rate.providerPricing?.generationCostMicros ?? zeroCreditMicros;
+ (rate.providerPricing?.generationCostMicros ?? zeroCreditMicros) *
+ generationCount;
if (
generationCostMicros > zeroCreditMicros &&
balanceMicros < generationCostMicros
diff --git a/site/src/lib/credits/provider-pricing.ts b/site/src/lib/credits/provider-pricing.ts
index 3658b9bc..e9d0155a 100644
--- a/site/src/lib/credits/provider-pricing.ts
+++ b/site/src/lib/credits/provider-pricing.ts
@@ -17,6 +17,7 @@ import {
type GeminiTtsModel,
} from "@/lib/inference/gemini-models";
import { browserUsePerStepUsd } from "@/lib/browser/pricing";
+import { falMatteModels, type FalMatteModel } from "@/lib/inference/matte-models";
export type ProviderCreditPricing = {
inputTokenCostMicrosPerMillion?: bigint;
@@ -65,6 +66,11 @@ export function providerCreditPricing(
if (normalizedProvider === "elevenlabs") {
return elevenLabsCreditPricing(normalizedModel);
}
+ // The matte segmenter id is hardcoded (matte-models.ts); a matte bills at
+ // submit, one generation per 16-frame chunk of the segment.
+ if (normalizedProvider === "fal") {
+ return falMatteCreditPricing(normalizedModel);
+ }
if (normalizedProvider === "browser-use") {
return browserUseCreditPricing();
}
@@ -324,6 +330,18 @@ function geminiOmniCreditPricing(model: string): ProviderCreditPricing | undefin
return geminiOmniModelPricing[model as GeminiOmniModel];
}
+// The matte segmenter bills upstream per 16 processed frames ($0.005 a
+// chunk); the adapter counts the submitted segment's frames and charges the
+// same unit, generationCount = started chunks. Keyed by FalMatteModel, so
+// adding a segmenter id without a price fails the build.
+const falMatteModelPricing: Record<FalMatteModel, ProviderCreditPricing> = {
+ [falMatteModels.segmenter]: { generationCostMicros: usdWithMargin("0.005") },
+};
+
+function falMatteCreditPricing(model: string): ProviderCreditPricing | undefined {
+ return falMatteModelPricing[model as FalMatteModel];
+}
+
// Generative music (Gemini/Lyria) bills flat per rendered clip: the render is a
// fixed-length interaction that carries no per-second usage, so a clip charges
// generationCount = 1 at completion. Rates are Lyria 3's published per-request
diff --git a/site/src/lib/inference/adapters/fal-matte.ts b/site/src/lib/inference/adapters/fal-matte.ts
new file mode 100644
index 00000000..e03c165b
--- /dev/null
+++ b/site/src/lib/inference/adapters/fal-matte.ts
@@ -0,0 +1,341 @@
+import { ALL_FORMATS, Input, UrlSource } from "mediabunny";
+
+import { providerCreditPricing } from "@/lib/credits/provider-pricing";
+import { falMatteModels, type FalMatteModel } from "@/lib/inference/matte-models";
+import { matteConceptPrompt, mattePointPrompts } from "@/lib/inference/matte-prompts";
+import { ensureConfigured } from "@/lib/inference/http";
+import { isJsonObject, toJsonObject } from "@/lib/inference/json";
+import {
+ InferenceProviderError,
+ type AssetGenerationProviderRequest,
+ type AssetGenerationProviderResult,
+ type InferenceModality,
+ type InferenceModel,
+ type InferenceProvider,
+ type JsonValue,
+ type StoredGenerationForProvider,
+} from "@/lib/inference/providers";
+
+// The matte provider: a queue-hosted segmenter for kind="matte". One
+// call takes a clip segment (by URL — the route resolves the caller's stored
+// blob into a read link) plus a text description of what to keep, click
+// prompts, or both together — points sharing an object id refine what the
+// words detected — and returns a mask video. The prediction is async: submit
+// here, poll through refresh; the charge bills at submit, counted in the
+// segmenter's own unit.
+const providerID = "fal";
+
+const QUEUE_BASE = "https://queue.fal.run";
+
+// The segmenter meters per 16 processed frames, so the charge counts the same
+// unit: the submitted segment is probed for its frame count and each started
+// chunk of 16 bills one generation at the per-chunk price.
+const CHUNK_FRAMES = 16;
+
+// The billable chunk count of the segment at `videoUrl` — the server's own
+// count, so the charge never rides a client-supplied number. The frame count
+// is the track's duration times its packet rate, both read from the
+// container's metadata: a bounded sample of packet headers settles the rate
+// (segments encode at one constant rate), and no media bytes download on the
+// billing path. An unreadable segment stops the submit.
+const RATE_SAMPLE_PACKETS = 100;
+
+async function billableChunks(videoUrl: string): Promise<number> {
+ const input = new Input({ source: new UrlSource(videoUrl), formats: ALL_FORMATS });
+ try {
+ const track = await input.getPrimaryVideoTrack();
+ if (track) {
+ const duration = await track.computeDuration();
+ const rate = (await track.computePacketStats(RATE_SAMPLE_PACKETS)).averagePacketRate;
+ const frames = Math.round(duration * rate);
+ if (frames > 0) {
+ return Math.ceil(frames / CHUNK_FRAMES);
+ }
+ }
+ } catch {
+ // Falls through to the one error shape below.
+ } finally {
+ input.dispose();
+ }
+ throw new InferenceProviderError("The clip segment could not be read for billing.", {
+ statusCode: 422,
+ code: "matte_segment_unreadable",
+ });
+}
+
+export type AdapterEnvironment = Record<string, string | undefined>;
+export type MatteFetch = typeof fetch;
+
+export function createFalMatteProvider(
+ environment: AdapterEnvironment = process.env,
+ fetchImpl: MatteFetch = fetch,
+): InferenceProvider {
+ const token = environment.FAL_KEY?.trim();
+ const configured = Boolean(token);
+ const defaultModel: FalMatteModel = falMatteModels.segmenter;
+
+ async function listModels(modalities: InferenceModality[]): Promise<InferenceModel[]> {
+ if (!modalities.includes("matte")) {
+ return [];
+ }
+ return [
+ {
+ id: defaultModel,
+ name: defaultModel,
+ provider: providerID,
+ inputModalities: ["video"],
+ outputModalities: ["video"],
+ contextLength: null,
+ pricing: null,
+ metadata: { provider: providerID },
+ },
+ ];
+ }
+
+ function resolveModel(requested?: string): string {
+ const model = requested?.trim() || defaultModel;
+ if (model !== defaultModel) {
+ throw new InferenceProviderError("Unknown matte model.", {
+ statusCode: 400,
+ code: "unknown_matte_model",
+ details: { model },
+ });
+ }
+ // Fail before spending: the resolved model must have a configured price.
+ if (!providerCreditPricing(providerID, model)) {
+ throw new InferenceProviderError(
+ "No credit price is configured for the selected matte model.",
+ { statusCode: 500, code: "matte_model_not_priced", details: { model } },
+ );
+ }
+ return model;
+ }
+
+ async function api(url: string, init?: RequestInit): Promise<JsonValue> {
+ const res = await fetchImpl(url, {
+ ...init,
+ headers: {
+ Authorization: `Key ${token}`,
+ "Content-Type": "application/json",
+ ...(init?.headers ?? {}),
+ },
+ });
+ const body = (await res.json().catch(() => null)) as JsonValue;
+ if (!res.ok) {
+ const detail =
+ isJsonObject(body) && typeof body.detail === "string" ? body.detail : undefined;
+ throw new InferenceProviderError(detail ?? "The matte service refused the request.", {
+ statusCode: res.status === 429 ? 429 : 502,
+ code: "matte_provider_error",
+ });
+ }
+ return body;
+ }
+
+ function segmentUrl(request: AssetGenerationProviderRequest["request"]): string {
+ const inputs = toJsonObject(request.inputs ?? {});
+ const video = isJsonObject(inputs.video) ? inputs.video : {};
+ const videoUrl = typeof video.url === "string" ? video.url : "";
+ if (!videoUrl) {
+ throw new InferenceProviderError("A matte needs the clip segment to track.", {
+ statusCode: 400,
+ code: "empty_matte_request",
+ });
+ }
+ return videoUrl;
+ }
+
+ // The route's pre-preflight count: the chunks this segment will bill, read
+ // from the resolved segment itself.
+ async function assetGenerationCountFor(
+ request: AssetGenerationProviderRequest["request"],
+ ): Promise<number> {
+ ensureConfigured(configured);
+ if (request.kind !== "matte") {
+ throw new InferenceProviderError("Provider does not support this asset kind.", {
+ statusCode: 400,
+ code: "unsupported_asset_kind",
+ });
+ }
+ return billableChunks(segmentUrl(request));
+ }
+
+ async function generateAsset({
+ request,
+ generationCount,
+ }: AssetGenerationProviderRequest): Promise<AssetGenerationProviderResult> {
+ ensureConfigured(configured);
+ if (request.kind !== "matte") {
+ throw new InferenceProviderError("Provider does not support this asset kind.", {
+ statusCode: 400,
+ code: "unsupported_asset_kind",
+ });
+ }
+ // The route settles the count through assetGenerationCountFor and the
+ // preflight holds the balance to it; a submit without one would bill a
+ // charge no preflight covered.
+ const chunks = Math.floor(generationCount ?? 0);
+ if (chunks < 1) {
+ throw new InferenceProviderError("The matte charge was not counted before submit.", {
+ statusCode: 500,
+ code: "matte_charge_uncounted",
+ });
+ }
+
+ const model = resolveModel(request.model);
+ const videoUrl = segmentUrl(request);
+ const parameters = toJsonObject(request.parameters ?? {});
+ const prompt = matteConceptPrompt(parameters.prompt);
+ const points = mattePointPrompts(parameters.points);
+ if (!prompt && points.length === 0) {
+ throw new InferenceProviderError("A matte needs a text prompt or point prompts.", {
+ statusCode: 400,
+ code: "empty_matte_prompts",
+ });
+ }
+ // The segmenter takes object ids as integers; each named brush object
+ // maps to a stable one in prompt order.
+ const objectIds = new Map<string, number>();
+ const objectId = (name: string) => {
+ const held = objectIds.get(name);
+ if (held !== undefined) return held;
+ const next = objectIds.size + 1;
+ objectIds.set(name, next);
+ return next;
+ };
+
+ const submitted = await api(`${QUEUE_BASE}/${model}`, {
+ method: "POST",
+ body: JSON.stringify({
+ video_url: videoUrl,
+ // The mask video, without the source picture under it.
+ apply_mask: false,
+ ...(prompt ? { prompt } : {}),
+ ...(points.length > 0
+ ? {
+ point_prompts: points.map((p) => ({
+ x: Math.round(p.x),
+ y: Math.round(p.y),
+ label: p.label,
+ object_id: objectId(p.object),
+ frame_index: Math.max(0, Math.round(p.frame)),
+ })),
+ }
+ : {}),
+ }),
+ });
+ const requestId =
+ isJsonObject(submitted) && typeof submitted.request_id === "string"
+ ? submitted.request_id
+ : "";
+ const responseUrl =
+ isJsonObject(submitted) && typeof submitted.response_url === "string"
+ ? submitted.response_url
+ : "";
+ if (!requestId || !responseUrl) {
+ return failedResult(model, "The matte service accepted nothing.");
+ }
+ // The track is committed, so the frame-counted charge bills now — one
+ // generation per 16-frame chunk; the polls that follow are free.
+ return {
+ ...inProgressResult(requestId, responseUrl, model),
+ usage: { generationCount: chunks },
+ };
+ }
+
+ async function refreshAsset(
+ generation: StoredGenerationForProvider,
+ ): Promise<AssetGenerationProviderResult> {
+ ensureConfigured(configured);
+ const responseUrl = generation.providerPollingUrl?.trim() ?? "";
+ const requestId = generation.providerJobId?.trim() || generation.providerGenerationId?.trim();
+ if (!responseUrl || !requestId) {
+ return failedResult(generation.model, "No matte prediction to poll.");
+ }
+ const status = await api(`${responseUrl}/status`);
+ const state =
+ isJsonObject(status) && typeof status.status === "string" ? status.status : "";
+ if (state === "IN_QUEUE" || state === "IN_PROGRESS") {
+ return inProgressResult(requestId, responseUrl, generation.model);
+ }
+ let result: JsonValue;
+ try {
+ result = await api(responseUrl);
+ } catch (error) {
+ const message =
+ error instanceof InferenceProviderError ? error.message : "The matte track failed.";
+ return failedResult(generation.model, message);
+ }
+ const url = maskUrl(result);
+ if (!url) {
+ return failedResult(generation.model, "The tracker returned no mask.");
+ }
+ return {
+ provider: providerID,
+ model: generation.model,
+ status: "completed",
+ outputs: [
+ {
+ id: `${generation.id}-matte-0`,
+ kind: "video",
+ contentType: "video/mp4",
+ filename: `${generation.id}-matte.mp4`,
+ url,
+ metadata: { source: "provider-output" },
+ },
+ ],
+ metadata: { provider: providerID },
+ };
+ }
+
+ function inProgressResult(
+ requestId: string,
+ responseUrl: string,
+ model: string,
+ ): AssetGenerationProviderResult {
+ return {
+ provider: providerID,
+ model,
+ status: "in_progress",
+ providerJobId: requestId,
+ providerGenerationId: requestId,
+ providerPollingUrl: responseUrl,
+ outputs: [],
+ metadata: { provider: providerID },
+ };
+ }
+
+ function failedResult(model: string, message: string): AssetGenerationProviderResult {
+ return {
+ provider: providerID,
+ model,
+ status: "failed",
+ outputs: [],
+ error: { message },
+ metadata: { provider: providerID },
+ };
+ }
+
+ return {
+ id: providerID,
+ configured,
+ capabilities: ["matte"],
+ listModels,
+ assetModelFor: (request) => resolveModel(request.model),
+ assetGenerationCountFor,
+ generateAsset,
+ refreshAsset,
+ };
+}
+
+// The finished run's mask video URL.
+function maskUrl(result: JsonValue): string | null {
+ if (!isJsonObject(result)) {
+ return null;
+ }
+ const video = result.video;
+ if (isJsonObject(video) && typeof video.url === "string" && video.url.startsWith("https://")) {
+ return video.url;
+ }
+ return null;
+}
diff --git a/site/src/lib/inference/blobs.ts b/site/src/lib/inference/blobs.ts
index 3119411a..931c9ec3 100644
--- a/site/src/lib/inference/blobs.ts
+++ b/site/src/lib/inference/blobs.ts
@@ -15,7 +15,7 @@
// what keeps this one function enough for every payload shape — asset inputs
// read `data`, content parts read `dataBase64` — with no consumer aware that
// media ever left the body.
-import { getObject } from "@/cut/server/cloud/r2";
+import { getObject, head, presignGet } from "@/cut/server/cloud/r2";
import { isJsonObject } from "@/lib/inference/json";
import { InferenceProviderError, type JsonValue } from "@/lib/inference/providers";
@@ -24,13 +24,20 @@ import { InferenceProviderError, type JsonValue } from "@/lib/inference/provider
// counted, so the caller hears about it here instead.
const MAX_RESOLVED_BYTES = 20 * 1024 * 1024;
-/** True for a value shaped like a stored-blob placeholder. */
-function blobRefOf(value: JsonValue): { key: string; field: string } | null {
+/** True for a value shaped like a stored-blob placeholder. `blobUrl: true`
+ * asks for a time-limited read URL in place of inline bytes — for a provider
+ * that fetches its input itself, where a long video would blow the inline
+ * ceiling. */
+function blobRefOf(value: JsonValue): { key: string; field: string; asUrl: boolean } | null {
if (!isJsonObject(value)) return null;
const key = value.blobRef;
const field = value.blobField;
if (typeof key !== "string" || !key) return null;
- return { key, field: typeof field === "string" && field ? field : "data" };
+ return {
+ key,
+ field: typeof field === "string" && field ? field : "data",
+ asUrl: value.blobUrl === true,
+ };
}
export type BlobReader = (key: string) => Promise<{ bytes: Buffer; mime: string } | null>;
@@ -77,6 +84,19 @@ export async function resolveInferenceBlobs(
code: "blob_not_owned",
});
}
+ if (ref.asUrl) {
+ if (!(await head(ref.key))) {
+ throw new InferenceProviderError(
+ "An attached file is no longer in storage. Send the request again.",
+ { statusCode: 400, code: "blob_missing" }
+ );
+ }
+ const rest = { ...value };
+ delete rest.blobRef;
+ delete rest.blobField;
+ delete rest.blobUrl;
+ return { ...rest, [ref.field]: await presignGet(ref.key) };
+ }
const object = await read(ref.key);
if (!object) {
throw new InferenceProviderError(
diff --git a/site/src/lib/inference/matte-models.ts b/site/src/lib/inference/matte-models.ts
new file mode 100644
index 00000000..73176868
--- /dev/null
+++ b/site/src/lib/inference/matte-models.ts
@@ -0,0 +1,13 @@
+// The hosted matte (video background-removal) model ids, hardcoded like the
+// Gemini registries: configuration is code, and pricing keys off these exact
+// strings.
+
+// The segmenter: one call takes a clip plus a text description, click
+// prompts, or both together (points refine what the words detected), and
+// returns the mask video (apply_mask: false renders the white-on-black mask;
+// the sibling video-rle endpoint returns RLE JSON with no video output).
+export const falMatteModels = {
+ segmenter: "fal-ai/sam-3/video",
+} as const;
+
+export type FalMatteModel = (typeof falMatteModels)[keyof typeof falMatteModels];
diff --git a/site/src/lib/inference/matte-prompts.ts b/site/src/lib/inference/matte-prompts.ts
new file mode 100644
index 00000000..180237f3
--- /dev/null
+++ b/site/src/lib/inference/matte-prompts.ts
@@ -0,0 +1,50 @@
+// The matte request's point prompts, shared by every matte adapter: the
+// client sends parameters.points = [{x, y, frame, label, object}] with x/y in
+// pixels of the uploaded segment and frame as the segment frame index.
+import { isJsonObject } from "@/lib/inference/json";
+
+export type MattePointPrompt = {
+ x: number;
+ y: number;
+ frame: number;
+ label: 0 | 1;
+ object: string;
+};
+
+export function mattePointPrompts(value: unknown): MattePointPrompt[] {
+ if (!Array.isArray(value)) {
+ return [];
+ }
+ const out: MattePointPrompt[] = [];
+ for (const item of value) {
+ if (!isJsonObject(item)) {
+ continue;
+ }
+ const x = Number(item.x);
+ const y = Number(item.y);
+ const frame = Number(item.frame ?? 0);
+ if (!Number.isFinite(x) || !Number.isFinite(y)) {
+ continue;
+ }
+ out.push({
+ x,
+ y,
+ frame: Number.isFinite(frame) ? frame : 0,
+ label: item.label === 0 ? 0 : 1,
+ object:
+ typeof item.object === "string" && /^[\w-]{1,32}$/.test(item.object)
+ ? item.object
+ : "subject",
+ });
+ }
+ return out.slice(0, 64);
+}
+
+/** The request's text prompt — what to matte, in a few words. A runaway
+ * description truncates: the leading words still say what to track, while an
+ * emptied prompt would fail the request with an error naming the wrong
+ * cause. */
+export function matteConceptPrompt(value: unknown): string {
+ const prompt = typeof value === "string" ? value.trim() : "";
+ return prompt.slice(0, 200);
+}
diff --git a/site/src/lib/inference/media-outputs.ts b/site/src/lib/inference/media-outputs.ts
index 234b283c..791f59d1 100644
--- a/site/src/lib/inference/media-outputs.ts
+++ b/site/src/lib/inference/media-outputs.ts
@@ -9,6 +9,7 @@ import { isJsonObject } from "@/lib/inference/json";
const extensionByKind: Record<AssetGenerationKind, RegExp> = {
image: /\.(png|jpe?g|webp|gif|heic|avif)(\?|#|$)/i,
video: /\.(mp4|webm|mov|m4v)(\?|#|$)/i,
+ matte: /\.(mp4|webm|mov|m4v)(\?|#|$)/i,
music: /\.(mp3|wav|m4a|aac|ogg|flac)(\?|#|$)/i,
speech: /\.(mp3|wav|m4a|aac|ogg|flac|pcm)(\?|#|$)/i,
};
@@ -16,12 +17,14 @@ const extensionByKind: Record<AssetGenerationKind, RegExp> = {
const contentTypeByKind: Record<AssetGenerationKind, string> = {
image: "image/png",
video: "video/mp4",
+ matte: "video/mp4",
music: "audio/mpeg",
speech: "audio/mpeg",
};
export function mediaKind(kind: AssetGenerationKind): InferenceModality {
- return kind === "music" || kind === "speech" ? "audio" : kind;
+ if (kind === "music" || kind === "speech") return "audio";
+ return kind === "matte" ? "video" : kind;
}
export function extractMediaOutputs(
diff --git a/site/src/lib/inference/providers.ts b/site/src/lib/inference/providers.ts
index 12b9c738..8ee006ba 100644
--- a/site/src/lib/inference/providers.ts
+++ b/site/src/lib/inference/providers.ts
@@ -5,8 +5,8 @@ import type {
} from "@/lib/inference/schemas";
import type { z } from "zod";
-export type InferenceModality = "text" | "image" | "video" | "audio" | "music" | "speech";
-export type AssetGenerationKind = "image" | "video" | "music" | "speech";
+export type InferenceModality = "text" | "image" | "video" | "audio" | "music" | "speech" | "matte";
+export type AssetGenerationKind = "image" | "video" | "music" | "speech" | "matte";
export type GenerationStatus =
| "pending"
| "in_progress"
@@ -87,6 +87,10 @@ export type ResponseStreamResult = {
export type AssetGenerationProviderRequest = {
generationId: string;
request: AssetGenerationRequest;
+ // The billable unit count the route settled through assetGenerationCountFor,
+ // for a provider whose charge scales with the input. The provider bills
+ // exactly this count, so the preflight and the charge agree by construction.
+ generationCount?: number;
};
export type AssetGenerationProviderResult = {
@@ -141,6 +145,12 @@ export type InferenceProvider = {
// which is what lets the preflight price the generation. Throws the same way generateAsset would
// for an unpriced model, only sooner.
assetModelFor?: (request: AssetGenerationRequest) => string;
+ // How many generation units this request will bill, for a provider whose
+ // charge scales with the input (the matte segmenter's frame chunks). The
+ // route calls it with resolved inputs before the credit preflight, so the
+ // balance check covers the whole charge, and hands the count back through
+ // generateAsset's generationCount.
+ assetGenerationCountFor?: (request: AssetGenerationRequest) => Promise<number>;
generateAsset?: (
request: AssetGenerationProviderRequest,
) => Promise<AssetGenerationProviderResult>;
diff --git a/site/src/lib/inference/router.ts b/site/src/lib/inference/router.ts
index aad4ecdf..396148f2 100644
--- a/site/src/lib/inference/router.ts
+++ b/site/src/lib/inference/router.ts
@@ -3,6 +3,7 @@ import { createGeminiImageAssetProvider } from "@/lib/inference/adapters/gemini-
import { createGeminiMusicAssetProvider } from "@/lib/inference/adapters/gemini-music";
import { createGeminiSpeechAssetProvider } from "@/lib/inference/adapters/gemini-speech";
import { createGeminiOmniVideoAssetProvider } from "@/lib/inference/adapters/gemini-omni-video";
+import { createFalMatteProvider } from "@/lib/inference/adapters/fal-matte";
import {
InferenceProviderError,
type AssetGenerationRequest,
@@ -230,6 +231,10 @@ export function createProviderRegistry() {
// adapter picks the clip or the longer pro model from the requested length so
// a bed can still span a longer video (see gemini-music.ts).
createGeminiMusicAssetProvider(),
+ // The matte segmenter serves kind="matte": Cut's background-removal
+ // quality bake, a clip segment plus a text description and/or point
+ // prompts in, a mask video out.
+ createFalMatteProvider(),
createGeminiResponsesProvider(),
]);
}
diff --git a/site/src/lib/inference/schemas.ts b/site/src/lib/inference/schemas.ts
index 9168a456..f733014f 100644
--- a/site/src/lib/inference/schemas.ts
+++ b/site/src/lib/inference/schemas.ts
@@ -12,9 +12,10 @@ export const inferenceModalitySchema = z.enum([
"audio",
"music",
"speech",
+ "matte",
]);
-export const assetGenerationKindSchema = z.enum(["image", "video", "music", "speech"]);
+export const assetGenerationKindSchema = z.enum(["image", "video", "music", "speech", "matte"]);
export const modelsQuerySchema = z.object({
output_modalities: z.string().optional(),
@@ -139,7 +140,7 @@ export function parseRequestedModalities(value: string | null) {
}
if (value === "all") {
- return ["text", "image", "video", "audio", "music", "speech"] as const;
+ return ["text", "image", "video", "audio", "music", "speech", "matte"] as const;
}
return value