Skip to content

Open source CapCut alternative.

TypeScript

feat(site): add a camera card for split layouts [rebuild]

David Le46f710a81298
diff --git a/site/src/clients/chatgpt/server/mcp.ts b/site/src/clients/chatgpt/server/mcp.ts
index 35f88a6d..0417c335 100644
--- a/site/src/clients/chatgpt/server/mcp.ts
+++ b/site/src/clients/chatgpt/server/mcp.ts
@@ -136,6 +136,7 @@ export function createChatgptServer(
try {
const { view, playback, download, editor } = await projects.withEditor(await run());
const media = liftMedia(view);
+ liftDisplay(view);
const text = describeProjectView(view) + (editor ? "\nThe Donkey Cut editor is open in the card." : "");
return {
content: [{ type: "text" as const, text }, ...media],
diff --git a/site/src/cut/components/CameraCardSection.tools.ts b/site/src/cut/components/CameraCardSection.tools.ts
new file mode 100644
index 00000000..dfff524a
--- /dev/null
+++ b/site/src/cut/components/CameraCardSection.tools.ts
@@ -0,0 +1,61 @@
+/**
+ * The assistant's camera-card tool: the talking-head split layout, in any
+ * frame shape, that the Inspector's Camera card section sets on a video
+ * clip. Kept beside the section; the catalog spreads this list into the
+ * model's toolset and `aiTools.ts` keys its handler on `CameraCardToolName`.
+ * Defaults and ranges come from the same constants the section renders from.
+ */
+
+import { bool, ids, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+import {
+ CARD_BLEED_MAX,
+ CARD_DEFAULTS,
+ CARD_FEATHER_MAX,
+ CARD_OFFSET_MAX,
+ CARD_RADIUS_MAX,
+ CARD_SCALE_MAX,
+ CARD_SCALE_MIN,
+ CARD_SIDES,
+ CARD_TOP_MAX,
+ CARD_TOP_MIN,
+ CARD_WIDTH_MAX,
+ CARD_WIDTH_MIN,
+ defaultCardWidth,
+} from "@/cut/lib/cameraCard";
+
+export const CAMERA_CARD_TOOLS = [
+ {
+ name: "set_camera_card",
+ description: `Lay a video clip out as a camera card, the split section of a talking-head video in any frame shape: a graphic fills the frame and the speaker shows in a rounded card that bleeds off the frame edge, with their head popping out above the card's top edge. \`side\` picks where the card sits: bottom runs it across the bottom, the layout for portrait frames; left or right stands it up that side, ${Math.round(defaultCardWidth(21 / 9) * 100)}–${Math.round(defaultCardWidth(1) * 100)}% of the frame wide by default, the layout for square and landscape frames, where a full-width card leaves the graphic a strip. Left out, the side follows the frame shape, so the card stays right when the project changes shape; pass \`side: "auto"\` to hand a picked side back to the frame. Put the side card opposite the graphic's subject. \`top\` and \`width\` default from the side, the frame and the source, and passing \`side\` lets them default again unless the call names them too. One clip draws both the card and the head from the same frame, so they never drift; the head is the clip's own footage keyed by the free on-device person matte, which this call starts baking; editor_state reports its readiness under \`card.matte\`. Until the matte lands the card shows without the head. The graphic, an image or b-roll, sits on a lower track than the speaker's clip and fills the frame behind the card. The card places the footage itself: the defaults scale and position it from the source's aspect so the head clears the card top, portrait and landscape alike, and \`scale\`/\`offset_x\`/\`offset_y\` adjust from there; the clip's framing zoom, pan and box style stand aside while the card is on. Only the fields you pass change; the first call fills the rest with the defaults. \`clear: true\` turns the card off and the clip frames as before.`,
+ inputSchema: obj({
+ clipId: str("Video clip id"),
+ ids: ids("clipId"),
+ clear: bool("Turn the card layout off"),
+ side: {
+ type: "string",
+ enum: [...CARD_SIDES, "auto"],
+ description: "Where the card sits: bottom, left or right; auto follows the frame shape (bottom in a portrait frame, right in a square or landscape one)",
+ },
+ top: num(
+ `Card top edge, fraction of the clip's box height from the top, ${CARD_TOP_MIN}..${CARD_TOP_MAX} (default from the side, the frame and the source)`
+ ),
+ width: num(
+ `A left or right card's width in view, fraction of the box width, ${CARD_WIDTH_MIN}..${CARD_WIDTH_MAX} (default from the frame shape)`
+ ),
+ radius: num(`Top corner radius, design px at the 1080 short side, 0..${CARD_RADIUS_MAX} (default ${CARD_DEFAULTS.radius})`),
+ side_bleed: num(
+ `How far the card reaches past the frame edges it bleeds off, design px, 0..${CARD_BLEED_MAX} (default ${CARD_DEFAULTS.sideBleed})`
+ ),
+ pop_out: bool(`The speaker's head shows above the card (default ${CARD_DEFAULTS.popOut})`),
+ feather: num(
+ `How far below the card's top edge the head fades out, design px, 0..${CARD_FEATHER_MAX} (default ${CARD_DEFAULTS.feather})`
+ ),
+ shadow: num(`Card shadow opacity 0..1, 0 for none (default ${CARD_DEFAULTS.shadow})`),
+ scale: num(`Footage size over the default placement, ${CARD_SCALE_MIN}..${CARD_SCALE_MAX} (default 1)`),
+ offset_x: num(`Footage shift right, fraction of the frame width, -${CARD_OFFSET_MAX}..${CARD_OFFSET_MAX} (default 0)`),
+ offset_y: num(`Footage shift down, fraction of the frame height, -${CARD_OFFSET_MAX}..${CARD_OFFSET_MAX} (default 0)`),
+ }),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type CameraCardToolName = (typeof CAMERA_CARD_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/CameraCardSection.tsx b/site/src/cut/components/CameraCardSection.tsx
new file mode 100644
index 00000000..ffd4400c
--- /dev/null
+++ b/site/src/cut/components/CameraCardSection.tsx
@@ -0,0 +1,235 @@
+"use client";
+
+import { useEffect } from "react";
+import { Switch } from "@/components/ui/switch";
+import { Button } from "@/components/ui/button";
+import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
+import {
+ CARD_BLEED_MAX,
+ CARD_FEATHER_MAX,
+ CARD_OFFSET_MAX,
+ CARD_RADIUS_MAX,
+ CARD_SCALE_MAX,
+ CARD_SCALE_MIN,
+ CARD_SIDES,
+ CARD_TOP_MAX,
+ CARD_TOP_MIN,
+ CARD_WIDTH_MAX,
+ CARD_WIDTH_MIN,
+ cardMatteKey,
+ newCard,
+ normalizeCard,
+ resolveCardShape,
+ type CameraCard,
+ type CardSide,
+} from "@/cut/lib/cameraCard";
+import { ensureCardMatte, retryCardMatte, useMatteBakes } from "@/cut/lib/removal/bakeJobs";
+import { useEditor } from "@/cut/lib/store";
+import { frameOf, rectOf, type VideoClip } from "@/cut/lib/types";
+import { ResetButton, Row, Section, useSliderCheckpoint } from "@/cut/components/panelBits";
+import { parseNumberInput, parsePercentInput, ScrubValue } from "@/cut/components/ScrubValue";
+import { ValueSlider } from "@/cut/components/ValueSlider";
+
+const formatPercent = (v: number) => `${Math.round(v * 100)}%`;
+const formatPx = (v: number) => String(Math.round(v));
+const SIDE_LABELS: Record<CardSide, string> = { bottom: "Bottom", left: "Left", right: "Right" };
+
+/**
+ * The camera card: the talking-head split layout on one clip, in any frame
+ * shape. The picture shows through a rounded card along the bottom or one
+ * side of the clip's box, and with Pop out on the speaker's head shows above
+ * the card's edge, keyed by the free person matte this section starts
+ * baking. Side, top and width show what the card draws at, defaults
+ * included. Every field is what set_camera_card writes; drafts stream
+ * through the transient updater under one checkpoint per gesture.
+ */
+export function CameraCardSection({ clip }: { clip: VideoClip }) {
+ const card = clip.card;
+ const st = () => useEditor.getState();
+ const ck = useSliderCheckpoint();
+ const job = useMatteBakes((s) => s.jobs[cardMatteKey(clip.id)]);
+ const aspect = useEditor((s) => s.aspect);
+ const asset = useEditor((s) => s.assets.find((a) => a.id === clip.assetId));
+ // The head's matte is owed while the card shows it; the doc sweep starts
+ // it too, so this only makes the start immediate.
+ const owes = !!card?.popOut && !card.matte;
+ useEffect(() => {
+ if (owes) ensureCardMatte(clip.id);
+ }, [owes, clip.id]);
+
+ const current = () => st().clips.find((c) => c.id === clip.id)?.card;
+ const draft = (patch: Partial<CameraCard>) => {
+ const c = current();
+ if (!c) return;
+ ck.begin();
+ st().updateClipTransient(clip.id, { card: normalizeCard({ ...c, ...patch }) });
+ };
+ const commit = (patch: Partial<CameraCard>) => {
+ draft(patch);
+ ck.end();
+ };
+
+ const slider = (
+ label: string,
+ key: "top" | "width" | "shadow" | "scale",
+ value: number,
+ min: number,
+ max: number,
+ snap?: number[],
+ reset?: { show: boolean; onClick: () => void }
+ ) => (
+ <Row label={label}>
+ {reset && <ResetButton title={`Reset ${label.toLowerCase()}`} show={reset.show} onClick={reset.onClick} />}
+ <ValueSlider
+ label={label}
+ sliderClassName="data-horizontal:w-24"
+ valueClassName="w-9 text-muted-foreground"
+ value={value}
+ min={min}
+ max={max}
+ step={0.01}
+ snap={snap}
+ format={formatPercent}
+ parse={parsePercentInput}
+ onDraft={(v) => draft({ [key]: v })}
+ onCommit={(v) => commit({ [key]: v })}
+ />
+ </Row>
+ );
+ const scrub = (
+ label: string,
+ key: "radius" | "sideBleed" | "feather" | "offsetX" | "offsetY",
+ value: number,
+ min: number,
+ max: number,
+ step: number,
+ format: (v: number) => string,
+ parse: (raw: string) => number | null
+ ) => (
+ <ScrubValue
+ label={label}
+ className="w-9 text-muted-foreground"
+ value={value}
+ min={min}
+ max={max}
+ step={step}
+ format={format}
+ parse={parse}
+ onScrub={(v) => draft({ [key]: v })}
+ onCommit={(v) => commit({ [key]: v })}
+ />
+ );
+
+ const fr = frameOf(aspect);
+ const rect = rectOf(clip);
+ const shape = card
+ ? resolveCardShape(card, rect.w * fr.w, rect.h * fr.h, asset?.width ?? 0, asset?.height ?? 0)
+ : null;
+ const sideItems = {
+ auto: `Auto (${SIDE_LABELS[shape?.side ?? "bottom"]})`,
+ ...SIDE_LABELS,
+ };
+
+ const matteLine = !card?.popOut
+ ? null
+ : card.matte
+ ? "Person matte ready"
+ : job?.status === "running"
+ ? `Person matte baking ${Math.round(job.progress * 100)}%`
+ : job?.status === "error"
+ ? (job.error ?? "The person matte could not be baked.")
+ : "Person matte starting";
+
+ return (
+ <Section
+ title="Camera card"
+ info="The speaker shows through a rounded card at the edge of the frame, and their head pops out above it. Auto puts the card across the bottom of a portrait frame and up the right side of a square or landscape one. Put the graphic on a lower track. The card places the footage and draws its own corners and shadow, so framing zoom, pan, border and shadow stand aside while it is on. Lengths are design pixels."
+ enabled={!!card}
+ onEnabledChange={(v) => st().updateClip(clip.id, { card: v ? newCard() : undefined })}
+ >
+ {card && shape && (
+ <>
+ <Row label="Side">
+ <Select
+ value={card.side ?? "auto"}
+ items={sideItems}
+ onValueChange={(v) =>
+ st().updateClip(clip.id, {
+ card: normalizeCard({
+ ...card,
+ side: CARD_SIDES.includes(v as CardSide) ? (v as CardSide) : undefined,
+ top: undefined,
+ width: undefined,
+ }),
+ })
+ }
+ >
+ <SelectTrigger className="h-8 w-36 text-[12px]">
+ <SelectValue />
+ </SelectTrigger>
+ <SelectContent>
+ {(Object.keys(sideItems) as (keyof typeof sideItems)[]).map((id) => (
+ <SelectItem key={id} value={id} className="text-[12px]">
+ {sideItems[id]}
+ </SelectItem>
+ ))}
+ </SelectContent>
+ </Select>
+ </Row>
+ {slider("Top", "top", shape.top, CARD_TOP_MIN, CARD_TOP_MAX, undefined, {
+ show: card.top !== undefined,
+ onClick: () => st().updateClip(clip.id, { card: { ...card, top: undefined } }),
+ })}
+ {shape.side !== "bottom" &&
+ slider("Width", "width", shape.width, CARD_WIDTH_MIN, CARD_WIDTH_MAX, undefined, {
+ show: card.width !== undefined,
+ onClick: () => st().updateClip(clip.id, { card: { ...card, width: undefined } }),
+ })}
+ <Row label="Radius">
+ {scrub("Card radius", "radius", card.radius, 0, CARD_RADIUS_MAX, 1, formatPx, parseNumberInput)}
+ </Row>
+ <Row label="Side bleed">
+ {scrub("Card side bleed", "sideBleed", card.sideBleed, 0, CARD_BLEED_MAX, 1, formatPx, parseNumberInput)}
+ </Row>
+ {slider("Shadow", "shadow", card.shadow, 0, 1)}
+ <Row label="Pop out">
+ <Switch
+ aria-label="Pop out"
+ checked={card.popOut}
+ onCheckedChange={(popOut) => st().updateClip(clip.id, { card: { ...card, popOut } })}
+ />
+ </Row>
+ {card.popOut && (
+ <Row label="Feather">
+ {scrub("Pop out feather", "feather", card.feather, 0, CARD_FEATHER_MAX, 1, formatPx, parseNumberInput)}
+ </Row>
+ )}
+ {matteLine && (
+ <div className="flex min-h-7 items-center justify-between gap-2 text-[12px] text-muted-foreground">
+ <span className="truncate">{matteLine}</span>
+ {job?.status === "error" && (
+ <Button size="sm" variant="outline" onClick={() => retryCardMatte(clip.id)}>
+ Retry
+ </Button>
+ )}
+ </div>
+ )}
+ {slider("Footage size", "scale", card.scale ?? 1, CARD_SCALE_MIN, CARD_SCALE_MAX, [1])}
+ <Row label="Footage offset">
+ {(
+ [
+ ["X", "offsetX", card.offsetX ?? 0],
+ ["Y", "offsetY", card.offsetY ?? 0],
+ ] as const
+ ).map(([axis, key, value]) => (
+ <span key={axis} className="flex items-center gap-1">
+ <span className="text-[11px] text-muted-foreground/70 uppercase">{axis}</span>
+ {scrub(`Footage ${axis} offset`, key, value, -CARD_OFFSET_MAX, CARD_OFFSET_MAX, 0.01, formatPercent, parsePercentInput)}
+ </span>
+ ))}
+ </Row>
+ </>
+ )}
+ </Section>
+ );
+}
diff --git a/site/src/cut/components/Inspector.tsx b/site/src/cut/components/Inspector.tsx
index 71a6e339..4d9cae4c 100644
--- a/site/src/cut/components/Inspector.tsx
+++ b/site/src/cut/components/Inspector.tsx
@@ -718,8 +718,10 @@ function ClipFramePanel({ clip }: { clip: VideoClip }) {
<SubviewHead title="Frame" />
<div className="flex flex-col gap-1 px-3.5 pb-4">
<ClipTransformSection clip={clip} />
- <ClipBorderSection clip={clip} />
- <ClipShadowSection clip={clip} />
+ <CameraCardSection clip={clip} />
+ {/* A camera card draws its own corners and shadow. */}
+ {!clip.card && <ClipBorderSection clip={clip} />}
+ {!clip.card && <ClipShadowSection clip={clip} />}
<ClipMaskSection clip={clip} />
</div>
</>
@@ -1520,6 +1522,8 @@ function ClipPanel({ clip }: { clip: VideoClip }) {
onCheckedChange={(v) => updateClip(clip.id, { hidden: v })}
/>
</Row>
+ {/* A camera card places the footage itself (Frame view). */}
+ {!clip.card && (<>
<Row label="Framing">
<div className="clip-framing flex rounded-lg border border-input p-0.5">
{(["fit", "fill"] as const).map((mode) => (
diff --git a/site/src/cut/components/Timeline.tsx b/site/src/cut/components/Timeline.tsx
index 17da3a01..8cc13bf4 100644
--- a/site/src/cut/components/Timeline.tsx
+++ b/site/src/cut/components/Timeline.tsx
@@ -63,6 +63,7 @@ import type { VideoTrackPlacement } from "@/cut/lib/store";
import { playheadAt, setSkim, skimAt, subscribePlayhead, usePlayhead, useSkim } from "@/cut/lib/playhead";
import { useBrushUi } from "@/cut/lib/removal/brushUi";
import { useMatteBakes } from "@/cut/lib/removal/bakeJobs";
+import { cardMatteKey } from "@/cut/lib/cameraCard";
import { laneHidden, subtitleLaneCount } from "@/cut/lib/subtitles";
import { formatTime, formatTimecode } from "@/cut/lib/time";
import { EFFECT_LABELS, hasSpeedCurve, headSrc, retimeOf, SPEED_CURVE_MAX, SPEED_CURVE_MIN, tailSrc, type EffectId, type Retime, type SpeedNode } from "@donkeycut/effects-kit";
diff --git a/site/src/cut/components/inspectorToolFields.test.ts b/site/src/cut/components/inspectorToolFields.test.ts
index 24f45401..be9ff5d3 100644
--- a/site/src/cut/components/inspectorToolFields.test.ts
+++ b/site/src/cut/components/inspectorToolFields.test.ts
@@ -3,8 +3,11 @@ import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { GRADE_BASIC_FIELDS, GRADE_DETAIL_FIELDS, WHEEL_LABELS, WHEEL_ZONES } from "@donkeycut/effects-kit";
import { OVERLAY_ANIMATION_TOOLS } from "./AnimationTiles.tools";
+import { CAMERA_CARD_TOOLS } from "./CameraCardSection.tools";
import { EFFECTS_TOOLS } from "./EffectsPanel.tools";
+import { GROUP_PANEL_TOOLS } from "./GroupPanel.tools";
import { INSPECTOR_TOOLS } from "./Inspector.tools";
+import { SUBTITLES_TOOLS } from "./SubtitlesPanel.tools";
import { TIMELINE_TOOLS } from "./Timeline.tools";
import { TRANSITIONS_TOOLS } from "./TransitionsPanel.tools";
@@ -61,8 +64,11 @@ const PANEL_SOURCES = [
"ColorWheel.tsx",
"SpeedCurveStrip.tsx",
"RemovalPanel.tsx",
+ "CameraCardSection.tsx",
"AnimationTiles.tsx",
"TransitionsPanel.tsx",
+ "GroupPanel.tsx",
+ "MotionControls.tsx",
];
/** Parameters that say which item, or where it sits on the timeline — the
@@ -71,7 +77,7 @@ const PLACEMENT = new Set([
"id", "ids", "clipId", "transitionId", "which", "kind", "effect",
"start", "end", "in", "out", "at", "t", "track", "lane", "layout",
"region", "region.x", "region.y", "region.w", "region.h",
- "clear", "reset", "reset_all", "auto", "assetId",
+ "clear", "reset", "reset_all", "auto", "assetId", "group_id",
]);
/** A parameter whose panel control reads the document under another name:
diff --git a/site/src/cut/hooks/usePlayback.ts b/site/src/cut/hooks/usePlayback.ts
index 565637ae..bba8da49 100644
--- a/site/src/cut/hooks/usePlayback.ts
+++ b/site/src/cut/hooks/usePlayback.ts
@@ -39,6 +39,7 @@ import {
import { registerSourceColor, registerSourceSampler } from "@/cut/lib/previewCanvas";
import { colorRead, previewFile, type ColorRead } from "@/cut/lib/sourceColor";
import { backdropStill } from "@/cut/lib/backdropStills";
+import { cardMatteKey } from "@/cut/lib/cameraCard";
import { createRasterCanvas, type RasterSurface } from "@/cut/lib/raster";
import { disposeDetailGpu } from "@/cut/lib/detailGpu";
@@ -284,7 +285,8 @@ class Engine {
private presentCanvas: HTMLCanvasElement | null
) {
this.comp = new FrameCompositor(canvas);
- this.comp.removalMatteProvider = (clip, at) => this.matteFor(clip, at);
+ this.comp.removalMatteProvider = (clip, at) => this.matteFor(clip.id, clip.removal?.matte, clip, at);
+ this.comp.cardMatteProvider = (clip, at) => this.matteFor(cardMatteKey(clip.id), clip.card?.matte, clip, at);
// A clip's color recipe describes the frames its source hands over — the
// file that source reads, read the way it reads it; a LUT built off the
// thread repaints when it lands.
@@ -509,12 +511,17 @@ class Engine {
}
/**
- * The alpha matte for a removal clip at timeline time `t`: the baked matte
- * video read like any other source, its luma turned into alpha. Null while
- * nothing has decoded yet — the picture draws plain until the matte lands.
+ * The alpha matte `m` — a removal clip's, or a camera card's — at timeline
+ * time `t`: the baked matte video read like any other source, its luma
+ * turned into alpha and held under `key`. Null while nothing has decoded
+ * yet — the picture draws plain until the matte lands.
*/
- private matteFor(clip: VideoClip, at: number): CanvasImageSource | null {
- const m = clip.removal?.matte;
+ private matteFor(
+ key: string,
+ m: { assetId: string; in: number } | undefined,
+ clip: VideoClip,
+ at: number
+ ): CanvasImageSource | null {
if (!m) return null;
const s = useEditor.getState();
const asset = s.assets.find((a) => a.id === m.assetId);
@@ -538,12 +545,12 @@ class Engine {
);
const frame = src.frameAt(mt, 0, dur);
if (!frame) return null;
- const held = this.matteAlpha.get(clip.id);
+ const held = this.matteAlpha.get(key);
// Re-insert on every touch so the cap below drops the clip drawn longest
// ago.
- this.matteAlpha.delete(clip.id);
+ this.matteAlpha.delete(key);
if (held && held.ts === frame.timestamp) {
- this.matteAlpha.set(clip.id, held);
+ this.matteAlpha.set(key, held);
return held.canvas as CanvasImageSource;
}
const canvas = held?.canvas ?? createRasterCanvas(frame.width, frame.height);
@@ -556,7 +563,7 @@ class Engine {
const img = ctx.getImageData(0, 0, canvas.width, canvas.height);
matteLumaToAlpha(img.data);
ctx.putImageData(img, 0, 0);
- this.matteAlpha.set(clip.id, { ts: frame.timestamp, canvas });
+ this.matteAlpha.set(key, { ts: frame.timestamp, canvas });
if (this.matteAlpha.size > Engine.MATTE_ALPHA_MAX) {
const oldest = this.matteAlpha.keys().next().value;
if (oldest !== undefined) this.matteAlpha.delete(oldest);
@@ -722,51 +729,50 @@ class Engine {
* The same ramps that dim the picture dim the sound: a clip fading out of a
* dissolve takes its audio with it, an upper-track clip's transition carries
* its own, and a live voiceover ducks the rest. A cross dissolve is the
- * ramp on its own — the picture cuts and only these gains move — and it is
- * the one thing here that makes a clip audible outside its own footprint,
- * so both sides of the crossing are really playing at once.
+ * ramp on its own — the picture cuts and only these gains move. It and a
+ * split edit are what make a clip audible outside its own footprint, so
+ * both sides of a crossing really play at once.
*/
private voicesAt(t: number, spans: ClipSpan[], master: ClipSpan | undefined): Voice[] {
const s = useEditor.getState();
const out: Voice[] = [];
const duck = duckGainAt(s.audioClips, t);
- if (master && !master.clip.muted && !master.clip.hidden && !assetIsSilent(master.asset)) {
+ if (master && !master.clip.muted && !master.clip.hidden && !spanSilent(master)) {
const plan = trackZeroPlan(master, spans, t);
- out.push({
- id: master.clip.id,
- ...voiceSpan(master),
- gain: plan.gain * soundCrossGain(spans, master, t) * duck * (master.clip.volume ?? 1),
- });
+ const v = voiceSpan(master);
+ if (v)
+ out.push({
+ id: master.clip.id,
+ ...v,
+ gain: plan.gain * soundCrossGain(spans, master, t) * duck * (master.clip.volume ?? 1),
+ });
}
- // The other half of a crossing: a neighbour playing into its handle, past
- // its own footprint, carried by the cross ramp alone.
+ // The other half of a crossing, and a split edit's lead or tail: a clip
+ // playing into its handle, past its own footprint, carried by the cross
+ // and split ramps alone.
for (const spans0 of [spans, ...this.overlaySpans()]) {
for (const { span, gain } of crossHandles(spans0, t)) {
- if (assetIsSilent(span.asset)) continue;
+ if (spanSilent(span)) continue;
// The clip's own id, not a second voice: the handle is the same voice
// carrying on past the cut, and a new id would stop and re-open it
// there — a decode gap in the middle of the crossing.
- out.push({
- id: span.clip.id,
- ...voiceSpan(span),
- gain: gain * (span.clip.volume ?? 1) * duck,
- });
+ const v = voiceSpan(span);
+ if (v) out.push({ id: span.clip.id, ...v, gain: gain * (span.clip.volume ?? 1) * duck });
}
}
- for (const { span, clip, asset, gain } of this.liveOverlays(t)) {
- if (clip.muted || clip.hidden || assetIsSilent(asset)) continue;
- out.push({
- id: clip.id,
- ...voiceSpan(span),
- gain: gain * (clip.volume ?? 1) * duck,
- });
+ for (const { span, clip, gain } of this.liveOverlays(t)) {
+ if (clip.muted || clip.hidden || spanSilent(span)) continue;
+ const v = voiceSpan(span);
+ if (v) out.push({ id: clip.id, ...v, gain: gain * (clip.volume ?? 1) * duck });
}
for (const a of s.audioClips) {
const asset = s.assets.find((x) => x.id === a.assetId);
- if (!asset || a.hidden || assetIsSilent(asset)) continue;
+ if (!asset || a.hidden) continue;
const rt = retimeOf(a);
const len = Math.max(0.1, rt.len);
if (t < a.start || t >= a.start + len) continue;
+ const v = audioVoice(a, asset, s.assets);
+ if (!v || (v.url === asset.url && assetIsSilent(asset))) continue;
// Fade envelope: linear ramps at either end of the clip.
const rel = t - a.start;
const fi = a.fadeIn ?? 0;
@@ -776,16 +782,7 @@ class Engine {
if (fo > 0 && rel > len - fo) g *= Math.max(0, (len - rel) / fo);
// A ducking voiceover never ducks itself, or the others that duck.
const dg = a.duck !== undefined && a.duck < 1 ? 1 : duck;
- out.push({
- id: a.id,
- url: asset.url,
- start: a.start,
- in: a.in,
- out: a.out,
- retime: rt,
- gain: a.volume * g * dg,
- sound: a.sound,
- });
+ out.push({ id: a.id, ...v, gain: a.volume * g * dg, sound: a.sound });
}
this.upcomingVoices(t, spans, duck, out);
return out;
@@ -795,7 +792,9 @@ class Engine {
* The clips that begin within `SOUND_LOOKAHEAD_S` of `t`, added to `out` so
* the mixer opens them before the playhead arrives. Each carries the gain it
* will have on its first frame; the frames inside the clip take over from
- * there. A clip already voiced — the far side of a crossing — is left as it is.
+ * there. A clip's sound opens from where it first plays, its handle or split
+ * edit lead included. A clip already voiced — the far side of a crossing —
+ * is left as it is.
*/
private upcomingVoices(t: number, spans: ClipSpan[], duck: number, out: Voice[]) {
const s = useEditor.getState();
@@ -803,31 +802,22 @@ class Engine {
const soon = (start: number) => start > t && start <= t + SOUND_LOOKAHEAD_S;
for (const spans0 of [spans, ...this.overlaySpans()]) {
for (const span of spans0) {
- if (!soon(span.start) || have.has(span.clip.id)) continue;
- if (span.clip.muted || span.clip.hidden || assetIsSilent(span.asset)) continue;
+ if (!soon(span.start - span.soundBack) || have.has(span.clip.id)) continue;
+ if (span.clip.muted || span.clip.hidden || spanSilent(span)) continue;
+ const v = voiceSpan(span);
+ if (!v) continue;
have.add(span.clip.id);
- out.push({
- id: span.clip.id,
- ...voiceSpan(span),
- gain: (span.clip.volume ?? 1) * duck,
- });
+ out.push({ id: span.clip.id, ...v, gain: span.soundLead ? 0 : (span.clip.volume ?? 1) * duck });
}
}
for (const a of s.audioClips) {
if (!soon(a.start) || have.has(a.id) || a.hidden) continue;
const asset = s.assets.find((x) => x.id === a.assetId);
- if (!asset || assetIsSilent(asset)) continue;
+ if (!asset) continue;
+ const v = audioVoice(a, asset, s.assets);
+ if (!v || (v.url === asset.url && assetIsSilent(asset))) continue;
const dg = a.duck !== undefined && a.duck < 1 ? 1 : duck;
- out.push({
- id: a.id,
- url: asset.url,
- start: a.start,
- in: a.in,
- out: a.out,
- retime: retimeOf(a),
- gain: (a.fadeIn ?? 0) > 0 ? 0 : a.volume * dg,
- sound: a.sound,
- });
+ out.push({ id: a.id, ...v, gain: (a.fadeIn ?? 0) > 0 ? 0 : a.volume * dg, sound: a.sound });
}
}
diff --git a/site/src/cut/lib/aiContext.ts b/site/src/cut/lib/aiContext.ts
index e7b33f2a..c456f5b5 100644
--- a/site/src/cut/lib/aiContext.ts
+++ b/site/src/cut/lib/aiContext.ts
@@ -90,6 +90,60 @@ function clipEffects(clip: VideoClip, live: boolean, frame?: CardFrame) {
...color,
...(clip.boxStyle ? { boxStyle: clip.boxStyle } : {}),
...describeRemoval(clip, live),
+ ...(clip.card ? { card: describeCameraCard(clip, live, frame) } : {}),
+ };
+}
+
+/** What a camera card's defaults read: the frame shape and the clip's
+ * source. */
+export interface CardFrame {
+ aspect: Aspect;
+ assetById: Map<string, MediaAsset>;
+}
+
+/** The clip's camera-card layout as set_camera_card speaks it — the side,
+ * top and width it shows at, with `auto` naming the ones that follow the
+ * frame shape and the source — with the head's person matte: "ready", a live bake's progress or error, "off" when
+ * the card hides the head, or owed until the editor bakes it — what a claim
+ * about the head showing must rest on. */
+export function describeCameraCard(clip: VideoClip, live: boolean, frame?: CardFrame) {
+ const c = clip.card;
+ if (!c) return null;
+ const fr = frameOf(frame?.aspect ?? "9:16");
+ const rect = rectOf(clip);
+ const asset = frame?.assetById.get(clip.assetId);
+ const shape = resolveCardShape(c, rect.w * fr.w, rect.h * fr.h, asset?.width ?? 0, asset?.height ?? 0);
+ const auto = [
+ ...(c.side === undefined ? ["side"] : []),
+ ...(c.top === undefined ? ["top"] : []),
+ ...(shape.side !== "bottom" && c.width === undefined ? ["width"] : []),
+ ];
+ const job = live ? useMatteBakes.getState().jobs[cardMatteKey(clip.id)] : undefined;
+ const matte = !c.popOut
+ ? "off"
+ : c.matte
+ ? "ready"
+ : job?.status === "running"
+ ? `baking (${Math.round(job.progress * 100)}%)`
+ : job?.status === "error"
+ ? `error: ${job.error ?? "failed"}`
+ : live && matteBakesAvailable
+ ? "starting"
+ : "owed: bakes when the project is open in the editor";
+ return {
+ side: shape.side,
+ top: r(shape.top),
+ ...(shape.side !== "bottom" ? { width: r(shape.width) } : {}),
+ ...(auto.length ? { auto } : {}),
+ radius: c.radius,
+ side_bleed: c.sideBleed,
+ pop_out: c.popOut,
+ feather: c.feather,
+ shadow: r(c.shadow),
+ ...(c.scale !== undefined ? { scale: r(c.scale) } : {}),
+ ...(c.offsetX !== undefined ? { offset_x: r(c.offsetX) } : {}),
+ ...(c.offsetY !== undefined ? { offset_y: r(c.offsetY) } : {}),
+ matte,
};
}
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index 10b6e1fe..f53fcd35 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -160,6 +160,8 @@ import { requestSidePanel } from "./panelRequest";
import {
confirmMatteBake,
ensureMatteBake,
+ ensureCardMatte,
+ jobTarget as matteJobTarget,
matteBakesAvailable,
useMatteBakes,
} from "./removal/bakeJobs";
@@ -464,6 +466,16 @@ const MAX_COMPARE = 4;
const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v));
const isNum = (v: unknown): v is number => typeof v === "number" && Number.isFinite(v);
+/** A camera card places its clip's footage itself, so a framing write on
+ * one of `fields` would change nothing anyone sees: refused, naming the
+ * card's own controls. */
+function refuseCardFraming(clip: VideoClip, input: Record<string, unknown>, fields: string[]): void {
+ if (!clip.card || !fields.some((f) => input[f] !== undefined)) return;
+ throw new ToolError(
+ "This clip is laid out as a camera card, which places its footage itself: set_camera_card scale, offset_x and offset_y move it (flips still apply through set_framing)."
+ );
+}
+
/** The row an element tool was aimed at, if it named one. Rows are how
* elements stack, row 0 at the front: two on the same row slide clear of each
* other, so a title that has to sit over a shape names a lower row than it. */
@@ -475,9 +487,11 @@ const aimedLane = (input: Record<string, unknown>): { lane?: number } | undefine
type BrowserToolName = Exclude<
| AiPanelToolName
| AudioToolName
+ | CameraCardToolName
| EditorToolName
| EffectsToolName
| ElementsToolName
+ | GroupPanelToolName
| ImageGenToolName
| InspectorToolName
| LibraryToolName
@@ -2862,6 +2876,10 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
s.updateClip(clip.id, { boxStyle: undefined });
return { id: clip.id, boxStyle: null };
}
+ if (clip.card)
+ throw new ToolError(
+ "This clip is laid out as a camera card, which draws its own corners and shadow: set_camera_card radius and shadow set them."
+ );
const bs = { ...clip.boxStyle };
if (isNum(input.radius)) bs.radius = Math.max(0, input.radius) || undefined;
if (isNum(input.border_width)) bs.borderWidth = Math.max(0, input.border_width) || undefined;
@@ -2897,6 +2915,52 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
return { id: clip.id, boxStyle: boxStyle ?? null };
},
+ set_camera_card: (s, input) => {
+ const clip = requireItem(s.clips, input.clipId, "video clip");
+ if (input.clear === true) {
+ s.updateClip(clip.id, { card: undefined });
+ return { id: clip.id, card: null };
+ }
+ // Only what the call names changes; a first call starts from the
+ // defaults. Naming a side hands top and width back to its defaults
+ // unless the call names them too.
+ const side = input.side === "auto" ? undefined : (CARD_SIDES as readonly unknown[]).includes(input.side) ? (input.side as CardSide) : null;
+ if (input.side !== undefined && side === null)
+ throw new ToolError(`side must be one of ${[...CARD_SIDES, "auto"].join(", ")}.`);
+ const card = normalizeCard({
+ ...(clip.card ?? newCard()),
+ ...(input.side !== undefined ? { side: side ?? undefined, top: undefined, width: undefined } : {}),
+ ...(isNum(input.top) ? { top: input.top } : {}),
+ ...(isNum(input.width) ? { width: input.width } : {}),
+ ...(isNum(input.radius) ? { radius: input.radius } : {}),
+ ...(isNum(input.side_bleed) ? { sideBleed: input.side_bleed } : {}),
+ ...(typeof input.pop_out === "boolean" ? { popOut: input.pop_out } : {}),
+ ...(isNum(input.feather) ? { feather: input.feather } : {}),
+ ...(isNum(input.shadow) ? { shadow: input.shadow } : {}),
+ ...(isNum(input.scale) ? { scale: input.scale } : {}),
+ ...(isNum(input.offset_x) ? { offsetX: input.offset_x } : {}),
+ ...(isNum(input.offset_y) ? { offsetY: input.offset_y } : {}),
+ });
+ s.updateClip(clip.id, { card });
+ if (card.popOut) ensureCardMatte(clip.id);
+ const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
+ const owed = !!next.card?.popOut && !next.card.matte;
+ return {
+ id: clip.id,
+ card: describeCameraCard(next, true, {
+ aspect: useEditor.getState().aspect,
+ assetById: new Map(useEditor.getState().assets.map((a) => [a.id, a])),
+ }),
+ ...(owed
+ ? {
+ note: matteBakesAvailable
+ ? "The head's person matte is baking in the background — wait_for_renders reports when it lands. Until then the card shows without the head."
+ : "The card is set. The head's person matte bakes the next time the project is open in the editor; until then the card shows without the head.",
+ }
+ : {}),
+ };
+ },
+
set_removal: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
const mode = String(input.mode ?? "");
@@ -3393,14 +3457,19 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
};
});
const bakes = useMatteBakes.getState().jobs;
- const cutouts = watchedBakes.map((clipId) => {
- const j = bakes[clipId];
+ const cutouts = watchedBakes.map((key) => {
+ // A camera card's person matte reports beside the clip's cutout.
+ const { clipId, slot } = matteJobTarget(key);
+ const forCard = slot === "card" ? { for: "camera card" } : {};
+ const j = bakes[key];
if (!j) {
- const m = cur.clips.find((c) => c.id === clipId)?.removal?.matte;
- return { clipId, status: "done" as const, ...(m ? { quality: m.quality } : {}) };
+ const c = cur.clips.find((x) => x.id === clipId);
+ const m = slot === "card" ? c?.card?.matte : c?.removal?.matte;
+ return { clipId, ...forCard, status: "done" as const, ...(m ? { quality: m.quality } : {}) };
}
return {
clipId,
+ ...forCard,
status: j.status,
quality: j.quality,
...(j.error ? { error: j.error } : {}),
@@ -5899,6 +5968,7 @@ const SWEEP_TARGETS: Readonly<Record<string, string>> = {
apply_saved_grade: "clipId",
set_speed: "clipId",
set_framing: "clipId",
+ set_camera_card: "clipId",
set_transition: "clipId",
delete_item: "id",
update_cue: "id",
diff --git a/site/src/cut/lib/cameraCard.test.ts b/site/src/cut/lib/cameraCard.test.ts
new file mode 100644
index 00000000..271da888
--- /dev/null
+++ b/site/src/cut/lib/cameraCard.test.ts
@@ -0,0 +1,217 @@
+import { describe, expect, test } from "bun:test";
+import {
+ aboveShare,
+ cameraCardLayout,
+ CARD_DEFAULTS,
+ defaultCardWidth,
+ newCard,
+ normalizeCard,
+ resolveCardShape,
+ traceCardPath,
+ type CardLayout,
+ type PathSink,
+} from "./cameraCard";
+import { frameOf } from "./types";
+
+// The reference layout at 1080×1920: card 1210×864 at x −65, y 1200, top
+// corners 200; a 16:9 source at 1890×1063 from x −405, y 1008; the head
+// fading out over 40px under the card's top edge.
+
+const FRAME = { x: 0, y: 0, w: 1080, h: 1920 };
+const near = (a: number, b: number, tol = 2) => expect(Math.abs(a - b)).toBeLessThanOrEqual(tol);
+
+describe("camera card geometry", () => {
+ test("the defaults reproduce the reference layout for a 16:9 source", () => {
+ const L = cameraCardLayout(newCard(), FRAME, 1920, 1080, 1);
+ expect(L.card.x).toBe(-65);
+ expect(L.card.y).toBe(1200);
+ expect(L.card.w).toBe(1210);
+ // The card runs past the frame bottom, so its bottom edge never shows.
+ expect(L.card.y + L.card.h).toBeGreaterThan(1920);
+ expect(L.radius).toBe(200);
+ near(L.picture.x, -405);
+ near(L.picture.y, 1008);
+ near(L.picture.w, 1890);
+ near(L.picture.h, 1063);
+ });
+
+ test("the pop-out band runs from the box top to the card edge and fades over the feather", () => {
+ const L = cameraCardLayout(newCard(), FRAME, 1920, 1080, 1);
+ expect(L.band).toEqual({ top: 0, edge: 1200, bottom: 1240, left: 0, right: 1080 });
+ const flat = cameraCardLayout({ ...newCard(), feather: 0 }, FRAME, 1920, 1080, 1);
+ expect(flat.band.bottom).toBe(flat.band.edge);
+ });
+
+ test("lengths scale with the frame's design scale", () => {
+ const L = cameraCardLayout(newCard(), { x: 0, y: 0, w: 540, h: 960 }, 1920, 1080, 0.5);
+ expect(L.card.x).toBe(-32.5);
+ expect(L.card.y).toBe(600);
+ expect(L.radius).toBe(100);
+ expect(L.band.bottom).toBe(620);
+ });
+
+ test("a portrait source covers the card and carries the head above its edge", () => {
+ const L = cameraCardLayout(newCard(), FRAME, 1080, 1920, 1);
+ // Wide enough that the card never shows past the picture's sides.
+ expect(L.picture.w).toBeGreaterThanOrEqual(FRAME.w - 1e-6);
+ near(L.picture.h / L.picture.w, 1920 / 1080, 0.001);
+ // The top 40% of a selfie — where the head sits — clears the card.
+ expect(L.picture.y + 0.4 * L.picture.h).toBeLessThanOrEqual(L.card.y);
+ // And the picture reaches the frame bottom.
+ expect(L.picture.y + L.picture.h).toBeGreaterThanOrEqual(1920);
+ expect(aboveShare(9 / 16)).toBeGreaterThan(aboveShare(1));
+ expect(aboveShare(1)).toBeGreaterThan(aboveShare(16 / 9));
+ });
+
+ test("scale keeps the same share of the picture above the card; offsets shift it", () => {
+ const base = cameraCardLayout(newCard(), FRAME, 1920, 1080, 1);
+ const big = cameraCardLayout({ ...newCard(), scale: 1.5 }, FRAME, 1920, 1080, 1);
+ near(big.picture.w, base.picture.w * 1.5, 0.01);
+ near((big.card.y - big.picture.y) / big.picture.h, (base.card.y - base.picture.y) / base.picture.h, 1e-6);
+ near(big.picture.x + big.picture.w / 2, 540, 1e-6);
+ const moved = cameraCardLayout({ ...newCard(), offsetX: 0.1, offsetY: -0.05 }, FRAME, 1920, 1080, 1);
+ near(moved.picture.x - base.picture.x, 108, 1e-6);
+ near(moved.picture.y - base.picture.y, -96, 1e-6);
+ });
+
+ test("the card follows its top inside a regioned box", () => {
+ const box = { x: 100, y: 200, w: 400, h: 800 };
+ const L = cameraCardLayout({ ...newCard(), top: 0.5 }, box, 1920, 1080, 1);
+ expect(L.card.y).toBe(600);
+ expect(L.band.top).toBe(200);
+ near(L.picture.x + L.picture.w / 2, 300, 1e-6);
+ });
+
+ test("normalize clamps every field and drops identity footage fields", () => {
+ const c = normalizeCard({ ...newCard(), top: 5, radius: -3, shadow: 2, scale: 1, offsetX: 0, feather: Number.NaN });
+ expect(c.top).toBe(0.9);
+ expect(c.radius).toBe(0);
+ expect(c.shadow).toBe(1);
+ expect(c.feather).toBe(CARD_DEFAULTS.feather);
+ expect(c.scale).toBeUndefined();
+ expect(c.offsetX).toBeUndefined();
+ });
+
+ test("the outline rounds only the top corners", () => {
+ const calls: string[] = [];
+ const sink: PathSink = {
+ moveTo: (x, y) => calls.push(`M${x},${y}`),
+ lineTo: (x, y) => calls.push(`L${x},${y}`),
+ bezierCurveTo: (_a, _b, _c, _d, x, y) => calls.push(`C${x},${y}`),
+ closePath: () => calls.push("Z"),
+ };
+ traceCardPath(sink, { x: 0, y: 100, w: 1000, h: 500 }, 100);
+ expect(calls).toEqual(["M0,600", "L0,230", "C130,100", "L870,100", "C1000,230", "L1000,600", "Z"]);
+ });
+});
+
+// Every frame shape, with a landscape and a portrait source: the card lands
+// on its side and bleeds off the frame there, the footage covers the card's
+// part in view, and the head clears the card's top edge inside the frame.
+
+const SHAPES = ["9:16", "4:5", "1:1", "16:9", "21:9"] as const;
+const SOURCES = [
+ { name: "16:9", w: 1920, h: 1080 },
+ { name: "9:16", w: 1080, h: 1920 },
+] as const;
+const EPS = 1e-6;
+
+function inView(L: CardLayout, box: { x: number; y: number; w: number; h: number }) {
+ return { left: Math.max(box.x, L.card.x), right: Math.min(box.x + box.w, L.card.x + L.card.w) };
+}
+
+describe("camera card in every frame shape", () => {
+ for (const aspect of SHAPES) {
+ for (const src of SOURCES) {
+ test(`${aspect} frame, ${src.name} source`, () => {
+ const fr = frameOf(aspect);
+ const box = { x: 0, y: 0, w: fr.w, h: fr.h };
+ const ds = Math.min(fr.w, fr.h) / 1080;
+ const L = cameraCardLayout(newCard(), box, src.w, src.h, ds);
+ const view = inView(L, box);
+ const portrait = fr.w < fr.h;
+
+ // The side follows the frame shape.
+ expect(L.side).toBe(portrait ? "bottom" : "right");
+ // The card bleeds off the bottom, and off both sides or its own side.
+ expect(L.card.y + L.card.h).toBeGreaterThan(box.h);
+ expect(L.card.x + L.card.w).toBeGreaterThan(box.w);
+ if (portrait) {
+ expect(L.card.x).toBeLessThan(0);
+ // The graphic keeps at least half the frame above the card.
+ expect(L.card.y).toBeGreaterThanOrEqual(box.h / 2);
+ } else {
+ // A side card leaves the graphic most of the frame: a third to two
+ // fifths of it wide, its inner edge in view.
+ const share = (view.right - view.left) / box.w;
+ expect(share).toBeGreaterThanOrEqual(1 / 3 - EPS);
+ expect(share).toBeLessThanOrEqual(0.4 + EPS);
+ expect(L.card.x).toBeGreaterThan(box.w / 2);
+ }
+ expect(L.card.y).toBeGreaterThan(box.h * 0.2);
+ expect(L.card.y).toBeLessThan(box.h);
+
+ // The footage covers the card's part in view, down past the frame.
+ expect(L.picture.x).toBeLessThanOrEqual(view.left + EPS);
+ expect(L.picture.x + L.picture.w).toBeGreaterThanOrEqual(view.right - EPS);
+ expect(L.picture.y + L.picture.h).toBeGreaterThanOrEqual(box.h - EPS);
+ // The speaker keeps a size the card holds: the middle three tenths
+ // of the source fit across it.
+ expect(0.3 * L.picture.w).toBeLessThanOrEqual(view.right - view.left + EPS);
+
+ // The head clears the card's top edge and stays inside the frame.
+ near(L.card.y - L.picture.y, aboveShare(src.w / src.h) * L.picture.h, 1e-6);
+ expect(L.picture.y).toBeGreaterThanOrEqual(-EPS);
+ // The pop-out band runs over the card's columns, top of frame to edge.
+ expect(L.band.left).toBeCloseTo(view.left, 6);
+ expect(L.band.right).toBeCloseTo(view.right, 6);
+ expect(L.band.top).toBe(0);
+ expect(L.band.edge).toBe(L.card.y);
+ });
+ }
+ }
+
+ test("a side card's default width runs from two fifths of a square to a third of 21:9", () => {
+ expect(defaultCardWidth(1)).toBeCloseTo(0.4, 6);
+ expect(defaultCardWidth(21 / 9)).toBeCloseTo(1 / 3, 6);
+ expect(defaultCardWidth(16 / 9)).toBeLessThan(defaultCardWidth(1));
+ expect(defaultCardWidth(16 / 9)).toBeGreaterThan(defaultCardWidth(21 / 9));
+ });
+
+ test("a picked side holds in any frame", () => {
+ const fr = frameOf("16:9");
+ const box = { x: 0, y: 0, w: fr.w, h: fr.h };
+ const left = cameraCardLayout({ ...newCard(), side: "left" }, box, 1920, 1080, 1);
+ expect(left.side).toBe("left");
+ expect(left.card.x).toBe(-CARD_DEFAULTS.sideBleed);
+ near(left.picture.x + left.picture.w / 2, inView(left, box).right / 2, 1e-6);
+ expect(left.band.left).toBe(0);
+
+ const bottom = cameraCardLayout({ ...newCard(), side: "bottom" }, box, 1920, 1080, 1);
+ expect(bottom.card.w).toBe(fr.w + 2 * CARD_DEFAULTS.sideBleed);
+ expect(bottom.picture.w).toBeGreaterThanOrEqual(fr.w - EPS);
+
+ const tall = frameOf("9:16");
+ const side = cameraCardLayout({ ...newCard(), side: "right" }, { x: 0, y: 0, w: tall.w, h: tall.h }, 1920, 1080, 1);
+ expect(side.card.x).toBeGreaterThan(0);
+ expect(side.picture.y).toBeGreaterThanOrEqual(-EPS);
+ expect(side.picture.y + side.picture.h).toBeGreaterThanOrEqual(tall.h - EPS);
+ });
+
+ test("a default top moves down for a source that needs the headroom; a named one holds", () => {
+ const fr = frameOf("21:9");
+ const auto = resolveCardShape({}, fr.w, fr.h, 1080, 1920);
+ expect(auto.top).toBeGreaterThan(0.45);
+ const named = resolveCardShape({ top: 0.4 }, fr.w, fr.h, 1080, 1920);
+ expect(named.top).toBe(0.4);
+ // Landscape footage keeps the side card's own default.
+ expect(resolveCardShape({}, fr.w, fr.h, 1920, 1080).top).toBe(0.45);
+ });
+
+ test("normalize keeps a known side and drops an unknown one", () => {
+ expect(normalizeCard({ ...newCard(), side: "left" }).side).toBe("left");
+ expect(normalizeCard({ ...newCard(), side: "top" as never }).side).toBeUndefined();
+ expect(normalizeCard({ ...newCard(), width: 5 }).width).toBe(0.7);
+ expect(normalizeCard(newCard()).top).toBeUndefined();
+ });
+});
diff --git a/site/src/cut/lib/cameraCard.ts b/site/src/cut/lib/cameraCard.ts
new file mode 100644
index 00000000..c5a1ca3d
--- /dev/null
+++ b/site/src/cut/lib/cameraCard.ts
@@ -0,0 +1,332 @@
+/**
+ * The camera card: the talking-head video's split layout as one clip
+ * property, in any frame shape.
+ *
+ * A graphic fills the frame on a lower track. Over it, the speaker's clip
+ * shows through a rounded card that bleeds off the frame: across the bottom
+ * of a portrait frame, or up one side of a square or landscape frame, where
+ * a full-width card would leave the graphic a strip. Above the card's top
+ * edge the same footage shows again, keyed by the clip's person matte, so the
+ * head breaks out of the card. Both parts are drawn from one decoded frame,
+ * so the head and the card can never drift apart.
+ *
+ * This module is the geometry alone: which side the card takes, and where
+ * the card, the footage and the pop-out band land inside the clip's box, in
+ * canvas pixels. Every default comes from the box's shape and the source's
+ * aspect. The compositor draws from it, the export's piece renderer draws
+ * through the compositor, and the Inspector and the chat tool read the same
+ * defaults and ranges.
+ */
+
+import type { ClipRemoval } from "@donkeycut/effects-kit";
+
+/** Where the card sits: across the bottom, or up the left or right side. */
+export type CardSide = "bottom" | "left" | "right";
+export const CARD_SIDES = ["bottom", "left", "right"] as const satisfies readonly CardSide[];
+
+export interface CameraCard {
+ /** The card's side. Absent: from the box shape, the bottom of a portrait
+ * box and the right side of a square or landscape one. */
+ side?: CardSide;
+ /** The card's top edge, as a fraction of the clip's box height from the
+ * box top. Absent: from the side, the box shape and the source aspect. */
+ top?: number;
+ /** A side card's width in view, as a fraction of the box width. Absent:
+ * from the box shape. A bottom card spans the box. */
+ width?: number;
+ /** Top corner radius, design px at the 1080 short side. */
+ radius: number;
+ /** How far the card reaches past the frame edges it bleeds off, design
+ * px: both sides of a bottom card, the outer side of a side card. */
+ sideBleed: number;
+ /** The speaker's head shows above the card, keyed by the person matte. */
+ popOut: boolean;
+ /** How far below the card's top edge the pop-out fades out, design px.
+ * It softens the shoulders where the card's corners curve away. */
+ feather: number;
+ /** Opacity of the soft shadow the card casts, 0..1; 0 casts none. */
+ shadow: number;
+ /** Footage size over the default placement; absent = 1. */
+ scale?: number;
+ /** Footage shift, as fractions of the box width and height; absent = 0. */
+ offsetX?: number;
+ offsetY?: number;
+ /** The baked person matte the pop-out reads: the same free on-device
+ * matte an auto cutout bakes, with the same shape. */
+ matte?: ClipRemoval["matte"];
+}
+
+/** The reference look: at 1080×1920 a bottom card 1210×864 at x −65, y 1200,
+ * top corners 200, a 40px fade under the card edge. */
+export const CARD_DEFAULTS = {
+ radius: 200,
+ sideBleed: 65,
+ popOut: true,
+ feather: 40,
+ shadow: 0.45,
+} as const satisfies Omit<CameraCard, "side" | "top" | "width" | "scale" | "offsetX" | "offsetY" | "matte">;
+
+export const CARD_TOP_MIN = 0.2;
+export const CARD_TOP_MAX = 0.9;
+export const CARD_WIDTH_MIN = 0.2;
+export const CARD_WIDTH_MAX = 0.7;
+export const CARD_RADIUS_MAX = 400;
+export const CARD_BLEED_MAX = 300;
+export const CARD_FEATHER_MAX = 120;
+export const CARD_SCALE_MIN = 0.5;
+export const CARD_SCALE_MAX = 3;
+export const CARD_OFFSET_MAX = 1;
+/** The shadow's blur, design px. */
+export const CARD_SHADOW_BLUR = 60;
+
+/** Boxes at least this wide for their height take a side card. */
+const SIDE_ASPECT = 0.999;
+/** The default top of a bottom card and of a side card, before the source
+ * asks for more headroom. */
+const BOTTOM_TOP = 0.625;
+const SIDE_TOP = 0.45;
+/** A side card's default width: two fifths of a square box, a third of a
+ * 21:9 one, on a log scale of the box aspect between and past them. */
+const WIDTH_SQUARE = 0.4;
+const WIDTH_WIDE = 1 / 3;
+const WIDE_ASPECT = 21 / 9;
+/** How much of the picture sits above the card's top edge by default, for a
+ * landscape source and for a portrait one. A 16:9 talking head fills the
+ * frame's height with the speaker, so the top of the head breaks the card;
+ * a phone-held portrait take carries the head higher up a taller picture,
+ * so more of it has to clear the card. Between the two, it follows the
+ * source aspect on a log scale. */
+const ABOVE_LANDSCAPE = 0.18;
+const ABOVE_PORTRAIT = 0.42;
+const LANDSCAPE_ASPECT = 16 / 9;
+const PORTRAIT_ASPECT = 9 / 16;
+/** How much picture sits below the card's top edge, as a multiple of the
+ * card's visible height: the rest hangs off the bottom of the box. */
+const BELOW_RATIO = 1.21;
+/** The widest the footage gets by default, in card widths in view, while it
+ * still reaches the box bottom: a narrow side card keeps the speaker at a
+ * size that fits it. */
+const SPAN_MAX = 2.6;
+/** The corner curve: it starts this many radii from the corner, and its
+ * handles sit this share of that reach from the corner. The result is
+ * tighter at the apex than a circle of the same reach, and it eases into
+ * the straight edges. */
+const CORNER_REACH = 1.3;
+const CORNER_HANDLE = 0.32;
+
+export interface Box {
+ x: number;
+ y: number;
+ w: number;
+ h: number;
+}
+
+export interface CardLayout {
+ /** The side the card took. */
+ side: CardSide;
+ /** The card. Only the top corners round; the bottom sits past the box. */
+ card: Box;
+ /** The top corners' radius, px. */
+ radius: number;
+ /** Where the footage draws, px. */
+ picture: Box;
+ /** The pop-out band over the card's columns inside the box: solid from
+ * the box top down to `edge` (the card's top), fading to nothing at
+ * `bottom`. */
+ band: { top: number; edge: number; bottom: number; left: number; right: number };
+ /** The shadow's blur, px. */
+ shadowBlur: number;
+}
+
+const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v));
+const finite = (v: unknown): v is number => typeof v === "number" && Number.isFinite(v);
+const isSide = (v: unknown): v is CardSide => CARD_SIDES.includes(v as CardSide);
+
+/** The card with every field in range, and the fields that hold their
+ * default left out. */
+export function normalizeCard(c: CameraCard): CameraCard {
+ const or = (v: number | undefined, d: number) => (finite(v) ? v : d);
+ const scale = clamp(or(c.scale, 1), CARD_SCALE_MIN, CARD_SCALE_MAX);
+ const offsetX = clamp(or(c.offsetX, 0), -CARD_OFFSET_MAX, CARD_OFFSET_MAX);
+ const offsetY = clamp(or(c.offsetY, 0), -CARD_OFFSET_MAX, CARD_OFFSET_MAX);
+ return {
+ ...c,
+ side: isSide(c.side) ? c.side : undefined,
+ top: finite(c.top) ? clamp(c.top, CARD_TOP_MIN, CARD_TOP_MAX) : undefined,
+ width: finite(c.width) ? clamp(c.width, CARD_WIDTH_MIN, CARD_WIDTH_MAX) : undefined,
+ radius: clamp(or(c.radius, CARD_DEFAULTS.radius), 0, CARD_RADIUS_MAX),
+ sideBleed: clamp(or(c.sideBleed, CARD_DEFAULTS.sideBleed), 0, CARD_BLEED_MAX),
+ popOut: !!c.popOut,
+ feather: clamp(or(c.feather, CARD_DEFAULTS.feather), 0, CARD_FEATHER_MAX),
+ shadow: clamp(or(c.shadow, CARD_DEFAULTS.shadow), 0, 1),
+ scale: Math.abs(scale - 1) > 1e-4 ? scale : undefined,
+ offsetX: offsetX !== 0 ? offsetX : undefined,
+ offsetY: offsetY !== 0 ? offsetY : undefined,
+ };
+}
+
+/** The share of the picture's height above the card's top edge, from the
+ * source aspect (width / height). */
+export function aboveShare(aspect: number): number {
+ if (!(aspect > 0)) return ABOVE_LANDSCAPE;
+ const span = Math.log(LANDSCAPE_ASPECT) - Math.log(PORTRAIT_ASPECT);
+ const t = clamp((Math.log(LANDSCAPE_ASPECT) - Math.log(aspect)) / span, 0, 1);
+ return ABOVE_LANDSCAPE + (ABOVE_PORTRAIT - ABOVE_LANDSCAPE) * t;
+}
+
+/** The side a card takes in a box of `boxAspect` (width / height) when it
+ * names none. */
+export const defaultCardSide = (boxAspect: number): CardSide => (boxAspect >= SIDE_ASPECT ? "right" : "bottom");
+
+/** A side card's default width share in a box of `boxAspect`. */
+export function defaultCardWidth(boxAspect: number): number {
+ const a = boxAspect > 0 ? boxAspect : 1;
+ const t = clamp(Math.log(a) / Math.log(WIDE_ASPECT), -2, 1);
+ return clamp(WIDTH_SQUARE + (WIDTH_WIDE - WIDTH_SQUARE) * t, CARD_WIDTH_MIN, CARD_WIDTH_MAX);
+}
+
+/** The default footage height for a card whose top edge sits `vis` px above
+ * the box bottom and shows `visW` px wide: the part below the edge fills the
+ * card past the box bottom, capped so a narrow card keeps the speaker at a
+ * size that fits it, and never narrower than the card. */
+function footageHeight(a: number, k: number, vis: number, visW: number): number {
+ const reach = Math.max(1, vis) / (1 - k);
+ let h = BELOW_RATIO * reach;
+ if (h * a > SPAN_MAX * visW) h = Math.max(reach, (SPAN_MAX * visW) / a);
+ if (h * a < visW) h = visW / a;
+ return h;
+}
+
+/** The card's width in view inside a box `w` wide. */
+const viewWidth = (side: CardSide, share: number, w: number) => (side === "bottom" ? w : share * w);
+
+/**
+ * The card's resolved side, top and width share for a box of `boxW`×`boxH`
+ * and a source of `srcW`×`srcH`: what the card names, and the shape's
+ * defaults for the rest. A default top sits lower when the source needs the
+ * headroom, so the top of the picture stays inside the box.
+ */
+export function resolveCardShape(
+ card: Pick<CameraCard, "side" | "top" | "width">,
+ boxW: number,
+ boxH: number,
+ srcW: number,
+ srcH: number
+): { side: CardSide; top: number; width: number } {
+ const boxAspect = boxW > 0 && boxH > 0 ? boxW / boxH : PORTRAIT_ASPECT;
+ const side = isSide(card.side) ? card.side : defaultCardSide(boxAspect);
+ const width = finite(card.width) ? clamp(card.width, CARD_WIDTH_MIN, CARD_WIDTH_MAX) : defaultCardWidth(boxAspect);
+ if (finite(card.top)) return { side, top: clamp(card.top, CARD_TOP_MIN, CARD_TOP_MAX), width };
+ const a = srcW > 0 && srcH > 0 ? srcW / srcH : LANDSCAPE_ASPECT;
+ const k = aboveShare(a);
+ const H = boxH > 0 ? boxH : 1;
+ const visW = viewWidth(side, width, boxW > 0 ? boxW : 1);
+ // Room left above the picture with the card's top at `top`. It only grows
+ // as the card moves down, so the first top with room is found by halving.
+ const room = (top: number) => top * H - k * footageHeight(a, k, (1 - top) * H, visW);
+ let lo: number = side === "bottom" ? BOTTOM_TOP : SIDE_TOP;
+ if (room(lo) >= 0) return { side, top: lo, width };
+ let hi = CARD_TOP_MAX;
+ if (room(hi) < 0) return { side, top: hi, width };
+ for (let i = 0; i < 24; i++) {
+ const mid = (lo + hi) / 2;
+ if (room(mid) >= 0) hi = mid;
+ else lo = mid;
+ }
+ return { side, top: hi, width };
+}
+
+/**
+ * The whole layout inside `box` (canvas px) for a source of `srcW`×`srcH`,
+ * with `ds` canvas px per design px. The footage centers on the card's part
+ * in view and scales about the point where the card's top edge crosses that
+ * center line, so the same share of the speaker stays above the card at any
+ * size; the offset then shifts it.
+ */
+export function cameraCardLayout(
+ card: CameraCard,
+ box: Box,
+ srcW: number,
+ srcH: number,
+ ds: number
+): CardLayout {
+ const c = normalizeCard(card);
+ const shape = resolveCardShape(c, box.w, box.h, srcW, srcH);
+ const aspect = srcW > 0 && srcH > 0 ? srcW / srcH : LANDSCAPE_ASPECT;
+ const shadowBlur = c.shadow > 0 ? CARD_SHADOW_BLUR * ds : 0;
+ const top = box.y + shape.top * box.h;
+ const bottom = box.y + box.h;
+ const bleed = c.sideBleed * ds;
+ const visW = viewWidth(shape.side, shape.width, box.w);
+ const right = shape.side === "right";
+ const cardBox = {
+ x: right ? box.x + box.w - visW : box.x - bleed,
+ y: top,
+ w: shape.side === "bottom" ? box.w + 2 * bleed : visW + bleed,
+ // Past the box bottom by the shadow's reach and a margin, so neither
+ // the card's bottom edge nor its shadow ever shows.
+ h: bottom - top + 2 * shadowBlur + 24 * ds,
+ };
+ const radius = Math.min(c.radius * ds, cardBox.w / 2, cardBox.h);
+ const k = aboveShare(aspect);
+ const h = footageHeight(aspect, k, bottom - top, visW) * (c.scale ?? 1);
+ const w = h * aspect;
+ const cx = shape.side === "bottom" ? box.x + box.w / 2 : right ? box.x + box.w - visW / 2 : box.x + visW / 2;
+ const picture = {
+ x: cx - w / 2 + (c.offsetX ?? 0) * box.w,
+ y: top - k * h + (c.offsetY ?? 0) * box.h,
+ w,
+ h,
+ };
+ return {
+ side: shape.side,
+ card: cardBox,
+ radius,
+ picture,
+ band: {
+ top: box.y,
+ edge: top,
+ bottom: Math.min(bottom, top + c.feather * ds),
+ left: Math.max(box.x, cardBox.x),
+ right: Math.min(box.x + box.w, cardBox.x + cardBox.w),
+ },
+ shadowBlur,
+ };
+}
+
+/** The path calls a card outline needs: a canvas context, or a recorder in
+ * a test. */
+export interface PathSink {
+ moveTo(x: number, y: number): void;
+ lineTo(x: number, y: number): void;
+ bezierCurveTo(c1x: number, c1y: number, c2x: number, c2y: number, x: number, y: number): void;
+ closePath(): void;
+}
+
+/** Trace the card's outline: eased top corners, square bottom corners.
+ * Path commands only, so drawing it allocates nothing. */
+export function traceCardPath(p: PathSink, card: Box, radius: number): void {
+ const { x, y, w, h } = card;
+ const e = Math.max(0, Math.min(radius * CORNER_REACH, w / 2, h));
+ const d = e * CORNER_HANDLE;
+ p.moveTo(x, y + h);
+ p.lineTo(x, y + e);
+ p.bezierCurveTo(x, y + d, x + d, y, x + e, y);
+ p.lineTo(x + w - e, y);
+ p.bezierCurveTo(x + w - d, y, x + w, y + d, x + w, y + e);
+ p.lineTo(x + w, y + h);
+ p.closePath();
+}
+
+/** The card a fresh layout starts from: side, top and width follow the
+ * frame shape and the source. */
+export function newCard(): CameraCard {
+ return { ...CARD_DEFAULTS };
+}
+
+/** Whether a clip lays out as a camera card. */
+export const cardActive = (c: { card?: CameraCard }): boolean => !!c.card;
+
+/** The key a card's matte job and staged matte frame go under, beside the
+ * clip's own cutout under its bare id. */
+export const cardMatteKey = (clipId: string): string => `card:${clipId}`;
diff --git a/site/src/cut/lib/cameraCardComposite.test.ts b/site/src/cut/lib/cameraCardComposite.test.ts
new file mode 100644
index 00000000..39dc4b59
--- /dev/null
+++ b/site/src/cut/lib/cameraCardComposite.test.ts
@@ -0,0 +1,127 @@
+import { beforeAll, describe, expect, test } from "bun:test";
+import { FrameCompositor, type Frame } from "./composite";
+import { cameraCardLayout, newCard } from "./cameraCard";
+import { setRasterFactory, type RasterSurface } from "./raster";
+import type { VideoClip } from "./types";
+
+// The compositor's camera-card pass, drawn onto canvases that record their
+// calls: where the picture lands, that the head reads the person matte only
+// when the layout shows it, and that the matte multiplies through the same
+// placement as the picture.
+
+type Call = { op: string; target: string; args: unknown[]; mode: string };
+
+const calls: Call[] = [];
+
+class RecordingCanvas {
+ constructor(
+ public width: number,
+ public height: number,
+ readonly name: string
+ ) {}
+ private ctx = new Proxy(
+ { globalCompositeOperation: "source-over", globalAlpha: 1, filter: "none" } as Record<string, unknown>,
+ {
+ get: (target, prop) => {
+ if (prop in target) return target[prop as string];
+ if (prop === "canvas") return this;
+ if (prop === "createLinearGradient") return () => ({ addColorStop: () => {} });
+ if (typeof prop !== "string") return undefined;
+ return (...args: unknown[]) => {
+ calls.push({ op: prop, target: this.name, args, mode: String(target.globalCompositeOperation) });
+ };
+ },
+ set: (target, prop, value) => ((target[prop as string] = value), true),
+ }
+ );
+ getContext() {
+ return this.ctx;
+ }
+}
+
+let made = 0;
+beforeAll(() => {
+ setRasterFactory({
+ createCanvas: (w, h) => new RecordingCanvas(w, h, `scratch${made++}`) as unknown as RasterSurface,
+ decodeImage: async () => null,
+ canvasToBlob: async () => new Blob(),
+ });
+});
+
+const W = 1080;
+const H = 1920;
+const picture = { name: "picture" } as unknown as CanvasImageSource;
+const matteImage = { name: "matte" } as unknown as CanvasImageSource;
+const frame: Frame = { kind: "ready", image: picture, width: 1920, height: 1080 };
+
+const clipOf = (over: Partial<VideoClip> = {}): VideoClip => ({
+ id: "speaker",
+ assetId: "a1",
+ track: 1,
+ start: 0,
+ in: 0,
+ out: 4,
+ muted: false,
+ card: newCard(),
+ ...over,
+});
+
+function draw(clip: VideoClip, matte: CanvasImageSource | null) {
+ calls.length = 0;
+ const comp = new FrameCompositor(new RecordingCanvas(W, H, "frame") as unknown as RasterSurface);
+ let asked = 0;
+ comp.cardMatteProvider = () => {
+ asked++;
+ return matte;
+ };
+ comp.drawLayer(frame, clip, false, 1, 1);
+ return { asked };
+}
+
+describe("camera card compositing", () => {
+ test("the picture lands at the card layout through the card's clip", () => {
+ draw(clipOf({ card: { ...newCard(), popOut: false } }), null);
+ const L = cameraCardLayout(newCard(), { x: 0, y: 0, w: W, h: H }, 1920, 1080, 1);
+ const pic = calls.find((c) => c.op === "drawImage" && c.target === "frame" && c.args[0] === picture);
+ expect(pic?.args.slice(1)).toEqual([L.picture.x, L.picture.y, L.picture.w, L.picture.h]);
+ expect(calls.some((c) => c.op === "clip" && c.target === "frame")).toBe(true);
+ });
+
+ test("the person matte is read only when the card shows the head", () => {
+ expect(draw(clipOf({ card: { ...newCard(), popOut: false } }), matteImage).asked).toBe(0);
+ expect(draw(clipOf(), matteImage).asked).toBe(1);
+ });
+
+ test("the head multiplies the matte through the picture's own placement", () => {
+ draw(clipOf(), matteImage);
+ const L = cameraCardLayout(newCard(), { x: 0, y: 0, w: W, h: H }, 1920, 1080, 1);
+ const keyed = calls.find((c) => c.op === "drawImage" && c.args[0] === matteImage);
+ expect(keyed?.mode).toBe("destination-in");
+ expect(keyed?.args.slice(1)).toEqual([L.picture.x, L.picture.y, L.picture.w, L.picture.h]);
+ // The head lands on the frame after the card's picture.
+ const onFrame = calls.filter((c) => c.op === "drawImage" && c.target === "frame");
+ expect(onFrame[0].args[0]).toBe(picture);
+ expect(onFrame.length).toBe(2);
+ });
+
+ test("a side card in a landscape frame keeps the head over the card's columns", () => {
+ calls.length = 0;
+ const comp = new FrameCompositor(new RecordingCanvas(1920, 1080, "wide") as unknown as RasterSurface);
+ comp.cardMatteProvider = () => matteImage;
+ comp.drawLayer(frame, clipOf(), false, 1, 1);
+ const L = cameraCardLayout(newCard(), { x: 0, y: 0, w: 1920, h: 1080 }, 1920, 1080, 1);
+ expect(L.side).toBe("right");
+ const pic = calls.find((c) => c.op === "drawImage" && c.target === "wide" && c.args[0] === picture);
+ expect(pic?.args.slice(1)).toEqual([L.picture.x, L.picture.y, L.picture.w, L.picture.h]);
+ const keyed = calls.find((c) => c.op === "drawImage" && c.args[0] === matteImage);
+ const band = calls.find((c) => c.op === "rect" && c.target === keyed?.target);
+ expect(band?.args[0]).toBe(L.band.left);
+ expect(band?.args[2]).toBe(L.band.right - L.band.left);
+ });
+
+ test("without a matte yet the card draws alone", () => {
+ draw(clipOf(), null);
+ const onFrame = calls.filter((c) => c.op === "drawImage" && c.target === "frame");
+ expect(onFrame.length).toBe(1);
+ });
+});
diff --git a/site/src/cut/lib/cameraCardVideo.ts b/site/src/cut/lib/cameraCardVideo.ts
new file mode 100644
index 00000000..ed8183ab
--- /dev/null
+++ b/site/src/cut/lib/cameraCardVideo.ts
@@ -0,0 +1,154 @@
+"use client";
+
+/**
+ * Camera-card pieces for the ffmpeg export: a card clip's whole layout —
+ * shadow, the picture through the card, the matted head above it — rendered
+ * by the compositor at the size of the clip's box, as the color/alpha video
+ * pair a removal clip ships. The engine merges the pair and lays it in the
+ * box like any picture, then runs the clip's mask, pose and fades over it,
+ * so it composites the very pixels the preview draws. Grade, look and any
+ * cutout bake in. Without a person matte yet the card renders without the
+ * head, the preview's own degrade.
+ */
+
+import { matteLumaToAlpha, removalActive, retimeOf, type ClipRemoval } from "@donkeycut/effects-kit";
+import { openCanvasVideo } from "./canvasVideo";
+import { FrameCompositor, type Frame } from "./composite";
+import { ensureClipLuts } from "./lutBuild";
+import { createRasterCanvas, decodeRasterImageUrl, type RasterSurface } from "./raster";
+import { splitFrame, type RemovalPieces } from "./removalVideo";
+import type { MediaAsset, VideoClip } from "./types";
+import { liveReader } from "./liveReader";
+import type { ClipReader } from "./exportRender";
+
+type Matte = NonNullable<ClipRemoval["matte"]>;
+
+/** One baked matte read alongside the clip: the frame staged per output
+ * frame, luma turned to alpha in a canvas kept for the whole render. */
+export function matteStage(m: Matte | undefined, assets: MediaAsset[]) {
+ const asset = m ? assets.find((a) => a.id === m.assetId) : undefined;
+ if (!m || !asset) return null;
+ const reader: ClipReader = liveReader(asset);
+ const canvas: RasterSurface = createRasterCanvas(2, 2);
+ const dur = Math.max(0.1, asset.duration || 0.1);
+ let ready = false;
+ return {
+ async stage(srcT: number) {
+ ready = false;
+ const f: Frame = await reader.frameAt(Math.min(Math.max(0, srcT - m.in), dur - 0.001));
+ if (f.kind !== "ready") return;
+ if (canvas.width !== f.width) canvas.width = f.width;
+ if (canvas.height !== f.height) canvas.height = f.height;
+ const ctx = canvas.getContext("2d", { willReadFrequently: true }) as CanvasRenderingContext2D;
+ ctx.clearRect(0, 0, f.width, f.height);
+ ctx.drawImage(f.image, 0, 0);
+ const px = ctx.getImageData(0, 0, f.width, f.height);
+ matteLumaToAlpha(px.data);
+ ctx.putImageData(px, 0, 0);
+ ready = true;
+ },
+ get: () => (ready ? (canvas as CanvasImageSource) : null),
+ dispose: () => reader.dispose(),
+ };
+}
+
+/**
+ * Render one card clip's layer over its segment at `fps`, `box` pixels wide
+ * and high — the clip's region box in the output frame, `frameW`×`frameH`.
+ * Null when the clip has no card.
+ */
+export async function renderCardPieces(
+ asset: MediaAsset,
+ clip: VideoClip,
+ assets: MediaAsset[],
+ opts: { fps: number; frameW: number; frameH: number; box: { w: number; h: number } }
+): Promise<RemovalPieces | null> {
+ if (!clip.card) return null;
+ const rt = retimeOf(clip);
+ const still = asset.type === "image";
+ const dur = Math.max(0.1, rt.len);
+ const frames = Math.max(1, Math.ceil(dur * opts.fps));
+ const w = opts.box.w;
+ const h = opts.box.h;
+
+ const reader = liveReader(asset);
+ const removal = matteStage(removalActive(clip.removal) ? clip.removal?.matte : undefined, assets);
+ const card = matteStage(clip.card.popOut ? clip.card.matte : undefined, assets);
+ try {
+ const px = w * h;
+ const rgbOut = await openCanvasVideo({
+ width: w,
+ height: h,
+ fps: opts.fps,
+ frames,
+ bitrate: Math.min(20_000_000, Math.max(2_000_000, Math.round(px * opts.fps * 0.12))),
+ });
+ const alphaOut = await openCanvasVideo({
+ width: w,
+ height: h,
+ fps: opts.fps,
+ frames,
+ bitrate: Math.min(4_000_000, Math.max(600_000, Math.round(px * opts.fps * 0.02))),
+ });
+ const readSurface = createRasterCanvas(w, h);
+ const readCtx = readSurface.getContext("2d", {
+ willReadFrequently: true,
+ }) as CanvasRenderingContext2D | null;
+ const layer = createRasterCanvas(w, h);
+ const layerCtx = layer.getContext("2d") as CanvasRenderingContext2D | null;
+ if (!readCtx || !layerCtx) throw new Error("No drawing surface for the camera card layer.");
+
+ // The compositor draws the box as its whole canvas, at the output
+ // frame's design scale, so lengths match the preview's frame-sized draw.
+ const comp = new FrameCompositor(layer);
+ comp.colorMode = "exact";
+ comp.designScale = Math.min(opts.frameW, opts.frameH) / 1080;
+ await reader.colorRead.settled();
+ comp.sourceProvider = () => reader.colorRead.recipe();
+ // The engine places, masks, poses and fades the layer; the card and
+ // everything that rides inside it draw here.
+ const plain: VideoClip = {
+ ...clip,
+ frame: undefined,
+ mask: undefined,
+ kf: undefined,
+ rotation: undefined,
+ opacity: undefined,
+ boxStyle: undefined,
+ hidden: undefined,
+ };
+ await ensureClipLuts([comp.recipeFor(plain)]);
+ const backdrops = new Map<string, CanvasImageSource | null>();
+ const bd = clip.removal?.backdrop;
+ if (removalActive(clip.removal) && bd?.kind === "image" && bd.assetId) {
+ const a = assets.find((x) => x.id === bd.assetId);
+ const img = a ? await decodeRasterImageUrl(a.url).catch(() => null) : null;
+ backdrops.set(bd.assetId, img ? img.source : null);
+ }
+ comp.backdropImageProvider = (assetId) => backdrops.get(assetId) ?? null;
+ comp.removalMatteProvider = () => removal?.get() ?? null;
+ comp.cardMatteProvider = () => card?.get() ?? null;
+ const alphaPlane = new ImageData(w, h);
+
+ for (let i = 0; i < frames; i++) {
+ const s = Math.min(dur - 1 / (opts.fps * 2), i / opts.fps);
+ const srcT = still ? 0 : rt.srcAt(s);
+ const frame = await reader.frameAt(srcT);
+ layerCtx.setTransform(1, 0, 0, 1, 0, 0);
+ layerCtx.clearRect(0, 0, w, h);
+ if (frame.kind === "ready") {
+ await removal?.stage(srcT);
+ await card?.stage(srcT);
+ comp.drawIntoRect(frame, { x: 0, y: 0, w: 1, h: 1 }, false, 1, clip.start + s, 1, plain);
+ }
+ splitFrame(layer, w, h, { surface: readSurface, ctx: readCtx }, alphaPlane, rgbOut.ctx, alphaOut.ctx, w, h);
+ await rgbOut.add(i / opts.fps, 1 / opts.fps);
+ await alphaOut.add(i / opts.fps, 1 / opts.fps);
+ }
+ return { rgb: await rgbOut.finish(), alpha: await alphaOut.finish() };
+ } finally {
+ reader.dispose();
+ removal?.dispose();
+ card?.dispose();
+ }
+}
diff --git a/site/src/cut/lib/composite.ts b/site/src/cut/lib/composite.ts
index 3b557b06..5af2cf65 100644
--- a/site/src/cut/lib/composite.ts
+++ b/site/src/cut/lib/composite.ts
@@ -20,6 +20,7 @@
*/
import { applyDetail, applyLutToImageData, applyMaskToCanvas, detailActive, grainTile, hexToHlgHex, lookCssFilter, lookPost, maskComposite, paintStrokeInk, removalActive, type ClipColorRecipe, type OutputSpace } from "@donkeycut/effects-kit";
+import { cameraCardLayout, cardActive, traceCardPath, type CardLayout } from "./cameraCard";
import { applyDetailGpu } from "./detailGpu";
import { applyLutGpu } from "./gradeGpu";
import { clipRecipe, peekClipLut, requestClipLut, type ClipLut, type RecipeSource } from "./lutBuild";
@@ -116,6 +117,27 @@ export class FrameCompositor {
/** The pixel pass of the grade's spatial controls, when the GPU cannot
* run it: a scratch the CPU reads and writes. */
private detailScratch: Surface | null = null;
+ /** The camera card: the pop-out builds in `cardPop` (rows down to the
+ * band's fade), and a card drawn at partial alpha or under a look's post
+ * passes flattens in `cardLayer` first, with the post copy in `cardPost`. */
+ private cardPop: Surface | null = null;
+ private cardLayer: Surface | null = null;
+ private cardPost: Surface | null = null;
+ /** The fade under the card edge, rebuilt only when the band moves. */
+ private cardFade: { ctx: Ctx; edge: number; bottom: number; fill: CanvasGradient } | null = null;
+ /** The card shadow's ink, rebuilt only when its opacity changes. */
+ private cardInk: { opacity: number; ink: string } | null = null;
+
+ /** Canvas px per design px for the card's lengths. Absent, the canvas's
+ * own short side sets it; the export's piece renderer draws one clip's box
+ * on a canvas of that box's size and names the frame's scale here. */
+ designScale: number | null = null;
+
+ /** Where a camera card's person matte comes from: the clip's baked matte
+ * at `at` timeline seconds as alpha coverage over its source frame, like
+ * `removalMatteProvider`. Null until the bake lands; the card draws
+ * without its pop-out until then. */
+ cardMatteProvider: ((clip: VideoClip, at: number) => CanvasImageSource | null) | null = null;
/** What a clip's code values mean — its source profile, and whatever the
* decode route adds — so the compositor can build the clip's color recipe.
@@ -641,7 +663,8 @@ export class FrameCompositor {
return (
!!clip &&
(clipPosed(clip) ||
- !!clip.boxStyle?.shadow ||
+ // A camera card casts its own shadow; the box style stands aside.
+ (!!clip.boxStyle?.shadow && !cardActive(clip)) ||
!!(clip.mask && (clip.mask.kind !== "subject" || this.subjectMatteProvider)))
);
}
@@ -749,7 +772,7 @@ export class FrameCompositor {
ctx.save();
ctx.translate(Math.round(fx.dx ?? 0), Math.round(fx.dy ?? 0));
}
- const shade = this.shadowOf(out, clip.boxStyle?.shadow);
+ const shade = cardActive(clip) ? null : this.shadowOf(out, clip.boxStyle?.shadow);
if (shade) ctx.drawImage(shade, 0, 0);
ctx.drawImage(out, 0, 0);
if (hasFx) ctx.restore();
@@ -806,6 +829,10 @@ export class FrameCompositor {
);
return;
}
+ if (clip && cardActive(clip)) {
+ this.drawCard(frame, rect, alpha, at, zoom, clip);
+ return;
+ }
const ctx = this.ctx();
if (!ctx) return;
const W = this.canvas.width;
@@ -871,6 +898,213 @@ export class FrameCompositor {
this.applyLookPost(clip, rx, ry, rw, rh, alpha, at);
}
+ /**
+ * Draw a camera-card clip into its box (see cameraCard.ts): the card's
+ * shadow, the picture through the card, and the matted head above the
+ * card's top edge, all from the one frame. A transition's zoom scales the
+ * whole layout about the box center inside the box, the way the export's
+ * engine zooms the card's rendered layer.
+ */
+ private drawCard(
+ frame: Extract<Frame, { kind: "ready" }>,
+ rect: FrameRect,
+ alpha: number,
+ at: number,
+ zoom: number,
+ clip: VideoClip
+ ) {
+ const ctx = this.ctx();
+ const card = clip.card;
+ if (!ctx || !card || alpha <= 0) return;
+ const W = this.canvas.width;
+ const H = this.canvas.height;
+ const ds = this.designScale ?? Math.min(W, H) / 1080;
+ const box = { x: rect.x * W, y: rect.y * H, w: rect.w * W, h: rect.h * H };
+ const L = cameraCardLayout(card, box, frame.width, frame.height, ds);
+ const src = this.removedSource(frame, clip, at);
+ // The matte is asked for only when the layout shows the head.
+ const matte = card.popOut ? (this.cardMatteProvider?.(clip, at) ?? null) : null;
+ const post = !!lookPost(clip.look, clip.lookAmount);
+ const a = Math.min(1, alpha);
+ if (a >= 0.999 && !post) {
+ this.paintCardShadow(ctx, L, box, zoom, card.shadow);
+ this.paintCardBody(ctx, L, box, zoom, src, matte, clip);
+ return;
+ }
+ // Flatten first: a fading card fades as one layer, the head over the
+ // card included, and a look's post passes stay inside what it shows.
+ const { surface: layer } = this.scratch("cardLayer", W, H);
+ const lctx = layer.getContext("2d") as Ctx | null;
+ if (!lctx) return;
+ lctx.setTransform(1, 0, 0, 1, 0, 0);
+ lctx.globalAlpha = 1;
+ lctx.globalCompositeOperation = "source-over";
+ lctx.clearRect(0, 0, W, H);
+ this.paintCardBody(lctx, L, box, zoom, src, matte, clip);
+ if (post) {
+ const { surface: copy } = this.scratch("cardPost", W, H);
+ const pctx = copy.getContext("2d") as Ctx | null;
+ if (pctx) {
+ pctx.setTransform(1, 0, 0, 1, 0, 0);
+ pctx.globalAlpha = 1;
+ pctx.globalCompositeOperation = "source-over";
+ pctx.clearRect(0, 0, W, H);
+ pctx.drawImage(layer, 0, 0);
+ this.applyLookPost(clip, box.x, box.y, box.w, box.h, 1, at, { canvas: copy, ctx: pctx });
+ lctx.globalCompositeOperation = "source-atop";
+ lctx.drawImage(copy, 0, 0);
+ }
+ }
+ // The shadow goes in under the finished picture.
+ lctx.globalCompositeOperation = "destination-over";
+ this.paintCardShadow(lctx, L, box, zoom, card.shadow);
+ lctx.globalCompositeOperation = "source-over";
+ const prev = ctx.globalAlpha;
+ ctx.globalAlpha = a;
+ ctx.drawImage(layer, 0, 0);
+ ctx.globalAlpha = prev;
+ }
+
+ /** Clip `c` to the box and scale the layout by `zoom` about its center. */
+ private enterCardBox(c: Ctx, box: { x: number; y: number; w: number; h: number }, zoom: number) {
+ c.beginPath();
+ c.rect(box.x, box.y, box.w, box.h);
+ c.clip();
+ if (zoom > 0 && Math.abs(zoom - 1) > 1e-4) {
+ const cx = box.x + box.w / 2;
+ const cy = box.y + box.h / 2;
+ c.translate(cx, cy);
+ c.scale(zoom, zoom);
+ c.translate(-cx, -cy);
+ }
+ }
+
+ /** The card's soft shadow, outside the card only: the card's own fill is
+ * clipped away and its shadow stays. */
+ private paintCardShadow(
+ c: Ctx,
+ L: CardLayout,
+ box: { x: number; y: number; w: number; h: number },
+ zoom: number,
+ opacity: number
+ ) {
+ if (!(opacity > 0) || !(L.shadowBlur > 0)) return;
+ let ink = this.cardInk;
+ if (!ink || ink.opacity !== opacity) {
+ ink = { opacity, ink: `rgba(0,0,0,${Math.min(1, opacity).toFixed(3)})` };
+ this.cardInk = ink;
+ }
+ const reach = 2 * L.shadowBlur;
+ c.save();
+ this.enterCardBox(c, box, zoom);
+ c.beginPath();
+ c.rect(L.card.x - reach, L.card.y - reach, L.card.w + 2 * reach, L.card.h + 2 * reach);
+ traceCardPath(c, L.card, L.radius);
+ c.clip("evenodd");
+ // A canvas shadow ignores the transform, so the blur follows the zoom
+ // by hand.
+ c.shadowColor = ink.ink;
+ c.shadowBlur = L.shadowBlur * (zoom > 0 ? zoom : 1);
+ c.shadowOffsetX = 0;
+ c.shadowOffsetY = 0;
+ c.fillStyle = "#000000";
+ c.beginPath();
+ traceCardPath(c, L.card, L.radius);
+ c.fill();
+ c.restore();
+ }
+
+ /** The picture through the card, then the head above it. */
+ private paintCardBody(
+ c: Ctx,
+ L: CardLayout,
+ box: { x: number; y: number; w: number; h: number },
+ zoom: number,
+ src: CanvasImageSource,
+ matte: CanvasImageSource | null,
+ clip: VideoClip
+ ) {
+ c.save();
+ this.enterCardBox(c, box, zoom);
+ c.save();
+ c.beginPath();
+ traceCardPath(c, L.card, L.radius);
+ c.clip();
+ this.drawCardPicture(c, src, L.picture, clip);
+ c.restore();
+ const pop = matte ? this.cardPopOut(L, src, matte, clip) : null;
+ if (pop) c.drawImage(pop, 0, 0);
+ c.restore();
+ }
+
+ /** The footage at its card placement, mirrored about its own center. */
+ private drawCardPicture(
+ c: Ctx,
+ img: CanvasImageSource,
+ P: { x: number; y: number; w: number; h: number },
+ clip: VideoClip
+ ) {
+ if (clip.flipH || clip.flipV) {
+ c.save();
+ c.translate(P.x + P.w / 2, P.y + P.h / 2);
+ c.scale(clip.flipH ? -1 : 1, clip.flipV ? -1 : 1);
+ c.drawImage(img, -P.w / 2, -P.h / 2, P.w, P.h);
+ c.restore();
+ } else {
+ c.drawImage(img, P.x, P.y, P.w, P.h);
+ }
+ }
+
+ /**
+ * The head above the card: the picture again, multiplied by the person
+ * matte drawn through the same placement, kept over the card's columns
+ * from the box top down to the card's top edge and faded out over the
+ * feather below it. Built on
+ * the rows the band reaches; null when the band has no rows on the canvas.
+ */
+ private cardPopOut(
+ L: CardLayout,
+ src: CanvasImageSource,
+ matte: CanvasImageSource,
+ clip: VideoClip
+ ): Surface | null {
+ const W = this.canvas.width;
+ const rows = Math.min(this.canvas.height, Math.ceil(L.band.bottom));
+ const top = Math.max(0, Math.floor(L.band.top));
+ if (rows <= top || L.band.edge <= L.band.top) return null;
+ const { surface: pop } = this.scratch("cardPop", W, rows);
+ const p = pop.getContext("2d") as Ctx | null;
+ if (!p) return null;
+ p.setTransform(1, 0, 0, 1, 0, 0);
+ p.globalAlpha = 1;
+ p.globalCompositeOperation = "source-over";
+ p.clearRect(0, 0, W, rows);
+ p.save();
+ p.beginPath();
+ p.rect(L.band.left, top, L.band.right - L.band.left, rows - top);
+ p.clip();
+ this.drawCardPicture(p, src, L.picture, clip);
+ p.globalCompositeOperation = "destination-in";
+ p.imageSmoothingEnabled = true;
+ this.drawCardPicture(p, matte, L.picture, clip);
+ p.restore();
+ if (L.band.bottom > L.band.edge) {
+ let fade = this.cardFade;
+ if (!fade || fade.ctx !== p || fade.edge !== L.band.edge || fade.bottom !== L.band.bottom) {
+ const fill = p.createLinearGradient(0, L.band.edge, 0, L.band.bottom);
+ fill.addColorStop(0, "rgba(0,0,0,0)");
+ fill.addColorStop(1, "rgba(0,0,0,1)");
+ fade = { ctx: p, edge: L.band.edge, bottom: L.band.bottom, fill };
+ this.cardFade = fade;
+ }
+ p.globalCompositeOperation = "destination-out";
+ p.fillStyle = fade.fill;
+ p.fillRect(L.band.left, L.band.edge, L.band.right - L.band.left, L.band.bottom - L.band.edge);
+ p.globalCompositeOperation = "source-over";
+ }
+ return pop;
+ }
+
/**
* Draw one clip's layer over the frame, honouring its region, fit, pan, zoom,
* alpha and any frame motion a transition puts on it.
@@ -929,7 +1163,7 @@ export class FrameCompositor {
// styled box (rounded corners, border) or a mirrored picture also routes
// through the rect path, which knows how to draw those at full frame too.
const rect = rectOf(clip ?? {});
- if (!isFullRect(rect) || clip?.boxStyle || clip?.flipH || clip?.flipV) {
+ if (!isFullRect(rect) || clip?.boxStyle || clip?.flipH || clip?.flipV || (clip && cardActive(clip))) {
this.drawIntoRect(frame, rect, !!clip && clipCovers(clip), alpha, at, zoom, clip);
if (hasFx) ctx.restore();
return;
diff --git a/site/src/cut/lib/exportClient.ts b/site/src/cut/lib/exportClient.ts
index 576056de..0dedafe1 100644
--- a/site/src/cut/lib/exportClient.ts
+++ b/site/src/cut/lib/exportClient.ts
@@ -20,10 +20,13 @@ import { exportsDir, projectDir, writeFileAt, readFileAt, saveExport } from "./b
import { holdRegistered, registerBlobFile, releaseRegistered, resolveRegisteredBlob } from "./backend/browser/registry";
import { captureCloudBackend } from "./backend/cloud";
import { downloadFile, downloadFromUrl } from "./download";
-import { bitrateFor, canRenderInBrowser, renderProjectToMp4, type SourceExportProfile } from "./exportRender";
+import { bitrateFor, canRenderInBrowser, renderProjectToMp4, renderStemsArchive, type RenderedExport, type SourceExportProfile } from "./exportRender";
+import { planStems, type StemDef } from "./stems";
import { putSigned } from "./media";
import { withAssetColors } from "./mediaRead";
import { renderRemovalPieces } from "./removalVideo";
+import { cardActive } from "./cameraCard";
+import { renderCardPieces } from "./cameraCardVideo";
import { drawBlock } from "./blockSource";
import { createRasterCanvas, rasterCanvasToPng } from "./raster";
import { clipLen, clipSpeed, getClipSpans, openedTimeline, overlayLayers, projectDuration, spanSequence, useEditor } from "./store";
@@ -815,6 +818,26 @@ export function specColor(
return { profile: asset.colorProfile ?? c.detected, matrix: c.matrix, fullRange: c.fullRange };
}
+/** A camera card draws its own corners and shadow, so its clip's box style
+ * stands aside in every painted picture (mask corners, border, shadow). */
+const cardBoxless = (c: VideoClip): VideoClip => (cardActive(c) && c.boxStyle ? { ...c, boxStyle: undefined } : c);
+
+/** The spec fields of a camera-card clip whose layer ships as `<tag>`
+ * pieces: the pair stands in for the source, already framed to the box,
+ * graded, looked and mirrored, so the engine frames it 1:1 and applies none
+ * of those again. */
+const cardEntry = (tag: string) => ({
+ removal: { rgb: `${tag}_rgb.mp4`, alpha: `${tag}_a.mp4` },
+ grade: undefined,
+ look: undefined,
+ fit: "fill" as const,
+ zoom: 1,
+ panX: 0,
+ panY: 0,
+ flipH: undefined,
+ flipV: undefined,
+});
+
/** Build the export spec + overlay PNGs from the cut. Media already lives in
* the project folder — the spec references it by file name; only overlay PNGs
* travel with the request. Shared by full exports and the low-res hover proxy. */
@@ -1007,7 +1030,20 @@ export async function buildExportPayload(
// matte yet — the export shows the plain picture, like the preview); a
// broken render throws and fails the export instead of silently dropping
// the cutout.
- if (removalActive(c.removal) && !c.hidden) {
+ if (cardActive(c) && !c.hidden) {
+ const rp = regionPx(c.frame, settings.width, settings.height);
+ const pieces = await renderCardPieces(spans[i].asset, spans[i].clip, doc.assets, {
+ fps: settings.fps,
+ frameW: settings.width,
+ frameH: settings.height,
+ box: rp ? { w: rp.rw, h: rp.rh } : { w: settings.width, h: settings.height },
+ });
+ if (pieces) {
+ pngs.push({ name: `card_c${i}_rgb.mp4`, blob: pieces.rgb });
+ pngs.push({ name: `card_c${i}_a.mp4`, blob: pieces.alpha });
+ Object.assign(clipEntries[i], cardEntry(`card_c${i}`));
+ }
+ } else if (removalActive(c.removal) && !c.hidden) {
const pieces = await renderRemovalPieces(spans[i].asset, spans[i].clip, doc.assets, {
fps: settings.fps,
maxShort: Math.min(
@@ -1203,7 +1239,19 @@ export async function buildExportPayload(
// applies no grade or look to it — both bake into the pieces here, the
// way image overlays already carry their pixels ready-made.
const oAsset = assetById.get(c.assetId);
- if (removalActive(c.removal) && oAsset) {
+ if (cardActive(c) && oAsset) {
+ const pieces = await renderCardPieces(oAsset, c, doc.assets, {
+ fps: settings.fps,
+ frameW: settings.width,
+ frameH: settings.height,
+ box: { w: box.w, h: box.h },
+ });
+ if (pieces) {
+ pngs.push({ name: `card_ov${i}_rgb.mp4`, blob: pieces.rgb });
+ pngs.push({ name: `card_ov${i}_a.mp4`, blob: pieces.alpha });
+ Object.assign(entry, cardEntry(`card_ov${i}`));
+ }
+ } else if (removalActive(c.removal) && oAsset) {
const pieces = await renderRemovalPieces(oAsset, c, doc.assets, {
fps: settings.fps,
maxShort: Math.min(
diff --git a/site/src/cut/lib/exportRender.ts b/site/src/cut/lib/exportRender.ts
index d80f91d4..9f97f2e8 100644
--- a/site/src/cut/lib/exportRender.ts
+++ b/site/src/cut/lib/exportRender.ts
@@ -1353,6 +1353,7 @@ export class FramePainter {
// provider is a synchronous read of that map. Backdrop stills decode once
// here, so mid-render reads are cache hits.
this.comp.removalMatteProvider = (clip) => this.matteFrames.get(clip.id) ?? null;
+ this.comp.cardMatteProvider = (clip) => this.matteFrames.get(cardMatteKey(clip.id)) ?? null;
this.comp.backdropImageProvider = (assetId) => backdropStill(this.doc.assets, assetId);
const backdropIds = new Set(
this.doc.clips
@@ -1460,13 +1461,23 @@ export class FramePainter {
}
}
- /** Pull a removal clip's baked matte frame for timeline time `t` and stage
- * it, luma turned to alpha, where the compositor's provider reads. A clip
+ /** Pull a clip's baked matte frames for timeline time `t` — its cutout's,
+ * and its camera card's when the card shows the head — and stage them,
+ * luma turned to alpha, where the compositor's providers read. A clip
* whose bake is still owed stages nothing. */
private async fetchRemovalMatte(span: ClipSpan, t: number): Promise<void> {
const clip = span.clip;
- const m = removalActive(clip.removal) ? clip.removal?.matte : undefined;
- this.matteFrames.delete(clip.id);
+ await this.stageMatte(clip.id, span, t, removalActive(clip.removal) ? clip.removal?.matte : undefined);
+ await this.stageMatte(cardMatteKey(clip.id), span, t, clip.card?.popOut ? clip.card.matte : undefined);
+ }
+
+ private async stageMatte(
+ key: string,
+ span: ClipSpan,
+ t: number,
+ m: { assetId: string; in: number } | undefined
+ ): Promise<void> {
+ this.matteFrames.delete(key);
if (!m) return;
const asset = this.doc.assets.find((a) => a.id === m.assetId);
if (!asset) return;
diff --git a/site/src/cut/lib/itemKinds.ts b/site/src/cut/lib/itemKinds.ts
index dc668044..91c17033 100644
--- a/site/src/cut/lib/itemKinds.ts
+++ b/site/src/cut/lib/itemKinds.ts
@@ -123,13 +123,16 @@ export const ITEM_KINDS: { [K in ItemKind]: ItemKindDef<K> } = {
},
}
: {}),
+ ...(c.card?.matte?.assetId ? { card: { ...c.card, matte: { ...c.card.matte, assetId: to(c.card.matte.assetId) } } } : {}),
}),
// A background removal's matte was baked from this project's footage,
// and a clip arriving with a removal and no matte would start a fresh
- // bake on its own.
+ // bake on its own. A camera card keeps its layout and bakes its own
+ // person matte where it lands.
crossProject: (c) => {
const out = { ...c };
delete out.removal;
+ if (out.card) out.card = { ...out.card, matte: undefined };
return out;
},
},
diff --git a/site/src/cut/lib/maskVideo.ts b/site/src/cut/lib/maskVideo.ts
index 9f89a481..12a65c28 100644
--- a/site/src/cut/lib/maskVideo.ts
+++ b/site/src/cut/lib/maskVideo.ts
@@ -1,6 +1,6 @@
"use client";
-import { retimeOf } from "@donkeycut/effects-kit";
+import { removalActive, retimeOf } from "@donkeycut/effects-kit";
/**
* The person-matte video for the ffmpeg export path. The page — the only
@@ -30,8 +30,10 @@ import { ClipReader, renderFile, settleRenderColor, WORKING_VIDEO_CODECS } from
import type { ExportDoc } from "./renderSnapshot";
import { createRasterCanvas } from "./raster";
import { getClipSpans } from "./store";
-import { clipCovers, frameOf, isEffectOverlay, isTextOverlay, rectOf, subjectMasked, type ClipSpan, type MediaAsset } from "./types";
+import { clipCovers, frameOf, isEffectOverlay, isTextOverlay, rectOf, subjectMasked, type ClipSpan, type MediaAsset, type VideoClip } from "./types";
import { liveReader } from "./liveReader";
+import { cardMatteKey } from "./cameraCard";
+import { matteStage } from "./cameraCardVideo";
/** Encoded mask rate — the server's fps filter duplicates frames up to the
* output rate, and a person moves little in 1/15s. */
@@ -111,6 +113,20 @@ export async function renderSubjectMask(
if (!r) readers.set(asset.id, (r = liveReader(asset)));
return r;
};
+ // Baked mattes ride along: a cutout keys its clip and a card pops its head
+ // out above the card, so the segmenter sees the picture the preview draws.
+ // Each matte is staged once per mask frame, before its clip draws.
+ const mattes = new Map<string, ReturnType<typeof matteStage>>();
+ const stageFor = (key: string, m: Parameters<typeof matteStage>[0]) => {
+ if (!mattes.has(key)) mattes.set(key, matteStage(m, doc.assets));
+ return mattes.get(key) ?? null;
+ };
+ const stageMattes = async (clip: VideoClip, srcT: number) => {
+ if (removalActive(clip.removal)) await stageFor(clip.id, clip.removal?.matte)?.stage(srcT);
+ if (clip.card?.popOut) await stageFor(cardMatteKey(clip.id), clip.card.matte)?.stage(srcT);
+ };
+ comp.removalMatteProvider = (clip) => mattes.get(clip.id)?.get() ?? null;
+ comp.cardMatteProvider = (clip) => mattes.get(cardMatteKey(clip.id))?.get() ?? null;
const output = new Output({ format: new Mp4OutputFormat(), target: new BufferTarget() });
const video = new CanvasSource(mask, { codec, quality: new Quality({ bitrate: 1_000_000 }) });
@@ -130,9 +146,11 @@ export async function renderSubjectMask(
const plan = trackZeroPlan(master, spans, t);
if (plan.backdrop && !subjectClip(plan.backdrop.span.clip)) {
const f = await readerFor(plan.backdrop.span.asset).frameAt(plan.backdrop.at);
+ await stageMattes(plan.backdrop.span.clip, plan.backdrop.at);
comp.drawLayer(f, plan.backdrop.span.clip, false, 1, t);
}
const f = await readerFor(master.asset).frameAt(sourceTimeAt(master, t));
+ await stageMattes(master.clip, sourceTimeAt(master, t));
comp.drawLayer(f ?? MISSING_FRAME, master.clip, false, 1, t);
}
for (const layer of overlayPlan(overlayTracks, (track) => byTrack.get(track) ?? [], t)) {
@@ -140,6 +158,7 @@ export async function renderSubjectMask(
const span = spanOfClip.get(layer.clip.id);
if (!span) continue;
const f = await readerFor(layer.asset).frameAt(sourceTimeAt(span, t));
+ await stageMattes(layer.clip, sourceTimeAt(span, t));
comp.drawIntoRect(f, rectOf(layer.clip), clipCovers(layer.clip), layer.alpha, t, layer.zoom, layer.clip);
}
// Luma mask: black frame, the subject's alpha silhouette drawn white.
@@ -159,5 +178,6 @@ export async function renderSubjectMask(
return null;
} finally {
for (const r of readers.values()) r.dispose();
+ for (const m of mattes.values()) m?.dispose();
}
}
diff --git a/site/src/cut/lib/removal/bakeJobs.ts b/site/src/cut/lib/removal/bakeJobs.ts
index 098e9be7..93521765 100644
--- a/site/src/cut/lib/removal/bakeJobs.ts
+++ b/site/src/cut/lib/removal/bakeJobs.ts
@@ -14,12 +14,20 @@
* asset (origin "matte") and repoints `clip.removal.matte` through a
* transient write: background landings stay off the undo stack, and the clip
* change still marks the doc for saving.
+ *
+ * A camera card that shows the speaker's head owes the same free person
+ * matte, held on `clip.card.matte`. Its job runs under `cardMatteKey(clipId)`
+ * beside the clip's cutout, starts as soon as the card asks for the head
+ * (the layout is the ask, and the free rung bills nothing), and never climbs
+ * to the paid rung. Either slot adopts the other's matte when the selection
+ * matches, so a clip with an auto cutout and a card bakes once.
*/
import { create } from "zustand";
import { LOCAL_MATTE_BAKE } from "@donkeycut/effects-kit";
import { importFileToProject } from "../media";
import { useEditor } from "../store";
+import { cardMatteKey } from "../cameraCard";
import { removalFingerprint, type ClipRemoval, type VideoClip } from "../types";
import { useBrushUi } from "./brushUi";
import { hostedBakeMatte, type HostedBakeTicket } from "./hostedBake";
@@ -27,6 +35,33 @@ import { localBakeMatte, MatteBakeCancelled } from "./localBake";
type MatteQuality = "local" | "hq";
+/** Where on a clip a matte lands: its cutout, or its camera card. */
+type MatteSlot = "removal" | "card";
+type StoredMatte = NonNullable<ClipRemoval["matte"]>;
+
+/** The job key a slot's bake runs under: the cutout keeps the bare clip id. */
+const jobKey = (clipId: string, slot: MatteSlot): string =>
+ slot === "removal" ? clipId : cardMatteKey(clipId);
+
+/** The clip and slot a job key names. */
+export function jobTarget(key: string): { clipId: string; slot: MatteSlot } {
+ const card = cardMatteKey("");
+ return key.startsWith(card)
+ ? { clipId: key.slice(card.length), slot: "card" }
+ : { clipId: key, slot: "removal" };
+}
+
+/** The matte a slot holds now. */
+const storedOf = (clip: VideoClip, slot: MatteSlot): StoredMatte | undefined =>
+ slot === "removal" ? clip.removal?.matte : clip.card?.matte;
+
+/** Point a slot at a matte, off the undo stack. */
+function writeMatte(clip: VideoClip, slot: MatteSlot, matte: StoredMatte): void {
+ const editor = useEditor.getState();
+ if (slot === "removal") editor.updateClipTransient(clip.id, { removal: { ...clip.removal!, matte } });
+ else editor.updateClipTransient(clip.id, { card: { ...clip.card!, matte } });
+}
+
export interface MatteBakeJob {
quality: MatteQuality;
status: "running" | "error";
@@ -131,8 +166,12 @@ const clipOf = (clipId: string): VideoClip | undefined =>
useEditor.getState().clips.find((c) => c.id === clipId);
/** The fingerprint an AI-mode clip's matte must carry, or null when the clip
- * owes no matte at all. */
-function owedFingerprint(clip: VideoClip | undefined): string | null {
+ * owes no matte at all. A card showing the head owes the auto person matte:
+ * the same selection an auto cutout of the same footage carries. */
+function owedFingerprint(clip: VideoClip | undefined, slot: MatteSlot = "removal"): string | null {
+ if (slot === "card") {
+ return clip?.card?.popOut ? removalFingerprint(clip.assetId, { mode: "auto" }) : null;
+ }
const r = clip?.removal;
if (!clip || !r || r.off) return null;
// Every bake waits for its explicit start — the panel's Apply or the chat
@@ -169,34 +208,40 @@ function matteCovers(matte: NonNullable<ClipRemoval["matte"]>, clip: VideoClip):
return clip.in >= matte.in - 0.05 && clip.out <= matte.in + dur + 0.2;
}
-/** A matte some other clip already carries for this exact selection, whose
- * baked range spans this clip's trims. hq wins when both tiers exist. */
+/** A matte some other slot already carries for this exact selection — another
+ * clip's, or this clip's other slot — whose baked range spans this clip's
+ * trims. hq wins when both tiers exist. */
function twinMatte(
clipId: string,
+ slot: MatteSlot,
fp: string,
clip: VideoClip
): NonNullable<ClipRemoval["matte"]> | null {
const s = useEditor.getState();
let found: NonNullable<ClipRemoval["matte"]> | null = null;
for (const c of s.clips) {
- if (c.id === clipId) continue;
- const m = c.removal?.matte;
- if (!m || m.fingerprint !== fp || !s.assets.some((a) => a.id === m.assetId)) continue;
- if (!matteFresh(m) || !matteCovers(m, clip)) continue;
- if (m.quality === "hq") return m;
- found ??= m;
+ for (const other of ["removal", "card"] as const) {
+ if (c.id === clipId && other === slot) continue;
+ const m = storedOf(c, other);
+ if (!m || m.fingerprint !== fp || !s.assets.some((a) => a.id === m.assetId)) continue;
+ if (!matteFresh(m) || !matteCovers(m, clip)) continue;
+ if (m.quality === "hq") return m;
+ found ??= m;
+ }
}
return found;
}
-export function cancelMatteBake(clipId: string): void {
- running.get(clipId)?.ctl.abort();
- running.delete(clipId);
- failed.delete(clipId);
+/** Stop a bake by its job key: a clip id for its cutout, `cardMatteKey` for
+ * its card. */
+export function cancelMatteBake(key: string): void {
+ running.get(key)?.ctl.abort();
+ running.delete(key);
+ failed.delete(key);
// The ticket stays: the paid track keeps running upstream, and an undo
// that brings the same selection back resumes it free. The sweep retires
// tickets that can never land.
- setJob(clipId, null);
+ setJob(key, null);
}
/** Clear a failed bake and run the ladder again — the panel's Retry, and the
@@ -207,6 +252,15 @@ export function retryMatteBake(clipId: string): void {
ensureMatteBake(clipId, { whileBrushing: true });
}
+/** The Camera card section's Retry: clear the card's failed bake and run it
+ * again. */
+export function retryCardMatte(clipId: string): void {
+ const key = jobKey(clipId, "card");
+ failed.delete(key);
+ setJob(key, null);
+ ensureCardMatte(clipId);
+}
+
/** The brush panel's Apply: start the tracked bake for what's painted so
* far, with the session still open for touch-ups. */
export function confirmMatteBake(clipId: string): void {
@@ -219,81 +273,96 @@ export function confirmMatteBake(clipId: string): void {
* call on every panel render.
*/
export function ensureMatteBake(clipId: string, opts: { whileBrushing?: boolean } = {}): void {
+ ensureBake(clipId, "removal", opts);
+}
+
+/** Make sure a camera card showing the head has its person matte, starting
+ * the free on-device bake when it has none. Safe to call on every render. */
+export function ensureCardMatte(clipId: string): void {
+ ensureBake(clipId, "card", {});
+}
+
+function ensureBake(clipId: string, slot: MatteSlot, opts: { whileBrushing?: boolean }): void {
if (!matteBakesAvailable) return;
+ const key = jobKey(clipId, slot);
const clip = clipOf(clipId);
- const fp = owedFingerprint(clip);
+ const fp = owedFingerprint(clip, slot);
if (!clip || !fp) {
- cancelMatteBake(clipId);
+ cancelMatteBake(key);
return;
}
// An open brush session re-fingerprints on every stroke; the bake (and the
// hosted tier's bill) waits until it closes or the panel's Apply confirms.
- if (!opts.whileBrushing && useBrushUi.getState().clipId === clipId) return;
- const r = clip.removal!;
+ if (slot === "removal" && !opts.whileBrushing && useBrushUi.getState().clipId === clipId) return;
// Settled: the stored matte matches the clip and its asset is still around
// (an undo can revive a matte pointer whose asset was collected).
- const matte = r.matte;
+ const matte = storedOf(clip, slot);
const stored =
matte?.fingerprint === fp && matteFresh(matte) && matteCovers(matte, clip) ? matte : null;
// A split or a paste leaves two clips owing the same selection. A twin
// matte another clip already carries is adopted whole — same asset, one
// bake, one bill — as long as its baked range spans this clip's trims. A
// stored local matte adopts a twin only for the hq upgrade it still owes.
- const twin = twinMatte(clipId, fp, clip);
+ const twin = twinMatte(clipId, slot, fp, clip);
const adopt = !stored ? twin : stored.quality === "local" && twin?.quality === "hq" ? twin : null;
if (adopt) {
- useEditor.getState().updateClipTransient(clipId, { removal: { ...r, matte: adopt } });
- ensureMatteBake(clipId, opts);
+ writeMatte(clip, slot, adopt);
+ ensureBake(clipId, slot, opts);
return;
}
// What the ladder owes next. Custom tracks an arbitrary object, so only the
// hosted tracker can bake it; auto gets the free local matte immediately,
// and the hosted pass upgrades it once `refine` asks — the paid rung runs
- // on that explicit click alone.
+ // on that explicit click alone. A card takes the free rung only.
+ const r = clip.removal;
const want: MatteQuality | null =
- r.mode === "custom"
+ slot === "card"
? stored
? null
- : "hq"
- : stored
- ? stored.quality === "local" && r.refine
- ? "hq"
- : null
- : "local";
+ : "local"
+ : r?.mode === "custom"
+ ? stored
+ ? null
+ : "hq"
+ : stored
+ ? stored.quality === "local" && r?.refine
+ ? "hq"
+ : null
+ : "local";
if (!want) {
- cancelMatteBake(clipId);
+ cancelMatteBake(key);
return;
}
- const stuck = failed.get(clipId);
+ const stuck = failed.get(key);
if (stuck && stuck.fp === fp && stuck.quality === want) return;
- const inFlight = running.get(clipId);
+ const inFlight = running.get(key);
if (inFlight && inFlight.fp === fp && inFlight.quality === want) return;
- // Another clip is already baking this exact selection — one submit, one
- // bill. When it lands, the sweep runs this clip's ensure again and the
- // twin adoption above picks the result up (or bakes for real when the
- // twin's range turns out not to cover this clip's trims).
- for (const [otherId, run] of running) {
- if (otherId !== clipId && run.fp === fp && (run.quality === want || run.quality === "hq"))
+ // Another bake is already making this exact selection — one submit, one
+ // bill. When it lands, the sweep runs this ensure again and the twin
+ // adoption above picks the result up (or bakes for real when the twin's
+ // range turns out not to cover this clip's trims).
+ for (const [otherKey, run] of running) {
+ if (otherKey !== key && run.fp === fp && (run.quality === want || run.quality === "hq"))
return;
}
inFlight?.ctl.abort();
const ctl = new AbortController();
- running.set(clipId, { fp, quality: want, ctl });
- failed.delete(clipId);
- setJob(clipId, { quality: want, status: "running", progress: 0, startedAt: Date.now() });
- void runBake(clipId, fp, want, ctl).catch((e: unknown) => {
- if (running.get(clipId)?.ctl !== ctl) return;
- running.delete(clipId);
+ running.set(key, { fp, quality: want, ctl });
+ failed.delete(key);
+ setJob(key, { quality: want, status: "running", progress: 0, startedAt: Date.now() });
+ void runBake(clipId, slot, fp, want, ctl).catch((e: unknown) => {
+ if (running.get(key)?.ctl !== ctl) return;
+ running.delete(key);
if (e instanceof MatteBakeCancelled) {
- setJob(clipId, null);
+ setJob(key, null);
return;
}
// The ticket stands through failure: the run already forgot any part
// whose track settled dead, so Retry resumes the paid parts and re-buys
// only what died.
- failed.set(clipId, { fp, quality: want });
- setJob(clipId, {
+ failed.set(key, { fp, quality: want });
+ setJob(key, {
quality: want,
status: "error",
progress: 0,
@@ -309,6 +378,7 @@ let bakeTurn: Promise<void> = Promise.resolve();
async function runBake(
clipId: string,
+ slot: MatteSlot,
fp: string,
quality: MatteQuality,
ctl: AbortController
@@ -318,7 +388,7 @@ async function runBake(
bakeTurn = new Promise((r) => (release = r));
try {
await prev;
- await runBakeNow(clipId, fp, quality, ctl);
+ await runBakeNow(clipId, slot, fp, quality, ctl);
} finally {
release();
}
@@ -326,11 +396,13 @@ async function runBake(
async function runBakeNow(
clipId: string,
+ slot: MatteSlot,
fp: string,
quality: MatteQuality,
ctl: AbortController
): Promise<void> {
if (ctl.signal.aborted) throw new MatteBakeCancelled();
+ const key = jobKey(clipId, slot);
const state = useEditor.getState();
const clip = clipOf(clipId);
const asset = clip && state.assets.find((a) => a.id === clip.assetId);
@@ -341,13 +413,13 @@ async function runBakeNow(
const onProgress = (f: number) => {
if (f - shown < 0.03 && f < 1) return;
shown = f;
- if (running.get(clipId)?.ctl !== ctl) return;
+ if (running.get(key)?.ctl !== ctl) return;
const elapsed = (Date.now() - startedAt) / 1000;
const secondsLeft =
f >= 0.05 && elapsed >= 3
? Math.max(0, Math.round((elapsed * (1 - f)) / f))
: undefined;
- setJob(clipId, {
+ setJob(key, {
quality,
status: "running",
progress: f,
@@ -374,43 +446,38 @@ async function runBakeNow(
matteAsset.name = `${asset.name} matte`;
matteAsset.origin = "matte";
- // The doc may have moved while the bake ran — land only onto the removal
- // the matte was baked for.
+ // The doc may have moved while the bake ran — land only onto the slot the
+ // matte was baked for.
const editor = useEditor.getState();
const latest = clipOf(clipId);
- const owed = owedFingerprint(latest);
+ const owed = owedFingerprint(latest, slot);
editor.addAsset(matteAsset);
- dropTicket(clipId);
+ if (slot === "removal") dropTicket(clipId);
if (ctl.signal.aborted || !latest || owed !== fp) {
editor.removeAsset(matteAsset.id);
- running.delete(clipId);
- setJob(clipId, null);
+ running.delete(key);
+ setJob(key, null);
// Trims or seeds changed under the bake: the clip still owes a matte.
- if (owed) ensureMatteBake(clipId);
+ if (owed) ensureBake(clipId, slot, {});
return;
}
- const removal: ClipRemoval = {
- ...latest.removal!,
- matte: {
- assetId: matteAsset.id,
- fingerprint: fp,
- quality,
- in: baked.in,
- ...(quality === "local" ? { bake: LOCAL_MATTE_BAKE } : {}),
- },
- };
- editor.updateClipTransient(clipId, { removal });
+ writeMatte(latest, slot, {
+ assetId: matteAsset.id,
+ fingerprint: fp,
+ quality,
+ in: baked.in,
+ ...(quality === "local" ? { bake: LOCAL_MATTE_BAKE } : {}),
+ });
// The matte this replaced may still be another clip's (a split, a paste);
// the sweep collects it once nothing points at it.
- running.delete(clipId);
- setJob(clipId, null);
+ running.delete(key);
+ setJob(key, null);
// Re-settle: an auto matte that landed with `refine` set owes the quality
// pass next; anything else just clears its job.
- ensureMatteBake(clipId);
+ ensureBake(clipId, slot, {});
}
-/** Drop a cleared removal's matte asset (and its file) from the project. */
export function dropMatteAsset(assetId: string | undefined): void {
if (!assetId) return;
const editor = useEditor.getState();
@@ -424,8 +491,14 @@ export function dropMatteAsset(assetId: string | undefined): void {
function sweepMattes(): void {
const s = useEditor.getState();
if (!s.loaded || s.readOnly || !s.projectId) return;
- for (const id of [...running.keys()]) if (!owedFingerprint(clipOf(id))) cancelMatteBake(id);
- for (const c of s.clips) if (owedFingerprint(c)) ensureMatteBake(c.id);
+ for (const key of [...running.keys()]) {
+ const { clipId, slot } = jobTarget(key);
+ if (!owedFingerprint(clipOf(clipId), slot)) cancelMatteBake(key);
+ }
+ for (const c of s.clips) {
+ if (owedFingerprint(c)) ensureMatteBake(c.id);
+ if (owedFingerprint(c, "card")) ensureCardMatte(c.id);
+ }
// Tickets retire by age alone. A cutout that is cleared, a clip that is
// deleted, a selection that moved on — each keeps its ticket, because an
// undo brings the old fingerprint back and resumes the paid track for
@@ -438,7 +511,10 @@ function sweepMattes(): void {
// A clip on a closed timeline still owns its matte.
const pointed = new Set<string>();
for (const body of [s, ...Object.values(s.timelines)])
- for (const c of body.clips) if (c.removal?.matte) pointed.add(c.removal.matte.assetId);
+ for (const c of body.clips) {
+ if (c.removal?.matte) pointed.add(c.removal.matte.assetId);
+ if (c.card?.matte) pointed.add(c.card.matte.assetId);
+ }
for (const a of s.assets) if (a.origin === "matte" && !pointed.has(a.id)) s.removeAsset(a.id);
}
diff --git a/site/src/cut/lib/store.ts b/site/src/cut/lib/store.ts
index 2a6c2181..38cee33b 100644
--- a/site/src/cut/lib/store.ts
+++ b/site/src/cut/lib/store.ts
@@ -145,6 +145,7 @@ const clipTreatment = (c: VideoClip) => ({
...(c.boxStyle ? { boxStyle: c.boxStyle } : {}),
...(c.kf?.length ? { kf: c.kf } : {}),
...(c.hidden ? { hidden: true } : {}),
+ ...(c.card ? { card: { ...c.card, matte: undefined } } : {}),
});
/** One clip as a template layer, timed from the template's start. A track-0
@@ -183,6 +184,7 @@ const templateTreatment = (l: TemplateLayer) => ({
...(l.boxStyle ? { boxStyle: l.boxStyle } : {}),
...(l.kf?.length ? { kf: l.kf } : {}),
...(l.hidden ? { hidden: true } : {}),
+ ...(l.card ? { card: { ...l.card, matte: undefined } } : {}),
});
const MIN_LEN = 0.1;
@@ -2877,6 +2879,9 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {
// subject matte is computed from whatever plays, so it stays.
...(clip.mask && clip.mask.kind !== "subject" ? { mask: undefined } : {}),
removal: undefined,
+ // The card's layout stays; its matte belongs to the old
+ // picture, and the new one bakes its own.
+ ...(clip.card ? { card: { ...clip.card, matte: undefined } } : {}),
}
: c
),
diff --git a/site/src/cut/lib/types.ts b/site/src/cut/lib/types.ts
index 1970cc57..7f15ce8d 100644
--- a/site/src/cut/lib/types.ts
+++ b/site/src/cut/lib/types.ts
@@ -26,6 +26,7 @@ import {
type WordEffectId,
} from "@donkeycut/effects-kit";
import type { CodeFormat, SourceProfile } from "@donkeycut/effects-kit";
+import type { CameraCard } from "./cameraCard";
import { getBackend, type CutBackend } from "./backend";
import type { VideoProject } from "./genvideo/types";
@@ -697,6 +698,19 @@ export interface VideoClip {
* a backdrop filled in behind it (see the kit's removal.ts). Absent = the
* whole picture shows. */
removal?: ClipRemoval;
+ /** Split edit: seconds the clip's sound starts before its picture (a
+ * J-cut), drawn from the source ahead of `in`. Absent = the sound starts
+ * with the picture. */
+ audioLead?: number;
+ /** Seconds the clip's sound carries on past its picture's end (an L-cut),
+ * drawn from the source past `out`. Absent = it stops with the picture. */
+ audioTail?: number;
+ /** The camera-card layout (see cameraCard.ts): the picture shows through a
+ * rounded card along the bottom or one side of the clip's box, and with
+ * `popOut` the speaker's matted head shows above it. The card places the footage
+ * itself, so the clip's fit, zoom, pan and box style stand aside while it
+ * is on. Absent = the clip frames as usual. */
+ card?: CameraCard;
}
// Color grading (the dual-renderer math) lives in the effects kit; the model
@@ -1172,6 +1186,9 @@ export interface TemplateLayer {
boxStyle?: BoxStyle;
kf?: OverlayKey[];
hidden?: boolean;
+ /** The camera-card layout, without its matte: the clip it stands up
+ * bakes its own. */
+ card?: CameraCard;
track: number;
/** Came from video track 0 — re-materializes as a timeline clip, not an
* overlay, so a template stands up its own footage. */
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index a483caf0..96e6836d 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -27,17 +27,19 @@ import {
MAX_WORDS_PER_CUE,
MIN_WORDS_PER_CUE,
} from "@/cut/lib/cueChunk";
-import { CAPTION_STYLES } from "@/cut/lib/subtitles";
+import { CAPTION_EMPHASIS_DEFAULT, CAPTION_STYLES } from "@/cut/lib/subtitles";
import { OUTPUT_SPACES, SOUND_PRESETS, wordEffectCatalog, WORD_EFFECT_IDS } from "@donkeycut/effects-kit";
import { textMoveCatalog } from "@/cut/lib/textMotion";
import { OVERLAY_ANIMATION_TOOLS } from "@/cut/components/AnimationTiles.tools";
import { AUDIO_TOOLS } from "@/cut/components/AudioPanel.tools";
+import { CAMERA_CARD_TOOLS } from "@/cut/components/CameraCardSection.tools";
import { EDITOR_TOOLS } from "@/cut/components/Editor.tools";
import { EFFECT_TREATMENTS, EFFECTS_TOOLS } from "@/cut/components/EffectsPanel.tools";
import { ELEMENTS_TOOLS } from "@/cut/components/ElementsPanel.tools";
import { VIDEO_GEN_TOOLS } from "@/cut/components/GeneratePanel.tools";
import { IMAGE_GEN_TOOLS } from "@/cut/components/ImageGenPanel.tools";
import { INSPECTOR_TOOLS } from "@/cut/components/Inspector.tools";
+import { GROUP_PANEL_TOOLS } from "@/cut/components/GroupPanel.tools";
import { LIBRARY_TOOLS } from "@/cut/components/LibraryView.tools";
import { PREVIEW_TOOLS } from "@/cut/components/Preview.tools";
import { REMOVAL_TOOLS } from "@/cut/components/RemovalPanel.tools";
@@ -264,6 +266,7 @@ Times are in seconds on the shared timeline. The playhead is currentTime; a skim
- set_framing: per-clip Fit (letterbox) vs Fill (crop to cover the box), plus zoom 1..4 to push further in. Whenever the picture overflows, panX/panY position the crop window; in the preview a grey outline shows the part being cropped away, and the user drags that outline to slide the crop (dragging the clip itself moves its box). flipH mirrors the picture left for right and flipV top for bottom — a front-camera take that reads backward (mirrored text, a watch on the wrong wrist) wants flipH. The controls live in the Inspector under "Framing" when a video clip is selected, the mirror beside Rotation. Landscape footage usually wants fill + a pan that keeps the subject.
- update_overlay_video also carries an overlay clip's resting transform: rotation, opacity and the mirror, alongside its region (layout/region) and zoom. Use these for a tilted or faded layer; set_clip_keyframes is for a transform that moves over time.
- set_clip_style rounds a clip's corners, strokes a border ring, and casts a drop shadow from the clip's shape — mask included, so a circle-masked webcam gets a circular shadow. The floating-cam look is one recipe: put the clip on an upper track, region it small (update_overlay_video region or the preview handles), mask it with kind circle, and give it a soft shadow.
+- set_camera_card is a talking-head video's split section in one call, in any frame shape: a graphic fills the frame on a lower track, and the speaker's clip shows through a rounded card with their head popping out above the card's edge — across the bottom of a portrait frame, up one side of a square or landscape one, and \`side\` picks. The card and the head come from the same clip and the same frame; the head's person matte bakes for free and editor_state reports it under \`card.matte\`.
- The user can copy/paste any selected segment (video, overlay video, audio, title) with ⌘C/⌘V; pastes aim for the playhead and slide right to free space.
- Zoom: set_view pxPerSec (12..800) or fit. The timeline panel height: set_view timelineH (170..600).`,
@@ -393,6 +396,7 @@ Composite builds: these primitives compose. Stacked overlay video copies, masks
4. Work out the geometry as numbers before calling any tool. Any screen region converts to mask parameters by plain arithmetic: w and h are fractions of frame width and height — rect takes both, so it expresses any region on any aspect; a square's side rides frame width alone, and a shape meant to hold exact pixel proportions folds in project.frame's pixels — and x/y are the region's center measured from the item's own center. Write every piece's region and timing as numbers, check that together they lay out what the reference shows (coverage, spacing, alignment), then place them.
5. Build one representative piece and capture_frame it against the reference. Then build the rest and capture_frame again, mid-reveal and at the end: every cell present, slices aligned into the whole picture, the target region covered edge to edge. Fix what's off at the right level: a wrong number re-sets in place (set_mask and the keyframe tools overwrite), a mistimed copy retimes, and a wrong cell plan means deleting the copies you added (delete_item) and laying them again from the corrected list — the pieces you placed are yours to tear down. The build is done when the captured frame matches the reference.
All of it is regular timeline editing: free, live, and undoable.
+A speaker popping out of a card over a graphic is one tool: set_camera_card on the speaker's clip, with the graphic on a lower track. It draws the card and the head from one frame of one clip, so the two stay in register.
Good TikTok titles: short punchy lines, high contrast (white/yellow + shadow or plate), size 72–110, keep inside the middle 80% of the frame (x 0.1..0.9, y 0.1..0.9), avoid the caption band (y≈0.8) when subtitles are on.
Tasteful graphics: shapes read best as accents (a highlight box behind a stat, an arrow pointing at the subject, a color bar under a title) — semi-transparent fills (fill_opacity 0.2–0.5) sit better over footage than solid blocks. One or two stickers a scene; size 0.15–0.3 of the frame width.
In the UI: the timeline toolbar adds Text; the Elements side-panel tab browses stickers and shapes and creates sticker images, and the Effects tab holds the effects — a click in any panel picks a tile, and dragging one onto the timeline is what places it; dragging in the preview places an element, a corner handle scales it, a side handle stretches one axis with the far side planted, the top handle rotates; the Inspector edits every field.`,