Skip to content

Open source CapCut alternative.

TypeScript

feat(site): assistant tools and doctrine for masks and clip keyframes [rebuild]

David Le09f1cb00286d
diff --git a/site/src/cut/components/Inspector.tools.ts b/site/src/cut/components/Inspector.tools.ts
index e0ef9198..8c0abfa6 100644
--- a/site/src/cut/components/Inspector.tools.ts
+++ b/site/src/cut/components/Inspector.tools.ts
@@ -30,7 +30,6 @@ export const INSPECTOR_TOOLS = [
line_height: num("Line height multiplier (titles; default 1.25)"),
shadow: bool("Drop shadow (titles)"),
plate: bool("Backdrop plate (titles)"),
- behind_subject: bool("Sit the text behind the person in the shot (titles)"),
w: num("Width, fraction of frame width (shapes/stickers)"),
h: num("Height, fraction of frame height (shapes)"),
fill: str("Fill color (shapes)"),
@@ -43,6 +42,70 @@ export const INSPECTOR_TOOLS = [
hidden: bool("Hide the element without deleting it"),
}, ["id"]),
},
+ {
+ name: "set_mask",
+ description:
+ "Mask an overlay element or a video clip — trim its picture to a shape or to the person in the shot. Kinds: rect (rounded box), circle (ellipse), linear (half-plane: at rotation 0 the top side stays), mirror (band across the item), subject (the person matte: plain keeps the picture only on the person; invert keeps it off the person — behind the speaker). Position is the mask center's offset from the item's own center (a clip's region center), in frame fractions; w/h size it in frame fractions; feather softens the edge; invert keeps what the shape leaves out. Subject masks use only feather and invert. Pass kind \"none\" to remove the mask. `keys` animates the geometry over time — each key is the full geometry at a moment, moving linearly between keys; pass an empty list to clear the keys and keep the mask still.",
+ inputSchema: obj({
+ id: str("Overlay element id or video clip id"),
+ kind: {
+ type: "string",
+ enum: ["rect", "circle", "linear", "mirror", "subject", "none"],
+ description: 'Mask shape, the person matte ("subject"), or "none" to remove the mask',
+ },
+ x: num("Center offset x from the element center, fraction of frame width (default 0)"),
+ y: num("Center offset y, fraction of frame height (default 0)"),
+ w: num("Width, fraction of frame width (default 0.5; linear ignores)"),
+ h: num("Height, fraction of frame height (mirror: band height; linear ignores)"),
+ rotation: num("Degrees clockwise, -180..180"),
+ feather: num("Edge softness, px at the 1080 design short side (0..200)"),
+ invert: bool("Keep the pixels outside the shape"),
+ radius: num("Rect corner radius, px at 1080 short side"),
+ keys: {
+ type: "array",
+ description:
+ "Mask keyframes, seconds from the element's start; omitted fields on a key take the mask's value at that moment",
+ items: obj(
+ {
+ t: num("Seconds from the element's start"),
+ x: num("Center offset x, fraction of frame width"),
+ y: num("Center offset y, fraction of frame height"),
+ w: num("Width, fraction of frame width"),
+ h: num("Height, fraction of frame height"),
+ rotation: num("Degrees clockwise, -180..180"),
+ feather: num("Edge softness, px at 1080 short side"),
+ },
+ ["t"]
+ ),
+ },
+ }, ["id"]),
+ },
+ {
+ name: "set_clip_keyframes",
+ description:
+ "Give a video clip (any track) a keyframed pose track: each key is a whole pose at a time measured in seconds from the clip's own start — center position in frame fractions, scale multiplier on its fitted size, rotation, opacity — moving linearly between keys and holding outside them. Omitted fields on a key take the clip's pose at that moment. Pass an empty list to clear the track and return the clip to its region. Use for pans, push-ins, picture-in-picture flights, spins, and fades on video itself.",
+ inputSchema: obj(
+ {
+ clipId: str("Video clip id"),
+ keys: {
+ type: "array",
+ description: "Keys in any order; two at the same time collapse to one",
+ items: obj(
+ {
+ t: num("Seconds from the clip's start"),
+ x: num("Center x, fraction of frame width"),
+ y: num("Center y, fraction of frame height"),
+ scale: num("Size multiplier, 1 = the clip's fitted size (0.1..4)"),
+ rotation: num("Degrees clockwise, -180..180"),
+ opacity: num("0..1"),
+ },
+ ["t"]
+ ),
+ },
+ },
+ ["clipId", "keys"]
+ ),
+ },
{
name: "update_audio",
description:
diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.ts
index d18bc3b4..488c2d2b 100644
--- a/site/src/cut/components/Timeline.tools.ts
+++ b/site/src/cut/components/Timeline.tools.ts
@@ -176,7 +176,6 @@ export const TIMELINE_TOOLS = [
stroke_width: num("Text outline width in em (0..0.15; 0 removes it)"),
shadow: bool("Drop shadow (default true)"),
plate: bool("Translucent plate behind text (default false)"),
- behind_subject: bool("Sit the text behind the person in the shot (needs a clearly separated speaker)"),
rotation: num("Degrees clockwise, -180..180"),
opacity: num("Whole-element opacity 0..1"),
}, ["text"]),
diff --git a/site/src/cut/lib/aiContext.ts b/site/src/cut/lib/aiContext.ts
index f62b5585..dd52ef7f 100644
--- a/site/src/cut/lib/aiContext.ts
+++ b/site/src/cut/lib/aiContext.ts
@@ -195,6 +195,10 @@ export function buildAiContext(opts?: { fullCues?: boolean; chatId?: string | nu
? { panX: r(sp.clip.panX ?? 0), panY: r(sp.clip.panY ?? 0) }
: {}),
...(sp.clip.grade ? { colorGrade: sp.clip.grade } : {}),
+ ...(sp.clip.mask ? { mask: sp.clip.mask } : {}),
+ ...(sp.clip.kf?.length
+ ? { keyframes: sp.clip.kf.map((k) => ({ ...k, t: r(k.t), x: r(k.x), y: r(k.y) })) }
+ : {}),
...clipEffects(sp.clip),
})),
// Video layers composited over track 0 in track order (the topmost
@@ -293,6 +297,10 @@ function describeOverlayClip(c: VideoClip, assets: Map<string, { name: string }>
fit: c.fit ?? "fit",
...(speed !== 1 ? { speed: r(speed) } : {}),
...(c.grade ? { colorGrade: c.grade } : {}),
+ ...(c.mask ? { mask: c.mask } : {}),
+ ...(c.kf?.length
+ ? { keyframes: c.kf.map((k) => ({ ...k, t: r(k.t), x: r(k.x), y: r(k.y) })) }
+ : {}),
};
}
@@ -309,6 +317,7 @@ function describeOverlay(o: Overlay) {
...(o.kf?.length
? { keyframes: o.kf.map((k) => ({ ...k, t: r(k.t), x: r(k.x), y: r(k.y) })) }
: {}),
+ ...(o.mask ? { mask: o.mask } : {}),
...(o.lane ? { lane: o.lane } : {}),
...(o.hidden ? { hidden: true } : {}),
};
@@ -346,7 +355,6 @@ function describeOverlay(o: Overlay) {
color: o.color,
shadow: o.shadow,
plate: o.plate,
- ...(o.behindSubject ? { behindSubject: true } : {}),
...(o.groupId ? { groupId: o.groupId } : {}),
...(o.plateRadius !== undefined && { plateRadius: r(o.plateRadius) }),
};
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index 4e02859b..19b38831 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -5,6 +5,7 @@ import {
EFFECT_IDS,
normalizeGrade,
type EffectId,
+ type Mask,
OVERLAY_ANIM_DEFAULT_SECONDS,
OVERLAY_ANIM_MAX_SECONDS,
OVERLAY_ANIM_MIN_SECONDS,
@@ -85,6 +86,7 @@ import {
ANIM_STYLE_IDS,
frameOf,
IMAGE_CLIP_SECONDS,
+ isEffectOverlay,
isTextOverlay,
LAYOUTS,
MAX_SUBTITLE_LANES,
@@ -351,6 +353,103 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
return { id: o.id, keys: (kf ?? []).map((k) => ({ ...k, t: round2(k.t) })) };
},
+ set_clip_keyframes: (s, input) => {
+ const clip = requireItem(s.clips, input.clipId, "video clip");
+ const raw = input.keys;
+ if (!Array.isArray(raw)) throw new ToolError("keys must be a list.");
+ const speed = clip.speed && clip.speed > 0 ? clip.speed : 1;
+ const dur = Math.max(0.1, (clip.out - clip.in) / speed);
+ let kf = clip.kf;
+ if (raw.length === 0) {
+ s.clearClipKeys(clip.id);
+ kf = undefined;
+ } else {
+ // Each key builds on the pose already at its time, so a key that only
+ // names `x` keeps whatever the clip was doing otherwise.
+ s.pushHistory();
+ for (const k of raw as Record<string, unknown>[]) {
+ if (!isNum(k.t)) throw new ToolError("Every key needs a time `t`.");
+ s.setClipKey(
+ clip.id,
+ clamp(k.t, 0, dur),
+ {
+ ...(isNum(k.x) ? { x: clamp(k.x, -0.5, 1.5) } : {}),
+ ...(isNum(k.y) ? { y: clamp(k.y, -0.5, 1.5) } : {}),
+ ...(isNum(k.scale) ? { scale: clamp(k.scale, 0.1, 4) } : {}),
+ ...(isNum(k.rotation) ? { rotation: clamp(Math.round(k.rotation), -180, 180) } : {}),
+ ...(isNum(k.opacity) ? { opacity: clamp(k.opacity, 0, 1) } : {}),
+ },
+ { transient: true }
+ );
+ }
+ kf = useEditor.getState().clips.find((c) => c.id === clip.id)?.kf;
+ }
+ return { id: clip.id, keys: (kf ?? []).map((k) => ({ ...k, t: round2(k.t) })) };
+ },
+
+ set_mask: (s, input) => {
+ // One id space: an overlay element or a video clip on any track.
+ const o = s.overlays.find((x) => x.id === input.id);
+ const clip = o ? undefined : s.clips.find((c) => c.id === input.id);
+ if (!o && !clip)
+ throw new ToolError("No overlay element or video clip with that id.");
+ if (o && isEffectOverlay(o)) throw new ToolError("Effect elements have no pixels to mask.");
+ const current = o ? o.mask : clip!.mask;
+ const setMask = (mask: Mask | undefined) =>
+ o ? s.updateOverlay(o.id, { mask }) : s.updateClip(clip!.id, { mask });
+ if (input.kind === "none") {
+ setMask(undefined);
+ return { id: input.id, mask: null };
+ }
+ const kinds = ["rect", "circle", "linear", "mirror", "subject"];
+ const kind =
+ typeof input.kind === "string"
+ ? input.kind
+ : (current?.kind ?? "rect");
+ if (!kinds.includes(kind)) throw new ToolError(`Unknown mask kind. Use one of: ${kinds.join(", ")}, none.`);
+ const geom = (k: Record<string, unknown>) => ({
+ ...(isNum(k.x) ? { x: clamp(k.x, -1, 1) } : {}),
+ ...(isNum(k.y) ? { y: clamp(k.y, -1, 1) } : {}),
+ ...(isNum(k.w) ? { w: clamp(k.w, 0.01, 2) } : {}),
+ ...(isNum(k.h) ? { h: clamp(k.h, 0.01, 2) } : {}),
+ ...(isNum(k.rotation) ? { rotation: clamp(Math.round(k.rotation), -180, 180) } : {}),
+ ...(isNum(k.feather) ? { feather: clamp(k.feather, 0, 200) } : {}),
+ });
+ const mask: Mask = {
+ ...(current ?? {}),
+ ...geom(input),
+ kind: kind as Mask["kind"],
+ ...(typeof input.invert === "boolean" ? { invert: input.invert || undefined } : {}),
+ ...(isNum(input.radius) ? { radius: clamp(input.radius, 0, 400) || undefined } : {}),
+ };
+ setMask(mask);
+ // Keys layer on after the mask lands, so each one captures against the
+ // geometry the call just set.
+ const raw = input.keys;
+ if (Array.isArray(raw)) {
+ const st = useEditor.getState();
+ const dur = o
+ ? Math.max(0.1, o.end - o.start)
+ : Math.max(0.1, (clip!.out - clip!.in) / (clip!.speed && clip!.speed > 0 ? clip!.speed : 1));
+ if (raw.length === 0) {
+ if (o) st.clearOverlayMaskKeys(o.id);
+ else st.clearClipMaskKeys(clip!.id);
+ } else {
+ for (const k of raw as Record<string, unknown>[]) {
+ if (!isNum(k.t)) throw new ToolError("Every mask key needs a time `t`.");
+ const t = clamp(k.t, 0, dur);
+ if (o) st.setOverlayMaskKey(o.id, t, geom(k), { transient: true });
+ else st.setClipMaskKey(clip!.id, t, geom(k), { transient: true });
+ }
+ }
+ }
+ const after = useEditor.getState();
+ const next = o
+ ? after.overlays.find((x) => x.id === o.id)?.mask
+ : after.clips.find((c) => c.id === clip!.id)?.mask;
+ return { id: input.id, mask: next ?? null };
+ },
+
set_transition: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
// A transition joins a clip to the next one on its own track.
@@ -2633,7 +2732,6 @@ function overlayPatch(input: Record<string, unknown>, kind: "text" | "shape" | "
patch.lineHeight = Math.abs(input.line_height - 1.25) < 0.01 ? undefined : clamp(input.line_height, 0.7, 2.5);
if (typeof input.shadow === "boolean") patch.shadow = input.shadow;
if (typeof input.plate === "boolean") patch.plate = input.plate;
- if (typeof input.behind_subject === "boolean") patch.behindSubject = input.behind_subject || undefined;
if (isNum(input.plateRadius)) patch.plateRadius = clamp(input.plateRadius, 0, 1);
if (typeof input.stroke_color === "string" || isNum(input.stroke_width)) {
const width = isNum(input.stroke_width) ? clamp(input.stroke_width, 0, 0.3) : 0.04;
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index b1364012..25a2e1e1 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -128,8 +128,8 @@ Custom stickers (create_sticker): one call generates the image, removes the back
Effects (add_effect): time-ranged treatments over the finished picture — the footage and everything laid over it. Two families, one list: the treatments (${EFFECT_TREATMENTS.join(", ")}) and the graded looks (${LOOK_EFFECTS.join(", ")}). Placed like any element and tuned with \`amount\` (update_overlay); an effect added over a clip opens covering that clip, and trims like anything else. One effect at a time reads best, amounts under ~0.5.
Advanced text (update_overlay): italic, align (multi-line), letter_spacing (em), line_height, stroke_color/stroke_width (em outline behind the fill), and richer shadows. Bundled Google families join the system set — inter, montserrat, poppins, oswald, space-grotesk, playfair, caveat, bebas, anton, archivo-black, bangers, lobster, pacifico, permanent-marker, dm-serif — and the user can upload .ttf/.otf fonts (they appear as font ids like "asset:<id>").
Animation (set_overlay_animation): every element takes preset In/Out ramps (${OVERLAY_ANIM_STYLE_IDS.join(", ")} — typewriter for titles only) and a Loop (${OVERLAY_LOOP_STYLE_IDS.join(", ")}) that runs its whole duration. Entrances 0.3–0.6s read well; one loop per scene, sparingly.
-Keyframes (set_overlay_keyframes): for motion no preset covers — a path across the frame, a slow push in, a drift that holds first. A key is a whole pose (position, scale, rotation, opacity) at a time in seconds from the element's start; the pose moves linearly between keys and holds outside them, and presets still compose on top. Two or three keys carry almost everything; reach for a preset first and keyframes when the user describes a specific path or timing. In the UI the diamond in the inspector adds a key at the playhead, and dragging the element then records into it.
-Behind the speaker: a title with behind_subject sits behind the person in the shot (on-device person segmentation; preview and export match). It reads well when the speaker is clearly separated from the background; with no detectable person it degrades to a normal front title. Groups: the user can group elements (select several, Group in the inspector) — selecting one selects all, and moves/resize/rotate/timing ride together.
+Keyframes (set_overlay_keyframes): for motion no preset covers — a path across the frame, a slow push in, a drift that holds first. A key is a whole pose (position, scale, rotation, opacity) at a time in seconds from the element's start; the pose moves linearly between keys and holds outside them, and presets still compose on top. Two or three keys carry almost everything; reach for a preset first and keyframes when the user describes a specific path or timing. In the UI the diamond in the inspector adds a key at the playhead, and dragging the element then records into it. set_clip_keyframes is the same track on a video clip (any track): pans, push-ins, picture-in-picture flights, spins, and fades on the footage itself.
+Masks (set_mask): trim any element or video clip to a shape — rect, circle, linear, mirror — with feather, invert, and its own keyframe track for reveals and wipes. The subject kind trims by the person in the shot (on-device segmentation; preview and export match): inverted sits the item behind the speaker, plain keeps it only on the speaker. Behind-speaker reads well when the speaker is clearly separated from the background; with no detectable person it degrades to the plain picture. Groups: the user can group elements (select several, Group in the inspector) — selecting one selects all, and moves/resize/rotate/timing ride together.
Good TikTok titles: short punchy lines, high contrast (white/yellow + shadow or plate), size 72–110, keep inside the middle 80% of the frame (x 0.1..0.9, y 0.1..0.9), avoid the caption band (y≈0.8) when subtitles are on.
Tasteful graphics: shapes read best as accents (a highlight box behind a stat, an arrow pointing at the subject, a color bar under a title) — semi-transparent fills (fill_opacity 0.2–0.5) sit better over footage than solid blocks. One or two stickers a scene; size 0.15–0.3 of the frame width.
In the UI: the timeline toolbar adds Text; the Elements side-panel tab browses stickers and shapes and creates sticker images, and the Effects tab holds the effects — a click in any panel picks a tile, and dragging one onto the timeline is what places it; dragging in the preview places an element, its corner handle resizes, the top handle rotates; the Inspector edits every field.`,