Open source CapCut alternative.
TypeScript
feat(site): assistant tools and doctrine for masks and clip keyframes [rebuild]
David Le09f1cb00286d
diff --git a/site/src/cut/components/Inspector.tools.ts b/site/src/cut/components/Inspector.tools.tsindex e0ef9198..8c0abfa6 100644--- a/site/src/cut/components/Inspector.tools.ts+++ b/site/src/cut/components/Inspector.tools.ts@@ -30,7 +30,6 @@ export const INSPECTOR_TOOLS = [line_height: num("Line height multiplier (titles; default 1.25)"),shadow: bool("Drop shadow (titles)"),plate: bool("Backdrop plate (titles)"),- behind_subject: bool("Sit the text behind the person in the shot (titles)"),w: num("Width, fraction of frame width (shapes/stickers)"),h: num("Height, fraction of frame height (shapes)"),fill: str("Fill color (shapes)"),@@ -43,6 +42,70 @@ export const INSPECTOR_TOOLS = [hidden: bool("Hide the element without deleting it"),}, ["id"]),},+ {+ name: "set_mask",+ description:+ "Mask an overlay element or a video clip — trim its picture to a shape or to the person in the shot. Kinds: rect (rounded box), circle (ellipse), linear (half-plane: at rotation 0 the top side stays), mirror (band across the item), subject (the person matte: plain keeps the picture only on the person; invert keeps it off the person — behind the speaker). Position is the mask center's offset from the item's own center (a clip's region center), in frame fractions; w/h size it in frame fractions; feather softens the edge; invert keeps what the shape leaves out. Subject masks use only feather and invert. Pass kind \"none\" to remove the mask. `keys` animates the geometry over time — each key is the full geometry at a moment, moving linearly between keys; pass an empty list to clear the keys and keep the mask still.",+ inputSchema: obj({+ id: str("Overlay element id or video clip id"),+ kind: {+ type: "string",+ enum: ["rect", "circle", "linear", "mirror", "subject", "none"],+ description: 'Mask shape, the person matte ("subject"), or "none" to remove the mask',+ },+ x: num("Center offset x from the element center, fraction of frame width (default 0)"),+ y: num("Center offset y, fraction of frame height (default 0)"),+ w: num("Width, fraction of frame width (default 0.5; linear ignores)"),+ h: num("Height, fraction of frame height (mirror: band height; linear ignores)"),+ rotation: num("Degrees clockwise, -180..180"),+ feather: num("Edge softness, px at the 1080 design short side (0..200)"),+ invert: bool("Keep the pixels outside the shape"),+ radius: num("Rect corner radius, px at 1080 short side"),+ keys: {+ type: "array",+ description:+ "Mask keyframes, seconds from the element's start; omitted fields on a key take the mask's value at that moment",+ items: obj(+ {+ t: num("Seconds from the element's start"),+ x: num("Center offset x, fraction of frame width"),+ y: num("Center offset y, fraction of frame height"),+ w: num("Width, fraction of frame width"),+ h: num("Height, fraction of frame height"),+ rotation: num("Degrees clockwise, -180..180"),+ feather: num("Edge softness, px at 1080 short side"),+ },+ ["t"]+ ),+ },+ }, ["id"]),+ },+ {+ name: "set_clip_keyframes",+ description:+ "Give a video clip (any track) a keyframed pose track: each key is a whole pose at a time measured in seconds from the clip's own start — center position in frame fractions, scale multiplier on its fitted size, rotation, opacity — moving linearly between keys and holding outside them. Omitted fields on a key take the clip's pose at that moment. Pass an empty list to clear the track and return the clip to its region. Use for pans, push-ins, picture-in-picture flights, spins, and fades on video itself.",+ inputSchema: obj(+ {+ clipId: str("Video clip id"),+ keys: {+ type: "array",+ description: "Keys in any order; two at the same time collapse to one",+ items: obj(+ {+ t: num("Seconds from the clip's start"),+ x: num("Center x, fraction of frame width"),+ y: num("Center y, fraction of frame height"),+ scale: num("Size multiplier, 1 = the clip's fitted size (0.1..4)"),+ rotation: num("Degrees clockwise, -180..180"),+ opacity: num("0..1"),+ },+ ["t"]+ ),+ },+ },+ ["clipId", "keys"]+ ),+ },{name: "update_audio",description:diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.tsindex d18bc3b4..488c2d2b 100644--- a/site/src/cut/components/Timeline.tools.ts+++ b/site/src/cut/components/Timeline.tools.ts@@ -176,7 +176,6 @@ export const TIMELINE_TOOLS = [stroke_width: num("Text outline width in em (0..0.15; 0 removes it)"),shadow: bool("Drop shadow (default true)"),plate: bool("Translucent plate behind text (default false)"),- behind_subject: bool("Sit the text behind the person in the shot (needs a clearly separated speaker)"),rotation: num("Degrees clockwise, -180..180"),opacity: num("Whole-element opacity 0..1"),}, ["text"]),diff --git a/site/src/cut/lib/aiContext.ts b/site/src/cut/lib/aiContext.tsindex f62b5585..dd52ef7f 100644--- a/site/src/cut/lib/aiContext.ts+++ b/site/src/cut/lib/aiContext.ts@@ -195,6 +195,10 @@ export function buildAiContext(opts?: { fullCues?: boolean; chatId?: string | nu? { panX: r(sp.clip.panX ?? 0), panY: r(sp.clip.panY ?? 0) }: {}),...(sp.clip.grade ? { colorGrade: sp.clip.grade } : {}),+ ...(sp.clip.mask ? { mask: sp.clip.mask } : {}),+ ...(sp.clip.kf?.length+ ? { keyframes: sp.clip.kf.map((k) => ({ ...k, t: r(k.t), x: r(k.x), y: r(k.y) })) }+ : {}),...clipEffects(sp.clip),})),// Video layers composited over track 0 in track order (the topmost@@ -293,6 +297,10 @@ function describeOverlayClip(c: VideoClip, assets: Map<string, { name: string }>fit: c.fit ?? "fit",...(speed !== 1 ? { speed: r(speed) } : {}),...(c.grade ? { colorGrade: c.grade } : {}),+ ...(c.mask ? { mask: c.mask } : {}),+ ...(c.kf?.length+ ? { keyframes: c.kf.map((k) => ({ ...k, t: r(k.t), x: r(k.x), y: r(k.y) })) }+ : {}),};}@@ -309,6 +317,7 @@ function describeOverlay(o: Overlay) {...(o.kf?.length? { keyframes: o.kf.map((k) => ({ ...k, t: r(k.t), x: r(k.x), y: r(k.y) })) }: {}),+ ...(o.mask ? { mask: o.mask } : {}),...(o.lane ? { lane: o.lane } : {}),...(o.hidden ? { hidden: true } : {}),};@@ -346,7 +355,6 @@ function describeOverlay(o: Overlay) {color: o.color,shadow: o.shadow,plate: o.plate,- ...(o.behindSubject ? { behindSubject: true } : {}),...(o.groupId ? { groupId: o.groupId } : {}),...(o.plateRadius !== undefined && { plateRadius: r(o.plateRadius) }),};diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.tsindex 4e02859b..19b38831 100644--- a/site/src/cut/lib/aiTools.ts+++ b/site/src/cut/lib/aiTools.ts@@ -5,6 +5,7 @@ import {EFFECT_IDS,normalizeGrade,type EffectId,+ type Mask,OVERLAY_ANIM_DEFAULT_SECONDS,OVERLAY_ANIM_MAX_SECONDS,OVERLAY_ANIM_MIN_SECONDS,@@ -85,6 +86,7 @@ import {ANIM_STYLE_IDS,frameOf,IMAGE_CLIP_SECONDS,+ isEffectOverlay,isTextOverlay,LAYOUTS,MAX_SUBTITLE_LANES,@@ -351,6 +353,103 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {return { id: o.id, keys: (kf ?? []).map((k) => ({ ...k, t: round2(k.t) })) };},+ set_clip_keyframes: (s, input) => {+ const clip = requireItem(s.clips, input.clipId, "video clip");+ const raw = input.keys;+ if (!Array.isArray(raw)) throw new ToolError("keys must be a list.");+ const speed = clip.speed && clip.speed > 0 ? clip.speed : 1;+ const dur = Math.max(0.1, (clip.out - clip.in) / speed);+ let kf = clip.kf;+ if (raw.length === 0) {+ s.clearClipKeys(clip.id);+ kf = undefined;+ } else {+ // Each key builds on the pose already at its time, so a key that only+ // names `x` keeps whatever the clip was doing otherwise.+ s.pushHistory();+ for (const k of raw as Record<string, unknown>[]) {+ if (!isNum(k.t)) throw new ToolError("Every key needs a time `t`.");+ s.setClipKey(+ clip.id,+ clamp(k.t, 0, dur),+ {+ ...(isNum(k.x) ? { x: clamp(k.x, -0.5, 1.5) } : {}),+ ...(isNum(k.y) ? { y: clamp(k.y, -0.5, 1.5) } : {}),+ ...(isNum(k.scale) ? { scale: clamp(k.scale, 0.1, 4) } : {}),+ ...(isNum(k.rotation) ? { rotation: clamp(Math.round(k.rotation), -180, 180) } : {}),+ ...(isNum(k.opacity) ? { opacity: clamp(k.opacity, 0, 1) } : {}),+ },+ { transient: true }+ );+ }+ kf = useEditor.getState().clips.find((c) => c.id === clip.id)?.kf;+ }+ return { id: clip.id, keys: (kf ?? []).map((k) => ({ ...k, t: round2(k.t) })) };+ },++ set_mask: (s, input) => {+ // One id space: an overlay element or a video clip on any track.+ const o = s.overlays.find((x) => x.id === input.id);+ const clip = o ? undefined : s.clips.find((c) => c.id === input.id);+ if (!o && !clip)+ throw new ToolError("No overlay element or video clip with that id.");+ if (o && isEffectOverlay(o)) throw new ToolError("Effect elements have no pixels to mask.");+ const current = o ? o.mask : clip!.mask;+ const setMask = (mask: Mask | undefined) =>+ o ? s.updateOverlay(o.id, { mask }) : s.updateClip(clip!.id, { mask });+ if (input.kind === "none") {+ setMask(undefined);+ return { id: input.id, mask: null };+ }+ const kinds = ["rect", "circle", "linear", "mirror", "subject"];+ const kind =+ typeof input.kind === "string"+ ? input.kind+ : (current?.kind ?? "rect");+ if (!kinds.includes(kind)) throw new ToolError(`Unknown mask kind. Use one of: ${kinds.join(", ")}, none.`);+ const geom = (k: Record<string, unknown>) => ({+ ...(isNum(k.x) ? { x: clamp(k.x, -1, 1) } : {}),+ ...(isNum(k.y) ? { y: clamp(k.y, -1, 1) } : {}),+ ...(isNum(k.w) ? { w: clamp(k.w, 0.01, 2) } : {}),+ ...(isNum(k.h) ? { h: clamp(k.h, 0.01, 2) } : {}),+ ...(isNum(k.rotation) ? { rotation: clamp(Math.round(k.rotation), -180, 180) } : {}),+ ...(isNum(k.feather) ? { feather: clamp(k.feather, 0, 200) } : {}),+ });+ const mask: Mask = {+ ...(current ?? {}),+ ...geom(input),+ kind: kind as Mask["kind"],+ ...(typeof input.invert === "boolean" ? { invert: input.invert || undefined } : {}),+ ...(isNum(input.radius) ? { radius: clamp(input.radius, 0, 400) || undefined } : {}),+ };+ setMask(mask);+ // Keys layer on after the mask lands, so each one captures against the+ // geometry the call just set.+ const raw = input.keys;+ if (Array.isArray(raw)) {+ const st = useEditor.getState();+ const dur = o+ ? Math.max(0.1, o.end - o.start)+ : Math.max(0.1, (clip!.out - clip!.in) / (clip!.speed && clip!.speed > 0 ? clip!.speed : 1));+ if (raw.length === 0) {+ if (o) st.clearOverlayMaskKeys(o.id);+ else st.clearClipMaskKeys(clip!.id);+ } else {+ for (const k of raw as Record<string, unknown>[]) {+ if (!isNum(k.t)) throw new ToolError("Every mask key needs a time `t`.");+ const t = clamp(k.t, 0, dur);+ if (o) st.setOverlayMaskKey(o.id, t, geom(k), { transient: true });+ else st.setClipMaskKey(clip!.id, t, geom(k), { transient: true });+ }+ }+ }+ const after = useEditor.getState();+ const next = o+ ? after.overlays.find((x) => x.id === o.id)?.mask+ : after.clips.find((c) => c.id === clip!.id)?.mask;+ return { id: input.id, mask: next ?? null };+ },+set_transition: (s, input) => {const clip = requireItem(s.clips, input.clipId, "video clip");// A transition joins a clip to the next one on its own track.@@ -2633,7 +2732,6 @@ function overlayPatch(input: Record<string, unknown>, kind: "text" | "shape" | "patch.lineHeight = Math.abs(input.line_height - 1.25) < 0.01 ? undefined : clamp(input.line_height, 0.7, 2.5);if (typeof input.shadow === "boolean") patch.shadow = input.shadow;if (typeof input.plate === "boolean") patch.plate = input.plate;- if (typeof input.behind_subject === "boolean") patch.behindSubject = input.behind_subject || undefined;if (isNum(input.plateRadius)) patch.plateRadius = clamp(input.plateRadius, 0, 1);if (typeof input.stroke_color === "string" || isNum(input.stroke_width)) {const width = isNum(input.stroke_width) ? clamp(input.stroke_width, 0, 0.3) : 0.04;diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.tsindex b1364012..25a2e1e1 100644--- a/site/src/cut/server/ai/catalog.ts+++ b/site/src/cut/server/ai/catalog.ts@@ -128,8 +128,8 @@ Custom stickers (create_sticker): one call generates the image, removes the backEffects (add_effect): time-ranged treatments over the finished picture — the footage and everything laid over it. Two families, one list: the treatments (${EFFECT_TREATMENTS.join(", ")}) and the graded looks (${LOOK_EFFECTS.join(", ")}). Placed like any element and tuned with \`amount\` (update_overlay); an effect added over a clip opens covering that clip, and trims like anything else. One effect at a time reads best, amounts under ~0.5.Advanced text (update_overlay): italic, align (multi-line), letter_spacing (em), line_height, stroke_color/stroke_width (em outline behind the fill), and richer shadows. Bundled Google families join the system set — inter, montserrat, poppins, oswald, space-grotesk, playfair, caveat, bebas, anton, archivo-black, bangers, lobster, pacifico, permanent-marker, dm-serif — and the user can upload .ttf/.otf fonts (they appear as font ids like "asset:<id>").Animation (set_overlay_animation): every element takes preset In/Out ramps (${OVERLAY_ANIM_STYLE_IDS.join(", ")} — typewriter for titles only) and a Loop (${OVERLAY_LOOP_STYLE_IDS.join(", ")}) that runs its whole duration. Entrances 0.3–0.6s read well; one loop per scene, sparingly.-Keyframes (set_overlay_keyframes): for motion no preset covers — a path across the frame, a slow push in, a drift that holds first. A key is a whole pose (position, scale, rotation, opacity) at a time in seconds from the element's start; the pose moves linearly between keys and holds outside them, and presets still compose on top. Two or three keys carry almost everything; reach for a preset first and keyframes when the user describes a specific path or timing. In the UI the diamond in the inspector adds a key at the playhead, and dragging the element then records into it.-Behind the speaker: a title with behind_subject sits behind the person in the shot (on-device person segmentation; preview and export match). It reads well when the speaker is clearly separated from the background; with no detectable person it degrades to a normal front title. Groups: the user can group elements (select several, Group in the inspector) — selecting one selects all, and moves/resize/rotate/timing ride together.+Keyframes (set_overlay_keyframes): for motion no preset covers — a path across the frame, a slow push in, a drift that holds first. A key is a whole pose (position, scale, rotation, opacity) at a time in seconds from the element's start; the pose moves linearly between keys and holds outside them, and presets still compose on top. Two or three keys carry almost everything; reach for a preset first and keyframes when the user describes a specific path or timing. In the UI the diamond in the inspector adds a key at the playhead, and dragging the element then records into it. set_clip_keyframes is the same track on a video clip (any track): pans, push-ins, picture-in-picture flights, spins, and fades on the footage itself.+Masks (set_mask): trim any element or video clip to a shape — rect, circle, linear, mirror — with feather, invert, and its own keyframe track for reveals and wipes. The subject kind trims by the person in the shot (on-device segmentation; preview and export match): inverted sits the item behind the speaker, plain keeps it only on the speaker. Behind-speaker reads well when the speaker is clearly separated from the background; with no detectable person it degrades to the plain picture. Groups: the user can group elements (select several, Group in the inspector) — selecting one selects all, and moves/resize/rotate/timing ride together.Good TikTok titles: short punchy lines, high contrast (white/yellow + shadow or plate), size 72–110, keep inside the middle 80% of the frame (x 0.1..0.9, y 0.1..0.9), avoid the caption band (y≈0.8) when subtitles are on.Tasteful graphics: shapes read best as accents (a highlight box behind a stat, an arrow pointing at the subject, a color bar under a title) — semi-transparent fills (fill_opacity 0.2–0.5) sit better over footage than solid blocks. One or two stickers a scene; size 0.15–0.3 of the frame width.In the UI: the timeline toolbar adds Text; the Elements side-panel tab browses stickers and shapes and creates sticker images, and the Effects tab holds the effects — a click in any panel picks a tile, and dragging one onto the timeline is what places it; dragging in the preview places an element, its corner handle resizes, the top handle rotates; the Inspector edits every field.`,