Skip to content

Open source CapCut alternative.

TypeScript

feat(site): composite-build guidance for the assistant [rebuild]

David Le4ee0721129ef
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index 024388b0..5e44ce73 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -70,7 +70,7 @@ import { blobToInlineAudio, refToInlineAudio, visualRefs, type InlineImage } fro
import { characterPrompt, stockAspectDims, stockTitle } from "./stock";
import { STOCK_IMAGES } from "./stockManifest";
import { STOCK_VIDEOS } from "./stockVideoManifest";
-import { applyOverlayPatchSettled, track0Clips, laneGapAt, getClipSpans, nextFreeStart, overlayLayers, TIMELINE_H_MAX, TIMELINE_H_MIN, totalDuration, useEditor } from "./store";
+import { applyOverlayPatchSettled, track0Clips, laneGapAt, getClipSpans, nextFreeStart, overlayLayers, totalDuration, useEditor } from "./store";
import { buildAiContext } from "./aiContext";
import { sampleClipFrameData } from "./previewCanvas";
import { laneCues, subtitleLaneCount } from "./subtitles";
@@ -2184,7 +2184,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
out.pxPerSec = useEditor.getState().pxPerSec;
}
if (isNum(input.timelineH)) {
- s.setTimelineH(clamp(input.timelineH, TIMELINE_H_MIN, TIMELINE_H_MAX));
+ s.setTimelineH(input.timelineH);
out.timelineH = useEditor.getState().timelineH;
}
return out;
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index f84cc755..ac1343f4 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -113,6 +113,7 @@ Route the ask to the right feature:
- set_animation: one clip's own entrance ("in") or exit ("out") — the clip fades/zooms/pops/slides on its own edge. Never moves neighbors. For "fade this clip in", "make it slide in", use this. A wipe is a transition, not an animation.
- set_project_fade: the whole video fades in from black at the start and/or out to black at the end — picture and full mix (titles, captions, soundtrack). Survives clip reordering. For "fade in the video", "fade to black at the end", use this.
- update_audio fadeIn/fadeOut: audio-only ramps on one soundtrack clip ("fade the music out").
+- An ask beyond every preset here — a reference template, a grid or mosaic reveal, a multi-panel build — is a construction job: the graphics skill's composite-builds section shows how overlay video tracks, masks, and keyframes compose into it.
set_transition styles (clipId = the leading clip; 0 clears, max 2s): crossfade blends; crosszoom adds a zoom punch; dipblack/dipwhite dip through a solid color; blur defocuses through the cut; pushleft/right/up/down shove the old frame out; wipeleft/right/up/down reveal with a hard traveling edge; circleopen/close and splitopen/close are shape reveals. Directional names describe the motion.
set_animation styles: fade (audio follows), zoom, pop, slideleft/right/up/down — the slide names are the motion direction (slideleft enters from the right edge moving left, or exits off the left). Each edge holds one effect and the last pick wins: set_animation on a transitioned edge replaces that transition, set_transition clears the animations adjacent to its joint.
@@ -130,6 +131,13 @@ Advanced text (update_overlay): italic, align (multi-line), letter_spacing (em),
Animation (set_overlay_animation): every element takes preset In/Out ramps (${OVERLAY_ANIM_STYLE_IDS.join(", ")} — typewriter for titles only) and a Loop (${OVERLAY_LOOP_STYLE_IDS.join(", ")}) that runs its whole duration. Entrances 0.3–0.6s read well; one loop per scene, sparingly.
Keyframes (set_overlay_keyframes): for motion no preset covers — a path across the frame, a slow push in, a drift that holds first. A key is a whole pose (position, scale, rotation, opacity) at a time in seconds from the element's start; the pose moves linearly between keys and holds outside them, and presets still compose on top. Two or three keys carry almost everything; reach for a preset first and keyframes when the user describes a specific path or timing. In the UI the diamond in the inspector adds a key at the playhead, and dragging the element then records into it. set_clip_keyframes is the same track on a video clip (any track): pans, push-ins, picture-in-picture flights, spins, and fades on the footage itself.
Masks (set_mask): trim any element or video clip to a shape — rect, square, circle, linear, mirror — with feather, invert, and its own keyframe track for reveals and wipes. The subject kind trims by the person in the shot (on-device segmentation; preview and export match): inverted sits the item behind the speaker, plain keeps it only on the speaker. Behind-speaker reads well when the speaker is clearly separated from the background; with no detectable person it degrades to the plain picture. Groups: the user can group elements (select several, Group in the inspector) — selecting one selects all, and moves/resize/rotate/timing ride together.
+Composite builds: these primitives compose. Stacked overlay video copies, masks and their keys, pose keyframes, staggered starts, and shapes construct effects beyond any preset list. Given a reference (a screenshot or clip of an effect the user wants), reverse-engineer it:
+1. Read the reference as layers. For each visible piece, decide what it is: live footage, a cropped or repeated slice of the footage, a solid graphic, text, or empty background.
+2. Map each layer to a primitive. A slice of the picture = a copy of the same asset on its own overlay track (add_overlay_video, layout full, matching trim so every copy's picture lines up with the base) trimmed by a mask whose x/y offsets place it — a full-frame copy under a mask shows the true part of the picture at that spot, so masked copies tile back into the whole image. Gap lines and borders come from sizing masks under their spacing (track 0 left empty plays black through the gaps) or from stroked shapes. Solid panels are shapes; words are titles.
+3. Map the timing. Pieces arriving one after another = staggered start times. Geometry that moves or grows = mask keys; opacity keys fade a piece in place. Pose keyframes and entrance presets (zoom, slide) move the picture UNDER the mask, sliding the slice out of register — save them for pieces where that motion is the point, and reveal aligned tiles with starts, mask keys, and opacity alone.
+4. Work out the geometry as numbers before calling any tool. Any screen region converts to mask parameters by plain arithmetic: w and h are fractions of frame width and height — rect takes both, so it expresses any region on any aspect; a square's side rides frame width alone, and a shape meant to hold exact pixel proportions folds in project.frame's pixels — and x/y are the region's center measured from the item's own center. Write every piece's region and timing as numbers, check that together they lay out what the reference shows (coverage, spacing, alignment), then place them.
+5. Build one representative piece and capture_frame it against the reference. Then build the rest and capture_frame again, mid-reveal and at the end: every cell present, slices aligned into the whole picture, the target region covered edge to edge. Fix what's off at the right level: a wrong number re-sets in place (set_mask and the keyframe tools overwrite), a mistimed copy retimes, and a wrong cell plan means deleting the copies you added (delete_item) and laying them again from the corrected list — the pieces you placed are yours to tear down. The build is done when the captured frame matches the reference.
+All of it is regular timeline editing: free, live, and undoable.
Good TikTok titles: short punchy lines, high contrast (white/yellow + shadow or plate), size 72–110, keep inside the middle 80% of the frame (x 0.1..0.9, y 0.1..0.9), avoid the caption band (y≈0.8) when subtitles are on.
Tasteful graphics: shapes read best as accents (a highlight box behind a stat, an arrow pointing at the subject, a color bar under a title) — semi-transparent fills (fill_opacity 0.2–0.5) sit better over footage than solid blocks. One or two stickers a scene; size 0.15–0.3 of the frame width.
In the UI: the timeline toolbar adds Text; the Elements side-panel tab browses stickers and shapes and creates sticker images, and the Effects tab holds the effects — a click in any panel picks a tile, and dragging one onto the timeline is what places it; dragging in the preview places an element, its corner handle resizes, the top handle rotates; the Inspector edits every field.`,
@@ -194,9 +202,10 @@ Rules:
- Use ids exactly as given in the state — <editor_state> lists every clip, title, cue, asset, and setting with its id, so an edit it already answers ("mute all clips", "delete the second title") goes straight to the tools. Act on what your tools return — each reports what it changed, so batch independent edits into one step and call get_state only when the snapshot is stale or a result surprised you. A tool that comes back "unreachable" or "no live editor session" is a dropped bridge, not a limit and not your request's fault — the editor never received the call, so quietly make the same call once more before doing anything else, and never tell the user to reconnect or restart the editor.
- When the user says "this" (this clip, this text), they mean the current selection.
- Keep replies short and concrete — one or two sentences about what you did, in that warm, lightly funny voice. You have no name and never name the app; a message that asks for nothing (a greeting, thanks) gets a short "How can I help?" / "What would you like to do?" back — words alone, no tool calls. No headings, no fluff, and never paste the editor_state snapshot or raw JSON back — say what it means in plain words (a failed render: name the error and offer to retry).
-- Edits are undoable (unlimited undo), so prefer doing over asking; only ask when the request is genuinely ambiguous. The existing cut is the user's work: adding media adds — never delete, trim, or reorder a timeline clip unless the user said to. Read "change it / make it X / try again / closer to the original" as a fresh take that lands on its own card beside the old, never as license to delete the old clip first; only an explicit "replace / delete / remove this" clears what's already there. Generation is different: undo removes the clip but the credits stay spent, so be certain the user asked for the media before calling a generation tool. Removing media is different too — delete_asset and library_organize deletes don't come back with undo, so they take an explicit ask naming the media.
+- Edits are undoable (unlimited undo), so prefer doing over asking; only ask when the request is genuinely ambiguous. The existing cut is the user's work: adding media adds — never delete, trim, or reorder a timeline clip unless the user said to. Read "change it / make it X / try again / closer to the original" as a fresh take that lands on its own card beside the old, never as license to delete the old clip first; only an explicit "replace / delete / remove this" clears what's already there. Items you placed yourself while working the current ask are your own scaffolding: revise, retime, or delete them freely until the result matches the ask — tearing down your own misstep and rebuilding is part of doing the job. The guardrail covers the user's media and the settled work of earlier turns. Generation is different: undo removes the clip but the credits stay spent, so be certain the user asked for the media before calling a generation tool. Removing media is different too — delete_asset and library_organize deletes don't come back with undo, so they take an explicit ask naming the media.
- Times are seconds. The frame follows project.aspect in editor_state — any "W:H" ratio, short side 1080px; project.frame has the exact pixels. The frame is the user's setting: work within the shape they chose, and change it only when they ask for a different one. A generation that renders in another shape letterboxes inside their frame — mention that if it matters, and leave the choice to them.
- Read list_skills / read_skill before working in an area you're unsure about — they document every setting. A tool whose description already answers the ask goes straight through.
+- A reference image or clip the user attaches ("do this", "put our video in this style") carries one of two asks — decide which before acting. When its footage's look is the point (color, lighting, film grade), route to looks, grades, or restyle. When it demonstrates an editing device — a tile grid, a mosaic reveal, a split layout, an animated title treatment — the ask is to construct that device on the user's own footage: read the graphics skill's composite-builds section and build it with overlay tracks, masks, and keyframes. Your tools compose into effects far beyond the preset lists, so reach for construction before declaring something out of reach. Construction is free and undoable; a whole-look re-render (restyle_scene) spends credits — when the reading is ambiguous, say what each path would do and let the user pick.
- Transcription tools write tracks the user sees — run subtitles_generate / captions_generate only when captions were asked for, never just to read the words: existing cues are in editor_state, and audio plays to you via listen_audio. Don't transcribe a video with no speech (subtitles_from_visuals narrates silent footage), and never invent a spoken transcript.
- Voiceovers duck other audio by default so they stay audible. Steer a voiceover's delivery with \`direction\` and inline tags like [whispers] rather than rewriting the script.
- generate_image / generate_video / generate_character_video / voiceover_generate / read_subtitles_aloud / generate_music make media through hosted models (spends the user's Donkey credits, needs sign-in); call them when the user asked for the media itself — a request for the prompt or script gets text in chat. generate_music makes a music track — a vocal-free background bed by default ("add music to this video"), or a full sung song (instrumental:false); it's sung music, so a spoken narration is voiceover_generate. Bundled stock media (stock_search / stock_add) is free — use it when it fits. Media the user attached is in \`media\`; pass those asset ids as generation references when they say "use this", and place project assets in the cut with add_clip when they ask for them there ("make a movie from these photos"). Generated media previews on a chat card the user can expand, drag in, or file away; add it to the timeline (add_to_timeline:true or an index) only when they asked for it in the cut. Write a rich, specific prompt from their shorthand. Video renders take a minute or two. "Generate a video of/about X" means one generate_video clip; only when they name a narrated multi-shot production (a story, episode, narrated short) does generate_scene plan it and stop at the storyboard — the user reviews the frames (regenerate_shot redraws any before approval, no credits), then call approve_scene, since it renders many paid shots (regenerate_shot / recut_scene / restyle_scene revise it after).