Open source CapCut alternative.
TypeScript
fix(site): land blockout cuts on the frames detect_cuts read [rebuild]
David Le846791efcf5f
diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.tsindex 2793de72..dbb75b07 100644--- a/site/src/cut/components/Timeline.tools.ts+++ b/site/src/cut/components/Timeline.tools.ts@@ -118,7 +118,7 @@ export const TIMELINE_TOOLS = [{name: "add_clip",description:- `Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at \`start\`, inserted at \`index\`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at \`start\`, default the playhead). Asset ids come from \`media\` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut ("add my beach photo", "stitch these into a movie"); otherwise media stays on its card or panel for them to drag. Pass \`blocks\` instead of an asset to lay out a cut whose footage does not exist yet: one block per shot on track 0, each the length that shot runs, labelled with what belongs there. That is how a video you watched becomes a timeline — the cuts land at their real times, titles and sound go on top, and the person fills each block later. Blocks own no file and store nothing; footage dropped on one takes its place and its length, and replace_item does the same from here. A block runs one frame (1/60 s) or longer, so detect_cuts' \`shots\` go down as they are, flash cuts included; one call takes up to ${BLOCKOUT_MAX}, and a longer list goes in more calls with no start, each appending after the last block. Pass \`spans\` to cut one source into a run of clips — the moments of a talk worth keeping, with everything between them left out, or a video cut at its own shot boundaries when its picture is what plays. add_clip lays down the whole file; an item another timeline already has ("Main's music", "the same intro as Main") is copied with copy_from_timeline, which keeps its trim and mix.`,+ `Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at \`start\`, inserted at \`index\`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at \`start\`, default the playhead). Asset ids come from \`media\` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut ("add my beach photo", "stitch these into a movie"); otherwise media stays on its card or panel for them to drag. Pass \`blocks\` instead of an asset to lay out a cut whose footage does not exist yet: one block per shot on track 0, each the length that shot runs, labelled with what belongs there. That is how a video you watched becomes a timeline — the cuts land at their real times, titles and sound go on top, and the person fills each block later. Blocks own no file and store nothing; footage dropped on one takes its place and its length, and replace_item does the same from here. A block runs one frame (1/60 s) or longer, so detect_cuts' \`shots\` go down as they are, flash cuts included — pass each as its \`from\` and \`to\`, which land every cut on its frame, where lengths summed from times written to the millisecond drift a frame late; one call takes up to ${BLOCKOUT_MAX}, and a longer list goes in more calls with no start, each appending after the last block. Pass \`spans\` to cut one source into a run of clips — the moments of a talk worth keeping, with everything between them left out, or a video cut at its own shot boundaries when its picture is what plays. add_clip lays down the whole file; an item another timeline already has ("Main's music", "the same intro as Main") is copied with copy_from_timeline, which keeps its trim and mix.`,inputSchema: obj({asset_id: str("Project asset id from `media` in editor_state"),blocks: {@@ -126,11 +126,13 @@ export const TIMELINE_TOOLS = [description: "Shots with no footage yet, in order — one block each on track 0",items: obj({- seconds: num("How long this shot runs on the timeline"),+ from: num("The shot's first second, as detect_cuts' shots read it"),+ to: num("The second the shot ends, as detect_cuts' shots read it; with from it sets the block's length exactly"),+ seconds: num("How long this shot runs on the timeline, when there is no from/to"),label: str("What the person puts here. It names the slot on its timeline chip; the frame itself is the flat colour"),color: str("The shot's backdrop as #rrggbb, so the slot carries the look it sits in"),},- ["seconds"]+ []),},spans: {diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.tsindex 5fa23445..0fe278f5 100644--- a/site/src/cut/lib/aiTools.ts+++ b/site/src/cut/lib/aiTools.ts@@ -7690,7 +7690,14 @@ function placeBlocks(s: Editor, raw: unknown[], input: Record<string, unknown>)const shots = raw.filter((x): x is Record<string, unknown> => typeof x === "object" && x !== null).map((x) => ({- seconds: isNum(x.seconds) ? Math.min(x.seconds, BLOCK_MAX_SECONDS) : 0,+ // A shot as detect_cuts reads it, from and to each on the frame it+ // names, runs exactly between them; a length taken from two times+ // written to the millisecond can be a millisecond off, and a run of+ // them drifts a cut onto the frame after.+ seconds: Math.min(+ isNum(x.from) && isNum(x.to) ? x.to - x.from : isNum(x.seconds) ? x.seconds : 0,+ BLOCK_MAX_SECONDS+ ),label: typeof x.label === "string" ? x.label.trim() : "",// The shot's own backdrop, so a cut waiting on its footage still shows// the look it is standing in for.@@ -7698,7 +7705,7 @@ function placeBlocks(s: Editor, raw: unknown[], input: Record<string, unknown>)})).filter((x) => x.seconds > 0);if (shots.length === 0)- throw new ToolError("Pass blocks: [{seconds, label}] — one entry per shot, in order.");+ throw new ToolError("Pass blocks: [{from, to, label}] (a shot as detect_cuts reads it) or [{seconds, label}] — one entry per shot, in order.");if (shots.length > BLOCKOUT_MAX)throw new ToolError(`That is ${shots.length} shots — ${BLOCKOUT_MAX} is the most one call lays down. Pass the first ${BLOCKOUT_MAX} now, then the other ${shots.length - BLOCKOUT_MAX} in a second call with no start: they append right after the last block.`diff --git a/site/src/cut/lib/blockoutReference.test.ts b/site/src/cut/lib/blockoutReference.test.tsindex 1d0e7bb2..9f11b53c 100644--- a/site/src/cut/lib/blockoutReference.test.ts+++ b/site/src/cut/lib/blockoutReference.test.ts@@ -210,3 +210,22 @@ test("a block under one frame is refused with what to do, and a long list says hexpect(many).toContain("other 3");expect(useEditor.getState().clips).toHaveLength(0);});++test("detect_cuts' shots go down as they read, each cut on its own frame", async () => {+ // Shots read to the millisecond: lengths taken from them drift up to a+ // millisecond each, past the snap, and the cut would show the shot before+ // for a frame. Each block's from and to land on the frames they name.+ const shots = [+ { from: 11.8, to: 12.733 },+ { from: 12.733, to: 13.667 },+ { from: 13.667, to: 14.367 },+ { from: 14.367, to: 15.333 },+ ];+ await runAiTool("add_clip", { blocks: shots.map((x) => ({ ...x, color: "#253b33" })), start: 11.8 });+ const starts = useEditor+ .getState()+ .clips.filter((c) => c.track === 0)+ .map((c) => c.start)+ .sort((a, b) => a - b);+ expect(starts.map((t) => Math.round(t * 30 * 1e6) / 1e6)).toEqual([354, 382, 410, 431]);+});diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.tsindex 46017d39..bb43f579 100644--- a/site/src/cut/server/ai/catalog.ts+++ b/site/src/cut/server/ai/catalog.ts@@ -356,7 +356,7 @@ Someone points at a video — a file they imported, a link you brought in, a cli**Measure it.** A contact sheet tells you what happens; the numbers come from the source's pixels. A reference recorded off a screen — a phone filming an editing app, a letterboxed film — has its picture inside a bigger frame: find_picture finds that box and keeps it on the source (detect_cuts does on its first read), every measuring tool and compare_to_source then reads inside it, and their x, y, w, h are fractions of it, so they land on the cut's frame as they are; set_aspect to its \`aspect\`. detect_cuts reads every frame: each cut on its frame, the \`shots\` between them, and the \`changes\` inside a shot — a word landing, a word swapping fill for outline, a shape popping in. Run it over the whole source before the read passes, which then land a frame in every shot and change, and time each title from them. Its times are to the millisecond, and every tool lands such a time on the frame it names: write them back as read. sample_color reads the hex of any region at any second. track_element gives an element's box frame by frame and the ease of each move; with match ink it gives a title's ink box.-**Lay down the sound, then block out the cuts.** Cuts timed to a reference are timed to its sound: detach_audio with the reference's \`assetId\` puts its whole track on the soundtrack and none of its picture. add_clip with \`blocks\` lays the skeleton from \`shots\` in one call — one block per shot on track 0, flash cuts included, \`color\` what sample_color reads behind it and \`label\` naming what the person drops there. The label rides the timeline chip; the frame shows the colour alone. Footage dropped on a block, or replace_item, takes its place and its length, so everything built on top keeps its timing. When the ask is the reference's own picture playing, add_clip \`spans\` lays its shots down instead, one span per shot at those same times.+**Lay down the sound, then block out the cuts.** Cuts timed to a reference are timed to its sound: detach_audio with the reference's \`assetId\` puts its whole track on the soundtrack and none of its picture. add_clip with \`blocks\` lays the skeleton from \`shots\` in one call — one block per shot on track 0, each its shot's \`from\` and \`to\` as read so every cut lands on its frame, flash cuts included, \`color\` what sample_color reads behind it and \`label\` naming what the person drops there. The label rides the timeline chip; the frame shows the colour alone. Footage dropped on a block, or replace_item, takes its place and its length, so everything built on top keeps its timing. When the ask is the reference's own picture playing, add_clip \`spans\` lays its shots down instead, one span per shot at those same times.**Rebuild the text as elements.** One title per card, a stretch of cards per add_title call — the words exactly as they read, size in px at 1080 wide, and x/y the centre of its ink box (track_element, match ink, the text's colour). A run of lines timed one after another is add_text_sequence in one call. Match the case the reference uses, and its colour from sample_color. The face comes off the frame: suggest_font with the card's moment, a region close around its words and the words themselves ranks every face Donkey has and hands back the font, weight, size and tracking to set. Call it once per distinct face — a title that cycles through typefaces is one call per face it shows. A no_match verdict means the reference uses a face Donkey does not bundle: say so, and set the closest one it named. Without a suggestion, size comes off the ink: capitals stand about 0.7 of the size tall, so size is the ink height in px at 1080 wide ÷ 0.7 (h × 1920 ÷ 0.7 on 9:16). A cycle whose faces change on the music's beats times itself with set_overlay_animation font_cycle_beats on the soundtrack clip after detect_beats; one whose changes follow the reference's own rhythm takes font_cycle_switches, the timeline seconds detect_cuts reads each change at.