Skip to content

Open source CapCut alternative.

TypeScript

feat(cut): let a block run one frame, 160 to a call [rebuild]

David Le7c247bf19468
diff --git a/docs/guides/cut/ai-assistant.md b/docs/guides/cut/ai-assistant.md
index 6c2fab46..92a7e35e 100644
--- a/docs/guides/cut/ai-assistant.md
+++ b/docs/guides/cut/ai-assistant.md
@@ -131,7 +131,7 @@ A thread saves one session slot per provider, so switching models mid-thread kee
| Inline media per model call | 16MB, trimmed oldest-first |
| Browser tool execution | 120s |
| One watch-video call | 600s of source, 150 decoded candidates, a 90s decode budget, and 36/16/6 kept frames by detail; the result says where coverage stopped |
-| One blockout call | 40 shots, each 0.2-600s |
+| One blockout call | 160 shots, each 1/60-600s |
| Snapshot caps | 60 media items, 60 cues |
| One queued turn job | 1MB of messages, one live turn per project |
| Saved threads per project | 30 |
diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.ts
index 2ed319dd..73cb0909 100644
--- a/site/src/cut/components/Timeline.tools.ts
+++ b/site/src/cut/components/Timeline.tools.ts
@@ -18,6 +18,7 @@ import {
import { TEXT_MOVE_IDS, TEXT_MOVE_NOTES } from "@/cut/lib/textMotion";
import { LETTER_SPACING_MAX, LETTER_SPACING_MIN, OVERLAY_ANIM_STYLE_IDS, TEXT_TEXTURES, TEXT_WEIGHTS } from "@donkeycut/effects-kit";
import { REGION_MAX_SCALE, TIMELINE_IDS, TIMELINE_LABELS } from "@/cut/lib/types";
+import { BLOCKOUT_MAX } from "@/cut/lib/blockSource";
/** The title sizes the assistant and the inspector set, px at a 1080 short
* side. */
@@ -88,7 +89,7 @@ export const TIMELINE_TOOLS = [
{
name: "add_clip",
description:
- "Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at `start`, inserted at `index`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at `start`, default the playhead). Asset ids come from `media` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut (\"add my beach photo\", \"stitch these into a movie\"); otherwise media stays on its card or panel for them to drag. Pass `blocks` instead of an asset to lay out a cut whose footage does not exist yet: one block per shot on track 0, each the length that shot runs, labelled with what belongs there. That is how a video you watched becomes a timeline — the cuts land at their real times, titles and sound go on top, and the person fills each block later. Blocks own no file and store nothing; footage dropped on one takes its place and its length, and replace_item does the same from here. Pass `spans` to cut one source into a run of clips — the moments of a talk worth keeping, with everything between them left out, or a video cut at its own shot boundaries when its picture is what plays.",
+ `Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at \`start\`, inserted at \`index\`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at \`start\`, default the playhead). Asset ids come from \`media\` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut ("add my beach photo", "stitch these into a movie"); otherwise media stays on its card or panel for them to drag. Pass \`blocks\` instead of an asset to lay out a cut whose footage does not exist yet: one block per shot on track 0, each the length that shot runs, labelled with what belongs there. That is how a video you watched becomes a timeline — the cuts land at their real times, titles and sound go on top, and the person fills each block later. Blocks own no file and store nothing; footage dropped on one takes its place and its length, and replace_item does the same from here. A block runs one frame (1/60 s) or longer, so detect_cuts' \`shots\` go down as they are, flash cuts included; one call takes up to ${BLOCKOUT_MAX}, and a longer list goes in more calls with no start, each appending after the last block. Pass \`spans\` to cut one source into a run of clips — the moments of a talk worth keeping, with everything between them left out, or a video cut at its own shot boundaries when its picture is what plays.`,
inputSchema: obj({
asset_id: str("Project asset id from `media` in editor_state"),
blocks: {
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index e1dbc651..efa34b87 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -292,7 +292,7 @@ import { hostedFetch, hostedPost } from "./hosted";
import { readHostedError } from "./hostedError";
import { creditPriceQuote, type CreditPriceSheet } from "@/lib/credits/price-sheet";
import { creditsFromDollars } from "@/lib/credits/format-credits";
-import { blockAsset } from "./blockSource";
+import { BLOCK_MAX_SECONDS, BLOCK_MIN_SECONDS, BLOCKOUT_MAX, blockAsset } from "./blockSource";
import { cutClip, cutJudge } from "./chatRuntime";
import { FOLLOW_MODES } from "./trackKeys";
import { trackMotion } from "./trackMotion";
@@ -311,6 +311,7 @@ import {
DEFAULT_BACKGROUND,
clampKeyPos,
clampOverlayPos,
+ CLIP_MIN_SECONDS,
previewUrl,
clipZoom,
fontStack,
@@ -714,10 +715,9 @@ const MEASURE_BUDGET_MS = 90_000;
* enough to ride a turn. */
const CAPTURE_LONG_SIDE = 640;
-/** The most shots one blockout lays down. Every block in a call lands as one
- * undo step, and a whole feature's worth of them in a single step is not
- * something anyone can step back through; a caller with more comes back. */
-const MAX_BLOCKS = 40;
+/** A trim that leaves a clip under one frame. */
+const CLIP_TOO_SHORT = `A clip runs at least one frame (${Math.round(CLIP_MIN_SECONDS * 1000) / 1000}s at 60 fps).`;
+
/** What a comparison's reply teaches about its own numbers. */
const COMPARE_NOTE =
"Source on the left, your cut on the right, one picture per moment. diff is how far your frame is from the source's: the mean difference per pixel, 0 (the same) to 255, and under " +
@@ -2593,11 +2593,11 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
const dur = asset?.type === "image" ? Infinity : asset?.duration ?? c.out;
const patch: Partial<VideoClip> = {};
if (isNum(input.start)) patch.start = Math.max(0, input.start);
- if (isNum(input.in)) patch.in = clamp(input.in, 0, dur - 0.1);
- if (isNum(input.out)) patch.out = clamp(input.out, 0.1, dur);
+ if (isNum(input.in)) patch.in = clamp(input.in, 0, dur - CLIP_MIN_SECONDS);
+ if (isNum(input.out)) patch.out = clamp(input.out, CLIP_MIN_SECONDS, dur);
if (patch.in !== undefined || patch.out !== undefined) {
- if ((patch.out ?? c.out) - (patch.in ?? c.in) < 0.1)
- throw new ToolError("Clip must stay at least 0.1s long.");
+ if ((patch.out ?? c.out) - (patch.in ?? c.in) < CLIP_MIN_SECONDS - 1e-9)
+ throw new ToolError(CLIP_TOO_SHORT);
}
if (isNum(input.track)) {
const track = Math.round(input.track);
@@ -2661,9 +2661,9 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
const asset = s.assets.find((a) => a.id === clip.assetId);
// A still has no source bound, so its clip can stretch to any length.
const dur = asset?.type === "image" ? Infinity : asset?.duration ?? clip.out;
- const nextIn = isNum(input.in) ? clamp(input.in, 0, dur - 0.1) : clip.in;
- const nextOut = isNum(input.out) ? clamp(input.out, 0.1, dur) : clip.out;
- if (nextOut - nextIn < 0.1) throw new ToolError("Clip must stay at least 0.1s long.");
+ const nextIn = isNum(input.in) ? clamp(input.in, 0, dur - CLIP_MIN_SECONDS) : clip.in;
+ const nextOut = isNum(input.out) ? clamp(input.out, CLIP_MIN_SECONDS, dur) : clip.out;
+ if (nextOut - nextIn < CLIP_MIN_SECONDS - 1e-9) throw new ToolError(CLIP_TOO_SHORT);
// Trim through the store's resize path so track 0 keeps its no-overlap
// invariant: extending a clip pushes the following run right.
s.setClipTrim(clip.id, nextIn, nextOut);
@@ -7457,7 +7457,7 @@ function placeBlocks(s: Editor, raw: unknown[], input: Record<string, unknown>)
const shots = raw
.filter((x): x is Record<string, unknown> => typeof x === "object" && x !== null)
.map((x) => ({
- seconds: isNum(x.seconds) ? clamp(x.seconds, 0.2, 600) : 0,
+ seconds: isNum(x.seconds) ? Math.min(x.seconds, BLOCK_MAX_SECONDS) : 0,
label: typeof x.label === "string" ? x.label.trim() : "",
// The shot's own backdrop, so a cut waiting on its footage still shows
// the look it is standing in for.
@@ -7466,9 +7466,15 @@ function placeBlocks(s: Editor, raw: unknown[], input: Record<string, unknown>)
.filter((x) => x.seconds > 0);
if (shots.length === 0)
throw new ToolError("Pass blocks: [{seconds, label}] — one entry per shot, in order.");
- if (shots.length > MAX_BLOCKS)
+ if (shots.length > BLOCKOUT_MAX)
+ throw new ToolError(
+ `That is ${shots.length} shots — ${BLOCKOUT_MAX} is the most one call lays down. Pass the first ${BLOCKOUT_MAX} now, then the other ${shots.length - BLOCKOUT_MAX} in a second call with no start: they append right after the last block.`
+ );
+ // A block is at least one frame: the shortest clip any renderer plays.
+ const short = shots.findIndex((x) => x.seconds < BLOCK_MIN_SECONDS - 1e-9);
+ if (short >= 0)
throw new ToolError(
- `That is ${shots.length} shots — ${MAX_BLOCKS} is the most one call lays down. Lay the first stretch and call again for the rest.`
+ `Block ${short + 1} runs ${roundT(shots[short].seconds)}s, under one frame (${roundT(BLOCK_MIN_SECONDS)}s at 60 fps). Give it at least that, or fold it into the shot beside it.`
);
const frame = frameOf(s.aspect);
const made: { clipId: string; assetId: string; start: number; seconds: number; label: string }[] = [];
@@ -7486,7 +7492,10 @@ function placeBlocks(s: Editor, raw: unknown[], input: Record<string, unknown>)
useEditor.getState().updateClip(clipId, { out: shot.seconds });
const placed = useEditor.getState().clips.find((c) => c.id === clipId);
if (!placed) continue;
- at = undefined; // the rest follow the row
+ // The next block starts where this one ends, on the frame it names: a
+ // sum of frame lengths drifts off k/fps, and a start a hair late shows
+ // the shot before for one frame.
+ at = onFrame(placed.start + shot.seconds);
made.push({
clipId: placed.id,
assetId: asset.id,
@@ -7520,10 +7529,10 @@ function placeAssetSpans(
const dur = asset.duration > 0 ? asset.duration : Infinity;
const cuts = spans
.map((sp) => ({
- from: clamp(Math.min(sp.from, sp.to), 0, dur - 0.1),
- to: clamp(Math.max(sp.from, sp.to), 0.1, dur),
+ from: clamp(Math.min(sp.from, sp.to), 0, dur - CLIP_MIN_SECONDS),
+ to: clamp(Math.max(sp.from, sp.to), CLIP_MIN_SECONDS, dur),
}))
- .filter((sp) => sp.to - sp.from >= 0.1)
+ .filter((sp) => sp.to - sp.from >= CLIP_MIN_SECONDS - 1e-9)
.sort((a, b) => a.from - b.from);
if (cuts.length === 0) throw new ToolError("Every span was empty or outside the source.");
const s = useEditor.getState();
diff --git a/site/src/cut/lib/blockSource.ts b/site/src/cut/lib/blockSource.ts
index 4f5dbf9b..bf57798b 100644
--- a/site/src/cut/lib/blockSource.ts
+++ b/site/src/cut/lib/blockSource.ts
@@ -1,5 +1,5 @@
import { createRasterCanvas, type RasterSurface } from "./raster";
-import type { MediaAsset, StoredAsset } from "./types";
+import { CLIP_MIN_SECONDS, type MediaAsset, type StoredAsset } from "./types";
// A block is a shot with nothing in it yet: a source that holds a slot on the
// timeline and draws itself as the colour that shot sits on. It owns no file,
@@ -16,6 +16,14 @@ import type { MediaAsset, StoredAsset } from "./types";
export const BLOCK_COLOR = "#161A22";
+/** The most shots one blockout call lays down: two full detect_cuts passes,
+ * so a template's whole shot list goes down as one undo step. */
+export const BLOCKOUT_MAX = 160;
+/** The shortest block: the shortest clip every renderer plays, one frame. */
+export const BLOCK_MIN_SECONDS = CLIP_MIN_SECONDS;
+/** The longest block. */
+export const BLOCK_MAX_SECONDS = 600;
+
/** The source a shot stands on until its footage exists. It takes the
* project's own frame, so a block fills the picture the way the footage that
* replaces it will. */
diff --git a/site/src/cut/lib/blockoutReference.test.ts b/site/src/cut/lib/blockoutReference.test.ts
index ff1575ec..1d0e7bb2 100644
--- a/site/src/cut/lib/blockoutReference.test.ts
+++ b/site/src/cut/lib/blockoutReference.test.ts
@@ -176,3 +176,37 @@ test("shorthand is a colour too", async () => {
color: "#ff5500",
});
});
+
+test("a shot one or two frames long goes down as a block, a span and a trim", async () => {
+ // 60 fps: one frame is 1/60 s, two are 1/30 s — the flash cuts of a montage.
+ const out = (await runAiTool("add_clip", { blocks: [{ seconds: 1 / 60 }, { seconds: 2 / 60 }, { seconds: 1 }] })) as {
+ placed: { seconds: number }[];
+ };
+ expect(out.placed.map((p) => p.seconds)).toEqual([0.017, 0.033, 1]);
+ useEditor.setState({ clips: [] });
+ const spans = (await runAiTool("add_clip", { asset_id: "ref", spans: [{ from: 1, to: 1 + 2 / 60 }, { from: 2, to: 3 }] })) as {
+ clips: { len: number }[];
+ };
+ expect(spans.clips.map((c) => c.len)).toEqual([0.033, 1]);
+ const id = useEditor.getState().clips[1]!.id;
+ await runAiTool("trim_clip", { clipId: id, in: 2, out: 2 + 1 / 60 });
+ const c = useEditor.getState().clips.find((x) => x.id === id)!;
+ expect(c.out - c.in).toBeCloseTo(1 / 60, 9);
+});
+
+test("a block under one frame is refused with what to do, and a long list says how to split", async () => {
+ const short = await runAiTool("add_clip", { blocks: [{ seconds: 1 }, { seconds: 0.005 }] }).then(
+ () => "",
+ (e: Error) => e.message
+ );
+ expect(short).toContain("Block 2");
+ expect(short).toContain("one frame");
+ const { BLOCKOUT_MAX } = await import("./blockSource");
+ const many = await runAiTool("add_clip", { blocks: Array.from({ length: BLOCKOUT_MAX + 3 }, () => ({ seconds: 0.5 })) }).then(
+ () => "",
+ (e: Error) => e.message
+ );
+ expect(many).toContain(`first ${BLOCKOUT_MAX}`);
+ expect(many).toContain("other 3");
+ expect(useEditor.getState().clips).toHaveLength(0);
+});
diff --git a/site/src/cut/lib/frameSnap.test.ts b/site/src/cut/lib/frameSnap.test.ts
index f3c992ac..3376d029 100644
--- a/site/src/cut/lib/frameSnap.test.ts
+++ b/site/src/cut/lib/frameSnap.test.ts
@@ -1,6 +1,6 @@
import { beforeEach, expect, test } from "bun:test";
import { runAiTool } from "./aiTools";
-import { useEditor } from "./store";
+import { clipLen, useEditor } from "./store";
// The tools report times to the millisecond — frame 22 at 60 fps is 0.367 —
// and the model writes them back as it read them. A time that sits a third of
@@ -85,3 +85,17 @@ test("the snap never moves a time between frames at 24, 25, 30, 50 or 60 fps", a
expect(await shapeStart(t)).toBe(t);
}
});
+
+test("blocks written from a shot list land every cut on its frame", async () => {
+ // A fast montage at 60 fps: shots of 22, 2, 3, 1 and 40 frames, each length
+ // written to the millisecond.
+ const lengths = [22, 2, 3, 1, 40];
+ await runAiTool("add_clip", { blocks: lengths.map((n) => ({ seconds: ms(frame(n)) })) });
+ const clips = [...useEditor.getState().clips].sort((a, b) => a.start - b.start);
+ let at = 0;
+ for (const [i, n] of lengths.entries()) {
+ expect(clips[i]!.start).toBe(frame(at));
+ expect(clipLen(clips[i]!)).toBeCloseTo(frame(n), 9);
+ at += n;
+ }
+});
diff --git a/site/src/cut/lib/types.ts b/site/src/cut/lib/types.ts
index 0eb96014..ecefdede 100644
--- a/site/src/cut/lib/types.ts
+++ b/site/src/cut/lib/types.ts
@@ -47,10 +47,10 @@ export const isLinkedAssetType = (type: string): type is LinkedAssetType =>
* an image has no intrinsic duration, so the clip carries this as its `out`. */
export const IMAGE_CLIP_SECONDS = 8;
-/** The shortest clip, seconds: one frame at 30 fps, a flash frame. Every
+/** The shortest clip, seconds: one frame at 60 fps, a flash frame. Every
* renderer floors a clip's footprint here, so a one-frame clip plays one
* frame in the preview and in every export. */
-export const CLIP_MIN_SECONDS = 1 / 30;
+export const CLIP_MIN_SECONDS = 1 / 60;
/** Project output frame ratio as "W:H". Presets cover common platforms;
* any ratio that passes `parseRatio` (e.g. "9:5") is valid. */
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index d97b6168..3a36c31d 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -356,7 +356,7 @@ Someone points at a video — a file they imported, a link you brought in, a cli
**Measure it.** A contact sheet tells you what happens; the numbers come from the source's pixels. A reference recorded off a screen — a phone filming an editing app, a letterboxed film — has its picture inside a bigger frame: find_picture finds that box and keeps it on the source (detect_cuts does on its first read), every measuring tool and compare_to_source then reads inside it, and their x, y, w, h are fractions of it, so they land on the cut's frame as they are; set_aspect to its \`aspect\`. detect_cuts reads every frame and returns each cut on its frame and the \`shots\` between them with their lengths — run it over the whole source, continuing from \`coveredTo\`. Its times are to the millisecond, and every tool lands such a time on the frame it names: write them back as read. sample_color reads the hex of any region at any second. track_element gives an element's box frame by frame and the ease of each move; with match ink it gives a title's ink box.
-**Block out the cuts.** add_clip with \`blocks\` lays the skeleton from detect_cuts' \`shots\` in one call — one block per shot on track 0, each that shot's length, \`color\` what sample_color reads behind it and \`label\` naming what the person drops there. The label rides the timeline chip and the editor state; the frame itself shows the colour alone, so a shot never plays a sentence describing itself. A block is filled by dropping footage on it — it takes the block's place and its length — or with replace_item, so everything you build on top keeps its timing. Cuts timed to a reference are timed to what it sounds like, so that sound belongs on the soundtrack under them: detach_audio with the reference's \`assetId\` puts its whole track there and places none of its picture. When the ask is the reference's own picture playing, add_clip \`spans\` lays its shots down instead, one span per shot at those same times.
+**Block out the cuts.** add_clip with \`blocks\` lays the skeleton from detect_cuts' \`shots\` in one call — one block per shot on track 0, flash cuts included (a block can be one frame), each that shot's length, \`color\` what sample_color reads behind it and \`label\` naming what the person drops there. The label rides the timeline chip and the editor state; the frame itself shows the colour alone, so a shot never plays a sentence describing itself. A block is filled by dropping footage on it — it takes the block's place and its length — or with replace_item, so everything you build on top keeps its timing. Cuts timed to a reference are timed to what it sounds like, so that sound belongs on the soundtrack under them: detach_audio with the reference's \`assetId\` puts its whole track there and places none of its picture. When the ask is the reference's own picture playing, add_clip \`spans\` lays its shots down instead, one span per shot at those same times.
**Rebuild the text as elements.** One add_title per card — the words exactly as they read, size in px at 1080 wide, and x/y the centre of its ink box (track_element, match ink, the text's colour). A run of lines timed one after another is add_text_sequence in one call. Match the case the reference uses, and its colour from sample_color. The face comes off the frame: suggest_font with the card's moment, a region close around its words and the words themselves ranks every face Donkey has and hands back the font, weight, size and tracking to set. Call it once per distinct face — a title that cycles through typefaces is one call per face it shows. A no_match verdict means the reference uses a face Donkey does not bundle: say so, and set the closest one it named. Without a suggestion, size comes off the ink: capitals stand about 0.7 of the size tall, so size is the ink height in px at 1080 wide ÷ 0.7 (h × 1920 ÷ 0.7 on 9:16). A cycle whose faces change on the music's beats times itself with set_overlay_animation font_cycle_beats on the soundtrack clip after detect_beats; one whose changes follow the reference's own rhythm takes font_cycle_switches, the timeline seconds detect_cuts reads each change at.