Skip to content

Open source CapCut alternative.

TypeScript

Merge branch 'gap/fonts' into gap/all

David Lef209226e692b
diff --cc site/src/clients/chatgpt/server/catalog.ts
index de4f1b8d,ad4539b8..9f2f3c3f
--- a/site/src/clients/chatgpt/server/catalog.ts
+++ b/site/src/clients/chatgpt/server/catalog.ts
@@@ -58,9 -58,7 +58,10 @@@ export const READ_COMMANDS = new Set(
"watch_video",
"capture_frame",
"compare_to_source",
+ "detect_cuts",
+ "sample_color",
+ "track_element",
+ "suggest_font",
"listen_audio",
"detect_silence",
"detect_beats",
diff --cc site/src/cut/components/AiPanel.tools.ts
index 361063f6,6bea6f45..56d89217
--- a/site/src/cut/components/AiPanel.tools.ts
+++ b/site/src/cut/components/AiPanel.tools.ts
@@@ -189,75 -155,33 +189,98 @@@ export const AI_PANEL_TOOLS =
],
},
},
+ offset: num("Source second minus timeline second, for every moment without its own source and for the sweep (default 0)"),
+ from: num("sweep: timeline second to start at (default 0)"),
+ to: num("sweep: timeline second to stop at (default the cut's end)"),
+ step: num("sweep: seconds between samples (default one frame, widened so a long span fits the call)"),
+ },
+ ["asset_id"]
+ ),
+ },
+ {
+ name: "detect_cuts",
+ description:
+ `Find every hard cut in a stretch of a video source, to the frame. Reads every frame between from and to at a few dozen pixels, scores how much of the picture changed since the frame before, and reports each frame that opens a new shot: \`t\` in SOURCE seconds to the millisecond (the new shot's first frame), its \`frame\` index at the source's \`fps\`, and \`change\`, the percent of the picture that changed — near 100 is a new shot, 10-30 a card or a block of type swapping over the same background. \`shots\` lists the stretches between the cuts with their length in frames. watch_video's sceneChanges come from samples a step apart and lose a shot shorter than the step; this reads every frame, so a five-frame shot in a fast montage is found. Run it before blocking out a reference: add_clip \`blocks\` takes one block per shot at that shot's length. Hard cuts only — a dissolve or a fade changes a little each frame and reads as none. Up to ${DETECT_CUTS_SPAN_MAX}s and ${DETECT_CUTS_MAX} cuts a call; when \`truncated\`, call again from \`coveredTo\`.`,
+ inputSchema: obj(
+ {
+ asset_id: str("Project asset id of the video"),
+ from: num("Source start s (default 0)"),
+ to: num(`Source end s (default the source's end, at most ${DETECT_CUTS_SPAN_MAX}s past from)`),
+ },
+ ["asset_id"]
+ ),
+ },
+ {
+ name: "sample_color",
+ description:
+ `Read the colour of a source's pixels: the hex of a region or a point of a frame at a source second, so a block, a shape, a title or a background takes the reference's colour as a number. Contact-sheet cells are JPEGs and shift colour; this reads the decoded frame as the preview shows it. Each sample is a moment \`t\` (SOURCE seconds) and a box — x/y its centre and w/h its size as frame fractions, the box add_shape takes; leave out w/h to read a point. Returns per sample \`hex\`, the region's median colour, and \`spread\`, how far its pixels stray from it on average (0-255: under 10 is a flat fill, over 40 a mix — read \`dominant\` there). \`colors\` 2-${SAMPLE_COLORS_K_MAX} adds \`dominant\`, the colours the region is made of with their shares, largest first: a card and the type on it, a bar and the shot behind it. Read a point inside a flat fill for its colour, a box over a title for its ink and its ground. Up to ${SAMPLE_COLOR_MAX} samples a call, at one moment or each at its own. Works on a still image too (t is ignored).`,
+ inputSchema: obj(
+ {
+ asset_id: str("Project asset id (video or image)"),
+ samples: {
+ type: "array",
+ description: `Up to ${SAMPLE_COLOR_MAX} regions to read`,
+ items: obj(
+ {
+ t: num("Source second"),
+ x: num("Centre x, fraction of frame width"),
+ y: num("Centre y, fraction of frame height"),
+ w: num("Width, fraction of frame width (omit with h for a point)"),
+ h: num("Height, fraction of frame height"),
+ },
+ ["x", "y"]
+ ),
+ },
+ colors: num(`Also list this many dominant colours per region, 2..${SAMPLE_COLORS_K_MAX} (default: none)`),
+ },
+ ["asset_id", "samples"]
+ ),
+ },
+ {
+ name: "track_element",
+ description:
+ `Measure where an element of a source's picture sits, frame by frame, and how it moves, so a grow, a slide or a pop is keyed with set_overlay_keyframes to the reference's own numbers. Give a stretch of a video (SOURCE seconds) and a \`seed\` box on the element at \`from\`; boxes in and out are x/y centre and w/h size as frame fractions — what add_shape, add_title and the keys take. The element is found by colour: \`color\` (sample_color reads it), or else the seed's median colour on the first frame, so draw the seed inside the element's edges at a moment it is on screen. \`match\` shape follows the one connected piece of that colour the seed sits on — a card, a bar, a sticker — as it moves and changes size; ink takes every pixel of that colour inside the seed (or \`within\`), the ink of a line of type whose letters are separate pieces: its centre is where the title sits and its height is about 0.7 of the font size when the line is capitals. Leave out \`to\` to measure one frame. \`moves\` lists each move of x, y, w and h with from/to (the source seconds it starts and lands), start/end values, the ease that fits it (${EASE_IDS.join(", ")}) and \`error\`, the RMS miss as a share of the move: under 0.03 is that curve, over 0.1 none fits and the move keys from \`path\` point by point. A size move keys as scale: w over the element's resting w. \`path\` lists the box wherever it changes (a hold is one entry); a null box is a frame where the element was not found. Up to ${TRACK_FRAMES_MAX} frames a call; when \`truncated\`, continue from \`coveredTo\`. Works on a still image too (one frame).`,
+ inputSchema: obj(
+ {
+ asset_id: str("Project asset id (video or image)"),
+ from: num("Source second the seed shows the element at"),
+ to: num("Source end s; omit to measure the one frame at from"),
+ seed: frameBox("A box on the element at from — inside its edges for shape, around the line for ink"),
+ within: frameBox("Only look inside this box (default: the whole frame for shape, the seed for ink)"),
+ color: str("The element's colour as #RRGGBB (default: the seed's median colour)"),
+ tolerance: num(`Per-channel distance from color that still counts as it, 0..255 (default ${TRACK_TOLERANCE_DEFAULT}); raise it for a gradient or a soft edge`),
+ match: {
+ type: "string",
+ enum: [...TRACK_MATCHES],
+ description: "shape: the one connected piece the seed sits on (default). ink: every pixel of the colour inside the seed or within — type",
+ },
},
- ["asset_id", "times"]
+ ["asset_id", "from", "seed"]
),
},
+ {
+ name: "suggest_font",
+ description:
+ "Name the face a reference's type is set in: reads the source's frame at t, lifts the ink inside region off its ground, and ranks every face Donkey can set a title in — each bundled family at each bundled weight, the system set and the account's own fonts — by how closely its glyphs for the same words match, tracking solved so each runs as wide as the reference. Call it for each distinct face while rebuilding a reference's titles, so the font, weight, size and tracking come from the frame. Pass the words exactly as they read, case and line breaks included, and a region close around them read off watch_video at detail original. Returns the best faces with score (0..1), size (px at the 1080 short side, for the same ink height) and tracking (em), plus the ink's box in frame fractions. verdict no_match means nothing bundled is close: the reference uses a face Donkey does not bundle, so say so and name the stand-in you use.",
+ inputSchema: obj(
+ {
+ asset_id: str("Project asset id of the reference (video or image)"),
+ t: num("Source second the type shows at (default 0)"),
+ region: obj(
+ {
+ x: num("Left edge, frame fraction"),
+ y: num("Top edge, frame fraction"),
+ w: num("Width, frame fraction"),
+ h: num("Height, frame fraction"),
+ },
+ ["x", "y", "w", "h"]
+ ),
+ text: str("The words the region shows, as they read, with a line break where the type breaks"),
+ count: num("How many faces to return, 1..10 (default 5)"),
+ },
+ ["asset_id", "region", "text"]
+ ),
+ },
{
name: "find_highlights",
description:
diff --cc site/src/cut/lib/aiTools.ts
index 81794399,3b336cf5..389ac53c
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@@ -232,14 -233,12 +232,16 @@@ import { loadLibraryLut, lutIdOf, lutLa
import { type RawFrame, type ReadGrade, rawFrameAt, sampleClipRawFrame, sourceProfileOf, toBaseRendering, toGradedRendering } from "./baseFrame";
import { applyOverlayPatchSettled, assetClipUses, assetIdsOnAnyTimeline, clipLen, footprints, nextFreeStart, track0Clips, laneGapAt, getClipSpans, overlayLaneOrder, overlayLayers, parkedTransitions, projectDuration, resolveTransitions, totalDuration, useEditor } from "./store";
import { playheadAt } from "./playhead";
-import { mixSpecFor, renderProjectFrame, renderProjectFrames } from "./exportRender";
+import { mixSpecFor, renderProjectFrame } from "./exportRender";
import { renderMix } from "./audioMix";
import { measureLoudnessSliced, roundedLoudness, type LoudnessMeasure } from "./loudness";
- import { framesAt, readMediaFileSize } from "./mediaRead";
+ import { frameAt, framesAt, readMediaFileSize } from "./mediaRead";
+ import { bundledFaces, inkMaskOf, rankFaces } from "./fontMatch";
+ import { cutFontMatch } from "./fontMatchSettings";
+import { COMPARE_MODES, MAX_COMPARE, SWEEP_TARGET } from "./frameDiff";
+import { compareMoments, sweepSource } from "./sourceCompare";
+import { sampleColors, scanCuts, trackElement, type FrameBox } from "./measure/sourceMeasure";
+import { DETECT_CUTS_MAX, DETECT_CUTS_SPAN_MAX, SAMPLE_COLOR_MAX, SAMPLE_COLORS_K_MAX, TRACK_FRAMES_MAX, TRACK_MATCHES, TRACK_PATH_MAX, TRACK_TOLERANCE_DEFAULT } from "./measure/limits";
import { renderStageFrame, storeStageStill } from "./stageFrame";
import { createRasterCanvas, decodeRasterImageUrl, rasterCanvasToDataUrl } from "./raster";
import { buildAiContext, describeCameraCard, describeDoc, hiddenFromChat, OTHER_CHATS_MEDIA_NOTE, roundT } from "./aiContext";
@@@ -635,19 -576,17 +638,27 @@@ const CAPTURE_LONG_SIDE = 640
* undo step, and a whole feature's worth of them in a single step is not
* something anyone can step back through; a caller with more comes back. */
const MAX_BLOCKS = 40;
-// Each moment is two frames drawn and one picture composed; a handful at a
-// time keeps the call inside the bridge's deadline and the reply readable.
-const MAX_COMPARE = 4;
+/** What a comparison's reply teaches about its own numbers. */
+const COMPARE_NOTE =
+ "Source on the left, your cut on the right, one picture per moment. diff is how far your frame is from the source's: the mean difference per pixel, 0 (the same) to 255, and under " +
+ `${SWEEP_TARGET} reads as the same picture. grid is that difference over the frame in 4×4 cells, top row first, so the hot cells are where to look. ` +
+ "shift means the source matches your frame better that many seconds away: positive, your cut shows it early, so move what is there later by that much; negative, earlier. " +
+ "block names a block standing in for footage, which keeps a difference no edit removes, so judge those moments by the cells over your titles and shapes. " +
+ "Name what differs — colour, the words and their face, where a thing sits, its size, what is moving and how far along it is — fix it with the tools and check again. A moment that matches needs no edit.";
+const SWEEP_NOTE =
+ "diff is the mean difference per pixel between your frame and the source's, 0 (the same) to 255; median is the typical frame, p95 the bad tail, max the worst. " +
+ "worst lists the stretches most off, worst first, each with its peak frame (at on the cut, source on the source) and that frame's 4×4 grid, top row first. " +
+ "Look at them: compare_to_source with times [{at, source}] at those peaks, fix what differs, then sweep again, until the median is under target. " +
+ "shift on a stretch, or timing for the whole span, means the source matches better that many seconds away: positive, move it later by that much; negative, earlier. " +
+ "onBlocks is the share of samples over blocks standing in for footage; those keep a difference no edit removes.";
+ /** Faces suggest_font hands back by default, and at most. */
+ const FONT_SUGGESTIONS = 5;
+ const FONT_SUGGESTIONS_MAX = 10;
+ /** The longest side a suggest_font crop is read at, px: type this tall is
+ * already far finer than the grid it is scored on. */
+ const FONT_CROP_MAX = 1600;
+ /** Design px on a frame's short side, the unit a title's size is in. */
+ const DESIGN_SHORT_SIDE = 1080;
const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v));
const isNum = (v: unknown): v is number => typeof v === "number" && Number.isFinite(v);
@@@ -5957,9 -5871,7 +6051,10 @@@ export const MEDIA_RUNTIME_TOOLS: Reado
"sync_audio",
"capture_frame",
"compare_to_source",
+ "detect_cuts",
+ "sample_color",
+ "track_element",
+ "suggest_font",
"render_preview",
"freeze_frame",
"create_sticker",
diff --cc site/src/cut/lib/pi/mutationLedger.ts
index 7f59caf4,d88f979e..c57f9e24
--- a/site/src/cut/lib/pi/mutationLedger.ts
+++ b/site/src/cut/lib/pi/mutationLedger.ts
@@@ -13,9 -13,7 +13,10 @@@ const READ_ONLY = new Set(
"request_tools",
"capture_frame",
"compare_to_source",
+ "detect_cuts",
+ "sample_color",
+ "track_element",
+ "suggest_font",
"watch_video",
"note_source",
"detect_silence",
diff --cc site/src/cut/server/ai/catalog.ts
index 12f21073,c3fabf5e..9410157a
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@@ -354,15 -352,13 +354,15 @@@ Someone points at a video — a file th
**Clipping a long source.** A talk, an interview or a stream is mostly material that only makes sense in place. find_highlights ranks its stretches from the transcript, so an hour is read for its handful of moments without watching all of it; watch_video the spans it returns before committing, since the ranking never saw the picture. add_clip \`spans\` lays a pick down — several stretches of one source as one run of clips, the material between them left out — and a two-span result is one clip whose halves were never next to each other.
**Watch the whole thing first.** Every decision below comes off the record, so the record has to be complete: watch from 0 and keep going until \`unwatchedSeconds\` is 0, writing each pass into note_source before the next one. Anything over a couple of minutes gets a coarse pass end to end first (\`interval_seconds\` 15-30) to learn its shape, then a dense pass (0.5) over the stretches that carry text or cut fast. Read every card word for word — its text, its face, where it sits in the frame, when it comes and goes — because a turn later the sheets are gone and the notes are all you have. Reading type is what \`detail\` is for: the dense pass over a stretch carrying words runs at \`detail\` read, and a face or treatment you are about to name runs at original, where the frame arrives at the source's own resolution. A scan cell is a quarter of the frame's width; a small caption in one is a smudge, and a smudge read as a word is how a card ends up saying something the video never said. When the cards ARE the narration — a caption track of what the speaker is saying — the transcript already holds every word, and the close passes are for how they are set: face, weight, size, colour, placement, how many words ride a card, how it enters and leaves. Read a few of those at original and take the rest from the transcript.
-**Block out the cuts.** \`sceneChanges\` are the cut times: shot 1 runs from 0 to the first, shot n from one to the next, the last to the end. add_clip with \`blocks\` lays the whole skeleton down in one call — one block per shot on track 0, each the length that shot runs, \`color\` set to what the reference has behind that shot and \`label\` naming what the person drops there. The label rides the timeline chip and the editor state; the frame itself shows the colour alone, so a shot never plays a sentence describing itself. A block is filled by dropping footage on it — it takes the block's place and its length — or with replace_item, so everything you build on top keeps its timing. Cuts timed to a reference are timed to what it sounds like, so that sound belongs on the soundtrack under them: detach_audio with the reference's \`assetId\` puts its whole track there and places none of its picture. When the ask is the reference's own picture playing, add_clip \`spans\` lays its shots down instead, one span per shot at those same times.
+**Measure it.** A contact sheet tells you what happens; the numbers come from the source's pixels. detect_cuts reads every frame and returns each cut on its frame and the \`shots\` between them with their lengths — run it over the whole source, continuing from \`coveredTo\`. sample_color reads the hex of any region at any second. track_element gives an element's box frame by frame and the ease of each move; with match ink it gives a title's ink box.
-**Rebuild the text as elements.** One add_title per card — the words exactly as they read, size in px at 1080 wide, and x/y as the fraction of the frame it sits at (a lower-third card is around y 0.75, a centered statement 0.5). A run of lines timed one after another is add_text_sequence in one call. Match the case the reference uses, and its color against the frame behind it. The face comes off the frame: suggest_font with the card's moment, a region close around its words and the words themselves ranks every face Donkey has and hands back the font, weight, size and tracking to set. Call it once per distinct face — a title that cycles through typefaces is one call per face it shows. A no_match verdict means the reference uses a face Donkey does not bundle: say so, and set the closest one it named. A cycle whose faces change on the music's beats times itself with set_overlay_animation font_cycle_beats on the soundtrack clip after detect_beats; one whose changes follow the reference's own rhythm takes font_cycle_switches, the timeline seconds you read each change at.
+**Block out the cuts.** add_clip with \`blocks\` lays the skeleton from detect_cuts' \`shots\` in one call — one block per shot on track 0, each that shot's length, \`color\` what sample_color reads behind it and \`label\` naming what the person drops there. The label rides the timeline chip and the editor state; the frame itself shows the colour alone, so a shot never plays a sentence describing itself. A block is filled by dropping footage on it — it takes the block's place and its length — or with replace_item, so everything you build on top keeps its timing. Cuts timed to a reference are timed to what it sounds like, so that sound belongs on the soundtrack under them: detach_audio with the reference's \`assetId\` puts its whole track there and places none of its picture. When the ask is the reference's own picture playing, add_clip \`spans\` lays its shots down instead, one span per shot at those same times.
- **Rebuild the text as elements.** One add_title per card — the words exactly as they read, the face from the graphics skill's font ids, size in px at 1080 wide, and x/y the centre of its ink box (track_element, match ink, the text's colour); capitals stand about 0.7 of the size tall, so size is the ink height in px at 1080 wide ÷ 0.7 (h × 1920 ÷ 0.7 on 9:16). A run of lines timed one after another is add_text_sequence in one call. Match the case and weight the reference uses, and its colour from sample_color.
-**Then the motion.** set_overlay_keyframes gives each card its own pose track — where it sits, at what scale, at what second — which is how the timing lands on the reference's own frames. set_overlay_animation composes on top for the entrance, the exit, a loop, and what a line does while it holds; 0.3-0.6s reads as an entrance, and a per-letter style is what makes a card read as typed. add_shape is the rest of the graphics: bars, cards, rules, a block of color behind a line.
++**Rebuild the text as elements.** One add_title per card — the words exactly as they read, size in px at 1080 wide, and x/y the centre of its ink box (track_element, match ink, the text's colour). A run of lines timed one after another is add_text_sequence in one call. Match the case the reference uses, and its colour from sample_color. The face comes off the frame: suggest_font with the card's moment, a region close around its words and the words themselves ranks every face Donkey has and hands back the font, weight, size and tracking to set. Call it once per distinct face — a title that cycles through typefaces is one call per face it shows. A no_match verdict means the reference uses a face Donkey does not bundle: say so, and set the closest one it named. Without a suggestion, size comes off the ink: capitals stand about 0.7 of the size tall, so size is the ink height in px at 1080 wide ÷ 0.7 (h × 1920 ÷ 0.7 on 9:16). A cycle whose faces change on the music's beats times itself with set_overlay_animation font_cycle_beats on the soundtrack clip after detect_beats; one whose changes follow the reference's own rhythm takes font_cycle_switches, the timeline seconds detect_cuts reads each change at.
-**Look at what you made.** Watching told you what the reference does; it does not tell you whether what you made does it too, and a build that reads right from your own notes is still a guess until you see it. compare_to_source puts the source's frame and yours side by side at the same moment, a few at a time. Run it as you build, not once at the end: the first few shots, then each stretch as you finish it — an error in the first card is in every card after it. Spread the moments over what you laid down, including the middle of a move, and read each pair for colour, the words and their face, where a thing sits, its size, and how far along a move is. A title the same colour as the block behind it, or a shape covering the thing it was meant to sit beside, reads as correct in the document and is only visible here. Fix what differs, check the moments you changed, and keep going until a pass turns up nothing.
+**Then the motion.** set_overlay_keyframes gives each card its own pose track — where it sits, at what scale, at what second. track_element over a move hands you the keys: each move's from/to, start/end and ease, with scale as w over the resting w. set_overlay_animation composes on top for the entrance, the exit, a loop, and what a line does while it holds; 0.3-0.6s reads as an entrance, and a per-letter style is what makes a card read as typed. add_shape is the rest of the graphics: bars, cards, rules, a block of color behind a line.
+
+**Look at what you made.** Watching told you what the reference does; it does not tell you whether what you made does it too, and a build that reads right from your own notes is still a guess until you measure it. compare_to_source does: mode sweep scores every frame of the cut against the source — 0 is the same picture, under ${SWEEP_TARGET} reads as one — and names the worst stretches with a grid of where each differs; mode moments shows those peaks beside the source's frames. Sweep as you build, since an error in the first card is in every card after it: look at the worst stretches, fix what differs, sweep again, and stop when the median is under ${SWEEP_TARGET} and no stretch stands out. A \`shift\` means the right thing lands early or late by that much. Read each picture for colour, the words and their face, where a thing sits, its size, and how far along a move is; a title the same colour as the block behind it reads as correct in the document and shows up here. A block keeps its footage's difference, so judge a shot by the cells over what you built.
**Say what plays.** Close by naming what fills the shots, and what you checked. A shell gets what to drop in: how many shots, how long each one runs, and that the titles and timing are built around them.`,
diff --cc site/src/cut/server/ai/headlessTools.ts
index fabc4e68,a38c41dd..0635c95a
--- a/site/src/cut/server/ai/headlessTools.ts
+++ b/site/src/cut/server/ai/headlessTools.ts
@@@ -39,9 -39,7 +39,10 @@@ const PAGE_MEDIA_TOOLS: ReadonlySet<str
"freeze_frame",
"capture_frame",
"compare_to_source",
+ "detect_cuts",
+ "sample_color",
+ "track_element",
+ "suggest_font",
"render_preview",
"create_sticker",
"subtitles_generate",