Skip to content

Open source CapCut alternative.

TypeScript

feat(site): find a strobe as one flicker over its run [rebuild]

David Le01aeeef3ae16
diff --git a/site/src/cut/components/AiPanel.tools.ts b/site/src/cut/components/AiPanel.tools.ts
index 4197a485..e006185c 100644
--- a/site/src/cut/components/AiPanel.tools.ts
+++ b/site/src/cut/components/AiPanel.tools.ts
@@ -241,7 +241,7 @@ export const AI_PANEL_TOOLS = [
{
name: "detect_cuts",
description:
- `Find every hard cut in a stretch of a video source, to the frame. Reads every frame between from and to at a few dozen pixels, scores how much of the picture changed since the frame before, and reports each frame that opens a new shot: \`t\` in SOURCE seconds to the millisecond (the new shot's first frame), its \`frame\` index at the source's \`fps\`, and \`change\`, the percent of the picture that changed — near 100 is a new shot, 10-30 a card or a block of type swapping over the same background. A flat colour cut to a near one counts. \`shots\` lists the stretches between the cuts with their length in frames. \`changes\` lists what changes inside a shot over a picture that stays: a word appearing, a word swapping fill for outline, a shape popping in, a face switching — each its first frame (\`t\`, \`frame\`), how many frames it runs (a fade-in runs several), its \`change\` in percent and its \`box\`. A fade too faint for any one frame to show — small type easing in or out over half a second — still counts: its \`frame\` is the first step it takes and \`frames\` runs to where it settles, so a title's fade in, fade out and their lengths come from here. Time titles, their fades, outline swaps, shape pops and font_cycle_switches from these. watch_video lands a frame in every shot and change this reads, so read type after this pass. watch_video's sceneChanges come from samples a step apart and lose a shot shorter than the step; this reads every frame, so a five-frame shot in a fast montage is found. Run it before blocking out a reference: add_clip \`blocks\` takes one block per shot at that shot's length. Hard cuts only — a dissolve or a fade between two shots changes a little each frame and reads as no cut. Up to ${DETECT_CUTS_SPAN_MAX}s, ${DETECT_CUTS_MAX} cuts and ${DETECT_CHANGES_MAX} changes a call; when \`truncated\`, call again from \`coveredTo\`. A source with no picture box yet has it found first (find_picture) and kept, and every frame is read inside it, so a card that changes a small part of a recorded frame still counts as the change to the picture it is.`,
+ `Find every hard cut in a stretch of a video source, to the frame. Reads every frame between from and to at a few dozen pixels, scores how much of the picture changed since the frame before, and reports each frame that opens a new shot: \`t\` in SOURCE seconds to the millisecond (the new shot's first frame), its \`frame\` index at the source's \`fps\`, and \`change\`, the percent of the picture that changed — near 100 is a new shot, 10-30 a card or a block of type swapping over the same background. A flat colour cut to a near one counts. \`shots\` lists the stretches between the cuts with their length in frames. \`changes\` lists what changes inside a shot over a picture that stays: a word appearing, a word swapping fill for outline, a shape popping in, a face switching — each its first frame (\`t\`, \`frame\`), how many frames it runs (a fade-in runs several), its \`change\` in percent and its \`box\`. A change marked \`flicker\` flips back and forth frame by frame over its run — a strobe, a cutout flashing to its negative — so every frame of it differs from the one before and returns to a look two or three frames old; read its frames one at a time. A fade too faint for any one frame to show — small type easing in or out over half a second — still counts: its \`frame\` is the first step it takes and \`frames\` runs to where it settles, so a title's fade in, fade out and their lengths come from here. Time titles, their fades, outline swaps, shape pops and font_cycle_switches from these. watch_video lands a frame in every shot and change this reads, so read type after this pass. watch_video's sceneChanges come from samples a step apart and lose a shot shorter than the step; this reads every frame, so a five-frame shot in a fast montage is found. Run it before blocking out a reference: add_clip \`blocks\` takes one block per shot at that shot's length. Hard cuts only — a dissolve or a fade between two shots changes a little each frame and reads as no cut. Up to ${DETECT_CUTS_SPAN_MAX}s, ${DETECT_CUTS_MAX} cuts and ${DETECT_CHANGES_MAX} changes a call; when \`truncated\`, call again from \`coveredTo\`. A source with no picture box yet has it found first (find_picture) and kept, and every frame is read inside it, so a card that changes a small part of a recorded frame still counts as the change to the picture it is.`,
inputSchema: obj(
{
asset_id: str("Project asset id of the video"),
diff --git a/site/src/cut/lib/measure/cuts.test.ts b/site/src/cut/lib/measure/cuts.test.ts
index dae01bc1..df5b85e2 100644
--- a/site/src/cut/lib/measure/cuts.test.ts
+++ b/site/src/cut/lib/measure/cuts.test.ts
@@ -150,3 +150,28 @@ test("a flash frame inside fast streaking footage keeps both of its cuts", () =>
const { cuts } = scoreFrames(frames);
expect([6, 7].filter((i) => cuts.includes(i))).toEqual([6, 7]);
});
+
+test("a cutout flipping to its negative frame by frame is one flicker over its run", () => {
+ // A white figure over a blue shot, flipped to black on alternate frames,
+ // once held for two frames, the way a strobed cutout plays.
+ const bg: [number, number, number] = [40, 120, 200];
+ const plain = card(128, 128, bg, [240, 240, 240], [40, 72, 30, 100]);
+ const negative = card(128, 128, bg, [15, 15, 15], [40, 72, 30, 100]);
+ const shot = solid(128, 128, bg);
+ const run = [plain, negative, plain, negative, plain, plain, negative, plain, negative];
+ const { changes } = scoreFrames([shot, shot, shot, shot, ...run, shot, shot, shot]);
+ const flicker = changes.filter((c) => c.flicker);
+ expect(flicker).toHaveLength(1);
+ expect(flicker[0].frame).toBe(5);
+ expect(flicker[0].frames).toBe(8);
+ expect(flicker[0].box.x).toBeCloseTo(56 / 128, 1);
+ expect(changes.some((c) => !c.flicker && c.frame > 5 && c.frame < 13)).toBe(false);
+});
+
+test("a shape that pops in and out every few frames is no flicker", () => {
+ const bg: [number, number, number] = [40, 120, 200];
+ const on = card(128, 128, bg, [240, 240, 240], [40, 72, 30, 100]);
+ const off = solid(128, 128, bg);
+ const frames = Array.from({ length: 30 }, (_, i) => (Math.floor(i / 3) % 2 ? on : off));
+ expect(scoreFrames(frames).changes.some((c) => c.flicker)).toBe(false);
+});
diff --git a/site/src/cut/lib/measure/cuts.ts b/site/src/cut/lib/measure/cuts.ts
index 1d386a7f..0bb9e9a1 100644
--- a/site/src/cut/lib/measure/cuts.ts
+++ b/site/src/cut/lib/measure/cuts.ts
@@ -38,6 +38,13 @@ export const CUT_TUNING = {
* and the share that marks its last. */
SLOW_START: 0.05,
SLOW_END: 0.95,
+ /** % of the fine grid a flicker frame changes from the frame before. */
+ FLICKER_MIN: 2,
+ /** A flicker frame sits at most this share of that change away from the
+ * frame two or three back: it returns to a look it just left. */
+ FLICKER_ECHO: 0.6,
+ /** Flicker frames a run holds at least; one held frame may part two. */
+ FLICKER_RUN: 3,
};
/** Per-cell channel means of an RGBA frame on a GRID×GRID grid. */
@@ -120,6 +127,8 @@ export interface ShotChange {
frames: number;
change: number;
box: ChangeBox;
+ /** The change flips back and forth frame by frame over its run. */
+ flicker?: true;
}
export const gridsOf = (px: Pixels): FrameGrids => ({
@@ -241,10 +250,53 @@ export function pickShotEdges(steps: FrameStep[], fines?: ArrayLike<number>[]):
if (!fines) {
return { cuts, changes };
}
- const slow = gradualChanges(fines, cuts).filter(
- (g) => !changes.some((c) => g.frame < c.frame + c.frames && c.frame < g.frame + g.frames)
- );
- return { cuts, changes: [...changes, ...slow].sort((a, b) => a.frame - b.frame) };
+ const overlaps = (a: ShotChange, b: ShotChange) => a.frame < b.frame + b.frames && b.frame < a.frame + a.frames;
+ const flicks = flickers(fines);
+ const slow = gradualChanges(fines, cuts).filter((g) => !changes.some((c) => overlaps(g, c)));
+ const rest = [...changes, ...slow].filter((c) => !flicks.some((f) => overlaps(c, f)));
+ return { cuts, changes: [...rest, ...flicks].sort((a, b) => a.frame - b.frame) };
+}
+
+/** Runs of frames that flip back and forth — a strobe, a cutout flashing to
+ * its negative — each frame changed from the one before and back to a look
+ * two or three frames old. Every such frame matches its neighbours' change,
+ * so no one of them stands out as a change of its own. A shake that returns
+ * to rest echoes once; a run takes FLICKER_RUN. The run opens on the frame
+ * after the first look it returns to. */
+export function flickers(fines: ArrayLike<number>[]): ShotChange[] {
+ const { FLICKER_MIN, FLICKER_ECHO, FLICKER_RUN } = CUT_TUNING;
+ const cells = fines.length > 0 ? fines[0].length / 3 : 0;
+ const hits: { i: number; echo: number; moved: number[] }[] = [];
+ for (let i = 2; i < fines.length; i++) {
+ const moved = cellsApart(fines[i - 1], fines[i]);
+ const step = (100 * moved.length) / cells;
+ if (step < FLICKER_MIN) {
+ continue;
+ }
+ const back = [2, 3].find((k) => i - k >= 0 && (100 * cellsApart(fines[i - k], fines[i]).length) / cells <= step * FLICKER_ECHO);
+ if (back) {
+ hits.push({ i, echo: i - back, moved });
+ }
+ }
+ const out: ShotChange[] = [];
+ let run: typeof hits = [];
+ const close = () => {
+ if (run.length >= FLICKER_RUN) {
+ const frame = run[0].echo + 1;
+ const moved = [...new Set(run.flatMap((h) => h.moved))];
+ const change = Math.max(...run.map((h) => (100 * h.moved.length) / cells));
+ out.push({ frame, frames: run[run.length - 1].i - frame + 1, change, box: boxOfCells(moved), flicker: true });
+ }
+ run = [];
+ };
+ for (const h of hits) {
+ if (run.length > 0 && h.i - run[run.length - 1].i > 2) {
+ close();
+ }
+ run.push(h);
+ }
+ close();
+ return out;
}
/** The fine cells that differ between two grids past the hard tolerance. */
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index cfb3e0ee..e0af2dd2 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -358,7 +358,7 @@ Someone points at a video — a file they imported, a link you brought in, a cli
**Lay down the sound, then block out the cuts.** Cuts timed to a reference are timed to its sound: detach_audio with the reference's \`assetId\` puts its whole track on the soundtrack and none of its picture. add_clip with \`blocks\` lays the skeleton from \`shots\` in one call — one block per shot on track 0, each its shot's \`from\` and \`to\` as read so every cut lands on its frame, flash cuts included, \`color\` what sample_color reads behind it and \`label\` naming what the person drops there. The label rides the timeline chip; the frame shows the colour alone. Footage dropped on a block, or replace_item, takes its place and its length, so everything built on top keeps its timing. When the ask is the reference's own picture playing, add_clip \`spans\` lays its shots down instead, one span per shot at those same times.
-**Fill the shots from Pexels, when asked.** The reference's own frames never go in the cut; Pexels' free footage does. Search it with WebFetch — the one web tool you have, and it reaches only Pexels: https://www.pexels.com/search/videos/<words>/?orientation=landscape (portrait for a 9:16 cut), asking for every result's link, id and title. Search per shot by what it shows — the subject and what they wear, the action, the angle, the light, the time of day — and look again with other words when a page has nothing close; one creator's series often covers a whole template, so a good hit's neighbours are worth a search of their own. import_url \`urls\` takes the video page links (pexels.com/video/<slug>-<id>/) and brings each down as its 1080p file, the Pexels id in its name. Watch the candidates beside the reference shot and keep the one that matches subject, angle and motion — a candidate is footage to judge, not a source to replicate: one scan pass over the stretch you would use settles it, and the whole-watch and word-for-word reading above are the reference's alone; replace_item puts it on that shot's block, trim_clip sets its in so the motion lands where the reference's does, set_framing zooms and pans to the reference's framing, and match_color_grade with ref_asset_id the reference and ref_at a moment inside the shot grades it to the reference's frame. Pexels files carry their own sound: when the reference's music is the soundtrack, set_clip_muted with \`ids\` mutes every footage clip in one call, so only the music plays. A shot whose camera moves — a push, a punch-in on the beat, a shake, a tilt — is measured with track_camera over that shot of the reference and keyed on its clip with set_clip_keyframes from the \`path\` and \`moves\` it returns. When it says \`still\`, the reference froze a frame and moved it: freeze_frame at a timeline second inside the shot's clip, for the shot's length, at that clip's index, delete_item the moving clip, and key the moves on the still. An overlay that shows the next shot's subject over the current one is that shot's clip again on a track above, its subject extracted with set_removal. Such a cutout is usually flashed in on its own picture, a frame at a time: a solid white silhouette, then the subject, then blown out, then the subject. Read those frames one by one, split the cutout a clip per frame, and give each its set_clip_effects flash: fade \"from\" at amount 1 for the silhouette, a pop at 0.6-0.9 for a blown-out frame, none for a plain one. A cutout that flickers between the subject and its negative every frame is one clip with a negative effect strobing at half the timeline's frame rate, which flips it each frame. Name every Pexels id you used when you close.
+**Fill the shots from Pexels, when asked.** The reference's own frames never go in the cut; Pexels' free footage does. Search it with WebFetch — the one web tool you have, and it reaches only Pexels: https://www.pexels.com/search/videos/<words>/?orientation=landscape (portrait for a 9:16 cut), asking for every result's link, id and title. Search per shot by what it shows — the subject and what they wear, the action, the angle, the light, the time of day — and look again with other words when a page has nothing close; one creator's series often covers a whole template, so a good hit's neighbours are worth a search of their own. import_url \`urls\` takes the video page links (pexels.com/video/<slug>-<id>/) and brings each down as its 1080p file, the Pexels id in its name. Watch the candidates beside the reference shot and keep the one that matches subject, angle and motion — a candidate is footage to judge, not a source to replicate: one scan pass over the stretch you would use settles it, and the whole-watch and word-for-word reading above are the reference's alone; replace_item puts it on that shot's block, trim_clip sets its in so the motion lands where the reference's does, set_framing zooms and pans to the reference's framing, and match_color_grade with ref_asset_id the reference and ref_at a moment inside the shot grades it to the reference's frame. Pexels files carry their own sound: when the reference's music is the soundtrack, set_clip_muted with \`ids\` mutes every footage clip in one call, so only the music plays. A shot whose camera moves — a push, a punch-in on the beat, a shake, a tilt — is measured with track_camera over that shot of the reference and keyed on its clip with set_clip_keyframes from the \`path\` and \`moves\` it returns. When it says \`still\`, the reference froze a frame and moved it: freeze_frame at a timeline second inside the shot's clip, for the shot's length, at that clip's index, delete_item the moving clip, and key the moves on the still. An overlay that shows the next shot's subject over the current one is that shot's clip again on a track above, its subject extracted with set_removal. Such a cutout is usually flashed in on its own picture, a frame at a time: a solid white silhouette, then the subject, then blown out, then the subject. Read those frames one by one, split the cutout a clip per frame, and give each its set_clip_effects flash: fade \"from\" at amount 1 for the silhouette, a pop at 0.6-0.9 for a blown-out frame, none for a plain one. A cutout that flickers between the subject and its negative every frame — detect_cuts marks such a run \`flicker\` — is one clip with a negative effect strobing at half the timeline's frame rate, which flips it each frame. Name every Pexels id you used when you close.
**Rebuild the text as elements.** One title per card, a stretch of cards per add_title call — the words exactly as they read, size in px at 1080 wide, and x/y where suggest_font sets it on that card — a title is set from the middle of its line, so the centre of its ink box (track_element, match ink, the text's colour) sits off it by the face's own offset. A run of lines timed one after another is add_text_sequence in one call. Match the case the reference uses, and its colour from sample_color. The face comes off the frame: suggest_font with the card's moment, a region close around its words and the words themselves ranks every face Donkey has and hands back the font, weight, size, tracking and x/y to set. Call it on each card: cards in one face share its weight, size and tracking, but each sits its own way in its space, so its x/y comes from its own read, never from a neighbour's place. A title that cycles through typefaces is one call per face it shows. A no_match verdict means the reference uses a face Donkey does not bundle: say so, and set the closest one it named. Without a suggestion, size comes off the ink: capitals stand about 0.7 of the size tall, so size is the ink height in px at 1080 wide ÷ 0.7 (h × 1920 ÷ 0.7 on 9:16). A cycle whose faces change on the music's beats times itself with set_overlay_animation font_cycle_beats on the soundtrack clip after detect_beats; one whose changes follow the reference's own rhythm takes font_cycle_switches, the timeline seconds detect_cuts reads each change at.