Open source CapCut alternative.
TypeScript
feat(site): name titles a sweep reads in another colour than the source's [rebuild]
David Le92adf4deadb5
diff --git a/site/src/cut/components/AiPanel.tools.ts b/site/src/cut/components/AiPanel.tools.tsindex eb35d4d2..755030a9 100644--- a/site/src/cut/components/AiPanel.tools.ts+++ b/site/src/cut/components/AiPanel.tools.ts@@ -202,6 +202,7 @@ export const AI_PANEL_TOOLS = ["Check the cut against the thing it was built from, in numbers and in pictures. Every frame is scored by `diff`: the mean difference per pixel between your frame and the source's, 0 (the same) to 255, read at one small size so a score means the same thing in every call; under " +`${SWEEP_TARGET} reads as the same picture. ` +"mode sweep scores every step of the cut (or from/to) with no pictures and returns the median, p95 and max with the worst stretches, each with its peak frame and a 4×4 grid of where in the frame it differs, and `edges`: stretches whose difference runs along whole edges of the frame while the middle matches — a border, band or letterbox — listed under target too, since thin bars barely move the mean. The picture box is already cut out of any app or bars around it, so a band inside it is part of the picture: build it, with shapes or a smaller block. " ++ "`titles` names each title whose letters read another colour than the source's under it (`ink` yours, `sourceInk` the source's, read at the title's middle), which a frame score at this size barely moves for. " +"A decoder that fails mid-read is replaced and the read goes on: `lost` names source frames no decoder read, and a source that stops decoding ends the sweep at `coveredTo` with `stopped` saying where and why. " +`mode moments (the default) takes up to ${MAX_COMPARE} \`times\` and hands back each as the source's frame and yours side by side, with its score and grid. ` +"Either mode says when a frame matches the source better a few frames away: `shift` is that distance, and a positive one means move what is there later by that much. " +diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.tsindex 8ca3a0ba..7cc18c76 100644--- a/site/src/cut/lib/aiTools.ts+++ b/site/src/cut/lib/aiTools.ts@@ -839,6 +839,7 @@ const SWEEP_NOTE ="diff is the mean difference per pixel between your frame and the source's, 0 (the same) to 255; median is the typical frame, p95 the bad tail, max the worst. " +"worst lists the stretches most off, worst first, each with its peak frame (at on the cut, source on the source) and that frame's 4×4 grid, top row first. " +"edges lists stretches whose difference runs along whole edges of the frame (top, bottom, left, right) while the middle matches: a border, band or letterbox one side has and the other lacks. Thin bars barely move the mean, so these come back under target too. The picture box is already cut out of any app around it, so a band inside it is part of the picture, never the recording's: build it. " ++ "titles lists titles whose letters read another colour than the source's under them, ink yours and sourceInk the source's: a small label in the wrong colour barely moves the score, so set each one's color to what the source draws. " +"Look at them: compare_to_source with times [{at, source}] at those peaks, fix what differs, then sweep again, until the median is under target and each stretch in worst and edges is fixed or is a difference no edit removes. " +"shift on a stretch, or timing for the whole span, means the source matches better that many seconds away: positive, move it later by that much; negative, earlier. " +"onBlocks is the share of samples over blocks standing in for footage; those keep a difference no edit removes.";diff --git a/site/src/cut/lib/devHooks.ts b/site/src/cut/lib/devHooks.tsindex 2149e2be..f1c3e555 100644--- a/site/src/cut/lib/devHooks.ts+++ b/site/src/cut/lib/devHooks.ts@@ -16,6 +16,7 @@ import { awaitingFrame, startTrace, stopTrace, traceReport } from "./perfTrace";import { markSignedBatch } from "./mediaLinks";import { prefetchCloudMedia } from "./mediaSync";import { playheadAt } from "./playhead";+import { sweepSource } from "./sourceCompare";import { projectDuration, useEditor } from "./store";export function installDevHooks(): void {@@ -49,6 +50,9 @@ export function installDevHooks(): void {// sees what the cut actually draws at a moment.renderProjectFrame,projectDuration,+ // A replica check sweeps the cut against its source the way+ // compare_to_source does, and reads the numbers back.+ sweepSource,// The export eval starts a real export through the store, the way the// dialog does, and watches the dock's feed for the file.useExports,diff --git a/site/src/cut/lib/frameDiff.test.ts b/site/src/cut/lib/frameDiff.test.tsindex f362f130..5e4649d2 100644--- a/site/src/cut/lib/frameDiff.test.ts+++ b/site/src/cut/lib/frameDiff.test.ts@@ -1,5 +1,5 @@import { expect, test } from "bun:test";-import { bestShift, diffPixels, edgeBand, meanDiff, runShift, sweepStats, worstStretches } from "./frameDiff";+import { bestShift, diffPixels, edgeBand, inkColor, inkDiff, inkOff, meanDiff, runShift, sweepStats, worstStretches } from "./frameDiff";// The numbers a replica is judged by. A score is only useful to the model if// it means the same thing every call: 0 for identical frames, 255 for black@@ -140,3 +140,55 @@ test("a difference along a whole edge while the middle matches is a band, under// A faint edge is noise.expect(edgeBand([[3, 3, 3, 3], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]], 5)).toBeNull();});++/** A w×h frame of `ground` with a `rgb` block of letters-like strokes at x 4..5, 8..9 of rows 3..5. */+const lettered = (w: number, h: number, ground: [number, number, number], rgb: [number, number, number]) => {+ const px = solid(w, h, ground);+ for (const x of [4, 5, 8, 9]) for (let y = 3; y <= 5; y++) px.set([...rgb, 255], (y * w + x) * 4);+ return px;+};++test("a title's ink is the colour its letters stand off the ground in", () => {+ const box = { x: 2, y: 2, w: 10, h: 5 };+ expect(inkColor(lettered(16, 9, [237, 105, 87], [0, 0, 0]), 16, 9, box)).toEqual({ ink: [0, 0, 0], ground: [237, 105, 87] });+ expect(inkColor(lettered(16, 9, [237, 105, 87], [255, 255, 255]), 16, 9, box)?.ink).toEqual([255, 255, 255]);+ // Nothing stands off a flat ground: no letters to read.+ expect(inkColor(solid(16, 9, [237, 105, 87]), 16, 9, box)).toBeNull();+});++test("white letters where the source has black ones are off; the same letters a shade apart aren't", () => {+ const box = { x: 0, y: 0, w: 16, h: 9 };+ const black = inkColor(lettered(16, 9, [237, 105, 87], [0, 0, 0]), 16, 9, box)!;+ const white = inkColor(lettered(16, 9, [237, 105, 87], [255, 255, 255]), 16, 9, box)!;+ const softer = inkColor(lettered(16, 9, [237, 105, 87], [20, 14, 12]), 16, 9, box)!;+ expect(inkOff(white, black)).toBeGreaterThan(150);+ expect(inkOff(softer, black)).toBeLessThan(30);+ expect(inkDiff(white.ink, black.ink)).toBe(255);+});++test("thinner letters of the same colour, blended into their ground, are not off", () => {+ // White on purple, and the source's white read through its thin strokes:+ // part way from the ground to white.+ const ours = { ink: [255, 255, 255] as [number, number, number], ground: [184, 3, 239] as [number, number, number] };+ const thin = { ink: [219, 155, 231] as [number, number, number], ground: [184, 3, 239] as [number, number, number] };+ expect(inkOff(ours, thin)).toBeLessThan(30);+});++test("tan letters where the source's are pale cyan are off, though both stand bright off black", () => {+ const tan = { ink: [204, 166, 151] as [number, number, number], ground: [0, 0, 0] as [number, number, number] };+ const cyan = { ink: [187, 231, 245] as [number, number, number], ground: [0, 0, 0] as [number, number, number] };+ expect(inkOff(tan, cyan)).toBeGreaterThan(64);+});++test("letters over another ground are left to the frame score", () => {+ // Ours reads the red card's edge off black; the source's white label sits on the card.+ const edge = { ink: [243, 47, 48] as [number, number, number], ground: [0, 0, 0] as [number, number, number] };+ const label = { ink: [230, 186, 185] as [number, number, number], ground: [242, 46, 48] as [number, number, number] };+ expect(inkOff(edge, label)).toBeNull();+});++test("a box past the frame's edge reads only what is inside it", () => {+ const px = lettered(16, 9, [0, 0, 0], [255, 255, 255]);+ expect(inkColor(px, 16, 9, { x: 6, y: -4, w: 20, h: 20 })?.ink).toEqual([255, 255, 255]);+ expect(inkColor(px, 16, 9, { x: 20, y: 20, w: 4, h: 4 })).toBeNull();+});diff --git a/site/src/cut/lib/frameDiff.ts b/site/src/cut/lib/frameDiff.tsindex b62d3d82..b36e2473 100644--- a/site/src/cut/lib/frameDiff.ts+++ b/site/src/cut/lib/frameDiff.ts@@ -248,3 +248,92 @@ export function edgeBand(grid: number[][], target: number): { edges: Edge[]; difconst cells = hot.flatMap(([, c]) => c);return { edges: hot.map(([e]) => e), diff: round1(cells.reduce((a, b) => a + b, 0) / cells.length) };}++/** How far a pixel stands off a title box's ground before it counts as ink. */+const INK_MIN = 30;++/** The share of a title box's pixels its ink is read from, the furthest off+ * the ground: the cores of the strokes, not their soft edges. */+const INK_SHARE = 0.08;++/** How much of the way from its ground to the other's ink a read may sit:+ * thinner or smaller letters read part way, never past the colour itself. */+const INK_BLEND = [0.3, 1.1] as const;++/** How far apart the grounds under a title may be for its letters to be+ * compared: past it the title sits on something else in one frame — a card's+ * edge, another shape — which is the frame score's to find, not the ink's. */+const GROUND_SLACK = 48;++type Rgb3 = [number, number, number];++/** A title's letters as a frame shows them: their colour and the ground's. */+export type Ink = { ink: Rgb3; ground: Rgb3 };++/** How far apart two colours are: the largest of their channel differences. */+export function inkDiff(a: Rgb3, b: Rgb3): number {+ return Math.max(Math.abs(a[0] - b[0]), Math.abs(a[1] - b[1]), Math.abs(a[2] - b[2]));+}++/**+ * How far the source's letters are from ours in colour. Letters a little+ * thinner, smaller or softer than ours read as our colour part way into their+ * ground, so the source's ink is held against the nearest blend of ours into+ * the source's ground: white letters read pale over purple are the same+ * white, black ones where ours are white are not. Null when the two sit on+ * different grounds.+ */+export function inkOff(cut: Ink, source: Ink): number | null {+ if (inkDiff(cut.ground, source.ground) > GROUND_SLACK) {+ return null;+ }+ const dir = cut.ink.map((v, c) => v - cut.ground[c]);+ const seen = source.ink.map((v, c) => v - source.ground[c]);+ const len = dir.reduce((a, v) => a + v * v, 0);+ const k = len > 0 ? Math.min(INK_BLEND[1], Math.max(INK_BLEND[0], dir.reduce((a, v, c) => a + v * seen[c], 0) / len)) : 1;+ return inkDiff(source.ground.map((v, c) => v + dir[c] * k) as Rgb3, source.ink);+}++/**+ * The colour a title's letters are drawn in, read off a frame: inside the+ * box, the ground is the median colour and the ink is the mean of the pixels+ * furthest from it. A title's box is mostly ground, so this holds on a flat+ * card and on footage alike, and it reads the letters' colour where a frame+ * score at a sweep's size barely moves for it. Null when nothing in the box+ * stands off the ground.+ */+export function inkColor(+ px: Uint8ClampedArray,+ w: number,+ h: number,+ box: { x: number; y: number; w: number; h: number }+): Ink | null {+ const x0 = Math.max(0, Math.floor(box.x));+ const y0 = Math.max(0, Math.floor(box.y));+ const x1 = Math.min(w, Math.ceil(box.x + box.w));+ const y1 = Math.min(h, Math.ceil(box.y + box.h));+ if (x1 <= x0 || y1 <= y0) {+ return null;+ }+ const at: number[] = [];+ const ch: number[][] = [[], [], []];+ for (let y = y0; y < y1; y++) {+ for (let x = x0; x < x1; x++) {+ const i = (y * w + x) * 4;+ at.push(i);+ for (let c = 0; c < 3; c++) ch[c].push(px[i + c]);+ }+ }+ const ground = ch.map((v) => v.sort((a, b) => a - b)[v.length >> 1]) as Rgb3;+ const off = at+ .map((i) => ({ i, d: inkDiff([px[i], px[i + 1], px[i + 2]], ground) }))+ .filter((p) => p.d > INK_MIN)+ .sort((a, b) => b.d - a.d)+ .slice(0, Math.max(1, Math.ceil(at.length * INK_SHARE)));+ if (off.length === 0) {+ return null;+ }+ const sum = [0, 0, 0];+ for (const { i } of off) for (let c = 0; c < 3; c++) sum[c] += px[i + c];+ return { ink: sum.map((v) => Math.round(v / off.length)) as Rgb3, ground };+}diff --git a/site/src/cut/lib/sourceCompare.ts b/site/src/cut/lib/sourceCompare.tsindex a518b23b..06490ebe 100644--- a/site/src/cut/lib/sourceCompare.ts+++ b/site/src/cut/lib/sourceCompare.ts@@ -1,14 +1,17 @@import { renderProjectFrames } from "./exportRender";import { FULL_FRAME, isFullFrame } from "./measure/picture";import type { FrameBox } from "./measure/sourceMeasure";-import { bestShift, diffPixels, edgeBand, meanDiff, runShift, SCORE_LONG_SIDE, SWEEP_TARGET, sweepStats, worstStretches, type Edge } from "./frameDiff";+import { bestShift, diffPixels, edgeBand, inkColor, inkOff, meanDiff, runShift, SCORE_LONG_SIDE, SWEEP_TARGET, sweepStats, worstStretches, type Edge } from "./frameDiff";+import { hexOf } from "./measure/color";import { composeComparison, makeStillFrame } from "./media";import type { WrappedCanvas } from "mediabunny";import { framesAt, type FrameSize } from "./mediaRead";import { createRasterCanvas, decodeRasterImageUrl, rasterCanvasToDataUrl, type RasterSurface } from "./raster";import type { ExportDoc } from "./renderSnapshot";import { clipLen } from "./store";-import { DEFAULT_BACKGROUND, frameOf, type MediaAsset } from "./types";+import { textBoxSize } from "./textFit";+import { DEFAULT_BACKGROUND, frameOf, isTextOverlay, type MediaAsset, type TextOverlay } from "./types";+import { poseAt } from "@donkeycut/effects-kit";// Holding a replica up against the video it copies, as pictures and as// numbers. The pictures are for the eye; the numbers say how far off each@@ -50,6 +53,14 @@ const READ_LOST_RUN = 3;* comes back once the contention passes. */const READ_RETRY_MS = 100;+/** How far a title's ink may sit from the source's (inkOff) before the sweep+ * names it: a white label where the source's is black reads near 230, the+ * same label a shade off or a little thinner under 30. */+const INK_TOLERANCE = 64;++/** The most titles one sweep names. */+const MAX_TITLES = 8;+/** Aspect ratios this close draw edge to edge; anything further letterboxes* or fills, as the call asks. */const ASPECT_SLACK = 0.01;@@ -120,8 +131,8 @@ const readHeight = (area: FrameBox, height: number) => Math.round(height / area.* full-bleed replica in another shape fills its frame — and the scorer* remembers it did so the result can say so.*/-function makeScorer(doc: ExportDoc, area: FrameBox, fit: SourceFit) {- const size = sizeOf(doc, SCORE_LONG_SIDE);+function makeScorer(doc: ExportDoc, area: FrameBox, fit: SourceFit, longSide = SCORE_LONG_SIDE) {+ const size = sizeOf(doc, longSide);const canvas = createRasterCanvas(size.width, size.height);const ctx = canvas.getContext("2d", { willReadFrequently: true }) as CanvasRenderingContext2D | null;if (!ctx) {@@ -407,6 +418,15 @@ type Stretch = {block?: string;};+export type TitleInk = {+ id: string;+ text: string;+ at: number;+ source: number;+ ink: string;+ sourceInk: string;+};+export type SweepResult = {mode: "sweep";from: number;@@ -423,6 +443,9 @@ export type SweepResult = {/** Stretches whose difference runs along whole edges of the frame while* the middle matches — a border or band one side lacks — under target too. */edges?: (Omit<Stretch, "shift"> & { edges: Edge[] })[];+ /** Titles whose letters read another colour than the source's under them,+ * each read once at the sample nearest its middle. */+ titles?: TitleInk[];timing?: { shift: number; diff: number };onBlocks?: number;frame?: string;@@ -473,6 +496,11 @@ export async function sweepSource(const size = sizeOf(doc, SWEEP_LONG_SIDE);const scorer = makeScorer(doc, area, fit);+ // Titles read at the render's own size, where letters still have cores.+ const reads = titleReads(doc, samples);+ const fine = reads.size > 0 ? makeScorer(doc, area, fit, SWEEP_LONG_SIDE) : null;+ const fineSources = new Map<number, Uint8ClampedArray>();+ const titles: (TitleInk & { off: number })[] = [];const width = SHIFT_REACH * 2 + 1;const started = performance.now();const outOfTime = (j: number) => j > 0 && performance.now() - started > SWEEP_BUDGET_MS;@@ -487,12 +515,19 @@ export async function sweepSource(const still = asset.type === "image" ? (await stillPixels(asset, scorer, area)).pixels : null;if (still) {sources.push(...samples.map(() => still));+ if (fine) {+ const px = (await stillPixels(asset, fine, area)).pixels;+ for (const j of reads.keys()) fineSources.set(j, px);+ }} else {try {for await (const c of steadyFrames(asset, samples.map((s) => s.source), { height: readHeight(area, size.height) }, lost)) {if (outOfTime(sources.length)) {break;}+ if (c && fine && reads.has(sources.length)) {+ fineSources.set(sources.length, fine.pixels(c.canvas, c.canvas.width, c.canvas.height));+ }sources.push(c ? scorer.pixels(c.canvas, c.canvas.width, c.canvas.height) : null);}} catch (e) {@@ -536,6 +571,20 @@ export async function sweepSource(}cuts.set(j, cut);cuts.delete(j - SHIFT_REACH);++ const fineSrc = fineSources.get(j);+ if (fine && fineSrc) {+ const fineCut = fine.cut(canvas, canvas.width, canvas.height);+ for (const o of reads.get(j) ?? []) {+ const box = titleBox(doc, o, samples[j].at, fine.size);+ const ink = inkColor(fineCut, fine.size.width, fine.size.height, box);+ const sourceInk = inkColor(fineSrc, fine.size.width, fine.size.height, box);+ const off = ink && sourceInk ? inkOff(ink, sourceInk) : null;+ if (off !== null && off > INK_TOLERANCE) {+ titles.push({ id: o.id, text: o.text, at: samples[j].at, source: samples[j].source, ink: hexOf(...ink!.ink), sourceInk: hexOf(...sourceInk!.ink), off });+ }+ }+ }});} catch (e) {if (!(e instanceof SweepOutOfTime)) {@@ -603,6 +652,9 @@ export async function sweepSource(target: SWEEP_TARGET,worst,...(edges.length > 0 ? { edges } : {}),+ ...(titles.length > 0+ ? { titles: titles.sort((a, b) => b.off - a.off).slice(0, MAX_TITLES).map(({ off: _off, ...t }) => t) }+ : {}),...(timing ? { timing } : {}),...(onBlocks > 0 ? { onBlocks: Math.round(onBlocks * 100) / 100 } : {}),...(frame ? { frame } : {}),@@ -610,3 +662,37 @@ export async function sweepSource(...(stopped ? { stopped: `${stopped.message} Cut ${samples[stopped.index].at}s.` } : {}),};}++/** The titles a sweep reads the ink of, by the sample each is read at: the+ * one nearest the title's middle, so it is read settled, past its way in. */+function titleReads(doc: ExportDoc, samples: { at: number }[]): Map<number, TextOverlay[]> {+ const reads = new Map<number, TextOverlay[]>();+ const first = samples[0].at;+ const last = samples[samples.length - 1].at;+ for (const o of doc.overlays) {+ if (!isTextOverlay(o) || o.hidden || !o.text.trim() || o.end <= first || o.start > last) {+ continue;+ }+ const mid = clamp((o.start + o.end) / 2, Math.max(first, o.start), Math.min(last, o.end));+ let j = 0;+ for (let i = 1; i < samples.length; i++) {+ if (Math.abs(samples[i].at - mid) < Math.abs(samples[j].at - mid)) j = i;+ }+ if (samples[j].at < o.start || samples[j].at >= o.end || poseAt(o, samples[j].at - o.start).opacity < 0.5) {+ continue;+ }+ reads.set(j, [...(reads.get(j) ?? []), o]);+ }+ return reads;+}++/** Where a title sits in a frame of `size` at timeline second `at`. */+function titleBox(doc: ExportDoc, o: TextOverlay, at: number, size: { width: number; height: number }) {+ const design = frameOf(doc.aspect);+ const pose = poseAt(o, at - o.start);+ const box = textBoxSize(o, design.w, design.h);+ const k = (size.width / design.w) * pose.scale;+ const w = box.width * k;+ const h = box.height * k;+ return { x: pose.x * size.width - w / 2, y: pose.y * size.height - h / 2, w, h };+}diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.tsindex 62a6f349..0312bde5 100644--- a/site/src/cut/server/ai/catalog.ts+++ b/site/src/cut/server/ai/catalog.ts@@ -365,7 +365,7 @@ Someone points at a video — a file they imported, a link you brought in, a cli**Then the motion.** set_overlay_keyframes gives each card its own pose track — where it sits, at what scale, at what second. track_element over a move hands you the keys: each move's from/to, start/end and ease, with scale as w over the resting w. set_overlay_animation composes on top for the entrance, the exit, a loop, and what a line does while it holds; 0.3-0.6s reads as an entrance, and a per-letter style is what makes a card read as typed. A line whose letters land one at a time is a per-letter entrance read off detect_cuts' \`changes\`, and it is read in one call, never by eye from sheets: track_element with match letters, the line's \`text\`, a seed close around the landed line, \`from\` before the first letter shows and \`to\` just after the last one lands. Its \`entrance\` is the set_overlay_animation call — in_style, in_seconds, in_stagger, in_sequence, in_sequence_at, in_distance, in_ease, in_fade, in_blur — and \`starts\`, the source second the title's ramp begins; pass them as they come. Its \`crosses\` says whether the letters pass in front of or behind other ink on their way in — a smaller line above them — and so which of the two takes the lower row. motion_blur on the element (update_overlay) is for letters that streak along their fall; one that comes in soft is in_blur. add_shape, a stretch per call, is the rest of the graphics: bars, cards, rules, a block of color behind a line.-**Look at what you made.** A build that reads right from your own notes is still a guess until you measure it. compare_to_source does: mode sweep scores every frame of the cut against the source — 0 is the same picture, under ${SWEEP_TARGET} reads as one — and names the worst stretches with a grid of where each differs; mode moments shows those peaks beside the source's frames. Sweep as you build, since an error in the first card is in every card after it: look at the worst stretches, fix what differs, sweep again, and stop when the median is under ${SWEEP_TARGET} and no stretch stands out. A \`shift\` means the right thing lands early or late by that much. Read each picture for colour, the words and their face, where a thing sits, its size, and how far along a move is; a title the same colour as the block behind it reads as correct in the document and shows up here. A block keeps its footage's difference, so judge a shot by the cells over what you built.+**Look at what you made.** A build that reads right from your own notes is still a guess until you measure it. compare_to_source does: mode sweep scores every frame of the cut against the source — 0 is the same picture, under ${SWEEP_TARGET} reads as one — and names the worst stretches with a grid of where each differs; mode moments shows those peaks beside the source's frames. Sweep as you build, since an error in the first card is in every card after it: look at the worst stretches, fix what differs, sweep again, and stop when the median is under ${SWEEP_TARGET} and no stretch stands out. A \`shift\` means the right thing lands early or late by that much. Read each picture for colour, the words and their face, where a thing sits, its size, and how far along a move is; a title the same colour as the block behind it reads as correct in the document and shows up here, and a sweep's \`titles\` names each title whose letters are another colour than the source's. A block keeps its footage's difference, so judge a shot by the cells over what you built.**Say what plays.** Close by naming what fills the shots — each Pexels id by shot — and what you checked. A shell gets what to drop in: how many shots, how long each one runs, and that the titles and timing are built around them.`,