Open source CapCut alternative.
TypeScript
feat(cut): read a recorded source inside its picture [rebuild]
David Le798d611721a4
diff --git a/site/src/clients/chatgpt/server/guidance.ts b/site/src/clients/chatgpt/server/guidance.tsindex 4e09df06..5f0c5c66 100644--- a/site/src/clients/chatgpt/server/guidance.ts+++ b/site/src/clients/chatgpt/server/guidance.ts@@ -41,7 +41,7 @@ Inspect the project for asset ids, transcripts, trims, notes and existing coverawatch_video returns timestamped contact sheets, sceneChanges and coverage. Stamps are source seconds. Continue from coveredTo when truncated. For a request about the whole source, cover it until unwatchedSeconds reaches zero. For long footage, sweep coarsely to learn its structure, then inspect relevant spans closely. A source map or a transcript alone does not establish visual coverage. Use detail read or original to read text and match typography; scan is for structure. Check unreadSeconds and unreadFrom when legible coverage matters.After every watch, record what you observed through edit_project with note_source: source span, actions, cuts, and exact on-screen words. A further watch can be refused until those observations are recorded. Notes persist on the asset; returned contact sheets belong to this conversation. Read observed, recorded and unnoted to resume from existing evidence.listen_audio returns playable source audio. Use the transcript for words and listen for delivery, music and timing. Respect coveredTo; continue from unheardFrom until unheardSeconds reaches zero when judging a whole source. Record observations with note_source. A clip_id scopes listening to its source trim; from and to are source seconds.-Use signal measurements for editing decisions: detect_silence locates quiet spans; detect_beats returns a beat grid and timelineBeats for a clip; measure_level measures loudness, including integratedLufs and truePeakDbtp per clip and for the whole mix, and returns volumeToMatch against a target. read_color_stats measures exposure, tonal range and color before grading. compare_to_source checks the edit against a reference; capture_frame shows the composited picture. detect_cuts, sample_color and track_element measure a source's cuts, colours and moving elements as numbers. These read commands belong in inspect_project's commands array.+Use signal measurements for editing decisions: detect_silence locates quiet spans; detect_beats returns a beat grid and timelineBeats for a clip; measure_level measures loudness, including integratedLufs and truePeakDbtp per clip and for the whole mix, and returns volumeToMatch against a target. read_color_stats measures exposure, tonal range and color before grading. compare_to_source checks the edit against a reference; capture_frame shows the composited picture. detect_cuts, sample_color and track_element measure a source's cuts, colours and moving elements as numbers, inside the picture box find_picture keeps for a recorded or letterboxed source. These read commands belong in inspect_project's commands array.find_filler identifies transcript fillers; find_highlights ranks transcript spans. Their results need the relevant visual or audio check before cutting. Use edit_project for these commands unless list_commands marks them reads. Caption creation is a separate edit requested by the user.Plan cuts around complete thoughts and measured pauses. Split and delete in batches, read the resulting ids, then place clips to close intended gaps. Leave room around speech boundaries and run refine_speech_cuts with the requested pace. Listen around adjusted edges, widen a clipped word's trim, and check the changed cut visually. Source seconds drive watches, listens and trims; timeline seconds drive placement and splits. Use the clip timing data to map between them.Report only the coverage you inspected and the changes tool results confirm. render_preview produces a playable saved revision for the user to review.`,diff --git a/site/src/cut/components/AiPanel.tools.ts b/site/src/cut/components/AiPanel.tools.tsindex 56d89217..027a4762 100644--- a/site/src/cut/components/AiPanel.tools.ts+++ b/site/src/cut/components/AiPanel.tools.ts@@ -38,6 +38,24 @@ const frameBox = (what: string) => ({description: what,});+/** The part of a source's frame the source-reading tools read. */+const pictureArg = {+ anyOf: [+ { type: "string", enum: ["full"] },+ obj(+ {+ x: num("Centre x, fraction of the whole frame's width"),+ y: num("Centre y, fraction of the whole frame's height"),+ w: num("Width, fraction of the whole frame's width"),+ h: num("Height, fraction of the whole frame's height"),+ },+ ["x", "y", "w", "h"]+ ),+ ],+ description:+ 'Read inside this box of the source\'s frame, or "full" for the whole frame. Default: the picture box find_picture keeps on the source, else the whole frame. Every x, y, w, h the call takes and returns is a fraction of it.',+};+export const AI_PANEL_TOOLS = [{name: "get_state",@@ -174,7 +192,8 @@ export const AI_PANEL_TOOLS = ["This is how a replica is verified. The loop: sweep, look at the worst stretches' peaks in moments mode, fix what differs, sweep again — until the median is under " +`${SWEEP_TARGET} and no stretch stands out. ` +"Blocks standing in for footage keep a difference no edit removes, so read those by the grid cells over your titles and shapes. " +- "Pass `times` as timeline seconds, or {at, source} pairs when the cut sits at a different second than the source does; `offset` maps every bare second and the whole sweep (source second = timeline second + offset).",+ "Pass `times` as timeline seconds, or {at, source} pairs when the cut sits at a different second than the source does; `offset` maps every bare second and the whole sweep (source second = timeline second + offset). " ++ "Only the source's picture box is held against your frame (find_picture), so a reference recorded inside an app scores its picture alone.",inputSchema: obj({asset_id: str("Project asset id of the source the cut was built from"),@@ -193,6 +212,24 @@ export const AI_PANEL_TOOLS = [from: num("sweep: timeline second to start at (default 0)"),to: num("sweep: timeline second to stop at (default the cut's end)"),step: num("sweep: seconds between samples (default one frame, widened so a long span fits the call)"),+ picture: pictureArg,+ },+ ["asset_id"]+ ),+ },+ {+ name: "find_picture",+ description:+ "Find where a video's picture sits inside its frame: the part that changes over time, apart from letterbox bars or the app a screen recording shows it in — a phone recording of an editing app has its picture across the middle, with buttons and a timeline around it. Reads frames spread over the source (or from/to) and returns `picture`, the box as centre and size in frame fractions, its `aspect`, and `active`, the share of the box that moved. The box is kept on the source: detect_cuts, sample_color, track_element, suggest_font and compare_to_source read inside it from then on, and the x, y, w, h they take and return are fractions of it, so they land on the cut's frame as they are. detect_cuts finds it on its first read of a source. Pass `box` to set it yourself, or \"full\" for the whole frame, when the one found is off.",+ inputSchema: obj(+ {+ asset_id: str("Project asset id of the video (or image, with box)"),+ from: num("Source start s to read frames from (default 0)"),+ to: num("Source end s (default the source's end)"),+ box: {+ anyOf: [{ type: "string", enum: ["full"] }, frameBox("The picture's box")],+ description: 'Set the picture box yourself: centre and size as frame fractions, or "full" for the whole frame',+ },},["asset_id"]),@@ -200,12 +237,13 @@ export const AI_PANEL_TOOLS = [{name: "detect_cuts",description:- `Find every hard cut in a stretch of a video source, to the frame. Reads every frame between from and to at a few dozen pixels, scores how much of the picture changed since the frame before, and reports each frame that opens a new shot: \`t\` in SOURCE seconds to the millisecond (the new shot's first frame), its \`frame\` index at the source's \`fps\`, and \`change\`, the percent of the picture that changed — near 100 is a new shot, 10-30 a card or a block of type swapping over the same background. \`shots\` lists the stretches between the cuts with their length in frames. watch_video's sceneChanges come from samples a step apart and lose a shot shorter than the step; this reads every frame, so a five-frame shot in a fast montage is found. Run it before blocking out a reference: add_clip \`blocks\` takes one block per shot at that shot's length. Hard cuts only — a dissolve or a fade changes a little each frame and reads as none. Up to ${DETECT_CUTS_SPAN_MAX}s and ${DETECT_CUTS_MAX} cuts a call; when \`truncated\`, call again from \`coveredTo\`.`,+ `Find every hard cut in a stretch of a video source, to the frame. Reads every frame between from and to at a few dozen pixels, scores how much of the picture changed since the frame before, and reports each frame that opens a new shot: \`t\` in SOURCE seconds to the millisecond (the new shot's first frame), its \`frame\` index at the source's \`fps\`, and \`change\`, the percent of the picture that changed — near 100 is a new shot, 10-30 a card or a block of type swapping over the same background. \`shots\` lists the stretches between the cuts with their length in frames. watch_video's sceneChanges come from samples a step apart and lose a shot shorter than the step; this reads every frame, so a five-frame shot in a fast montage is found. Run it before blocking out a reference: add_clip \`blocks\` takes one block per shot at that shot's length. Hard cuts only — a dissolve or a fade changes a little each frame and reads as none. Up to ${DETECT_CUTS_SPAN_MAX}s and ${DETECT_CUTS_MAX} cuts a call; when \`truncated\`, call again from \`coveredTo\`. A source with no picture box yet has it found first (find_picture) and kept, and every frame is read inside it, so a card that changes a small part of a recorded frame still counts as the change to the picture it is.`,inputSchema: obj({asset_id: str("Project asset id of the video"),from: num("Source start s (default 0)"),to: num(`Source end s (default the source's end, at most ${DETECT_CUTS_SPAN_MAX}s past from)`),+ picture: pictureArg,},["asset_id"]),@@ -232,6 +270,7 @@ export const AI_PANEL_TOOLS = [),},colors: num(`Also list this many dominant colours per region, 2..${SAMPLE_COLORS_K_MAX} (default: none)`),+ picture: pictureArg,},["asset_id", "samples"]),@@ -254,6 +293,7 @@ export const AI_PANEL_TOOLS = [enum: [...TRACK_MATCHES],description: "shape: the one connected piece the seed sits on (default). ink: every pixel of the colour inside the seed or within — type",},+ picture: pictureArg,},["asset_id", "from", "seed"]),@@ -277,6 +317,7 @@ export const AI_PANEL_TOOLS = [),text: str("The words the region shows, as they read, with a line break where the type breaks"),count: num("How many faces to return, 1..10 (default 5)"),+ picture: pictureArg,},["asset_id", "region", "text"]),diff --git a/site/src/cut/lib/aiContext.ts b/site/src/cut/lib/aiContext.tsindex 5ac5ceaa..718a9f8e 100644--- a/site/src/cut/lib/aiContext.ts+++ b/site/src/cut/lib/aiContext.ts@@ -552,6 +552,9 @@ function describeState(: { status: "uploading", percent: Math.round(a.upload.progress * 100) },}: {}),+ // Where the picture sits inside a recorded frame (find_picture): the+ // box the measuring tools read inside, centre and size as fractions.+ ...(a.picture && (a.picture.w < 1 || a.picture.h < 1) ? { picture: a.picture } : {}),// The recording bound to this video (sync_audio): every clip of it// plays that file, recording second = video second + offset....(a.soundFromdiff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.tsindex 389ac53c..771e3f74 100644--- a/site/src/cut/lib/aiTools.ts+++ b/site/src/cut/lib/aiTools.ts@@ -240,7 +240,8 @@ import { bundledFaces, inkMaskOf, rankFaces } from "./fontMatch";import { cutFontMatch } from "./fontMatchSettings";import { COMPARE_MODES, MAX_COMPARE, SWEEP_TARGET } from "./frameDiff";import { compareMoments, sweepSource } from "./sourceCompare";-import { sampleColors, scanCuts, trackElement, type FrameBox } from "./measure/sourceMeasure";+import { findPicture, sampleColors, scanCuts, trackElement, type FrameBox } from "./measure/sourceMeasure";+import { FULL_FRAME, isFullFrame } from "./measure/picture";import { DETECT_CUTS_MAX, DETECT_CUTS_SPAN_MAX, SAMPLE_COLOR_MAX, SAMPLE_COLORS_K_MAX, TRACK_FRAMES_MAX, TRACK_MATCHES, TRACK_PATH_MAX, TRACK_TOLERANCE_DEFAULT } from "./measure/limits";import { renderStageFrame, storeStageStill } from "./stageFrame";import { createRasterCanvas, decodeRasterImageUrl, rasterCanvasToDataUrl } from "./raster";@@ -622,6 +623,41 @@ function trackPath(boxes: { t: number; box: FrameBox | null }[]) {};}+/** The part of a source's frame a reading tool reads: `picture` as passed+ * ("full" for the whole frame, or a box), else the picture box stored on the+ * asset, else the whole frame. */+function pictureArea(asset: MediaAsset, input: Record<string, unknown>): FrameBox {+ if (input.picture === "full") return FULL_FRAME;+ if (typeof input.picture === "string")+ throw new ToolError(`picture is a box {x, y, w, h} in frame fractions, or "full" for the whole frame — got "${input.picture}".`);+ return frameBoxOf(input.picture, "picture") ?? asset.picture ?? FULL_FRAME;+}++/** Find a video's picture over [from, to) and keep it on the asset — the+ * whole frame when nothing around a picture holds still. */+async function storePicture(asset: MediaAsset, from: number, to: number) {+ const found = await withSweepPaused(() => findPicture(asset.url, { from, to })).catch((e) => {+ throw new ToolError(e instanceof Error ? e.message : `Could not read "${asset.name}".`, { cause: e });+ });+ useEditor.getState().updateAsset(asset.id, { picture: found?.box ?? FULL_FRAME });+ return found;+}++/** The picture box a reading read inside, for its result: the box and what+ * its numbers are fractions of. Nothing when it read the whole frame. */+function pictureReport(area: FrameBox) {+ if (isFullFrame(area)) return {};+ return {+ picture: { x: rF(area.x), y: rF(area.y), w: rF(area.w), h: rF(area.h) },+ pictureNote: PICTURE_NOTE,+ };+}++/** What a reading inside a picture box says about its own numbers. */+const PICTURE_NOTE =+ "This read only the source's picture box (`picture`, centre and size as fractions of the whole recorded frame) — the part the picture fills, apart from the bars or app around it. " ++ 'Every x, y, w, h passed in or handed back here is a fraction of that box, so it lands on the cut\'s frame as it is. picture: "full" reads the whole recorded frame; find_picture sets the box.';+/** Frame fractions to the 1/10,000: a pixel of a 1080 frame is 0.0009. */const rF = (n: number) => Math.round(n * 10000) / 10000;/** The side of the box a point is read over, as a frame fraction. */@@ -1385,6 +1421,8 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {const failed = (fallback: string) => (e: unknown) => {throw new ToolError(e instanceof Error ? e.message : fallback, { cause: e });};+ // Only the source's picture box is held against the cut's frame.+ const area = pictureArea(asset, input);// The sweep: numbers over a span, at a step, under a time budget.if (input.mode === "sweep") {@@ -1397,10 +1435,12 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {step: isNum(input.step) ? input.step : undefined,offset,},- total+ total,+ area).catch(failed(`Could not sweep "${asset.name}".`));return {...sweep,+ ...pictureReport(area),source: sourceRef(asset),note:(sweep.coveredTo !== undefined@@ -1423,9 +1463,10 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {throw new ToolError("Pass times: [seconds] — the moments of the cut to check, or [{at, source}] when the cut sits at a different second than the source does. To score the whole cut first, pass mode: \"sweep\".");if (moments.length > MAX_COMPARE)throw new ToolError(`That is ${moments.length} moments — ${MAX_COMPARE} is the most one call reads. Check these, fix what is off, then call again for the rest; mode: "sweep" scores every frame without pictures.`);- const out = await compareMoments(doc, asset, moments, total).catch(failed("Could not compare the frames."));+ const out = await compareMoments(doc, asset, moments, total, area).catch(failed("Could not compare the frames."));return {...out,+ ...pictureReport(area),// The same shape every look returns, so a record opened by a// comparison knows what the other senses may ask of this source.source: sourceRef(asset),@@ -1433,6 +1474,48 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {};},+ find_picture: async (s, input) => {+ // Where the picture sits inside the source's frame: found from what+ // changes over time, or set as the model sees it. Kept on the asset,+ // where every source-reading tool reads inside it from then on.+ const asset = measuredSource(s, input, "picture");+ let found: Awaited<ReturnType<typeof findPicture>> = null;+ let box: FrameBox;+ if (input.box === "full") {+ box = FULL_FRAME;+ } else if (input.box !== undefined && input.box !== null) {+ box = frameBoxOf(input.box, "box")!;+ } else {+ if (asset.type === "image")+ throw new ToolError(`"${asset.name}" is a still — nothing in it changes to find its picture by. Pass box: the picture's {x, y, w, h} as you see it.`);+ const from = clamp(isNum(input.from) ? input.from : 0, 0, asset.duration);+ const to = isNum(input.to) ? clamp(input.to, from, asset.duration) : asset.duration;+ if (!(to > from)) throw new ToolError("from/to describe an empty range of the source.");+ found = await storePicture(asset, from, to);+ box = found?.box ?? FULL_FRAME;+ }+ useEditor.getState().updateAsset(asset.id, { picture: box });++ // Its size in the source's pixels, when the source's size is known.+ const pw = asset.width ? box.w * asset.width : 0;+ const ph = asset.height ? box.h * asset.height : 0;+ const full = isFullFrame(box);+ return {+ picture: { x: rF(box.x), y: rF(box.y), w: rF(box.w), h: rF(box.h) },+ full,+ ...(found ? { active: found.active } : {}),+ ...(pw > 0 && ph > 0 ? { pixels: { width: Math.round(pw), height: Math.round(ph) }, aspect: round2(pw / ph) } : {}),+ source: sourceRef(asset),+ note:+ input.box === undefined && !found+ ? "Too little of the frame changes to tell the picture from what is around it, so the tools read the whole frame. Pass box once you can see where the picture sits (watch_video at detail original)."+ : full+ ? "The picture fills the whole frame; the tools read all of it."+ : "The picture fills this box of the recorded frame (centre and size as frame fractions). detect_cuts, sample_color, track_element, suggest_font and compare_to_source read inside it from now on, and their x, y, w, h are fractions of it, so they land on the cut's frame as they are. " ++ "aspect is its shape; set_aspect to match when the copy should. A box that is off — check it on a watch_video frame at detail original — is set with box.",+ };+ },+detect_cuts: async (s, input) => {// Every frame of the span, scored against the one before: the cuts a// sampled watch can step over, each on its own frame.@@ -1441,8 +1524,21 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {const asked = isNum(input.to) ? clamp(input.to, 0, asset.duration) : asset.duration;const to = Math.min(asked, from + DETECT_CUTS_SPAN_MAX);if (!(to > from)) throw new ToolError("from/to describe an empty range of the source.");+ // A source read for the first time has its picture found and kept, so+ // a change that fills the picture counts in full when the picture is a+ // small part of a recorded frame.+ if (input.picture === undefined && !asset.picture) {+ await storePicture(asset, 0, asset.duration);+ }+ const area = pictureArea(useEditor.getState().assets.find((a) => a.id === asset.id) ?? asset, input);const scan = await withSweepPaused(() =>- scanCuts(asset.url, { from, to, maxCuts: DETECT_CUTS_MAX, budgetMs: MEASURE_BUDGET_MS })+ scanCuts(asset.url, {+ from,+ to,+ maxCuts: DETECT_CUTS_MAX,+ budgetMs: MEASURE_BUDGET_MS,+ ...(isFullFrame(area) ? {} : { area }),+ })).catch((e) => {throw new ToolError(e instanceof Error ? e.message : `Could not read "${asset.name}".`, { cause: e });});@@ -1466,6 +1562,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {truncated,cuts: scan.cuts.map((c) => ({ t: roundT(c.t), frame: c.frame, change: c.change })),shots,+ ...pictureReport(area),source: sourceRef(asset),note:(scan.cuts.length === 0 ? "No hard cut in this span. " : "") +@@ -1498,7 +1595,8 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {return { t, box, point };});const k = isNum(input.colors) ? clamp(Math.round(input.colors), 1, SAMPLE_COLORS_K_MAX) : 1;- const read = await withSweepPaused(() => sampleColors(asset.url, asset.type === "image" ? "image" : "video", samples, k)).catch((e) => {+ const area = pictureArea(asset, input);+ const read = await withSweepPaused(() => sampleColors(asset.url, asset.type === "image" ? "image" : "video", samples, k, area)).catch((e) => {throw new ToolError(e instanceof Error ? e.message : `Could not read "${asset.name}".`, { cause: e });});return {@@ -1509,6 +1607,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {...(smp.point ? {} : { w: rF(smp.box.w), h: rF(smp.box.h) }),...read[i],})),+ ...pictureReport(area),source: sourceRef(asset),};},@@ -1526,8 +1625,10 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {const end = Math.max(0, asset.duration - 0.001);const from = asset.type === "video" ? clamp(isNum(input.from) ? input.from : 0, 0, end) : 0;const to = asset.type === "video" && isNum(input.to) && input.to > from ? clamp(input.to, 0, asset.duration) : undefined;+ const area = pictureArea(asset, input);const run = await withSweepPaused(() =>trackElement(asset.url, asset.type === "image" ? "image" : "video", {+ area,from,...(to !== undefined ? { to } : {}),seed,@@ -1565,6 +1666,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {error: Math.round(m.error * 1000) / 1000,})),...(to !== undefined ? { coveredTo: roundT(run.coveredTo), truncated: run.truncated } : {}),+ ...pictureReport(area),source: sourceRef(asset),};},@@ -1765,6 +1867,15 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {// Frame fractions, quietest first: where a title or caption can go// over this stretch without covering the picture or the platform's UI....(spots.length > 0 ? { textFits: spots } : {}),+ // The picture box found or set on this source, so what the sheets+ // show and what the measuring tools read are told apart.+ ...(asset.picture && !isFullFrame(asset.picture)+ ? {+ picture: { x: rF(asset.picture.x), y: rF(asset.picture.y), w: rF(asset.picture.w), h: rF(asset.picture.h) },+ pictureNote:+ "The source's picture fills this box of its frame (centre and size, frame fractions); the sheets show the whole frame. detect_cuts, sample_color, track_element, suggest_font and compare_to_source read inside the box and speak in its fractions.",+ }+ : {}),note: reference? tiled +"This is a look at a reference project for deciding which of this project's sources plays its part — nothing is written to the reference and no note is owed. " +@@ -1993,10 +2104,14 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {fw = frame.canvas.width;fh = frame.canvas.height;}- const x0 = clamp(r.x as number, 0, 1) * fw;- const y0 = clamp(r.y as number, 0, 1) * fh;- const sw = Math.max(1, Math.min(fw - x0, (r.w as number) * fw));- const sh = Math.max(1, Math.min(fh - y0, (r.h as number) * fh));+ // The region is in the picture box's fractions; the crop is the frame's.+ const area = pictureArea(asset, input);+ const left = area.x - area.w / 2;+ const top = area.y - area.h / 2;+ const x0 = (left + clamp(r.x as number, 0, 1) * area.w) * fw;+ const y0 = (top + clamp(r.y as number, 0, 1) * area.h) * fh;+ const sw = Math.max(1, Math.min(fw - x0, (r.w as number) * area.w * fw));+ const sh = Math.max(1, Math.min(fh - y0, (r.h as number) * area.h * fh));const k = Math.min(1, FONT_CROP_MAX / Math.max(sw, sh));const cw = Math.max(1, Math.round(sw * k));const ch = Math.max(1, Math.round(sh * k));@@ -2021,7 +2136,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {weight: m.weight,...(m.italic ? { italic: true } : {}),score: m.score,- size: m.typePx > 0 ? Math.round((m.typePx / k / Math.min(fw, fh)) * DESIGN_SHORT_SIDE) : null,+ size: m.typePx > 0 ? Math.round((m.typePx / k / Math.min(fw * area.w, fh * area.h)) * DESIGN_SHORT_SIDE) : null,tracking: m.tracking,}));const good = cutFontMatch().goodScore;@@ -2031,11 +2146,12 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {t: round2(t),text,ink: {- x: round3((x0 + ref.box.x / k) / fw),- y: round3((y0 + ref.box.y / k) / fh),- w: round3(ref.box.w / k / fw),- h: round3(ref.box.h / k / fh),+ x: round3(((x0 + ref.box.x / k) / fw - left) / area.w),+ y: round3(((y0 + ref.box.y / k) / fh - top) / area.h),+ w: round3(ref.box.w / k / fw / area.w),+ h: round3(ref.box.h / k / fh / area.h),},+ ...pictureReport(area),verdict: best && best.score >= good ? "match" : "no_match",matches,note:@@ -6051,6 +6167,7 @@ export const MEDIA_RUNTIME_TOOLS: ReadonlySet<string> = new Set(["sync_audio","capture_frame","compare_to_source",+ "find_picture","detect_cuts","sample_color","track_element",diff --git a/site/src/cut/lib/compareToSource.test.ts b/site/src/cut/lib/compareToSource.test.tsindex 762724a5..1f366241 100644--- a/site/src/cut/lib/compareToSource.test.ts+++ b/site/src/cut/lib/compareToSource.test.ts@@ -222,6 +222,35 @@ test("a source of another shape is scored letterboxed into the cut's frame, andexpect(out.frame).toContain("letterboxed");});+test("a recording's picture box is what is held against the cut, read at the cut's size", async () => {+ // A 9:16 phone recording with a 16:9 picture across its middle, against a+ // 16:9 cut: the box is the cut's shape, so nothing is letterboxed.+ useEditor.setState({ aspect: "16:9" });+ useEditor.getState().updateAsset("ref", { picture: { x: 0.5, y: 0.5, w: 1, h: 0.3164 } });+ const out = (await runAiTool("compare_to_source", { asset_id: "ref", times: [1] })) as {+ frame?: string;+ picture?: { h: number };+ pictureNote?: string;+ };+ expect(out.frame).toBeUndefined();+ expect(out.picture?.h).toBe(0.3164);+ expect(out.pictureNote).toContain("fraction");+ // The frame is read tall enough that its picture box is the cut's height.+ const height = (reads[0].size as { height: number }).height;+ expect(height * 0.3164).toBeCloseTo(270, -1);++ const sweep = (await runAiTool("compare_to_source", { asset_id: "ref", mode: "sweep", from: 0, to: 1, step: 0.5 })) as {+ frame?: string;+ picture?: unknown;+ };+ expect(sweep.frame).toBeUndefined();+ expect(sweep.picture).toBeDefined();++ // "full" scores the whole recorded frame, letterboxed.+ const whole = (await runAiTool("compare_to_source", { asset_id: "ref", times: [1], picture: "full" })) as { frame?: string };+ expect(whole.frame).toContain("letterboxed");+});+type Sweep = {images?: unknown;sampled: number;diff --git a/site/src/cut/lib/measure/picture.test.ts b/site/src/cut/lib/measure/picture.test.tsnew file mode 100644index 00000000..44f108be--- /dev/null+++ b/site/src/cut/lib/measure/picture.test.ts@@ -0,0 +1,105 @@+import { expect, test } from "bun:test";+import type { Pixels } from "./color";+import { areaOf, findPictureBox, intoArea } from "./picture";++// A source's picture is the part of its frame that changes over time. These+// frames are painted from a seed so each sample shows a different shot inside+// the picture, while whatever is around it — bars, an app's buttons — holds+// still, the way a screen recording of an editing app does.++/** A deterministic 0..255 noise from three integers. */+const noise = (a: number, b: number, c: number) => {+ let h = (a * 374761393 + b * 668265263 + c * 2147483647) | 0;+ h = Math.imul(h ^ (h >>> 13), 1274126177);+ return (h ^ (h >>> 16)) & 255;+};++type Painter = (x: number, y: number, f: number) => number;++function frames(w: number, h: number, n: number, paint: Painter): Pixels[] {+ return Array.from({ length: n }, (_, f) => {+ const data = new Uint8ClampedArray(w * h * 4);+ for (let y = 0; y < h; y++) {+ for (let x = 0; x < w; x++) {+ const v = paint(x, y, f);+ data.set([v, v, v, 255], (y * w + x) * 4);+ }+ }+ return { data, width: w, height: h };+ });+}++/** Each shot is a smooth field of its own, so neighbouring pixels agree the+ * way footage does. */+const shot = (x: number, y: number, f: number) => noise(Math.floor(x / 6), Math.floor(y / 6), f + 1);++test("a phone recording of an editing app reads as the picture in its middle", () => {+ // 90×160 (9:16). The picture is rows 26–83 (a 16:9 strip across the full+ // width); a timeline strip scrolls at rows 120–131; a clock ticks in a+ // corner; everything else is static chrome with buttons on it.+ const W = 90;+ const H = 160;+ const px = frames(W, H, 24, (x, y, f) => {+ if (y >= 26 && y < 84) return shot(x, y, f);+ if (y >= 120 && y < 132) return noise(Math.floor((x + f * 7) / 5), 3, 0);+ if (y < 6 && x > 70) return noise(x, y, f);+ return noise(Math.floor(x / 15), Math.floor(y / 15), 99) > 128 ? 230 : 40;+ });+ const found = findPictureBox(px)!;+ expect(found.full).toBe(false);+ expect(found.box.x).toBeCloseTo(0.5, 2);+ expect(found.box.w).toBeCloseTo(1, 2);+ expect(found.box.y * H).toBeCloseTo(55, 0);+ expect(found.box.h * H).toBeCloseTo(58, 0);+});++test("a pillarboxed picture reads as the box between the bars", () => {+ const W = 160;+ const H = 90;+ const px = frames(W, H, 16, (x, y, f) => (x >= 40 && x < 120 ? shot(x, y, f) : 0));+ const { box, full } = findPictureBox(px)!;+ expect(full).toBe(false);+ expect(box.x).toBeCloseTo(0.5, 2);+ expect(box.w * W).toBeCloseTo(80, 0);+ expect(box.h).toBeCloseTo(1, 2);+});++test("a picture that fills the frame reads as the whole frame", () => {+ const px = frames(64, 36, 12, shot);+ const { box, full } = findPictureBox(px)!;+ expect(full).toBe(true);+ expect(box).toEqual({ x: 0.5, y: 0.5, w: 1, h: 1 });+});++test("a picture with a static logo inside it keeps its whole box", () => {+ const W = 90;+ const H = 160;+ const px = frames(W, H, 16, (x, y, f) => {+ if (y < 40 || y >= 100) return 20;+ if (x >= 60 && x < 80 && y >= 45 && y < 55) return 255;+ return shot(x, y, f);+ });+ const { box } = findPictureBox(px)!;+ expect(box.y * H).toBeCloseTo(70, 0);+ expect(box.h * H).toBeCloseTo(60, 0);+ expect(box.w).toBeCloseTo(1, 2);+});++test("a source that never changes has no picture to find", () => {+ expect(findPictureBox(frames(32, 32, 8, (x, y) => noise(x, y, 0)))).toBeNull();+ expect(findPictureBox(frames(32, 32, 1, shot))).toBeNull();+});++test("a box inside the picture maps onto the whole frame and back", () => {+ const picture = { x: 0.5, y: 0.3, w: 1, h: 0.32 };+ const inside = { x: 0.25, y: 0.5, w: 0.1, h: 0.2 };+ const onFrame = intoArea(picture, inside);+ expect(onFrame.x).toBeCloseTo(0.25, 6);+ expect(onFrame.y).toBeCloseTo(0.3, 6);+ expect(onFrame.h).toBeCloseTo(0.064, 6);+ const back = areaOf(picture, onFrame);+ expect(back.x).toBeCloseTo(inside.x, 6);+ expect(back.y).toBeCloseTo(inside.y, 6);+ expect(back.w).toBeCloseTo(inside.w, 6);+ expect(back.h).toBeCloseTo(inside.h, 6);+});diff --git a/site/src/cut/lib/measure/picture.ts b/site/src/cut/lib/measure/picture.tsnew file mode 100644index 00000000..a6c3e84d--- /dev/null+++ b/site/src/cut/lib/measure/picture.ts@@ -0,0 +1,219 @@+/** Where a source's picture sits inside its frame. A reference is often a+ * recording of something else — a phone filming an editing app, a letterboxed+ * film — and only part of its frame is the picture being copied: the part+ * that changes over time. Bars, buttons and app chrome hold still. Frames+ * sampled across the source are reduced to a mask of the pixels that moved,+ * and the picture is the box of the largest connected stretch of them.+ * Pure data in, pure data out; sourceMeasure.ts feeds it decoded frames. */++import type { Pixels } from "./color";+import type { FrameBox } from "./sourceMeasure";++export const PICTURE_TUNING = {+ /** Luma range across the samples for a pixel to count as moving. */+ RANGE_MIN: 32,+ /** Samples a moving pixel changed in, each by STEP_MIN from the one before,+ * so a button that toggles once stays chrome. */+ MOVES_MIN: 2,+ STEP_MIN: 16,+ /** Pixels per side of the cells the connected stretch is found on. */+ CELL: 4,+ /** Share of a cell's pixels that must move for the cell to count. */+ CELL_SHARE: 0.25,+ /** Share of a row or column that must move for it to stay inside the box,+ * so a thin line touching the picture is trimmed off it. */+ EDGE_SHARE: 0.3,+ /** An edge this close to the frame's edge, as a fraction, is the frame's. */+ EDGE_SNAP: 0.015,+ /** The smallest picture, as a share of the frame. */+ AREA_MIN: 0.05,+ /** Fewer samples than this cannot show what changes. */+ SAMPLES_MIN: 3,+};++/** The whole frame as a box. */+export const FULL_FRAME: FrameBox = { x: 0.5, y: 0.5, w: 1, h: 1 };++export interface PictureFind {+ box: FrameBox;+ /** The box is the whole frame: nothing around the picture. */+ full: boolean;+ /** Share of the box's pixels that moved: near 1 is a busy picture, a low+ * share a picture that held still most of the time. */+ active: number;+}++/** A box given in `area`'s own fractions, as fractions of the whole frame. */+export const intoArea = (area: FrameBox, b: FrameBox): FrameBox => ({+ x: area.x - area.w / 2 + b.x * area.w,+ y: area.y - area.h / 2 + b.y * area.h,+ w: b.w * area.w,+ h: b.h * area.h,+});++/** A box on the whole frame, as fractions of `area`. */+export const areaOf = (area: FrameBox, b: FrameBox): FrameBox => ({+ x: (b.x - (area.x - area.w / 2)) / area.w,+ y: (b.y - (area.y - area.h / 2)) / area.h,+ w: b.w / area.w,+ h: b.h / area.h,+});++/** Whether a box is the whole frame. */+export const isFullFrame = (b: FrameBox): boolean =>+ Math.abs(b.x - 0.5) < 1e-6 && Math.abs(b.y - 0.5) < 1e-6 && b.w >= 1 - 1e-6 && b.h >= 1 - 1e-6;++/** The pixels that moved across `frames`, all of one size: 1 where moving. */+function movingMask(frames: Pixels[]): Uint8Array {+ const { RANGE_MIN, MOVES_MIN, STEP_MIN } = PICTURE_TUNING;+ const n = frames[0].width * frames[0].height;+ const lo = new Uint8Array(n).fill(255);+ const hi = new Uint8Array(n);+ const prev = new Uint8Array(n);+ const moves = new Uint16Array(n);+ frames.forEach((f, k) => {+ for (let p = 0; p < n; p++) {+ const i = p * 4;+ const l = (f.data[i] * 77 + f.data[i + 1] * 150 + f.data[i + 2] * 29) >> 8;+ if (l < lo[p]) {+ lo[p] = l;+ }+ if (l > hi[p]) {+ hi[p] = l;+ }+ if (k > 0 && Math.abs(l - prev[p]) >= STEP_MIN) {+ moves[p]++;+ }+ prev[p] = l;+ }+ });+ const mask = new Uint8Array(n);+ for (let p = 0; p < n; p++) {+ mask[p] = hi[p] - lo[p] >= RANGE_MIN && moves[p] >= MOVES_MIN ? 1 : 0;+ }+ return mask;+}++/** The cells of the largest 8-connected stretch of moving cells, as a cell+ * box, or null when no cell moved. */+function largestStretch(cells: Uint8Array, cw: number, ch: number) {+ const seen = new Uint8Array(cells.length);+ let best: { x0: number; y0: number; x1: number; y1: number; size: number } | null = null;+ const stack: number[] = [];+ for (let start = 0; start < cells.length; start++) {+ if (!cells[start] || seen[start]) {+ continue;+ }++ // Flood the stretch this cell opens, keeping its bounds and size.+ const run = { x0: cw, y0: ch, x1: -1, y1: -1, size: 0 };+ seen[start] = 1;+ stack.push(start);+ while (stack.length > 0) {+ const c = stack.pop()!;+ const cx = c % cw;+ const cy = (c - cx) / cw;+ run.size++;+ run.x0 = Math.min(run.x0, cx);+ run.x1 = Math.max(run.x1, cx);+ run.y0 = Math.min(run.y0, cy);+ run.y1 = Math.max(run.y1, cy);+ for (let dy = -1; dy <= 1; dy++) {+ for (let dx = -1; dx <= 1; dx++) {+ const nx = cx + dx;+ const ny = cy + dy;+ if (nx < 0 || ny < 0 || nx >= cw || ny >= ch) {+ continue;+ }+ const nc = ny * cw + nx;+ if (cells[nc] && !seen[nc]) {+ seen[nc] = 1;+ stack.push(nc);+ }+ }+ }+ }+ if (!best || run.size > best.size) {+ best = run;+ }+ }+ return best;+}++/**+ * The box a source's picture fills, from frames sampled across it. The+ * largest connected stretch of moving cells is the picture; its box is then+ * trimmed row by row and column by column to where the picture's own pixels+ * move, so its edges land on pixels. Null when too little moved to tell.+ */+export function findPictureBox(frames: Pixels[]): PictureFind | null {+ const { CELL, CELL_SHARE, EDGE_SHARE, EDGE_SNAP, AREA_MIN, SAMPLES_MIN } = PICTURE_TUNING;+ if (frames.length < SAMPLES_MIN) {+ return null;+ }+ const W = frames[0].width;+ const H = frames[0].height;+ const mask = movingMask(frames);++ // The mask on a coarse grid, so noise inside the picture never splits it.+ const cw = Math.ceil(W / CELL);+ const ch = Math.ceil(H / CELL);+ const counts = new Uint16Array(cw * ch);+ for (let y = 0; y < H; y++) {+ for (let x = 0; x < W; x++) {+ counts[Math.floor(y / CELL) * cw + Math.floor(x / CELL)] += mask[y * W + x];+ }+ }+ const cells = new Uint8Array(cw * ch);+ for (let c = 0; c < cells.length; c++) {+ cells[c] = counts[c] >= CELL * CELL * CELL_SHARE ? 1 : 0;+ }+ const stretch = largestStretch(cells, cw, ch);+ if (!stretch) {+ return null;+ }++ // The stretch's box in pixels, a cell wider each side, then trimmed from+ // each edge while the row (or column) is mostly still.+ let x0 = Math.max(0, (stretch.x0 - 1) * CELL);+ let x1 = Math.min(W, (stretch.x1 + 2) * CELL);+ let y0 = Math.max(0, (stretch.y0 - 1) * CELL);+ let y1 = Math.min(H, (stretch.y1 + 2) * CELL);+ const rowShare = (y: number) => {+ let n = 0;+ for (let x = x0; x < x1; x++) n += mask[y * W + x];+ return n / Math.max(1, x1 - x0);+ };+ const colShare = (x: number) => {+ let n = 0;+ for (let y = y0; y < y1; y++) n += mask[y * W + x];+ return n / Math.max(1, y1 - y0);+ };+ while (y0 < y1 && rowShare(y0) < EDGE_SHARE) y0++;+ while (y1 > y0 && rowShare(y1 - 1) < EDGE_SHARE) y1--;+ while (x0 < x1 && colShare(x0) < EDGE_SHARE) x0++;+ while (x1 > x0 && colShare(x1 - 1) < EDGE_SHARE) x1--;+ if ((x1 - x0) * (y1 - y0) < W * H * AREA_MIN) {+ return null;+ }++ // Edges within a hair of the frame's are the frame's.+ if (x0 <= W * EDGE_SNAP) x0 = 0;+ if (y0 <= H * EDGE_SNAP) y0 = 0;+ if (x1 >= W * (1 - EDGE_SNAP)) x1 = W;+ if (y1 >= H * (1 - EDGE_SNAP)) y1 = H;+ let moving = 0;+ for (let y = y0; y < y1; y++) {+ for (let x = x0; x < x1; x++) moving += mask[y * W + x];+ }+ const active = Math.round((moving / ((x1 - x0) * (y1 - y0))) * 100) / 100;+ const full = x0 === 0 && y0 === 0 && x1 === W && y1 === H;+ if (full) {+ return { box: { ...FULL_FRAME }, full, active };+ }+ return {+ box: { x: (x0 + x1) / 2 / W, y: (y0 + y1) / 2 / H, w: (x1 - x0) / W, h: (y1 - y0) / H },+ full,+ active,+ };+}diff --git a/site/src/cut/lib/measure/sourceMeasure.ts b/site/src/cut/lib/measure/sourceMeasure.tsindex 22c12df1..34c0d59b 100644--- a/site/src/cut/lib/measure/sourceMeasure.ts+++ b/site/src/cut/lib/measure/sourceMeasure.ts@@ -11,6 +11,7 @@ import type { WrappedCanvas } from "mediabunny";import { hexOf, regionColor, parseHex, type PixelRect, type Pixels, type Rgb } from "./color";import { changePct, gridMeans, pickCuts } from "./cuts";import { fitMoves, type Move } from "./easeFit";+import { FULL_FRAME, areaOf, findPictureBox, intoArea, type PictureFind } from "./picture";import { inkBox, shapeBox } from "./track";/** A box as the editor places one: centre and size, frame fractions. */@@ -30,6 +31,10 @@ const COLOR_LONG_SIDE = 1920;const TRACK_LONG_SIDE = 720;/** Reads a pass back off by this much so the frame on `from` is scored. */const LEAD_S = 0.001;+/** Long side the picture box is found at: a pixel is 0.16% of the frame. */+const PICTURE_LONG_SIDE = 640;+/** Frames sampled across a source to find its picture. */+const PICTURE_SAMPLES = 48;/** An open video source: its display size, first timestamp and frame reads. */interface SourceVideo {@@ -95,12 +100,27 @@ async function imagePixels(url: string, longSide: number): Promise<Pixels> {return { data: image.data, width: w, height: h };}-/** The read size for a source at `longSide`, never past its own pixels. */-function sizeAt(width: number, height: number, longSide: number): FrameSize {- const k = Math.min(1, longSide / Math.max(width, height, 1));+/** The read size for a source whose `area` is `longSide` on its long side,+ * never past its own pixels. */+function sizeAt(width: number, height: number, longSide: number, area: FrameBox = FULL_FRAME): FrameSize {+ const k = Math.min(1, longSide / Math.max(width * area.w, height * area.h, 1));return { width: Math.max(1, Math.round(width * k)), height: Math.max(1, Math.round(height * k)), fit: "fill" };}+/** The pixels of `px` inside `r`, as a picture of their own. */+function cropPixels(px: Pixels, r: PixelRect): Pixels {+ const x0 = Math.max(0, Math.min(px.width - 1, Math.round(r.x)));+ const y0 = Math.max(0, Math.min(px.height - 1, Math.round(r.y)));+ const w = Math.max(1, Math.min(px.width - x0, Math.round(r.w)));+ const h = Math.max(1, Math.min(px.height - y0, Math.round(r.h)));+ const data = new Uint8ClampedArray(w * h * 4);+ for (let y = 0; y < h; y++) {+ const from = ((y0 + y) * px.width + x0) * 4;+ data.set(px.data.subarray(from, from + w * 4), y * w * 4);+ }+ return { data, width: w, height: h };+}+/** A centre-and-size box in frame fractions, as pixels of `px`. */const toPixels = (px: Pixels, b: FrameBox): PixelRect => ({x: (b.x - b.w / 2) * px.width,@@ -140,21 +160,29 @@ export interface CutScan {* budget, and `coveredTo` says where the next pass picks up. */export async function scanCuts(url: string,- opts: { from: number; to: number; maxCuts: number; budgetMs: number }+ opts: { from: number; to: number; maxCuts: number; budgetMs: number; area?: FrameBox }): Promise<CutScan> {return withSourceVideo(url, async (v) => {+ // Inside a picture box, the frame is read big enough that the box alone+ // is the scan's size: a change that fills the picture fills the grid.+ const area = opts.area ?? FULL_FRAME;+ const size: FrameSize = {+ width: Math.min(v.width, Math.round(SCAN_SIZE.width! / area.w)),+ height: Math.min(v.height, Math.round(SCAN_SIZE.height! / area.h)),+ fit: "fill",+ };const deadline = Date.now() + opts.budgetMs;const times: number[] = [];const durations: number[] = [];const scores: number[] = [];let prev: Float32Array | null = null;let outOfTime = false;- for await (const f of v.frames(Math.max(0, opts.from - LEAD_S), opts.to, SCAN_SIZE)) {+ for await (const f of v.frames(Math.max(0, opts.from - LEAD_S), opts.to, size)) {if (Date.now() > deadline) {outOfTime = true;break;}- const grid = gridMeans(f);+ const grid = gridMeans(opts.area ? cropPixels(f, toPixels(f, area)) : f);scores.push(prev ? changePct(prev, grid) : NaN);times.push(f.t);durations.push(f.duration);@@ -194,15 +222,18 @@ export async function sampleColors(url: string,kind: "video" | "image",samples: ColorSample[],- k: number+ k: number,+ area: FrameBox = FULL_FRAME): Promise<ReturnType<typeof regionColor>[]> {+ // Each sample's box is in the area's fractions; the reads are the frame's.+ samples = samples.map((s) => ({ ...s, box: intoArea(area, s.box) }));if (kind === "image") {const px = await imagePixels(url, COLOR_LONG_SIDE);return samples.map((s) => regionColor(px, toPixels(px, s.box), k));}const times = [...new Set(samples.map((s) => s.t))].sort((a, b) => a - b);return withSourceVideo(url, async (v) => {- const size = sizeAt(v.width, v.height, COLOR_LONG_SIDE);+ const size = sizeAt(v.width, v.height, COLOR_LONG_SIDE, area);const out = new Array<ReturnType<typeof regionColor>>(samples.length);let at = 0;for await (const f of v.framesAt(times, size)) {@@ -245,8 +276,11 @@ export async function trackElement(tolerance: number;match: TrackMatch;maxFrames: number;+ /** The picture box the seed, within and every box out are fractions of. */+ area?: FrameBox;}): Promise<TrackRun> {+ const area = opts.area ?? FULL_FRAME;let truncated = false;let coveredTo = opts.to ?? opts.from;@@ -255,16 +289,16 @@ export async function trackElement(let near: PixelRect | null = null;const boxes: TrackRun["boxes"] = [];const read = (t: number, px: Pixels) => {- const seed = toPixels(px, opts.seed);+ const seed = toPixels(px, intoArea(area, opts.seed));if (!target) target = parseHex(regionColor(px, seed).hex)!;- const area = opts.within ? toPixels(px, opts.within) : opts.match === "ink" ? seed : { x: 0, y: 0, w: px.width, h: px.height };+ const look = opts.within ? toPixels(px, intoArea(area, opts.within)) : opts.match === "ink" ? seed : toPixels(px, area);const found =opts.match === "ink"- ? inkBox(px, target, opts.tolerance, area)- : (shapeBox(px, target, opts.tolerance, area, near ?? seed) ??- (near ? shapeBox(px, target, opts.tolerance, area, seed) : null));+ ? inkBox(px, target, opts.tolerance, look)+ : (shapeBox(px, target, opts.tolerance, look, near ?? seed) ??+ (near ? shapeBox(px, target, opts.tolerance, look, seed) : null));if (found) near = found;- boxes.push({ t, box: found ? toFrame(px, found) : null });+ boxes.push({ t, box: found ? areaOf(area, toFrame(px, found)) : null });};let size = { width: 1, height: 1 };@@ -276,7 +310,7 @@ export async function trackElement(read(0, px);} else {await withSourceVideo(url, async (v) => {- const at = sizeAt(v.width, v.height, TRACK_LONG_SIDE);+ const at = sizeAt(v.width, v.height, TRACK_LONG_SIDE, area);size = { width: at.width!, height: at.height! };if (opts.to === undefined || opts.to <= opts.from) {for await (const f of v.framesAt([opts.from], at)) {@@ -304,7 +338,7 @@ export async function trackElement(if (boxes.length > 2) {const ts = boxes.map((b) => b.t);for (const prop of ["x", "y", "w", "h"] as const) {- const step = 1 / (prop === "x" || prop === "w" ? size.width : size.height);+ const step = 1 / (prop === "x" || prop === "w" ? size.width * area.w : size.height * area.h);for (const m of fitMoves(ts, boxes.map((b) => (b.box ? b.box[prop] : null)), step)) moves.push({ ...m, prop });}moves.sort((a, b) => a.from - b.from);@@ -319,3 +353,18 @@ export async function trackElement(truncated,};}++/** The box a video's picture fills inside its frame, found from frames+ * sampled evenly over [from, to): the part that changes, apart from bars and+ * an app's chrome around it. Null when too little moves to tell. */+export async function findPicture(url: string, opts: { from: number; to: number }): Promise<PictureFind | null> {+ return withSourceVideo(url, async (v) => {+ const span = Math.max(0, opts.to - opts.from);+ const times = Array.from({ length: PICTURE_SAMPLES }, (_, i) => opts.from + ((i + 0.5) * span) / PICTURE_SAMPLES);+ const frames: Pixels[] = [];+ for await (const f of v.framesAt(times, sizeAt(v.width, v.height, PICTURE_LONG_SIDE))) {+ if (f) frames.push(f);+ }+ return findPictureBox(frames);+ });+}diff --git a/site/src/cut/lib/pi/mutationLedger.ts b/site/src/cut/lib/pi/mutationLedger.tsindex c57f9e24..64ea841f 100644--- a/site/src/cut/lib/pi/mutationLedger.ts+++ b/site/src/cut/lib/pi/mutationLedger.ts@@ -13,6 +13,7 @@ const READ_ONLY = new Set(["request_tools","capture_frame","compare_to_source",+ "find_picture","detect_cuts","sample_color","track_element",diff --git a/site/src/cut/lib/pictureTools.test.ts b/site/src/cut/lib/pictureTools.test.tsnew file mode 100644index 00000000..9e958474--- /dev/null+++ b/site/src/cut/lib/pictureTools.test.ts@@ -0,0 +1,197 @@+import { afterAll, beforeEach, expect, spyOn, test } from "bun:test";+import type { Input, InputVideoTrack, WrappedCanvas } from "mediabunny";+import * as mediaRead from "./mediaRead";+import { storedAssets, useEditor } from "./store";++// A reference recorded off a phone screen: a 9:16 frame of an editing app with+// the 16:9 picture across its middle, buttons and bars still around it. The+// source-reading tools find that picture, keep it on the asset, read only+// inside it, and speak in its fractions so their numbers land on the cut.++const FPS = 60;+const FRAMES = 120; // 2 s+const W = 1080;+const H = 1920;+/** The picture's rows, as frame fractions: 16:9 across the full width. */+const TOP = 0.342;+const BOTTOM = 0.658;++type Rgb = [number, number, number];+const RED: Rgb = [200, 30, 30];+const BLUE: Rgb = [30, 30, 200];+const GREEN: Rgb = [30, 160, 60];+const WHITE: Rgb = [255, 255, 255];+const CHROME: Rgb = [40, 40, 44];+const BUTTON: Rgb = [220, 220, 220];++/** Picture fractions of the card that lands at frame 90: 30% wide, 40% tall. */+const CARD = { x0: 0.35, x1: 0.65, y0: 0.3, y1: 0.7 };++/** The colour at frame f and frame fractions (fx, fy). Inside the picture:+ * red, then blue from frame 30, then green from 60, with a white card over+ * the green from frame 90 — a change to a small share of the whole frame. */+function colourAt(f: number, fx: number, fy: number): Rgb {+ if (fy < TOP || fy >= BOTTOM) {+ const button = (fx > 0.1 && fx < 0.2 && fy > 0.8 && fy < 0.85) || (fx > 0.7 && fx < 0.9 && fy > 0.05 && fy < 0.1);+ return button ? BUTTON : CHROME;+ }+ const px = fx;+ const py = (fy - TOP) / (BOTTOM - TOP);+ if (f < 30) return RED;+ if (f < 60) return BLUE;+ if (f >= 90 && px >= CARD.x0 && px < CARD.x1 && py >= CARD.y0 && py < CARD.y1) return WHITE;+ return GREEN;+}++function frameCanvas(f: number, w: number, h: number) {+ const data = new Uint8ClampedArray(w * h * 4);+ for (let y = 0; y < h; y++) {+ for (let x = 0; x < w; x++) {+ const [r, g, b] = colourAt(f, (x + 0.5) / w, (y + 0.5) / h);+ const i = (y * w + x) * 4;+ data[i] = r;+ data[i + 1] = g;+ data[i + 2] = b;+ data[i + 3] = 255;+ }+ }+ const ctx = { getImageData: () => ({ data, width: w, height: h }), drawImage() {}, putImageData() {} };+ return { width: w, height: h, getContext: () => ctx } as unknown as HTMLCanvasElement;+}++const wrap = (f: number, w: number, h: number) =>+ ({ canvas: frameCanvas(f, w, h), timestamp: f / FPS, duration: 1 / FPS }) as WrappedCanvas;+const frameShownAt = (t: number) => Math.max(0, Math.min(FRAMES - 1, Math.floor(t * FPS + 1e-6)));++/** The sizes frames were read at, newest last. */+const sizes: { width: number; height: number }[] = [];+const input = { dispose() {} } as unknown as Input;+const track = {+ getDisplayWidth: async () => W,+ getDisplayHeight: async () => H,+ getFirstTimestamp: async () => 0,+ displayWidth: W,+ displayHeight: H,+} as unknown as InputVideoTrack;+const spies = [+ spyOn(mediaRead, "openMedia").mockReturnValue(input),+ spyOn(mediaRead, "videoTrackOf").mockResolvedValue(track),+ spyOn(mediaRead, "frameSink").mockImplementation((_track, size) => {+ const w = size?.width ?? W;+ const h = size?.height ?? H;+ sizes.push({ width: w, height: h });+ return {+ getCanvas: async (t: number) => wrap(frameShownAt(t), w, h),+ async *canvases(start = 0, end = Infinity) {+ for (let f = frameShownAt(start); f < FRAMES && f / FPS < end; f++) yield wrap(f, w, h);+ },+ async *canvasesAtTimestamps(times: Iterable<number> | AsyncIterable<number>) {+ for await (const t of times) yield wrap(frameShownAt(t), w, h);+ },+ };+ }),+];+afterAll(() => {+ for (const spy of spies) spy.mockRestore();+});++const { runAiTool } = await import("./aiTools");++beforeEach(() => {+ sizes.length = 0;+ useEditor.setState({+ assets: [{ id: "ref", fileName: "ref.mp4", name: "Recording", type: "video", duration: FRAMES / FPS, url: "fixture.mp4", width: W, height: H }],+ });+});++const message = async (p: Promise<unknown>) =>+ p.then(() => "", (e: unknown) => (e instanceof Error ? e.message : String(e)));+const stored = () => useEditor.getState().assets.find((a) => a.id === "ref")!.picture;++type Box = { x: number; y: number; w: number; h: number };++test("find_picture finds the picture inside a recording and keeps it on the asset", async () => {+ const out = (await runAiTool("find_picture", { asset_id: "ref" })) as { picture: Box; full: boolean; aspect: number; note: string };+ expect(out.full).toBe(false);+ expect(out.picture.x).toBeCloseTo(0.5, 2);+ expect(out.picture.w).toBeCloseTo(1, 2);+ expect(out.picture.y).toBeCloseTo((TOP + BOTTOM) / 2, 2);+ expect(out.picture.h).toBeCloseTo(BOTTOM - TOP, 2);+ expect(out.aspect).toBeCloseTo(16 / 9, 1);+ expect(out.note).toContain("fractions");++ // Stored as a source fact, so it saves with the project.+ expect(stored()!.h).toBeCloseTo(BOTTOM - TOP, 2);+ expect(storedAssets(useEditor.getState().assets)[0].picture).toEqual(stored());+});++test("find_picture takes a box the model sets, the whole frame included", async () => {+ await runAiTool("find_picture", { asset_id: "ref", box: { x: 0.5, y: 0.4, w: 1, h: 0.3 } });+ expect(stored()).toEqual({ x: 0.5, y: 0.4, w: 1, h: 0.3 });+ const out = (await runAiTool("find_picture", { asset_id: "ref", box: "full" })) as { full: boolean };+ expect(out.full).toBe(true);+ expect(stored()).toEqual({ x: 0.5, y: 0.5, w: 1, h: 1 });+});++test("detect_cuts finds the picture first, then a card that changes a small share of the frame", async () => {+ const out = (await runAiTool("detect_cuts", { asset_id: "ref" })) as {+ cuts: { frame: number }[];+ picture: Box;+ pictureNote: string;+ };+ expect(out.cuts.map((c) => c.frame)).toEqual([30, 60, 90]);+ expect(out.picture.h).toBeCloseTo(BOTTOM - TOP, 2);+ expect(out.pictureNote).toContain("picture");+ expect(stored()).toBeDefined();++ // Over the whole recorded frame, the card is too small a change to call.+ const whole = (await runAiTool("detect_cuts", { asset_id: "ref", picture: "full" })) as { cuts: { frame: number }[]; picture?: Box };+ expect(whole.cuts.map((c) => c.frame)).toEqual([30, 60]);+ expect(whole.picture).toBeUndefined();+});++test("sample_color reads in the picture's fractions once a picture is stored", async () => {+ await runAiTool("find_picture", { asset_id: "ref" });+ const out = (await runAiTool("sample_color", {+ asset_id: "ref",+ samples: [+ { t: 1.2, x: 0.5, y: 0.1 },+ { t: 1.6, x: 0.5, y: 0.5 },+ ],+ })) as { samples: { hex: string; x: number; y: number }[]; picture: Box };+ // y 0.1 of the picture is green footage; of the whole frame it is chrome.+ expect(out.samples.map((s) => s.hex)).toEqual(["#1EA03C", "#FFFFFF"]);+ expect(out.samples[0].y).toBe(0.1);+ expect(out.picture).toBeDefined();+ const whole = (await runAiTool("sample_color", { asset_id: "ref", picture: "full", samples: [{ t: 1.2, x: 0.5, y: 0.1 }] })) as {+ samples: { hex: string }[];+ };+ expect(whole.samples[0].hex).toBe("#28282C");+});++test("track_element measures a card in the picture's fractions", async () => {+ await runAiTool("find_picture", { asset_id: "ref" });+ const out = (await runAiTool("track_element", {+ asset_id: "ref",+ from: 1.6,+ seed: { x: 0.5, y: 0.5, w: 0.05, h: 0.05 },+ })) as { path: Box[] };+ const [box] = out.path;+ expect(box.x).toBeCloseTo(0.5, 2);+ expect(box.y).toBeCloseTo(0.5, 2);+ expect(box.w).toBeCloseTo(CARD.x1 - CARD.x0, 2);+ expect(box.h).toBeCloseTo(CARD.y1 - CARD.y0, 2);+});++test("a picture passed on the call wins over the stored one", async () => {+ useEditor.getState().updateAsset("ref", { picture: { x: 0.5, y: 0.5, w: 1, h: 1 } });+ const out = (await runAiTool("sample_color", {+ asset_id: "ref",+ picture: { x: 0.5, y: (TOP + BOTTOM) / 2, w: 1, h: BOTTOM - TOP },+ samples: [{ t: 1.2, x: 0.5, y: 0.1 }],+ })) as { samples: { hex: string }[] };+ expect(out.samples[0].hex).toBe("#1EA03C");+ expect(await message(runAiTool("sample_color", { asset_id: "ref", picture: "half", samples: [{ t: 1, x: 0.5, y: 0.5 }] }))).toContain(+ "full"+ );+});diff --git a/site/src/cut/lib/sourceCompare.ts b/site/src/cut/lib/sourceCompare.tsindex de11338e..916f3145 100644--- a/site/src/cut/lib/sourceCompare.ts+++ b/site/src/cut/lib/sourceCompare.ts@@ -1,8 +1,10 @@import { renderProjectFrames } from "./exportRender";+import { FULL_FRAME, isFullFrame } from "./measure/picture";+import type { FrameBox } from "./measure/sourceMeasure";import { bestShift, diffPixels, meanDiff, runShift, SCORE_LONG_SIDE, SWEEP_TARGET, sweepStats, worstStretches } from "./frameDiff";import { composeComparison, makeStillFrame } from "./media";import { framesAt } from "./mediaRead";-import { createRasterCanvas, decodeRasterImageUrl, rasterCanvasToDataUrl } from "./raster";+import { createRasterCanvas, decodeRasterImageUrl, rasterCanvasToDataUrl, type RasterSurface } from "./raster";import type { ExportDoc } from "./renderSnapshot";import { clipLen } from "./store";import { DEFAULT_BACKGROUND, frameOf, type MediaAsset } from "./types";@@ -64,13 +66,42 @@ function blockAt(doc: ExportDoc, t: number): string | undefined {return undefined;}+/** `area` of a `w`×`h` frame, in its pixels. */+function areaPx(area: FrameBox, w: number, h: number) {+ return { x: (area.x - area.w / 2) * w, y: (area.y - area.h / 2) * h, w: area.w * w, h: area.h * h };+}++/** A source frame's picture box as a JPEG for the side-by-side: the frame+ * itself when the box is the whole of it. */+async function pictureUrl(area: FrameBox, source: CanvasImageSource, w: number, h: number): Promise<string> {+ if (isFullFrame(area)) {+ return rasterCanvasToDataUrl(source as RasterSurface, "image/jpeg", 0.82);+ }+ const r = areaPx(area, w, h);+ const canvas = createRasterCanvas(Math.max(1, Math.round(r.w)), Math.max(1, Math.round(r.h)));+ const ctx = canvas.getContext("2d") as CanvasRenderingContext2D | null;+ if (!ctx) {+ throw new Error("No 2d context to crop the source's picture on.");+ }+ ctx.drawImage(source, r.x, r.y, r.w, r.h, 0, 0, canvas.width, canvas.height);+ try {+ return await rasterCanvasToDataUrl(canvas, "image/jpeg", 0.82);+ } finally {+ canvas.width = 0;+ }+}++/** The read height that gives the source's picture box `height` rows. */+const readHeight = (area: FrameBox, height: number) => Math.round(height / area.h);+/**- * Reads any frame down to the score's size, in the cut's shape. A source of- * another shape is letterboxed into it over the cut's background, the way a- * clip of that shape would sit on the timeline, and the scorer remembers it- * did so the result can say so.+ * Reads any frame down to the score's size, in the cut's shape. Only the+ * source's picture box is read — the part of its frame the cut copies. A+ * picture of another shape is letterboxed into the cut's frame over its+ * background, the way a clip of that shape would sit on the timeline, and the+ * scorer remembers it did so the result can say so.*/-function makeScorer(doc: ExportDoc) {+function makeScorer(doc: ExportDoc, area: FrameBox) {const size = sizeOf(doc, SCORE_LONG_SIDE);const canvas = createRasterCanvas(size.width, size.height);const ctx = canvas.getContext("2d", { willReadFrequently: true }) as CanvasRenderingContext2D | null;@@ -87,42 +118,55 @@ function makeScorer(doc: ExportDoc) {get letterboxed() {return letterboxed;},- pixels(source: CanvasImageSource, w: number, h: number): Uint8ClampedArray {- ctx.fillStyle = doc.background ?? DEFAULT_BACKGROUND;- ctx.fillRect(0, 0, size.width, size.height);- if (Math.abs(w / h / (size.width / size.height) - 1) <= ASPECT_SLACK) {- ctx.drawImage(source, 0, 0, size.width, size.height);- } else {- letterboxed = { w, h };- const k = Math.min(size.width / w, size.height / h);- const dw = w * k;- const dh = h * k;- ctx.drawImage(source, (size.width - dw) / 2, (size.height - dh) / 2, dw, dh);- }- return ctx.getImageData(0, 0, size.width, size.height).data;+ /** A source frame, read inside its picture box. */+ pixels(source: CanvasImageSource, fw: number, fh: number): Uint8ClampedArray {+ return draw(source, areaPx(area, fw, fh));+ },+ /** One of the cut's own frames, read whole. */+ cut(source: CanvasImageSource, fw: number, fh: number): Uint8ClampedArray {+ return draw(source, { x: 0, y: 0, w: fw, h: fh });},};++ function draw(source: CanvasImageSource, r: { x: number; y: number; w: number; h: number }): Uint8ClampedArray {+ ctx!.fillStyle = doc.background ?? DEFAULT_BACKGROUND;+ ctx!.fillRect(0, 0, size.width, size.height);+ if (Math.abs(r.w / r.h / (size.width / size.height) - 1) <= ASPECT_SLACK) {+ ctx!.drawImage(source, r.x, r.y, r.w, r.h, 0, 0, size.width, size.height);+ } else {+ letterboxed = { w: r.w, h: r.h };+ const k = Math.min(size.width / r.w, size.height / r.h);+ const dw = r.w * k;+ const dh = r.h * k;+ ctx!.drawImage(source, r.x, r.y, r.w, r.h, (size.width - dw) / 2, (size.height - dh) / 2, dw, dh);+ }+ return ctx!.getImageData(0, 0, size.width, size.height).data;+ }}/** The line a result carries when the source was letterboxed to score it. */-function frameNote(doc: ExportDoc, shape: { w: number; h: number } | undefined): string | undefined {+function frameNote(doc: ExportDoc, shape: { w: number; h: number } | undefined, area: FrameBox): string | undefined {if (!shape) {return undefined;}return (- `The source's frame is ${round1(shape.w / shape.h)}:1 and the cut's is ${doc.aspect}, so scores read the source letterboxed into the cut's frame over its background. ` ++ `The source's ${isFullFrame(area) ? "frame" : "picture box"} is ${round1(shape.w / shape.h)}:1 and the cut's is ${doc.aspect}, so scores read the source letterboxed into the cut's frame over its background. ` +"Bars the cut does not have count as difference; set_aspect to the source's shape when the copy should match it.");}-/** A still source's one frame, as the scorer reads it. */-async function stillPixels(asset: MediaAsset, scorer: ReturnType<typeof makeScorer>) {+/** A still source's one frame, as the scorer reads it, and its picture box+ * as the side-by-side shows it. */+async function stillPixels(asset: MediaAsset, scorer: ReturnType<typeof makeScorer>, area: FrameBox) {const still = await makeStillFrame(asset.url, "read");const img = await decodeRasterImageUrl(still.frames[0].image);if (!img) {throw new Error(`Could not read "${asset.name}".`);}- return { image: still.frames[0].image, pixels: scorer.pixels(img.source, img.width, img.height) };+ const image = isFullFrame(area)+ ? still.frames[0].image+ : await pictureUrl(area, img.source, img.width, img.height);+ return { image, pixels: scorer.pixels(img.source, img.width, img.height) };}/** One moment checked: where it was read on each clock, the score, the grid,@@ -139,10 +183,11 @@ export async function compareMoments(doc: ExportDoc,asset: MediaAsset,moments: { at: number; sourceAt: number }[],- total: number+ total: number,+ area: FrameBox = FULL_FRAME): Promise<{ images: string[]; checked: Checked[]; frame?: string }> {const size = sizeOf(doc, COMPARE_LONG_SIDE);- const scorer = makeScorer(doc);+ const scorer = makeScorer(doc, area);const wanted = moments.map((m) => ({at: clamp(m.at, 0, Math.max(0, total - 0.001)),sourceAt: clamp(m.sourceAt, 0, Math.max(0, asset.duration - 0.001)),@@ -156,7 +201,7 @@ export async function compareMoments(const pixels = new Map<number, Uint8ClampedArray>();const pictures = new Map<number, string>();if (asset.type === "image") {- const still = await stillPixels(asset, scorer);+ const still = await stillPixels(asset, scorer, area);for (const m of wanted) {pixels.set(key(m.sourceAt), still.pixels);pictures.set(key(m.sourceAt), still.image);@@ -167,16 +212,18 @@ export async function compareMoments(.map((k) => k / 1000);// The source keeps its own aspect: a height alone lets a widescreen// reference stay widescreen beside a vertical cut, which is half of what- // the comparison is for. Asking for both dimensions squashes it.+ // the comparison is for. Asking for both dimensions squashes it. The+ // height is its picture box's, so a picture inside a bigger frame is+ // read at the size the cut's frame is.let seen = 0;- for await (const c of framesAt(asset.url, times, { height: size.height })) {+ for await (const c of framesAt(asset.url, times, { height: readHeight(area, size.height) })) {const k = key(times[seen++]);if (!c) {continue;}pixels.set(k, scorer.pixels(c.canvas, c.canvas.width, c.canvas.height));if (centers.has(k)) {- pictures.set(k, await rasterCanvasToDataUrl(c.canvas, "image/jpeg", 0.82));+ pictures.set(k, await pictureUrl(area, c.canvas, c.canvas.width, c.canvas.height));}}}@@ -193,7 +240,7 @@ export async function compareMoments(const checked: Checked[] = [];await renderProjectFrames(doc, wanted.map((m) => m.at), size, (a) => a.url, async (canvas, at) => {const m = wanted[pairs.length];- const cut = scorer.pixels(canvas, canvas.width, canvas.height);+ const cut = scorer.cut(canvas, canvas.width, canvas.height);const { diff, grid } = diffPixels(cut, pixels.get(key(m.sourceAt))!, scorer.size.width, scorer.size.height);const near: { shift: number; diff: number }[] = [];for (let k = -SHIFT_REACH; k <= SHIFT_REACH; k++) {@@ -222,7 +269,7 @@ export async function compareMoments(});});const images = await composeComparison(pairs);- const frame = frameNote(doc, scorer.letterboxed);+ const frame = frameNote(doc, scorer.letterboxed, area);return { images, checked, ...(frame ? { frame } : {}) };}@@ -271,7 +318,8 @@ export async function sweepSource(doc: ExportDoc,asset: MediaAsset,span: { from: number; to: number; step?: number; offset: number },- total: number+ total: number,+ area: FrameBox = FULL_FRAME): Promise<SweepResult> {const from = clamp(span.from, 0, total);const to = clamp(span.to, from, total);@@ -296,7 +344,7 @@ export async function sweepSource(}const size = sizeOf(doc, SWEEP_LONG_SIDE);- const scorer = makeScorer(doc);+ const scorer = makeScorer(doc, area);const width = SHIFT_REACH * 2 + 1;const near = new Float32Array(samples.length * width).fill(NaN);const diffs: number[] = [];@@ -306,8 +354,8 @@ export async function sweepSource(// The source side: a still answers every sample with its one frame; a// video is pulled a frame at a time, in step with the render.- const still = asset.type === "image" ? (await stillPixels(asset, scorer)).pixels : null;- const reader = still ? null : framesAt(asset.url, samples.map((s) => s.source), { height: size.height });+ const still = asset.type === "image" ? (await stillPixels(asset, scorer, area)).pixels : null;+ const reader = still ? null : framesAt(asset.url, samples.map((s) => s.source), { height: readHeight(area, size.height) });const started = performance.now();try {await renderProjectFrames(doc, samples.map((s) => s.at), size, (a) => a.url, async (canvas) => {@@ -321,7 +369,7 @@ export async function sweepSource(const c = next.done ? null : next.value;src = c ? scorer.pixels(c.canvas, c.canvas.width, c.canvas.height) : null;}- const cut = scorer.pixels(canvas, canvas.width, canvas.height);+ const cut = scorer.cut(canvas, canvas.width, canvas.height);// The sample's own score, then the timing read: this cut frame against// the source samples before it, and the cut frames before it against@@ -377,7 +425,7 @@ export async function sweepSource(});const timing = still ? undefined : runShift(near, SHIFT_REACH, 0, diffs.length - 1, step);const onBlocks = scored.filter((s) => blockAt(doc, s.at)).length / scored.length;- const frame = frameNote(doc, scorer.letterboxed);+ const frame = frameNote(doc, scorer.letterboxed, area);const reached = diffs.length < samples.length ? samples[diffs.length].at : undefined;return {mode: "sweep",diff --git a/site/src/cut/lib/store.ts b/site/src/cut/lib/store.tsindex 4a23253a..bc6298ad 100644--- a/site/src/cut/lib/store.ts+++ b/site/src/cut/lib/store.ts@@ -5321,14 +5321,14 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {};});-/** Whether any asset's beat grid or source-colour override was rewritten+/** Whether any asset's beat grid, picture box or source-colour override was rewritten* between two asset lists. An asset arriving or leaving is no edit; only a* value changing under an id that was already there counts. */function beatsEdited(prev: MediaAsset[], next: MediaAsset[]): boolean {const before = new Map(prev.map((a) => [a.id, a]));return next.some((a) => {const was = before.get(a.id);- return !!was && (was.beats !== a.beats || was.colorProfile !== a.colorProfile || was.soundFrom !== a.soundFrom);+ return !!was && (was.beats !== a.beats || was.picture !== a.picture || was.colorProfile !== a.colorProfile || was.soundFrom !== a.soundFrom);});}@@ -5493,7 +5493,7 @@ export const docTimelines = (() => {export function storedAssets(assets: MediaAsset[]): StoredAsset[] {return assets.filter((a) => !tabOnlyUpload(a))- .map(({ id, fileName, name, type, duration, sizeBytes, width, height, origin, chatId, folderId, language, watch, speech, beats, sceneCuts, copiedFrom, block, color, colorProfile, proxy, soundFrom }) => ({+ .map(({ id, fileName, name, type, duration, sizeBytes, width, height, origin, chatId, folderId, language, watch, speech, beats, picture, sceneCuts, copiedFrom, block, color, colorProfile, proxy, soundFrom }) => ({id,fileName,name,@@ -5509,6 +5509,7 @@ export function storedAssets(assets: MediaAsset[]): StoredAsset[] {...(watch !== undefined ? { watch } : {}),...(speech !== undefined ? { speech } : {}),...(beats !== undefined ? { beats } : {}),+ ...(picture !== undefined ? { picture } : {}),...(sceneCuts !== undefined ? { sceneCuts } : {}),...(copiedFrom !== undefined ? { copiedFrom } : {}),...(block !== undefined ? { block } : {}),diff --git a/site/src/cut/lib/suggestFontTool.test.ts b/site/src/cut/lib/suggestFontTool.test.tsindex a3671935..09d7ec03 100644--- a/site/src/cut/lib/suggestFontTool.test.ts+++ b/site/src/cut/lib/suggestFontTool.test.ts@@ -107,3 +107,19 @@ test("a region with no type, a sound, and missing words are refused", async () =expect(await message(runAiTool("suggest_font", { asset_id: "ref", t: 3, region: REGION, text: " " }))).toContain("text");expect(await message(runAiTool("suggest_font", { asset_id: "ref", t: 3, region: { x: 0.5, y: 0.5, w: 0, h: 0.1 }, text: "TRAVEL" }))).toContain("region");});++test("inside a picture box, the region, the ink and the size are the box's", async () => {+ // The picture is the frame's middle half: 1920×540. The same crop as REGION,+ // given in the box's fractions, and 72px type is 144px at the box's 1080.+ useEditor.getState().updateAsset("ref", { picture: { x: 0.5, y: 0.5, w: 1, h: 0.5 } });+ const out = (await runAiTool("suggest_font", {+ asset_id: "ref",+ t: 3,+ region: { x: 0.2, y: 0.3, w: 0.6, h: 0.4 },+ text: "TRAVEL DESTINATION",+ })) as Out & { picture?: unknown };+ expect(out.matches[0]).toMatchObject({ font: "oswald", weight: 700 });+ expect(Math.abs(out.matches[0].size - 144)).toBeLessThanOrEqual(6);+ expect(Math.abs(out.ink.y + out.ink.h / 2 - 0.5)).toBeLessThan(0.02);+ expect(out.picture).toBeDefined();+});diff --git a/site/src/cut/lib/types.ts b/site/src/cut/lib/types.tsindex 0639ede1..0eb96014 100644--- a/site/src/cut/lib/types.ts+++ b/site/src/cut/lib/types.ts@@ -252,6 +252,10 @@ export interface StoredAsset {speech?: AssetSpeech;/** The music's beat grid — dots on the clip, snap targets for every edge. */beats?: AssetBeats;+ /** Where the video's picture sits inside its frame, as centre and size+ * frame fractions: the part that changes, apart from letterbox bars or the+ * app a recording shows it in. The measuring tools read inside it. */+ picture?: { x: number; y: number; w: number; h: number };/** How this asset entered the project. Absent = the user imported it (drag,* drop, or upload), so it belongs in the Media panel. Any value marks media* Cut created or fetched — it lives where it was made (the timeline, adiff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.tsindex 9410157a..52c7917c 100644--- a/site/src/cut/server/ai/catalog.ts+++ b/site/src/cut/server/ai/catalog.ts@@ -330,7 +330,7 @@ Your eyes and ears:- detect_silence(clip_id | asset_id, threshold_db?, min_silence?): silent stretches in source seconds; with clip_id each also carries its timeline times.- detect_beats(clip_id | asset_id, regenerate?): the music's beat grid — bpm and beat times in source seconds, plus timelineBeats ready for split_at with clip_id. The grid persists on the asset, draws as yellow dots on its clips, and every drag or trim snaps to it. "Cut to the beat" / "sync to the music" means detect, then split or place at the timeline beat times; the stored grid carries the user's hand edits, so re-detect only when they ask.- capture_frame(t?): one composited frame, at the given time in seconds or the playhead — for checking the final look. Surveying footage is watch_video's job.-- detect_cuts(asset_id, from?, to?), sample_color(asset_id, samples), track_element(asset_id, from, to?, seed): a source in numbers — every hard cut on its frame, the hex of a region at a second, an element's box per frame with the ease of each move. Replicating a video runs on them.+- detect_cuts(asset_id, from?, to?), sample_color(asset_id, samples), track_element(asset_id, from, to?, seed): a source in numbers — every hard cut on its frame, the hex of a region at a second, an element's box per frame with the ease of each move. find_picture(asset_id) finds the box a recorded or letterboxed source's picture fills, and they read inside it. Replicating a video runs on them.The flow for "edit this for me":1. get_state — every clip, its trim (in/out), speed, gaps, and the other tracks.@@ -354,7 +354,7 @@ Someone points at a video — a file they imported, a link you brought in, a cli**Clipping a long source.** A talk, an interview or a stream is mostly material that only makes sense in place. find_highlights ranks its stretches from the transcript, so an hour is read for its handful of moments without watching all of it; watch_video the spans it returns before committing, since the ranking never saw the picture. add_clip \`spans\` lays a pick down — several stretches of one source as one run of clips, the material between them left out — and a two-span result is one clip whose halves were never next to each other.**Watch the whole thing first.** Every decision below comes off the record, so the record has to be complete: watch from 0 and keep going until \`unwatchedSeconds\` is 0, writing each pass into note_source before the next one. Anything over a couple of minutes gets a coarse pass end to end first (\`interval_seconds\` 15-30) to learn its shape, then a dense pass (0.5) over the stretches that carry text or cut fast. Read every card word for word — its text, its face, where it sits in the frame, when it comes and goes — because a turn later the sheets are gone and the notes are all you have. Reading type is what \`detail\` is for: the dense pass over a stretch carrying words runs at \`detail\` read, and a face or treatment you are about to name runs at original, where the frame arrives at the source's own resolution. A scan cell is a quarter of the frame's width; a small caption in one is a smudge, and a smudge read as a word is how a card ends up saying something the video never said. When the cards ARE the narration — a caption track of what the speaker is saying — the transcript already holds every word, and the close passes are for how they are set: face, weight, size, colour, placement, how many words ride a card, how it enters and leaves. Read a few of those at original and take the rest from the transcript.-**Measure it.** A contact sheet tells you what happens; the numbers come from the source's pixels. detect_cuts reads every frame and returns each cut on its frame and the \`shots\` between them with their lengths — run it over the whole source, continuing from \`coveredTo\`. sample_color reads the hex of any region at any second. track_element gives an element's box frame by frame and the ease of each move; with match ink it gives a title's ink box.+**Measure it.** A contact sheet tells you what happens; the numbers come from the source's pixels. A reference recorded off a screen — a phone filming an editing app, a letterboxed film — has its picture inside a bigger frame: find_picture finds that box and keeps it on the source (detect_cuts does on its first read), every measuring tool and compare_to_source then reads inside it, and their x, y, w, h are fractions of it, so they land on the cut's frame as they are; set_aspect to its \`aspect\`. detect_cuts reads every frame and returns each cut on its frame and the \`shots\` between them with their lengths — run it over the whole source, continuing from \`coveredTo\`. sample_color reads the hex of any region at any second. track_element gives an element's box frame by frame and the ease of each move; with match ink it gives a title's ink box.**Block out the cuts.** add_clip with \`blocks\` lays the skeleton from detect_cuts' \`shots\` in one call — one block per shot on track 0, each that shot's length, \`color\` what sample_color reads behind it and \`label\` naming what the person drops there. The label rides the timeline chip and the editor state; the frame itself shows the colour alone, so a shot never plays a sentence describing itself. A block is filled by dropping footage on it — it takes the block's place and its length — or with replace_item, so everything you build on top keeps its timing. Cuts timed to a reference are timed to what it sounds like, so that sound belongs on the soundtrack under them: detach_audio with the reference's \`assetId\` puts its whole track there and places none of its picture. When the ask is the reference's own picture playing, add_clip \`spans\` lays its shots down instead, one span per shot at those same times.diff --git a/site/src/cut/server/ai/headlessTools.ts b/site/src/cut/server/ai/headlessTools.tsindex 0635c95a..8822e01f 100644--- a/site/src/cut/server/ai/headlessTools.ts+++ b/site/src/cut/server/ai/headlessTools.ts@@ -39,6 +39,7 @@ const PAGE_MEDIA_TOOLS: ReadonlySet<string> = new Set(["freeze_frame","capture_frame","compare_to_source",+ "find_picture","detect_cuts","sample_color","track_element",