Open source CapCut alternative.
TypeScript
feat(cut): measure sources in numbers [rebuild]
David Le72fb6cf0adf3
diff --git a/site/src/clients/chatgpt/server/catalog.ts b/site/src/clients/chatgpt/server/catalog.tsindex f4d08107..de4f1b8d 100644--- a/site/src/clients/chatgpt/server/catalog.ts+++ b/site/src/clients/chatgpt/server/catalog.ts@@ -58,6 +58,9 @@ export const READ_COMMANDS = new Set(["watch_video","capture_frame","compare_to_source",+ "detect_cuts",+ "sample_color",+ "track_element","listen_audio","detect_silence","detect_beats",diff --git a/site/src/clients/chatgpt/server/guidance.ts b/site/src/clients/chatgpt/server/guidance.tsindex bdc96d01..4e09df06 100644--- a/site/src/clients/chatgpt/server/guidance.ts+++ b/site/src/clients/chatgpt/server/guidance.ts@@ -41,7 +41,7 @@ Inspect the project for asset ids, transcripts, trims, notes and existing coverawatch_video returns timestamped contact sheets, sceneChanges and coverage. Stamps are source seconds. Continue from coveredTo when truncated. For a request about the whole source, cover it until unwatchedSeconds reaches zero. For long footage, sweep coarsely to learn its structure, then inspect relevant spans closely. A source map or a transcript alone does not establish visual coverage. Use detail read or original to read text and match typography; scan is for structure. Check unreadSeconds and unreadFrom when legible coverage matters.After every watch, record what you observed through edit_project with note_source: source span, actions, cuts, and exact on-screen words. A further watch can be refused until those observations are recorded. Notes persist on the asset; returned contact sheets belong to this conversation. Read observed, recorded and unnoted to resume from existing evidence.listen_audio returns playable source audio. Use the transcript for words and listen for delivery, music and timing. Respect coveredTo; continue from unheardFrom until unheardSeconds reaches zero when judging a whole source. Record observations with note_source. A clip_id scopes listening to its source trim; from and to are source seconds.-Use signal measurements for editing decisions: detect_silence locates quiet spans; detect_beats returns a beat grid and timelineBeats for a clip; measure_level measures loudness, including integratedLufs and truePeakDbtp per clip and for the whole mix, and returns volumeToMatch against a target. read_color_stats measures exposure, tonal range and color before grading. compare_to_source checks the edit against a reference; capture_frame shows the composited picture. These read commands belong in inspect_project's commands array.+Use signal measurements for editing decisions: detect_silence locates quiet spans; detect_beats returns a beat grid and timelineBeats for a clip; measure_level measures loudness, including integratedLufs and truePeakDbtp per clip and for the whole mix, and returns volumeToMatch against a target. read_color_stats measures exposure, tonal range and color before grading. compare_to_source checks the edit against a reference; capture_frame shows the composited picture. detect_cuts, sample_color and track_element measure a source's cuts, colours and moving elements as numbers. These read commands belong in inspect_project's commands array.find_filler identifies transcript fillers; find_highlights ranks transcript spans. Their results need the relevant visual or audio check before cutting. Use edit_project for these commands unless list_commands marks them reads. Caption creation is a separate edit requested by the user.Plan cuts around complete thoughts and measured pauses. Split and delete in batches, read the resulting ids, then place clips to close intended gaps. Leave room around speech boundaries and run refine_speech_cuts with the requested pace. Listen around adjusted edges, widen a clipped word's trim, and check the changed cut visually. Source seconds drive watches, listens and trims; timeline seconds drive placement and splits. Use the clip timing data to map between them.Report only the coverage you inspected and the changes tool results confirm. render_preview produces a playable saved revision for the user to review.`,diff --git a/site/src/cut/components/AiPanel.tools.ts b/site/src/cut/components/AiPanel.tools.tsindex e6488525..906d0a62 100644--- a/site/src/cut/components/AiPanel.tools.ts+++ b/site/src/cut/components/AiPanel.tools.ts@@ -12,6 +12,30 @@ import { WATCH_DETAIL_NOTES, WATCH_DETAILS } from "@/cut/lib/types";import { FACT_CLAIM_MAX_CHARS, FACT_CLAIMS_MAX } from "@/lib/inference/factCheck";import { URL_BATCH_MAX } from "@/cut/lib/urlBatch";import { FIND_MEDIA_DEFAULT, FIND_MEDIA_MAX, FIND_MEDIA_SNAPSHOT_CAP } from "@/cut/lib/findMedia";+import { EASE_IDS } from "@donkeycut/effects-kit";+import {+ DETECT_CUTS_MAX,+ DETECT_CUTS_SPAN_MAX,+ SAMPLE_COLOR_MAX,+ SAMPLE_COLORS_K_MAX,+ TRACK_FRAMES_MAX,+ TRACK_MATCHES,+ TRACK_TOLERANCE_DEFAULT,+} from "@/cut/lib/measure/limits";++/** A box on a source's frame as the editor places one: centre and size. */+const frameBox = (what: string) => ({+ ...obj(+ {+ x: num("Centre x, fraction of frame width"),+ y: num("Centre y, fraction of frame height"),+ w: num("Width, fraction of frame width"),+ h: num("Height, fraction of frame height"),+ },+ ["x", "y", "w", "h"]+ ),+ description: what,+});export const AI_PANEL_TOOLS = [{@@ -42,7 +66,7 @@ export const AI_PANEL_TOOLS = [{name: "watch_video",description:- "Watch a video source with your own eyes: samples candidate frames on a dense steady floor plus one just after each line the source speaks — what a video shows changes when what it says changes, so a transcribed source is read on its own clock and a caption track lands a frame per card — keeps only the ones that actually differ, and tiles them into timestamped contact-sheet images — so every cell is a distinct moment and a gap between stamps means nothing changed there. Also returns sceneChanges, hard-cut times refined to about a third of a second of the true boundary (the time is where the new shot first appears) — though a shot shorter than the sampling step can fall between candidates, so where short cuts matter, re-watch narrow with a small interval_seconds. When the clip has captions or the source's own transcript exists, a fused timeline places each kept frame inside the speech it belongs to. Pass clip_id to watch a timeline clip's source (the result includes that clip's source↔timeline time math) or asset_id for any project video or image. `detail` decides how many pixels each kept frame is worth: scan tiles many small cells for coverage, read enlarges them for on-screen text, original hands back the source's own frames for matching type, colour and edges. A contact-sheet cell is a quarter of the frame's width, so a small caption arrives too small to read — when the job is words on screen or a look to reproduce, spend the pixels instead of guessing. `textFits` comes back with the frames: the parts of the picture that stay quiet across everything just watched, inside whatever the project's guides keep clear, quietest first — where a title or caption can go without covering the subject or the source's own on-screen text. The stamp burned into each cell is SOURCE seconds — what trim_clip's in/out use — not timeline seconds. Coverage ends where the result says it does: `coveredTo` is how far you have looked and `unwatchedSeconds` is how much of the source you have NOT — everything before coveredTo was seen, nothing after it was. Reading is measured apart from seeing: `unreadSeconds` is how much of the source no pass has looked at closely enough to read its type, and `unreadFrom` is where the next close pass starts. Both ride the asset, so they carry what every earlier chat read as well as this one. Continue from coveredTo until unwatchedSeconds is 0 before you describe, summarize, or reproduce the source as a whole; a `to` you chose yourself ends a call with truncated false and the rest of the video still unseen. Watching builds the source's frame map (moment times + cut candidates), persisted on the asset; media entries list those spans as `mapped`, and your first watch starts a quiet background sweep that maps the rest (`watching: true` while it runs). The map aims your watches — a mapped span is NOT footage you have seen; only sheets returned in this conversation are, so watch any span you need to actually look at. Every pass ends with note_source: the sheets are conversation-local and drop out as the chat grows, so write what this stretch showed against its source seconds and it stays on the asset for every later turn. The result carries `recorded` (what is already written about the stretch you just watched) and `unnoted` (the spans of this source nothing describes yet); watching a source that still owes a note is refused. Read the watching-and-cutting skill before editing footage by content.",+ "Watch a video source with your own eyes: samples candidate frames on a dense steady floor plus one just after each line the source speaks — what a video shows changes when what it says changes, so a transcribed source is read on its own clock and a caption track lands a frame per card — keeps only the ones that actually differ, and tiles them into timestamped contact-sheet images — so every cell is a distinct moment and a gap between stamps means nothing changed there. Also returns sceneChanges, hard-cut times on the first frame of the new shot, to the millisecond — though a shot shorter than the sampling step can fall between candidates, so where every cut matters, detect_cuts reads every frame. When the clip has captions or the source's own transcript exists, a fused timeline places each kept frame inside the speech it belongs to. Pass clip_id to watch a timeline clip's source (the result includes that clip's source↔timeline time math) or asset_id for any project video or image. `detail` decides how many pixels each kept frame is worth: scan tiles many small cells for coverage, read enlarges them for on-screen text, original hands back the source's own frames for matching type, colour and edges. A contact-sheet cell is a quarter of the frame's width, so a small caption arrives too small to read — when the job is words on screen or a look to reproduce, spend the pixels instead of guessing. `textFits` comes back with the frames: the parts of the picture that stay quiet across everything just watched, inside whatever the project's guides keep clear, quietest first — where a title or caption can go without covering the subject or the source's own on-screen text. The stamp burned into each cell is SOURCE seconds — what trim_clip's in/out use — not timeline seconds. Coverage ends where the result says it does: `coveredTo` is how far you have looked and `unwatchedSeconds` is how much of the source you have NOT — everything before coveredTo was seen, nothing after it was. Reading is measured apart from seeing: `unreadSeconds` is how much of the source no pass has looked at closely enough to read its type, and `unreadFrom` is where the next close pass starts. Both ride the asset, so they carry what every earlier chat read as well as this one. Continue from coveredTo until unwatchedSeconds is 0 before you describe, summarize, or reproduce the source as a whole; a `to` you chose yourself ends a call with truncated false and the rest of the video still unseen. Watching builds the source's frame map (moment times + cut candidates), persisted on the asset; media entries list those spans as `mapped`, and your first watch starts a quiet background sweep that maps the rest (`watching: true` while it runs). The map aims your watches — a mapped span is NOT footage you have seen; only sheets returned in this conversation are, so watch any span you need to actually look at. Every pass ends with note_source: the sheets are conversation-local and drop out as the chat grows, so write what this stretch showed against its source seconds and it stays on the asset for every later turn. The result carries `recorded` (what is already written about the stretch you just watched) and `unnoted` (the spans of this source nothing describes yet); watching a source that still owes a note is refused. Read the watching-and-cutting skill before editing footage by content.",inputSchema: obj({clip_id: str("Video clip id, track 0 or overlay (defaults from/to to its trimmed in/out)"),asset_id: str("Project asset id (video or image) — watch the source itself"),@@ -159,6 +183,67 @@ export const AI_PANEL_TOOLS = [["asset_id", "times"]),},+ {+ name: "detect_cuts",+ description:+ `Find every hard cut in a stretch of a video source, to the frame. Reads every frame between from and to at a few dozen pixels, scores how much of the picture changed since the frame before, and reports each frame that opens a new shot: \`t\` in SOURCE seconds to the millisecond (the new shot's first frame), its \`frame\` index at the source's \`fps\`, and \`change\`, the percent of the picture that changed — near 100 is a new shot, 10-30 a card or a block of type swapping over the same background. \`shots\` lists the stretches between the cuts with their length in frames. watch_video's sceneChanges come from samples a step apart and lose a shot shorter than the step; this reads every frame, so a five-frame shot in a fast montage is found. Run it before blocking out a reference: add_clip \`blocks\` takes one block per shot at that shot's length. Hard cuts only — a dissolve or a fade changes a little each frame and reads as none. Up to ${DETECT_CUTS_SPAN_MAX}s and ${DETECT_CUTS_MAX} cuts a call; when \`truncated\`, call again from \`coveredTo\`.`,+ inputSchema: obj(+ {+ asset_id: str("Project asset id of the video"),+ from: num("Source start s (default 0)"),+ to: num(`Source end s (default the source's end, at most ${DETECT_CUTS_SPAN_MAX}s past from)`),+ },+ ["asset_id"]+ ),+ },+ {+ name: "sample_color",+ description:+ `Read the colour of a source's pixels: the hex of a region or a point of a frame at a source second, so a block, a shape, a title or a background takes the reference's colour as a number. Contact-sheet cells are JPEGs and shift colour; this reads the decoded frame as the preview shows it. Each sample is a moment \`t\` (SOURCE seconds) and a box — x/y its centre and w/h its size as frame fractions, the box add_shape takes; leave out w/h to read a point. Returns per sample \`hex\`, the region's median colour, and \`spread\`, how far its pixels stray from it on average (0-255: under 10 is a flat fill, over 40 a mix — read \`dominant\` there). \`colors\` 2-${SAMPLE_COLORS_K_MAX} adds \`dominant\`, the colours the region is made of with their shares, largest first: a card and the type on it, a bar and the shot behind it. Read a point inside a flat fill for its colour, a box over a title for its ink and its ground. Up to ${SAMPLE_COLOR_MAX} samples a call, at one moment or each at its own. Works on a still image too (t is ignored).`,+ inputSchema: obj(+ {+ asset_id: str("Project asset id (video or image)"),+ samples: {+ type: "array",+ description: `Up to ${SAMPLE_COLOR_MAX} regions to read`,+ items: obj(+ {+ t: num("Source second"),+ x: num("Centre x, fraction of frame width"),+ y: num("Centre y, fraction of frame height"),+ w: num("Width, fraction of frame width (omit with h for a point)"),+ h: num("Height, fraction of frame height"),+ },+ ["x", "y"]+ ),+ },+ colors: num(`Also list this many dominant colours per region, 2..${SAMPLE_COLORS_K_MAX} (default: none)`),+ },+ ["asset_id", "samples"]+ ),+ },+ {+ name: "track_element",+ description:+ `Measure where an element of a source's picture sits, frame by frame, and how it moves, so a grow, a slide or a pop is keyed with set_overlay_keyframes to the reference's own numbers. Give a stretch of a video (SOURCE seconds) and a \`seed\` box on the element at \`from\`; boxes in and out are x/y centre and w/h size as frame fractions — what add_shape, add_title and the keys take. The element is found by colour: \`color\` (sample_color reads it), or else the seed's median colour on the first frame, so draw the seed inside the element's edges at a moment it is on screen. \`match\` shape follows the one connected piece of that colour the seed sits on — a card, a bar, a sticker — as it moves and changes size; ink takes every pixel of that colour inside the seed (or \`within\`), the ink of a line of type whose letters are separate pieces: its centre is where the title sits and its height is about 0.7 of the font size when the line is capitals. Leave out \`to\` to measure one frame. \`moves\` lists each move of x, y, w and h with from/to (the source seconds it starts and lands), start/end values, the ease that fits it (${EASE_IDS.join(", ")}) and \`error\`, the RMS miss as a share of the move: under 0.03 is that curve, over 0.1 none fits and the move keys from \`path\` point by point. A size move keys as scale: w over the element's resting w. \`path\` lists the box wherever it changes (a hold is one entry); a null box is a frame where the element was not found. Up to ${TRACK_FRAMES_MAX} frames a call; when \`truncated\`, continue from \`coveredTo\`. Works on a still image too (one frame).`,+ inputSchema: obj(+ {+ asset_id: str("Project asset id (video or image)"),+ from: num("Source second the seed shows the element at"),+ to: num("Source end s; omit to measure the one frame at from"),+ seed: frameBox("A box on the element at from — inside its edges for shape, around the line for ink"),+ within: frameBox("Only look inside this box (default: the whole frame for shape, the seed for ink)"),+ color: str("The element's colour as #RRGGBB (default: the seed's median colour)"),+ tolerance: num(`Per-channel distance from color that still counts as it, 0..255 (default ${TRACK_TOLERANCE_DEFAULT}); raise it for a gradient or a soft edge`),+ match: {+ type: "string",+ enum: [...TRACK_MATCHES],+ description: "shape: the one connected piece the seed sits on (default). ink: every pixel of the colour inside the seed or within — type",+ },+ },+ ["asset_id", "from", "seed"]+ ),+ },{name: "find_highlights",description:diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.tsindex 4b863d46..c30154c7 100644--- a/site/src/cut/lib/aiTools.ts+++ b/site/src/cut/lib/aiTools.ts@@ -237,6 +237,8 @@ import { mixSpecFor, renderProjectFrame, renderProjectFrames } from "./exportRenimport { renderMix } from "./audioMix";import { measureLoudnessSliced, roundedLoudness, type LoudnessMeasure } from "./loudness";import { framesAt, readMediaFileSize } from "./mediaRead";+import { sampleColors, scanCuts, trackElement, type FrameBox } from "./measure/sourceMeasure";+import { DETECT_CUTS_MAX, DETECT_CUTS_SPAN_MAX, SAMPLE_COLOR_MAX, SAMPLE_COLORS_K_MAX, TRACK_FRAMES_MAX, TRACK_MATCHES, TRACK_PATH_MAX, TRACK_TOLERANCE_DEFAULT } from "./measure/limits";import { renderStageFrame, storeStageStill } from "./stageFrame";import { createRasterCanvas, decodeRasterImageUrl, rasterCanvasToDataUrl } from "./raster";import { buildAiContext, describeCameraCard, describeDoc, hiddenFromChat, OTHER_CHATS_MEDIA_NOTE, roundT } from "./aiContext";@@ -565,6 +567,65 @@ function statsReport(stats: ColorStats) {};}+/** The measuring tools (detect_cuts, sample_color, track_element) read the+ * source's own pixels: a video, or for "picture" also a still image. */+function measuredSource(s: Editor, input: Record<string, unknown>, wants: "video" | "picture"): MediaAsset {+ const asset = requireItem(s.assets, input.asset_id, "project asset");+ if (asset.type === "audio")+ throw new ToolError(`"${asset.name}" is audio — listen_audio hears it, and it has no picture to measure.`);+ if (asset.type === "image" && wants === "video")+ throw new ToolError(`"${asset.name}" is a still image — one frame, no cuts.`);+ if (asset.type !== "image" && !(asset.duration > 0))+ throw new ToolError(`"${asset.name}" has no length yet — its import is still being read. Try again in a moment.`);+ return asset;+}++/** A {x, y, w, h} box from a tool argument, clamped to the frame. */+function frameBoxOf(raw: unknown, name: string): FrameBox | null {+ if (raw === undefined || raw === null) return null;+ const r = raw as Record<string, unknown>;+ if (!isNum(r.x) || !isNum(r.y) || !isNum(r.w) || !isNum(r.h) || r.w <= 0 || r.h <= 0)+ throw new ToolError(`${name} needs x, y (its centre) and w, h (its size), all frame fractions.`);+ return { x: clamp(r.x, 0, 1), y: clamp(r.y, 0, 1), w: clamp(r.w, 0, 1), h: clamp(r.h, 0, 1) };+}++/** A tracked path as a model reads it: an entry wherever the box changes by+ * a pixel or more, or the element comes or goes — a hold is one entry — and+ * evenly thinned past TRACK_PATH_MAX. */+function trackPath(boxes: { t: number; box: FrameBox | null }[]) {+ const listed: { t: number; box: FrameBox | null }[] = [];+ const PIXEL = 1 / 1080;+ for (let i = 0; i < boxes.length; i++) {+ const b = boxes[i];+ const last = listed[listed.length - 1];+ const changed =+ !last ||+ i === boxes.length - 1 ||+ !b.box !== !last.box ||+ (b.box && last.box && (["x", "y", "w", "h"] as const).some((k) => Math.abs(b.box![k] - last.box![k]) >= PIXEL));+ if (changed) listed.push(b);+ }+ const thinned = listed.length > TRACK_PATH_MAX;+ const kept = thinned+ ? Array.from({ length: TRACK_PATH_MAX }, (_, i) => listed[Math.round((i * (listed.length - 1)) / (TRACK_PATH_MAX - 1))])+ : listed;+ return {+ path: kept.map((b) => ({+ t: roundT(b.t),+ ...(b.box ? { x: rF(b.box.x), y: rF(b.box.y), w: rF(b.box.w), h: rF(b.box.h) } : { box: null }),+ })),+ thinned,+ };+}++/** Frame fractions to the 1/10,000: a pixel of a 1080 frame is 0.0009. */+const rF = (n: number) => Math.round(n * 10000) / 10000;+/** The side of the box a point is read over, as a frame fraction. */+const POINT_SIZE = 0.004;+/** A measuring pass gives back what it covered by this, inside the tool+ * bridge's deadline. */+const MEASURE_BUDGET_MS = 90_000;+/** Long side of a captured frame: enough for the model to read a title, small* enough to ride a turn. */const CAPTURE_LONG_SIDE = 640;@@ -1378,6 +1439,142 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {};},+ detect_cuts: async (s, input) => {+ // Every frame of the span, scored against the one before: the cuts a+ // sampled watch can step over, each on its own frame.+ const asset = measuredSource(s, input, "video");+ const from = clamp(isNum(input.from) ? input.from : 0, 0, asset.duration);+ const asked = isNum(input.to) ? clamp(input.to, 0, asset.duration) : asset.duration;+ const to = Math.min(asked, from + DETECT_CUTS_SPAN_MAX);+ if (!(to > from)) throw new ToolError("from/to describe an empty range of the source.");+ const scan = await withSweepPaused(() =>+ scanCuts(asset.url, { from, to, maxCuts: DETECT_CUTS_MAX, budgetMs: MEASURE_BUDGET_MS })+ ).catch((e) => {+ throw new ToolError(e instanceof Error ? e.message : `Could not read "${asset.name}".`, { cause: e });+ });+ const fps = Math.round(scan.fps * 100) / 100;+ // The shots between the cuts, from the span's start to where the pass+ // reached, each with its length in frames.+ const edges = [from, ...scan.cuts.map((c) => c.t), scan.coveredTo];+ const shots = edges.slice(1).map((end, i) => ({+ from: roundT(edges[i]),+ to: roundT(end),+ frames: Math.round((end - edges[i]) * scan.fps),+ }));+ // The span bound ends a pass the caller asked to run further.+ const truncated = scan.truncated || to < asked;+ const more = truncated || to < asset.duration - 0.001;+ return {+ fps,+ framesRead: scan.frames,+ from: roundT(from),+ coveredTo: roundT(scan.coveredTo),+ truncated,+ cuts: scan.cuts.map((c) => ({ t: roundT(c.t), frame: c.frame, change: c.change })),+ shots,+ source: sourceRef(asset),+ note:+ (scan.cuts.length === 0 ? "No hard cut in this span. " : "") ++ (more+ ? `The source runs on past ${roundT(scan.coveredTo)}s — call again with from: ${roundT(scan.coveredTo)} for the rest. The last shot listed runs on past it.`+ : "Shot n runs from one cut to the next; the first from `from`, the last to the end."),+ };+ },++ sample_color: async (s, input) => {+ const asset = measuredSource(s, input, "picture");+ const raw = Array.isArray(input.samples) ? input.samples : [];+ if (raw.length === 0)+ throw new ToolError("Pass samples: [{t, x, y, w?, h?}] — a source second and a box (or a point) on the frame, as fractions.");+ if (raw.length > SAMPLE_COLOR_MAX)+ throw new ToolError(`That is ${raw.length} samples — ${SAMPLE_COLOR_MAX} is the most one call reads. Read these, then call again for the rest.`);+ const samples = raw.map((x, i) => {+ const rec = (x ?? {}) as Record<string, unknown>;+ if (!isNum(rec.x) || !isNum(rec.y)) throw new ToolError(`Sample ${i + 1} needs x and y, the centre as frame fractions.`);+ if (asset.type === "video" && !isNum(rec.t)) throw new ToolError(`Sample ${i + 1} needs t, the source second to read.`);+ const point = !isNum(rec.w) || !isNum(rec.h);+ const t = asset.type === "video" ? clamp(rec.t as number, 0, Math.max(0, asset.duration - 0.001)) : 0;+ // A point is the few pixels around it: one decoded pixel is grain.+ const box: FrameBox = {+ x: clamp(rec.x, 0, 1),+ y: clamp(rec.y, 0, 1),+ w: point ? POINT_SIZE : clamp(rec.w as number, 0, 1),+ h: point ? POINT_SIZE : clamp(rec.h as number, 0, 1),+ };+ return { t, box, point };+ });+ const k = isNum(input.colors) ? clamp(Math.round(input.colors), 1, SAMPLE_COLORS_K_MAX) : 1;+ const read = await withSweepPaused(() => sampleColors(asset.url, asset.type === "image" ? "image" : "video", samples, k)).catch((e) => {+ throw new ToolError(e instanceof Error ? e.message : `Could not read "${asset.name}".`, { cause: e });+ });+ return {+ samples: samples.map((smp, i) => ({+ ...(asset.type === "video" ? { t: roundT(smp.t) } : {}),+ x: rF(smp.box.x),+ y: rF(smp.box.y),+ ...(smp.point ? {} : { w: rF(smp.box.w), h: rF(smp.box.h) }),+ ...read[i],+ })),+ source: sourceRef(asset),+ };+ },++ track_element: async (s, input) => {+ const asset = measuredSource(s, input, "picture");+ const seed = frameBoxOf(input.seed, "seed");+ if (!seed) throw new ToolError("Pass seed: {x, y, w, h} — a box on the element at from, centre and size as frame fractions.");+ const within = input.within === undefined || input.within === null ? undefined : frameBoxOf(input.within, "within");+ const color = typeof input.color === "string" && input.color.trim() ? input.color.trim() : undefined;+ if (color && !/^#?([0-9a-f]{3}|[0-9a-f]{6})$/i.test(color)) throw new ToolError(`color "${color}" is not a hex colour — pass #RRGGBB.`);+ const match = TRACK_MATCHES.includes(input.match as (typeof TRACK_MATCHES)[number])+ ? (input.match as (typeof TRACK_MATCHES)[number])+ : "shape";+ const end = Math.max(0, asset.duration - 0.001);+ const from = asset.type === "video" ? clamp(isNum(input.from) ? input.from : 0, 0, end) : 0;+ const to = asset.type === "video" && isNum(input.to) && input.to > from ? clamp(input.to, 0, asset.duration) : undefined;+ const run = await withSweepPaused(() =>+ trackElement(asset.url, asset.type === "image" ? "image" : "video", {+ from,+ ...(to !== undefined ? { to } : {}),+ seed,+ ...(within ? { within } : {}),+ ...(color ? { color } : {}),+ tolerance: isNum(input.tolerance) ? clamp(input.tolerance, 0, 255) : TRACK_TOLERANCE_DEFAULT,+ match,+ maxFrames: TRACK_FRAMES_MAX,+ })+ ).catch((e) => {+ throw new ToolError(e instanceof Error ? e.message : `Could not read "${asset.name}".`, { cause: e });+ });+ const found = run.boxes.filter((b) => b.box).length;+ if (found === 0)+ throw new ToolError(+ `Nothing of ${run.color} ${match === "ink" ? "inside the seed" : "under or near the seed"} in ${run.boxes.length} frame${run.boxes.length === 1 ? "" : "s"}. ` ++ "Check the colour with sample_color at a moment the element shows, pass it as color, and raise tolerance for a soft or shaded fill."+ );+ const { path, thinned } = trackPath(run.boxes);+ return {+ color: run.color,+ match,+ ...(run.fps > 0 ? { fps: Math.round(run.fps * 100) / 100 } : {}),+ framesRead: run.boxes.length,+ framesFound: found,+ path,+ ...(thinned ? { thinned: true } : {}),+ moves: run.moves.map((m) => ({+ prop: m.prop,+ from: roundT(m.from),+ to: roundT(m.to),+ start: rF(m.start),+ end: rF(m.end),+ ease: m.ease,+ error: Math.round(m.error * 1000) / 1000,+ })),+ ...(to !== undefined ? { coveredTo: roundT(run.coveredTo), truncated: run.truncated } : {}),+ source: sourceRef(asset),+ };+ },+watch_video: async (s, input) => {// A look at a reference project's footage decides a mapping and writes// nothing: no note owed, no frame map kept, no sweep started.@@ -1523,7 +1720,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {// the next close pass would start.unreadSeconds: unread,...(nextUnread ? { unreadFrom: round2(nextUnread.from) } : {}),- sceneChanges: body.sceneChanges.map(round2),+ sceneChanges: body.sceneChanges.map(roundT),coveredTo: round2(body.coveredTo),truncated: body.truncated,// What NOBODY has looked at, anywhere in the source — the union of@@ -5777,6 +5974,9 @@ export const MEDIA_RUNTIME_TOOLS: ReadonlySet<string> = new Set(["sync_audio","capture_frame","compare_to_source",+ "detect_cuts",+ "sample_color",+ "track_element","render_preview","freeze_frame","create_sticker",diff --git a/site/src/cut/lib/measure/color.test.ts b/site/src/cut/lib/measure/color.test.tsnew file mode 100644index 00000000..176c5a82--- /dev/null+++ b/site/src/cut/lib/measure/color.test.ts@@ -0,0 +1,43 @@+import { expect, test } from "bun:test";+import { hexOf, parseHex, regionColor } from "./color";++/** A w×h RGBA frame painted by `paint(x, y)`. */+const frame = (w: number, h: number, paint: (x: number, y: number) => [number, number, number]) => {+ const data = new Uint8ClampedArray(w * h * 4);+ for (let y = 0; y < h; y++) for (let x = 0; x < w; x++) data.set([...paint(x, y), 255], (y * w + x) * 4);+ return { data, width: w, height: h };+};++test("hex round-trips", () => {+ expect(hexOf(255, 90, 0)).toBe("#FF5A00");+ expect(parseHex("#ff5a00")).toEqual([255, 90, 0]);+ expect(parseHex("FF5A00")).toEqual([255, 90, 0]);+ expect(parseHex("#F50")).toEqual([255, 85, 0]);+ expect(parseHex("orange")).toBeNull();+});++test("a flat region reads as its colour with no spread", () => {+ const px = frame(40, 40, () => [255, 90, 0]);+ const out = regionColor(px, { x: 5, y: 5, w: 10, h: 10 });+ expect(out.hex).toBe("#FF5A00");+ expect(out.spread).toBe(0);+ expect(out.dominant).toBeUndefined();+});++test("a split region names both colours by their share", () => {+ // Left 3/4 orange, right 1/4 white.+ const px = frame(40, 40, (x) => (x < 30 ? [255, 90, 0] : [255, 255, 255]));+ const out = regionColor(px, { x: 0, y: 0, w: 40, h: 40 }, 2);+ expect(out.hex).toBe("#FF5A00");+ expect(out.spread).toBeGreaterThan(0);+ expect(out.dominant).toEqual([+ { hex: "#FF5A00", share: 0.75 },+ { hex: "#FFFFFF", share: 0.25 },+ ]);+});++test("a region running off the frame is clipped to it", () => {+ const px = frame(10, 10, () => [0, 0, 0]);+ const out = regionColor(px, { x: 8, y: 8, w: 10, h: 10 });+ expect(out.hex).toBe("#000000");+});diff --git a/site/src/cut/lib/measure/color.ts b/site/src/cut/lib/measure/color.tsnew file mode 100644index 00000000..777c07ee--- /dev/null+++ b/site/src/cut/lib/measure/color.ts@@ -0,0 +1,186 @@+/** Colour reads of a region of a frame: its median colour, how far the region+ * strays from it, and the few colours it is made of. Pure pixel math on RGBA+ * data; sourceMeasure.ts hands it decoded frames. */++/** An RGBA frame, as ImageData carries it. */+export interface Pixels {+ data: Uint8ClampedArray;+ width: number;+ height: number;+}++/** A rectangle in pixels: left, top, width, height. */+export interface PixelRect {+ x: number;+ y: number;+ w: number;+ h: number;+}++export type Rgb = [number, number, number];++/** Pixels a dominant-colour pass looks at, at most: a whole 1080p frame+ * strided down to this many still finds every colour that covers a percent. */+const CLUSTER_SAMPLES = 20_000;+const CLUSTER_ROUNDS = 10;+/** Dominant colours below this share of the region are noise: edges, grain. */+const MIN_SHARE = 0.01;++const hex2 = (v: number) => Math.round(Math.max(0, Math.min(255, v))).toString(16).padStart(2, "0").toUpperCase();+export const hexOf = (r: number, g: number, b: number): string => `#${hex2(r)}${hex2(g)}${hex2(b)}`;++/** "#RRGGBB", "RRGGBB" or "#RGB" as channels; null for anything else. */+export function parseHex(raw: string): Rgb | null {+ const s = raw.trim().replace(/^#/, "");+ if (/^[0-9a-f]{3}$/i.test(s)) {+ return [0, 1, 2].map((i) => parseInt(s[i] + s[i], 16)) as Rgb;+ }+ if (/^[0-9a-f]{6}$/i.test(s)) {+ return [0, 2, 4].map((i) => parseInt(s.slice(i, i + 2), 16)) as Rgb;+ }+ return null;+}++/** The rectangle clipped to the frame, at least one pixel. */+export function clipRect(px: Pixels, r: PixelRect): PixelRect {+ const x0 = Math.max(0, Math.min(px.width - 1, Math.floor(r.x)));+ const y0 = Math.max(0, Math.min(px.height - 1, Math.floor(r.y)));+ const x1 = Math.max(x0 + 1, Math.min(px.width, Math.ceil(r.x + r.w)));+ const y1 = Math.max(y0 + 1, Math.min(px.height, Math.ceil(r.y + r.h)));+ return { x: x0, y: y0, w: x1 - x0, h: y1 - y0 };+}++/** The colour of a region: the per-channel median as hex, `spread` — the+ * mean of each pixel's largest channel distance from that median, 0..255 —+ * and with `k` ≥ 2 the k colours it is made of, largest share first. */+export function regionColor(+ px: Pixels,+ rect: PixelRect,+ k = 1+): { hex: string; spread: number; dominant?: { hex: string; share: number }[] } {+ const r = clipRect(px, rect);++ // Per-channel histograms give an exact median in one pass.+ const hist = [new Uint32Array(256), new Uint32Array(256), new Uint32Array(256)];+ for (let y = r.y; y < r.y + r.h; y++) {+ for (let x = r.x; x < r.x + r.w; x++) {+ const i = (y * px.width + x) * 4;+ hist[0][px.data[i]]++;+ hist[1][px.data[i + 1]]++;+ hist[2][px.data[i + 2]]++;+ }+ }+ const n = r.w * r.h;+ const median = hist.map((h) => {+ let seen = 0;+ for (let v = 0; v < 256; v++) {+ seen += h[v];+ if (seen * 2 >= n) return v;+ }+ return 255;+ }) as Rgb;++ // How far the region strays from its median, on average.+ let dev = 0;+ for (let y = r.y; y < r.y + r.h; y++) {+ for (let x = r.x; x < r.x + r.w; x++) {+ const i = (y * px.width + x) * 4;+ dev += Math.max(+ Math.abs(px.data[i] - median[0]),+ Math.abs(px.data[i + 1] - median[1]),+ Math.abs(px.data[i + 2] - median[2])+ );+ }+ }+ const out = { hex: hexOf(...median), spread: Math.round(dev / n) };+ if (k < 2) return out;+ return { ...out, dominant: dominantColors(px, r, k) };+}++/** The k colours a region is made of, by k-means over a strided sample, each+ * with its share of the region. Seeded deterministically — the most common+ * coarse colour first, then each next seed the pixel farthest from the+ * seeds so far — so one region always reads the same. */+function dominantColors(px: Pixels, r: PixelRect, k: number): { hex: string; share: number }[] {+ const stride = Math.max(1, Math.floor(Math.sqrt((r.w * r.h) / CLUSTER_SAMPLES)));+ const pts: number[] = [];+ for (let y = r.y; y < r.y + r.h; y += stride) {+ for (let x = r.x; x < r.x + r.w; x += stride) {+ const i = (y * px.width + x) * 4;+ pts.push(px.data[i], px.data[i + 1], px.data[i + 2]);+ }+ }+ const count = pts.length / 3;+ const dist = (p: number, c: number[]) =>+ (pts[p * 3] - c[0]) ** 2 + (pts[p * 3 + 1] - c[1]) ** 2 + (pts[p * 3 + 2] - c[2]) ** 2;++ // Seeds: the most common 4-bit colour, then farthest-point picks.+ const coarse = new Map<number, number>();+ let top = 0;+ let topKey = -1;+ for (let p = 0; p < count; p++) {+ const key = ((pts[p * 3] >> 4) << 8) | ((pts[p * 3 + 1] >> 4) << 4) | (pts[p * 3 + 2] >> 4);+ const c = (coarse.get(key) ?? 0) + 1;+ coarse.set(key, c);+ if (c > top) {+ top = c;+ topKey = key;+ }+ }+ const first = (() => {+ for (let p = 0; p < count; p++) {+ const key = ((pts[p * 3] >> 4) << 8) | ((pts[p * 3 + 1] >> 4) << 4) | (pts[p * 3 + 2] >> 4);+ if (key === topKey) return p;+ }+ return 0;+ })();+ const centers: number[][] = [[pts[first * 3], pts[first * 3 + 1], pts[first * 3 + 2]]];+ while (centers.length < k) {+ let far = -1;+ let farD = 0;+ for (let p = 0; p < count; p++) {+ const d = Math.min(...centers.map((c) => dist(p, c)));+ if (d > farD) {+ farD = d;+ far = p;+ }+ }+ // Fewer distinct colours than asked for: stop at what is there.+ if (far < 0) break;+ centers.push([pts[far * 3], pts[far * 3 + 1], pts[far * 3 + 2]]);+ }++ // Lloyd rounds.+ const label = new Uint8Array(count);+ for (let round = 0; round < CLUSTER_ROUNDS; round++) {+ const sums = centers.map(() => [0, 0, 0, 0]);+ for (let p = 0; p < count; p++) {+ let best = 0;+ let bestD = Infinity;+ for (let c = 0; c < centers.length; c++) {+ const d = dist(p, centers[c]);+ if (d < bestD) {+ bestD = d;+ best = c;+ }+ }+ label[p] = best;+ const s = sums[best];+ s[0] += pts[p * 3];+ s[1] += pts[p * 3 + 1];+ s[2] += pts[p * 3 + 2];+ s[3]++;+ }+ for (let c = 0; c < centers.length; c++) {+ const s = sums[c];+ if (s[3] > 0) centers[c] = [s[0] / s[3], s[1] / s[3], s[2] / s[3]];+ }+ }++ const sizes = centers.map(() => 0);+ for (let p = 0; p < count; p++) sizes[label[p]]++;+ return centers+ .map((c, i) => ({ hex: hexOf(c[0], c[1], c[2]), share: Math.round((sizes[i] / count) * 100) / 100 }))+ .filter((c) => c.share >= MIN_SHARE)+ .sort((a, b) => b.share - a.share);+}diff --git a/site/src/cut/lib/measure/cuts.test.ts b/site/src/cut/lib/measure/cuts.test.tsnew file mode 100644index 00000000..c6eeade5--- /dev/null+++ b/site/src/cut/lib/measure/cuts.test.ts@@ -0,0 +1,40 @@+import { expect, test } from "bun:test";+import { changePct, gridMeans, pickCuts } from "./cuts";++/** A w×h RGBA frame of one colour. */+const solid = (w: number, h: number, rgb: [number, number, number]) => {+ const data = new Uint8ClampedArray(w * h * 4);+ for (let i = 0; i < w * h; i++) data.set([...rgb, 255], i * 4);+ return { data, width: w, height: h };+};++test("two frames of the same shot change nothing, a new shot changes every cell", () => {+ const red = gridMeans(solid(64, 64, [200, 20, 20]));+ const alsoRed = gridMeans(solid(64, 64, [204, 18, 22]));+ const blue = gridMeans(solid(64, 64, [20, 20, 200]));+ expect(changePct(red, alsoRed)).toBe(0);+ expect(changePct(red, blue)).toBe(100);+});++test("a cut is a spike against the frames around it", () => {+ // Steady footage, one hard cut at frame 5.+ const scores = [NaN, 1, 0, 2, 1, 95, 1, 0, 2, 1];+ expect(pickCuts(scores)).toEqual([5]);+});++test("a shot one frame long keeps both of its cuts", () => {+ // A flash frame at 4: the cut in and the cut out sit side by side, and+ // neither may hide the other.+ const scores = [NaN, 0, 1, 0, 90, 88, 1, 0, 0];+ expect(pickCuts(scores)).toEqual([4, 5]);+});++test("a pan changes every frame by the same amount and is not a cut", () => {+ const scores = [NaN, 30, 32, 31, 29, 33, 30, 31];+ expect(pickCuts(scores)).toEqual([]);+});++test("a small flicker under the floor is not a cut", () => {+ const scores = [NaN, 0, 0, 5, 0, 0];+ expect(pickCuts(scores)).toEqual([]);+});diff --git a/site/src/cut/lib/measure/cuts.ts b/site/src/cut/lib/measure/cuts.tsnew file mode 100644index 00000000..d9ab06fa--- /dev/null+++ b/site/src/cut/lib/measure/cuts.ts@@ -0,0 +1,77 @@+/** Hard-cut detection over every frame of a span: each frame is reduced to a+ * grid of cell means, scored by how much of the grid changed since the frame+ * before, and a cut is a score that stands out against the frames around it.+ * Pure data in, pure data out; sourceMeasure.ts feeds it decoded frames. */++import { diffCellPct } from "../watch/signatures";+import type { Pixels } from "./color";++export const CUT_TUNING = {+ /** Cells per side of the grid each frame is reduced to. */+ GRID: 16,+ /** Per-channel change for a cell to count as changed (the watch selector's). */+ CELL_TOL: 25,+ /** % of cells that must change for a frame to be a cut at all. */+ MIN_CHANGE: 8,+ /** A cut's change is at least this many times its neighbours' median. */+ SPIKE: 3,+ /** Frames either side a cut is judged against. */+ NEIGHBOURS: 3,+};++/** Per-cell channel means of an RGBA frame on a GRID×GRID grid. */+export function gridMeans(px: Pixels, grid = CUT_TUNING.GRID): Float32Array {+ const out = new Float32Array(grid * grid * 3);+ const counts = new Uint32Array(grid * grid);+ for (let y = 0; y < px.height; y++) {+ const gy = Math.min(grid - 1, Math.floor((y * grid) / px.height));+ for (let x = 0; x < px.width; x++) {+ const cell = gy * grid + Math.min(grid - 1, Math.floor((x * grid) / px.width));+ const i = (y * px.width + x) * 4;+ out[cell * 3] += px.data[i];+ out[cell * 3 + 1] += px.data[i + 1];+ out[cell * 3 + 2] += px.data[i + 2];+ counts[cell]++;+ }+ }+ for (let c = 0; c < counts.length; c++) {+ const n = Math.max(1, counts[c]);+ out[c * 3] /= n;+ out[c * 3 + 1] /= n;+ out[c * 3 + 2] /= n;+ }+ return out;+}++/** % of the grid that changed between two frames. */+export const changePct = (a: Float32Array, b: Float32Array): number => diffCellPct(a, b, CUT_TUNING.CELL_TOL);++/** The indices of the frames that open a new shot, given each frame's change+ * from the one before (index 0 has none: NaN). A cut is a frame whose change+ * clears the floor and towers over the median of its neighbours, so a pan —+ * the same large change every frame — is not one, and a one-frame shot keeps+ * both of its cuts because one high neighbour never moves the median. */+export function pickCuts(scores: number[]): number[] {+ const { MIN_CHANGE, SPIKE, NEIGHBOURS } = CUT_TUNING;+ const cuts: number[] = [];+ for (let i = 0; i < scores.length; i++) {+ const s = scores[i];+ if (!(s >= MIN_CHANGE)) {+ continue;+ }++ // The neighbours on both sides that have a score of their own.+ const around: number[] = [];+ for (let j = Math.max(0, i - NEIGHBOURS); j <= Math.min(scores.length - 1, i + NEIGHBOURS); j++) {+ if (j !== i && Number.isFinite(scores[j])) {+ around.push(scores[j]);+ }+ }+ around.sort((a, b) => a - b);+ const median = around.length === 0 ? 0 : around[Math.floor((around.length - 1) / 2)];+ if (s >= SPIKE * median) {+ cuts.push(i);+ }+ }+ return cuts;+}diff --git a/site/src/cut/lib/measure/easeFit.test.ts b/site/src/cut/lib/measure/easeFit.test.tsnew file mode 100644index 00000000..16782045--- /dev/null+++ b/site/src/cut/lib/measure/easeFit.test.ts@@ -0,0 +1,63 @@+import { expect, test } from "bun:test";+import { easeAt, type EaseId } from "@donkeycut/effects-kit";+import { fitMoves } from "./easeFit";++const FPS = 60;++/** A value holding at `v0`, moving to `v1` along `ease` over [t0, t1], then+ * holding — sampled every frame and rounded to `px` steps like a measured box. */+function series(ease: EaseId, t0: number, t1: number, v0: number, v1: number, end: number, px = 1 / 1080) {+ const times: number[] = [];+ const values: number[] = [];+ for (let f = 0; f / FPS <= end; f++) {+ const t = f / FPS;+ const p = (t - t0) / (t1 - t0);+ times.push(t);+ values.push(Math.round((v0 + (v1 - v0) * easeAt(ease, p)) / px) * px);+ }+ return { times, values };+}++test("a grow along power3.out is named power3.out with its ends and times", () => {+ const { times, values } = series("power3.out", 0.5, 1.0, 0.1, 0.6, 2);+ const [move] = fitMoves(times, values, 1 / 1080);+ expect(move.ease).toBe("power3.out");+ expect(move.start).toBeCloseTo(0.1, 3);+ expect(move.end).toBeCloseTo(0.6, 3);+ expect(move.from).toBeCloseTo(0.5, 2);+ expect(move.to).toBeCloseTo(1.0, 1);+ expect(move.error).toBeLessThan(0.02);+});++test("a linear slide is named linear", () => {+ const { times, values } = series("linear", 0.2, 0.7, 0.9, 0.3, 1.5);+ const [move] = fitMoves(times, values, 1 / 1080);+ expect(move.ease).toBe("linear");+ expect(move.from).toBeCloseTo(0.2, 2);+ expect(move.to).toBeCloseTo(0.7, 2);+});++test("an ease-in that starts too slowly to see still starts on time", () => {+ const { times, values } = series("power2.in", 0.3, 0.8, 0, 0.5, 1.5);+ const [move] = fitMoves(times, values, 1 / 1080);+ expect(move.ease).toBe("power2.in");+ expect(move.from).toBeCloseTo(0.3, 1);+});++test("a hold is no move, and two moves come back as two", () => {+ const still = fitMoves([0, 0.1, 0.2, 0.3], [0.5, 0.5, 0.5, 0.5], 1 / 1080);+ expect(still).toEqual([]);+ const a = series("linear", 0.2, 0.5, 0.2, 0.8, 1);+ const b = series("linear", 1.2, 1.5, 0.8, 0.2, 2);+ const times = [...a.times, ...b.times.filter((t) => t > 1 + 1e-9)];+ const values = [...a.values, ...b.values.filter((_, i) => b.times[i] > 1 + 1e-9)];+ const moves = fitMoves(times, values, 1 / 1080);+ expect(moves).toHaveLength(2);+ expect(moves[0].end).toBeCloseTo(0.8, 3);+ expect(moves[1].end).toBeCloseTo(0.2, 3);+});++test("a frame where the element is gone splits the series", () => {+ const values: (number | null)[] = [0.2, 0.2, null, null, 0.2, 0.2];+ expect(fitMoves([0, 0.1, 0.2, 0.3, 0.4, 0.5], values, 1 / 1080)).toEqual([]);+});diff --git a/site/src/cut/lib/measure/easeFit.ts b/site/src/cut/lib/measure/easeFit.tsnew file mode 100644index 00000000..9e709436--- /dev/null+++ b/site/src/cut/lib/measure/easeFit.ts@@ -0,0 +1,125 @@+/** Name the moves in a measured series: where a value holds, where it moves,+ * and which of the editor's eases carries each move from its start value to+ * its end value — so a grow or a slide measured off a reference keys as two+ * set_overlay_keyframes keys with the right curve between them. */++import { easeAt, EASE_IDS, type EaseId } from "@donkeycut/effects-kit";++export interface Move {+ /** Seconds the move starts and lands, on the series' clock. */+ from: number;+ to: number;+ /** The value held before and after. */+ start: number;+ end: number;+ ease: EaseId;+ /** RMS distance of the measured frames from the fitted curve, as a share of+ * the move's size: under 0.03 is the curve; over 0.1 none of them is. */+ error: number;+}++/** Still frames inside one move that do not end it: an ease's slow end moves+ * less than a pixel a frame and reads as a few frames of hold between steps. */+const MERGE_GAP = 4;+/** A move smaller than this many measurement steps is jitter. */+const MIN_STEPS = 3;+/** How far past the frames that visibly moved a move may really start or+ * land, as a share of its visible length: an ease-in's first frames and an+ * ease-out's last ones change by less than a pixel. */+const SLACK = 0.3;+const TIME_STEPS = 16;+/** The share of a step a frame must move by to count as moving: a whole-pixel+ * step does, and a centre swaying half a pixel as both edges grow does not. */+const MOVE_STEP = 0.75;++/** The moves in `values` sampled at `times`, each fitted with its ease. A null+ * value is a frame where the element was not found, and splits the series.+ * `step` is the size of one measurement unit (a pixel, as a fraction): a+ * change under MOVE_STEP of it is no change. */+export function fitMoves(times: number[], values: (number | null)[], step: number): Move[] {+ const moves: Move[] = [];+ let i = 0;+ while (i < values.length) {+ // One unbroken stretch of found frames.+ if (values[i] === null) {+ i++;+ continue;+ }+ let j = i;+ while (j + 1 < values.length && values[j + 1] !== null) j++;+ moves.push(...movesIn(times.slice(i, j + 1), values.slice(i, j + 1) as number[], step));+ i = j + 1;+ }+ return moves;+}++function movesIn(times: number[], v: number[], step: number): Move[] {+ const moved = (k: number) => Math.abs(v[k] - v[k - 1]) >= step * MOVE_STEP;+ const sign = (k: number) => Math.sign(v[k] - v[k - 1]);+ const moves: Move[] = [];+ let k = 1;+ while (k < v.length) {+ if (!moved(k)) {+ k++;+ continue;+ }++ // Extend the run over short holds that continue the same direction.+ const dir = sign(k);+ const a = k;+ let b = k;+ let m = k + 1;+ while (m < v.length && m - b <= MERGE_GAP + 1) {+ if (moved(m)) {+ if (sign(m) !== dir) break;+ b = m;+ }+ m++;+ }+ k = b + 1;+ const s = a - 1;+ if (Math.abs(v[b] - v[s]) < MIN_STEPS * step) continue;+ moves.push(fitOne(times, v, s, a, b));+ }+ return moves;+}++/** Fit one move: the frames s (last still) … b (last moving) with a the first+ * frame that moved. Searches each ease over start and landing times around+ * the visible ones and keeps the least error. */+function fitOne(times: number[], v: number[], s: number, a: number, b: number): Move {+ const v0 = v[s];+ const v1 = v[b];+ const size = Math.abs(v1 - v0);+ const span = times[b] - times[s];+ const t0Lo = times[s] - SLACK * span;+ const t0Hi = times[a];+ const t1Lo = times[Math.max(a, b - 1)];+ const t1Hi = times[b] + SLACK * span;++ // The frames that judge a fit: everything the search could move a move over.+ const judged: number[] = [];+ for (let k = 0; k < times.length; k++) {+ if (times[k] >= t0Lo - 1e-9 && times[k] <= t1Hi + 1e-9) judged.push(k);+ }++ let best: Move = { from: times[s], to: times[b], start: v0, end: v1, ease: "linear", error: Infinity };+ for (const ease of EASE_IDS) {+ for (let p = 0; p <= TIME_STEPS; p++) {+ const t0 = t0Lo + ((t0Hi - t0Lo) * p) / TIME_STEPS;+ for (let q = 0; q <= TIME_STEPS; q++) {+ const t1 = t1Lo + ((t1Hi - t1Lo) * q) / TIME_STEPS;+ if (t1 - t0 <= 1e-6) continue;+ let sq = 0;+ for (const k of judged) {+ const want = v0 + (v1 - v0) * easeAt(ease, (times[k] - t0) / (t1 - t0));+ sq += (want - v[k]) ** 2;+ }+ const error = Math.sqrt(sq / judged.length) / size;+ // A curve has to beat the simpler one by more than rounding to win.+ if (error < best.error - 1e-4) best = { from: t0, to: t1, start: v0, end: v1, ease, error };+ }+ }+ }+ return best;+}diff --git a/site/src/cut/lib/measure/limits.ts b/site/src/cut/lib/measure/limits.tsnew file mode 100644index 00000000..174fdc06--- /dev/null+++ b/site/src/cut/lib/measure/limits.ts@@ -0,0 +1,21 @@+/** What one call of each measuring tool reads at most, shared by the tool+ * schemas (AiPanel.tools.ts) and their handlers so the two never disagree.+ * Each bound keeps a call inside the tool bridge's deadline and its result+ * small enough for a model to read whole. */++/** detect_cuts: seconds scanned per call; a longer span resumes from coveredTo. */+export const DETECT_CUTS_SPAN_MAX = 120;+/** detect_cuts: cuts reported per call. */+export const DETECT_CUTS_MAX = 80;+/** sample_color: regions read per call. */+export const SAMPLE_COLOR_MAX = 24;+/** sample_color: dominant colours per region. */+export const SAMPLE_COLORS_K_MAX = 6;+/** track_element: frames read per call — six seconds of a 60 fps source. */+export const TRACK_FRAMES_MAX = 360;+/** track_element: per-channel distance from the colour that still counts as it. */+export const TRACK_TOLERANCE_DEFAULT = 40;+/** track_element: the element as one connected piece, or every pixel of it. */+export const TRACK_MATCHES = ["shape", "ink"] as const;+/** track_element: path entries listed per call; a longer path is thinned. */+export const TRACK_PATH_MAX = 60;diff --git a/site/src/cut/lib/measure/sourceMeasure.ts b/site/src/cut/lib/measure/sourceMeasure.tsnew file mode 100644index 00000000..22c12df1--- /dev/null+++ b/site/src/cut/lib/measure/sourceMeasure.ts@@ -0,0 +1,321 @@+/** Measurements of a source's own pixels at source seconds — the numbers a+ * replica is built from: every hard cut to its frame, the colour of a region,+ * and the box an element fills frame by frame with the ease of each move.+ * Frames come through the same base-rendering sink the watch and the colour+ * reads use, so a log or HDR source measures as the preview shows it. */++import { baseFrameSink, colorReadAt } from "../baseFrame";+import { openMedia, UnreadableMediaError, videoTrackOf, type FrameSize } from "../mediaRead";+import { createRasterCanvas, decodeRasterImageUrl } from "../raster";+import type { WrappedCanvas } from "mediabunny";+import { hexOf, regionColor, parseHex, type PixelRect, type Pixels, type Rgb } from "./color";+import { changePct, gridMeans, pickCuts } from "./cuts";+import { fitMoves, type Move } from "./easeFit";+import { inkBox, shapeBox } from "./track";++/** A box as the editor places one: centre and size, frame fractions. */+export interface FrameBox {+ x: number;+ y: number;+ w: number;+ h: number;+}++/** Every frame of a cut scan is this small: a cut changes the whole picture,+ * and 64² holds the 16-grid the scorer reads at four pixels a cell. */+const SCAN_SIZE: FrameSize = { width: 64, height: 64, fit: "fill" };+/** Long side colours are read at: a 1080p source's own pixels. */+const COLOR_LONG_SIDE = 1920;+/** Long side elements are tracked at: a pixel is 0.14% of the frame. */+const TRACK_LONG_SIDE = 720;+/** Reads a pass back off by this much so the frame on `from` is scored. */+const LEAD_S = 0.001;++/** An open video source: its display size, first timestamp and frame reads. */+interface SourceVideo {+ width: number;+ height: number;+ first: number;+ frames(from: number, to: number, size: FrameSize): AsyncGenerator<SourceFrame>;+ framesAt(times: number[], size: FrameSize): AsyncGenerator<SourceFrame | null>;+}++interface SourceFrame extends Pixels {+ t: number;+ duration: number;+}++function pixelsOf(w: WrappedCanvas): SourceFrame {+ const canvas = w.canvas;+ const ctx = canvas.getContext("2d") as CanvasRenderingContext2D | null;+ if (!ctx) throw new Error("Could not read the frame's pixels.");+ const image = ctx.getImageData(0, 0, canvas.width, canvas.height);+ return { data: image.data, width: image.width, height: image.height, t: w.timestamp, duration: w.duration };+}++/** Open `url`'s video track for the length of `fn`, closing it after. */+async function withSourceVideo<T>(url: string, fn: (v: SourceVideo) => Promise<T>): Promise<T> {+ const input = openMedia(url);+ try {+ const track = await videoTrackOf(input);+ if (!track) throw new UnreadableMediaError("This file has no readable video.");+ const reading = colorReadAt(url);+ const width = await track.getDisplayWidth();+ const height = await track.getDisplayHeight();+ const first = await track.getFirstTimestamp().catch(() => 0);+ return await fn({+ width,+ height,+ first,+ frames: async function* (from, to, size) {+ // A small pool: each frame is read and dropped before the next.+ for await (const w of baseFrameSink(track, size, reading, { poolSize: 2 }).canvases(from, to)) yield pixelsOf(w);+ },+ framesAt: async function* (times, size) {+ for await (const w of baseFrameSink(track, size, reading).canvasesAtTimestamps(times)) yield w ? pixelsOf(w) : null;+ },+ });+ } finally {+ input.dispose();+ }+}++/** A picture's pixels at `longSide`, for a still asset. */+async function imagePixels(url: string, longSide: number): Promise<Pixels> {+ const img = await decodeRasterImageUrl(url);+ if (!img || !img.width || !img.height) throw new Error("Could not read the image.");+ const k = Math.min(1, longSide / Math.max(img.width, img.height));+ const w = Math.max(1, Math.round(img.width * k));+ const h = Math.max(1, Math.round(img.height * k));+ const canvas = createRasterCanvas(w, h);+ const ctx = canvas.getContext("2d") as CanvasRenderingContext2D | null;+ if (!ctx) throw new Error("Could not read the image.");+ ctx.drawImage(img.source, 0, 0, w, h);+ const image = ctx.getImageData(0, 0, w, h);+ return { data: image.data, width: w, height: h };+}++/** The read size for a source at `longSide`, never past its own pixels. */+function sizeAt(width: number, height: number, longSide: number): FrameSize {+ const k = Math.min(1, longSide / Math.max(width, height, 1));+ return { width: Math.max(1, Math.round(width * k)), height: Math.max(1, Math.round(height * k)), fit: "fill" };+}++/** A centre-and-size box in frame fractions, as pixels of `px`. */+const toPixels = (px: Pixels, b: FrameBox): PixelRect => ({+ x: (b.x - b.w / 2) * px.width,+ y: (b.y - b.h / 2) * px.height,+ w: Math.max(1, b.w * px.width),+ h: Math.max(1, b.h * px.height),+});++/** A pixel box as centre-and-size frame fractions. */+const toFrame = (px: { width: number; height: number }, r: PixelRect): FrameBox => ({+ x: (r.x + r.w / 2) / px.width,+ y: (r.y + r.h / 2) / px.height,+ w: r.w / px.width,+ h: r.h / px.height,+});++/** The frame rate a run of decoded frames plays at: the median frame+ * duration, or the median gap between timestamps when durations are absent. */+function rateOf(times: number[], durations: number[]): number {+ const steps = durations.filter((d) => d > 0);+ const gaps = steps.length > 0 ? steps : times.slice(1).map((t, i) => t - times[i]).filter((d) => d > 0);+ if (gaps.length === 0) return 0;+ gaps.sort((a, b) => a - b);+ return 1 / gaps[Math.floor(gaps.length / 2)];+}++export interface CutScan {+ fps: number;+ frames: number;+ cuts: { t: number; frame: number; change: number }[];+ coveredTo: number;+ truncated: boolean;+}++/** Every hard cut in [from, to) of a video, found by scoring every frame+ * against the one before it. A pass stops at `maxCuts` or at the time+ * budget, and `coveredTo` says where the next pass picks up. */+export async function scanCuts(+ url: string,+ opts: { from: number; to: number; maxCuts: number; budgetMs: number }+): Promise<CutScan> {+ return withSourceVideo(url, async (v) => {+ const deadline = Date.now() + opts.budgetMs;+ const times: number[] = [];+ const durations: number[] = [];+ const scores: number[] = [];+ let prev: Float32Array | null = null;+ let outOfTime = false;+ for await (const f of v.frames(Math.max(0, opts.from - LEAD_S), opts.to, SCAN_SIZE)) {+ if (Date.now() > deadline) {+ outOfTime = true;+ break;+ }+ const grid = gridMeans(f);+ scores.push(prev ? changePct(prev, grid) : NaN);+ times.push(f.t);+ durations.push(f.duration);+ prev = grid;+ }+ if (times.length === 0) throw new Error("Could not read any frames in that span.");++ const fps = rateOf(times, durations);+ const picked = pickCuts(scores);+ const kept = picked.slice(0, opts.maxCuts);+ const capped = picked.length > kept.length;+ // A capped pass ends on the frame before the first cut it leaves out, so+ // the next pass scores that cut against it.+ const coveredTo = capped ? times[picked[kept.length] - 1] : outOfTime ? times[times.length - 1] : opts.to;+ return {+ fps,+ frames: times.length,+ cuts: kept.map((i) => ({+ t: times[i],+ frame: fps > 0 ? Math.round((times[i] - v.first) * fps) : i,+ change: Math.round(scores[i]),+ })),+ coveredTo,+ truncated: capped || outOfTime,+ };+ });+}++export interface ColorSample {+ t: number;+ box: FrameBox;+}++/** The colour of each sample's region at its moment, read in one ordered+ * pass over the source. `k` ≥ 2 adds the k colours each region is made of. */+export async function sampleColors(+ url: string,+ kind: "video" | "image",+ samples: ColorSample[],+ k: number+): Promise<ReturnType<typeof regionColor>[]> {+ if (kind === "image") {+ const px = await imagePixels(url, COLOR_LONG_SIDE);+ return samples.map((s) => regionColor(px, toPixels(px, s.box), k));+ }+ const times = [...new Set(samples.map((s) => s.t))].sort((a, b) => a - b);+ return withSourceVideo(url, async (v) => {+ const size = sizeAt(v.width, v.height, COLOR_LONG_SIDE);+ const out = new Array<ReturnType<typeof regionColor>>(samples.length);+ let at = 0;+ for await (const f of v.framesAt(times, size)) {+ const t = times[at++];+ if (!f) throw new Error(`Could not read the frame at ${t}s.`);+ samples.forEach((s, i) => {+ if (s.t === t) out[i] = regionColor(f, toPixels(f, s.box), k);+ });+ }+ return out;+ });+}++export type TrackMatch = "shape" | "ink";++export interface TrackRun {+ color: string;+ fps: number;+ /** Per frame read: its time and the element's box, or null where absent. */+ boxes: { t: number; box: FrameBox | null }[];+ moves: (Move & { prop: keyof FrameBox })[];+ coveredTo: number;+ truncated: boolean;+}++/** The box an element fills in every frame of [from, to) — one frame at+ * `from` when `to` is absent — and the moves of its centre and size, each+ * fitted with an ease. The element is the piece of `color` the seed points+ * at ("shape"), or every pixel of it inside the seed ("ink"); with no colour+ * the seed's own median colour on the first frame is the element's. */+export async function trackElement(+ url: string,+ kind: "video" | "image",+ opts: {+ from: number;+ to?: number;+ seed: FrameBox;+ within?: FrameBox;+ color?: string;+ tolerance: number;+ match: TrackMatch;+ maxFrames: number;+ }+): Promise<TrackRun> {+ let truncated = false;+ let coveredTo = opts.to ?? opts.from;++ // The element's box in each frame, carried forward as the next one's seed.+ let target: Rgb | null = opts.color ? parseHex(opts.color) : null;+ let near: PixelRect | null = null;+ const boxes: TrackRun["boxes"] = [];+ const read = (t: number, px: Pixels) => {+ const seed = toPixels(px, opts.seed);+ if (!target) target = parseHex(regionColor(px, seed).hex)!;+ const area = opts.within ? toPixels(px, opts.within) : opts.match === "ink" ? seed : { x: 0, y: 0, w: px.width, h: px.height };+ const found =+ opts.match === "ink"+ ? inkBox(px, target, opts.tolerance, area)+ : (shapeBox(px, target, opts.tolerance, area, near ?? seed) ??+ (near ? shapeBox(px, target, opts.tolerance, area, seed) : null));+ if (found) near = found;+ boxes.push({ t, box: found ? toFrame(px, found) : null });+ };++ let size = { width: 1, height: 1 };+ const times: number[] = [];+ const durations: number[] = [];+ if (kind === "image") {+ const px = await imagePixels(url, TRACK_LONG_SIDE);+ size = px;+ read(0, px);+ } else {+ await withSourceVideo(url, async (v) => {+ const at = sizeAt(v.width, v.height, TRACK_LONG_SIDE);+ size = { width: at.width!, height: at.height! };+ if (opts.to === undefined || opts.to <= opts.from) {+ for await (const f of v.framesAt([opts.from], at)) {+ if (!f) throw new Error(`Could not read the frame at ${opts.from}s.`);+ read(opts.from, f);+ }+ return;+ }+ for await (const f of v.frames(opts.from, opts.to, at)) {+ if (times.length >= opts.maxFrames) {+ truncated = true;+ coveredTo = times[times.length - 1];+ break;+ }+ times.push(f.t);+ durations.push(f.duration);+ read(f.t, f);+ }+ });+ }++ // Each of the four numbers fitted on its own: a grow moves w and h, a+ // slide moves x or y, and one fit per number keys each the way it moved.+ const moves: TrackRun["moves"] = [];+ if (boxes.length > 2) {+ const ts = boxes.map((b) => b.t);+ for (const prop of ["x", "y", "w", "h"] as const) {+ const step = 1 / (prop === "x" || prop === "w" ? size.width : size.height);+ for (const m of fitMoves(ts, boxes.map((b) => (b.box ? b.box[prop] : null)), step)) moves.push({ ...m, prop });+ }+ moves.sort((a, b) => a.from - b.from);+ }+ const rgb = target as Rgb | null;+ return {+ color: rgb ? hexOf(...rgb) : "",+ fps: rateOf(times, durations),+ boxes,+ moves,+ coveredTo,+ truncated,+ };+}diff --git a/site/src/cut/lib/measure/track.test.ts b/site/src/cut/lib/measure/track.test.tsnew file mode 100644index 00000000..f22d4189--- /dev/null+++ b/site/src/cut/lib/measure/track.test.ts@@ -0,0 +1,56 @@+import { expect, test } from "bun:test";+import { inkBox, shapeBox } from "./track";++type Rgb = [number, number, number];++/** A w×h RGBA frame on `bg`, with rectangles painted over it. */+const frame = (w: number, h: number, bg: Rgb, rects: { x: number; y: number; w: number; h: number; c: Rgb }[]) => {+ const data = new Uint8ClampedArray(w * h * 4);+ for (let i = 0; i < w * h; i++) data.set([...bg, 255], i * 4);+ for (const r of rects)+ for (let y = r.y; y < r.y + r.h; y++) for (let x = r.x; x < r.x + r.w; x++) data.set([...r.c, 255], (y * w + x) * 4);+ return { data, width: w, height: h };+};++const WHITE: Rgb = [255, 255, 255];+const BLACK: Rgb = [0, 0, 0];+const ORANGE: Rgb = [255, 90, 0];+const all = { x: 0, y: 0, w: 100, h: 100 };++test("the shape box is the piece of that colour at the seed, and only it", () => {+ // Two orange cards: the seed sits on the left one.+ const px = frame(100, 100, BLACK, [+ { x: 10, y: 20, w: 20, h: 10, c: ORANGE },+ { x: 60, y: 60, w: 30, h: 30, c: ORANGE },+ ]);+ expect(shapeBox(px, ORANGE, 40, all, { x: 12, y: 22, w: 4, h: 4 })).toEqual({ x: 10, y: 20, w: 20, h: 10 });+});++test("a shape that left the seed is found by its nearest piece", () => {+ const px = frame(100, 100, BLACK, [{ x: 40, y: 20, w: 20, h: 10, c: ORANGE }]);+ expect(shapeBox(px, ORANGE, 40, all, { x: 10, y: 20, w: 20, h: 10 })).toEqual({ x: 40, y: 20, w: 20, h: 10 });+});++test("no piece of that colour is no box", () => {+ const px = frame(100, 100, BLACK, []);+ expect(shapeBox(px, ORANGE, 40, all, { x: 10, y: 20, w: 20, h: 10 })).toBeNull();+});++test("the ink box spans every letter inside the region", () => {+ // Three white "letters" on black, and a white mark outside the region.+ const px = frame(100, 100, BLACK, [+ { x: 20, y: 40, w: 6, h: 12, c: WHITE },+ { x: 30, y: 44, w: 6, h: 8, c: WHITE },+ { x: 40, y: 40, w: 6, h: 14, c: WHITE },+ { x: 90, y: 5, w: 5, h: 5, c: WHITE },+ ]);+ expect(inkBox(px, WHITE, 40, { x: 10, y: 30, w: 50, h: 40 })).toEqual({ x: 20, y: 40, w: 26, h: 14 });+});++test("a stray pixel of the colour does not stretch the ink box", () => {+ const px = frame(100, 100, BLACK, [+ { x: 20, y: 40, w: 20, h: 10, c: WHITE },+ { x: 5, y: 90, w: 1, h: 1, c: WHITE },+ ]);+ expect(inkBox(px, WHITE, 40, all)).toEqual({ x: 20, y: 40, w: 20, h: 10 });+});diff --git a/site/src/cut/lib/measure/track.ts b/site/src/cut/lib/measure/track.tsnew file mode 100644index 00000000..2496bc4d--- /dev/null+++ b/site/src/cut/lib/measure/track.ts@@ -0,0 +1,131 @@+/** Where an element of one colour sits in a frame: the connected piece of that+ * colour a seed points at (a card, a bar, a sticker), or the box around every+ * pixel of it inside a region (the ink of a line of type, whose letters are+ * separate pieces). Pure pixel math; sourceMeasure.ts runs it per frame. */++import { clipRect, type PixelRect, type Pixels, type Rgb } from "./color";++/** Pieces smaller than this many pixels are compression noise and grain. */+const MIN_PIECE = 4;+/** A piece clear of the seed is still taken when its centre sits within this+ * many seed sizes of the seed's — an element that moved since the last frame. */+const REACH = 2;++interface Piece {+ x0: number;+ y0: number;+ x1: number;+ y1: number;+ count: number;+}++/** The pieces of `target` colour (within `tol` on every channel) inside+ * `area`, 8-connected, as pixel boxes with their sizes. */+function piecesOf(px: Pixels, target: Rgb, tol: number, area: PixelRect): Piece[] {+ const r = clipRect(px, area);+ const w = r.w;+ const h = r.h;++ // Union-find over the mask, one row at a time.+ const parent = new Int32Array(w * h).fill(-1);+ const find = (a: number): number => {+ while (parent[a] !== a) {+ parent[a] = parent[parent[a]];+ a = parent[a];+ }+ return a;+ };+ const join = (a: number, b: number) => {+ const ra = find(a);+ const rb = find(b);+ if (ra !== rb) parent[Math.max(ra, rb)] = Math.min(ra, rb);+ };+ for (let y = 0; y < h; y++) {+ for (let x = 0; x < w; x++) {+ const i = ((r.y + y) * px.width + r.x + x) * 4;+ const near =+ Math.abs(px.data[i] - target[0]) <= tol &&+ Math.abs(px.data[i + 1] - target[1]) <= tol &&+ Math.abs(px.data[i + 2] - target[2]) <= tol;+ if (!near) continue;+ const p = y * w + x;+ parent[p] = p;+ if (x > 0 && parent[p - 1] >= 0) join(p, p - 1);+ if (y === 0) continue;+ for (let dx = -1; dx <= 1; dx++) {+ const nx = x + dx;+ if (nx >= 0 && nx < w && parent[p - w + dx] >= 0) join(p, p - w + dx);+ }+ }+ }++ // Each root's bounding box, back in frame pixels.+ const boxes = new Map<number, Piece>();+ for (let p = 0; p < w * h; p++) {+ if (parent[p] < 0) continue;+ const root = find(p);+ const x = r.x + (p % w);+ const y = r.y + Math.floor(p / w);+ const b = boxes.get(root);+ if (!b) {+ boxes.set(root, { x0: x, y0: y, x1: x + 1, y1: y + 1, count: 1 });+ continue;+ }+ b.x0 = Math.min(b.x0, x);+ b.y0 = Math.min(b.y0, y);+ b.x1 = Math.max(b.x1, x + 1);+ b.y1 = Math.max(b.y1, y + 1);+ b.count++;+ }+ return [...boxes.values()].filter((b) => b.count >= MIN_PIECE);+}++const rectOf = (b: Piece): PixelRect => ({ x: b.x0, y: b.y0, w: b.x1 - b.x0, h: b.y1 - b.y0 });++/** The box around every piece of `target` inside `area` — the ink of a line+ * of type — or null when there is none. */+export function inkBox(px: Pixels, target: Rgb, tol: number, area: PixelRect): PixelRect | null {+ const pieces = piecesOf(px, target, tol, area);+ if (pieces.length === 0) return null;+ return rectOf({+ x0: Math.min(...pieces.map((b) => b.x0)),+ y0: Math.min(...pieces.map((b) => b.y0)),+ x1: Math.max(...pieces.map((b) => b.x1)),+ y1: Math.max(...pieces.map((b) => b.y1)),+ count: 0,+ });+}++/** The one piece of `target` inside `area` that `near` points at: the one+ * overlapping it most, else the closest within REACH of its size; null when+ * the element is not in this frame. */+export function shapeBox(px: Pixels, target: Rgb, tol: number, area: PixelRect, near: PixelRect): PixelRect | null {+ const pieces = piecesOf(px, target, tol, area);+ const overlap = (b: Piece) =>+ Math.max(0, Math.min(b.x1, near.x + near.w) - Math.max(b.x0, near.x)) *+ Math.max(0, Math.min(b.y1, near.y + near.h) - Math.max(b.y0, near.y));+ let best: Piece | null = null;+ let bestOverlap = 0;+ for (const b of pieces) {+ const o = overlap(b);+ if (o > bestOverlap) {+ bestOverlap = o;+ best = b;+ }+ }+ if (best) return rectOf(best);++ // Nothing under the seed: the closest piece, if it is within reach.+ const cx = near.x + near.w / 2;+ const cy = near.y + near.h / 2;+ const reach = REACH * Math.max(near.w, near.h, 1);+ let bestD = Infinity;+ for (const b of pieces) {+ const d = Math.hypot((b.x0 + b.x1) / 2 - cx, (b.y0 + b.y1) / 2 - cy);+ if (d <= reach && d < bestD) {+ bestD = d;+ best = b;+ }+ }+ return best ? rectOf(best) : null;+}diff --git a/site/src/cut/lib/measureTools.test.ts b/site/src/cut/lib/measureTools.test.tsnew file mode 100644index 00000000..580fffa1--- /dev/null+++ b/site/src/cut/lib/measureTools.test.ts@@ -0,0 +1,199 @@+import { afterAll, beforeEach, expect, spyOn, test } from "bun:test";+import type { Input, InputVideoTrack, WrappedCanvas } from "mediabunny";+import * as mediaRead from "./mediaRead";+import { useEditor } from "./store";++// The measuring tools read a source the way the other senses do — through the+// frame sink — so a synthetic 60 fps video stands in for the file: every+// frame's pixels are painted from its index, and each tool's numbers are+// checked against what was painted.++const FPS = 60;+const FRAMES = 180; // 3 s+const W = 1080;+const H = 1920;++type Rgb = [number, number, number];+const RED: Rgb = [200, 30, 30];+const BLUE: Rgb = [30, 30, 200];+const GREEN: Rgb = [30, 160, 60];+const ORANGE: Rgb = [255, 90, 0];+const WHITE: Rgb = [255, 255, 255];++/** The picture at frame f, as a colour at each frame fraction. Red until 1s,+ * a five-frame blue shot, then green. On the green, an orange bar centred at+ * (0.5, 0.3), 0.1 tall, grows linearly from 0.1 to 0.5 wide over 1.5–2.0s; a+ * white "title" block spans x 0.2–0.8, y 0.7–0.75. */+function colourAt(f: number, fx: number, fy: number): Rgb {+ if (f < 60) return RED;+ if (f < 65) return BLUE;+ const t = f / FPS;+ const p = Math.max(0, Math.min(1, (t - 1.5) / 0.5));+ const bw = 0.1 + 0.4 * p;+ if (Math.abs(fx - 0.5) <= bw / 2 && Math.abs(fy - 0.3) <= 0.05) return ORANGE;+ if (fx >= 0.2 && fx < 0.8 && fy >= 0.7 && fy < 0.75) return WHITE;+ return GREEN;+}++function frameCanvas(f: number, w: number, h: number) {+ const data = new Uint8ClampedArray(w * h * 4);+ for (let y = 0; y < h; y++)+ for (let x = 0; x < w; x++) data.set([...colourAt(f, (x + 0.5) / w, (y + 0.5) / h), 255], (y * w + x) * 4);+ const ctx = { getImageData: () => ({ data, width: w, height: h }), drawImage() {}, putImageData() {} };+ return { width: w, height: h, getContext: () => ctx } as unknown as HTMLCanvasElement;+}++const wrap = (f: number, w: number, h: number) =>+ ({ canvas: frameCanvas(f, w, h), timestamp: f / FPS, duration: 1 / FPS }) as WrappedCanvas;+const frameShownAt = (t: number) => Math.max(0, Math.min(FRAMES - 1, Math.floor(t * FPS + 1e-6)));++const decoded: number[] = [];+const input = { dispose() {} } as unknown as Input;+const track = {+ getDisplayWidth: async () => W,+ getDisplayHeight: async () => H,+ getFirstTimestamp: async () => 0,+ displayWidth: W,+ displayHeight: H,+} as unknown as InputVideoTrack;+const spies = [+ spyOn(mediaRead, "openMedia").mockReturnValue(input),+ spyOn(mediaRead, "videoTrackOf").mockResolvedValue(track),+ spyOn(mediaRead, "frameSink").mockImplementation((_track, size) => {+ const w = size?.width ?? W;+ const h = size?.height ?? H;+ return {+ getCanvas: async (t: number) => wrap(frameShownAt(t), w, h),+ async *canvases(start = 0, end = Infinity) {+ for (let f = frameShownAt(start); f < FRAMES && f / FPS < end; f++) {+ decoded.push(f);+ yield wrap(f, w, h);+ }+ },+ async *canvasesAtTimestamps(times: Iterable<number> | AsyncIterable<number>) {+ for await (const t of times) yield wrap(frameShownAt(t), w, h);+ },+ };+ }),+];+afterAll(() => {+ for (const spy of spies) spy.mockRestore();+});++const { runAiTool } = await import("./aiTools");++beforeEach(() => {+ decoded.length = 0;+ useEditor.setState({+ assets: [+ { id: "ref", fileName: "ref.mp4", name: "Reference", type: "video", duration: FRAMES / FPS, url: "fixture.mp4", width: W, height: H },+ { id: "song", fileName: "song.mp3", name: "Song", type: "audio", duration: 30, url: "" },+ ],+ });+});++const message = async (p: Promise<unknown>) =>+ p.then(() => "", (e: unknown) => (e instanceof Error ? e.message : String(e)));++test("detect_cuts finds every hard cut to the frame, a five-frame shot included", async () => {+ const out = (await runAiTool("detect_cuts", { asset_id: "ref" })) as {+ fps: number;+ cuts: { t: number; frame: number; change: number }[];+ shots: { from: number; to: number; frames: number }[];+ coveredTo: number;+ truncated: boolean;+ };+ expect(out.fps).toBe(60);+ expect(out.cuts.map((c) => [c.t, c.frame])).toEqual([+ [1, 60],+ [1.083, 65],+ ]);+ expect(out.cuts[0].change).toBe(100);+ expect(out.shots.map((s) => s.frames)).toEqual([60, 5, 115]);+ expect(out.truncated).toBe(false);+ // Every frame of the span was looked at.+ expect(decoded).toHaveLength(FRAMES);+});++test("detect_cuts scans only the span asked for, and a sound has no cuts", async () => {+ const out = (await runAiTool("detect_cuts", { asset_id: "ref", from: 1.05, to: 2 })) as { cuts: { t: number }[] };+ expect(out.cuts.map((c) => c.t)).toEqual([1.083]);+ expect(await message(runAiTool("detect_cuts", { asset_id: "song" }))).toContain("listen_audio");+});++test("sample_color reads a point and a region at their own moments", async () => {+ const out = (await runAiTool("sample_color", {+ asset_id: "ref",+ samples: [+ { t: 0.5, x: 0.5, y: 0.5 },+ { t: 1.03, x: 0.5, y: 0.5 },+ // The bar at full width, over its own box and a strip of green each side.+ { t: 2.5, x: 0.5, y: 0.3, w: 0.6, h: 0.1 },+ ],+ colors: 2,+ })) as { samples: { t: number; hex: string; spread: number; dominant?: { hex: string; share: number }[] }[] };+ expect(out.samples.map((s) => s.hex)).toEqual(["#C81E1E", "#1E1EC8", "#FF5A00"]);+ expect(out.samples[0].spread).toBe(0);+ const bar = out.samples[2].dominant!;+ expect(bar[0].hex).toBe("#FF5A00");+ expect(bar[0].share).toBeCloseTo(0.83, 1);+ expect(bar[1].hex).toBe("#1EA03C");+});++test("sample_color turns away a call with nothing to read", async () => {+ expect(await message(runAiTool("sample_color", { asset_id: "ref", samples: [] }))).toContain("samples");+});++test("track_element follows a growing bar and names its ease", async () => {+ const out = (await runAiTool("track_element", {+ asset_id: "ref",+ from: 1.3,+ to: 2.3,+ seed: { x: 0.5, y: 0.3, w: 0.08, h: 0.08 },+ })) as {+ color: string;+ path: { t: number; x: number; y: number; w: number; h: number }[];+ moves: { prop: string; from: number; to: number; start: number; end: number; ease: string; error: number }[];+ };+ expect(out.color).toBe("#FF5A00");+ expect(out.path[0].w).toBeCloseTo(0.1, 2);+ expect(out.path[out.path.length - 1].w).toBeCloseTo(0.5, 2);+ expect(out.path[0].y).toBeCloseTo(0.3, 2);+ const grow = out.moves.find((m) => m.prop === "w")!;+ expect(grow.ease).toBe("linear");+ expect(grow.from).toBeCloseTo(1.5, 1);+ expect(grow.to).toBeCloseTo(2.0, 1);+ expect(grow.start).toBeCloseTo(0.1, 2);+ expect(grow.end).toBeCloseTo(0.5, 2);+ // The bar grows about its centre: its position never moves.+ expect(out.moves.find((m) => m.prop === "x")).toBeUndefined();+});++test("track_element's ink box measures a title at one moment", async () => {+ const out = (await runAiTool("track_element", {+ asset_id: "ref",+ from: 2.5,+ match: "ink",+ color: "#FFFFFF",+ seed: { x: 0.5, y: 0.725, w: 0.9, h: 0.15 },+ })) as { path: { x: number; y: number; w: number; h: number }[] };+ expect(out.path).toHaveLength(1);+ const [box] = out.path;+ expect(box.x).toBeCloseTo(0.5, 2);+ expect(box.y).toBeCloseTo(0.725, 2);+ expect(box.w).toBeCloseTo(0.6, 2);+ expect(box.h).toBeCloseTo(0.05, 2);+});++test("detect_cuts says a pass ended at its span bound short of the source", async () => {+ // A long source: one pass covers DETECT_CUTS_SPAN_MAX seconds and hands+ // back where the next one starts.+ const { DETECT_CUTS_SPAN_MAX } = await import("./measure/limits");+ useEditor.setState({+ assets: [{ id: "long", fileName: "long.mp4", name: "Long", type: "video", duration: 500, url: "fixture.mp4", width: W, height: H }],+ });+ const out = (await runAiTool("detect_cuts", { asset_id: "long" })) as { truncated: boolean; note: string };+ expect(out.truncated).toBe(true);+ expect(out.note).toContain("call again");+ expect(DETECT_CUTS_SPAN_MAX).toBeLessThan(500);+});diff --git a/site/src/cut/lib/pi/mutationLedger.ts b/site/src/cut/lib/pi/mutationLedger.tsindex 3d7ce412..7f59caf4 100644--- a/site/src/cut/lib/pi/mutationLedger.ts+++ b/site/src/cut/lib/pi/mutationLedger.ts@@ -13,6 +13,9 @@ const READ_ONLY = new Set(["request_tools","capture_frame","compare_to_source",+ "detect_cuts",+ "sample_color",+ "track_element","watch_video","note_source","detect_silence",diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.tsindex c2d8e4c4..7ad19be3 100644--- a/site/src/cut/server/ai/catalog.ts+++ b/site/src/cut/server/ai/catalog.ts@@ -317,7 +317,7 @@ Filler words: find_filler judges every transcript word in its sentence and returWhen a request depends on what the footage actually contains — "cut the dead air", "clip the best moment", "remove the boring part", "split where the scene changes" — watch it first. Never guess at content you haven't seen.Your eyes and ears:-- watch_video(clip_id | asset_id, from?, to?, interval_seconds?, detail?): samples the SOURCE on a dense steady floor plus a frame just after each line it speaks — what a video shows changes when what it says changes, so a transcribed source is read on its own clock and a caption track lands a frame per card — keeps only the frames that actually differ, and returns them as timestamped contact sheets — cells read left→right, top→bottom, each burned stamp is source seconds, and a gap between stamps means nothing changed there — plus sceneChanges, hard-cut times refined to about a third of a second of the true boundary, and — when the clip has captions or the source's transcript exists — a fused timeline placing each frame inside the speech it belongs to. Coverage ends where the result says: when truncated (span bound, frame cap, or time budget), continue from coveredTo. A shot shorter than the sampling step can fall between candidates — where short cuts matter, re-watch narrow with interval_seconds 0.5–1. \`detail\` sets how many pixels each kept frame is worth: scan (default) tiles small cells for coverage, read enlarges them enough for on-screen text, original returns the source's own frames for matching type, colour and edges. A scan cell is a quarter of the frame's width, so small captions arrive unreadable — reading words off the screen or reproducing a look means spending the pixels, never guessing at what the type says. \`textFits\` rides back with them: frame fractions, quietest first, for where a card can sit over that stretch without covering the picture or the platform's UI — the busy parts are what the watch measured to pick these frames in the first place. Seeing and reading are counted apart: \`unwatchedSeconds\` is source nobody has looked at, \`unreadSeconds\` is source nobody has read at a legible size, and \`unreadFrom\` says where the next close pass starts. Both persist on the asset. Watching builds the source's frame map, persisted on the asset; each media entry's "mapped" spans list where that map exists, and the first watch or listen of a source starts a background sweep that maps the rest, transcript included ("watching": true marks it running; the spans grow as segments land). Use the map to aim your watches — a mapped span is NOT footage you have seen; only the sheets returned to you in this conversation are, so watch any span before you cut inside it.+- watch_video(clip_id | asset_id, from?, to?, interval_seconds?, detail?): samples the SOURCE on a dense steady floor plus a frame just after each line it speaks — what a video shows changes when what it says changes, so a transcribed source is read on its own clock and a caption track lands a frame per card — keeps only the frames that actually differ, and returns them as timestamped contact sheets — cells read left→right, top→bottom, each burned stamp is source seconds, and a gap between stamps means nothing changed there — plus sceneChanges, hard-cut times on the new shot's first frame, and — when the clip has captions or the source's transcript exists — a fused timeline placing each frame inside the speech it belongs to. Coverage ends where the result says: when truncated (span bound, frame cap, or time budget), continue from coveredTo. A shot shorter than the sampling step can fall between candidates; detect_cuts reads every frame. \`detail\` sets how many pixels each kept frame is worth: scan (default) tiles small cells for coverage, read enlarges them enough for on-screen text, original returns the source's own frames for matching type, colour and edges. A scan cell is a quarter of the frame's width, so small captions arrive unreadable — reading words off the screen or reproducing a look means spending the pixels, never guessing at what the type says. \`textFits\` rides back with them: frame fractions, quietest first, for where a card can sit over that stretch without covering the picture or the platform's UI — the busy parts are what the watch measured to pick these frames in the first place. Seeing and reading are counted apart: \`unwatchedSeconds\` is source nobody has looked at, \`unreadSeconds\` is source nobody has read at a legible size, and \`unreadFrom\` says where the next close pass starts. Both persist on the asset. Watching builds the source's frame map, persisted on the asset; each media entry's "mapped" spans list where that map exists, and the first watch or listen of a source starts a background sweep that maps the rest, transcript included ("watching": true marks it running; the spans grow as segments land). Use the map to aim your watches — a mapped span is NOT footage you have seen; only the sheets returned to you in this conversation are, so watch any span before you cut inside it.- find_highlights(clip_id | asset_id, from?, to?, count?): the moments of a long source worth cutting a short from, best first. It reads what is SAID — the transcript, or the project's cues where they cover the source — and ranks every stretch on whether it stands alone cold, whether its opening holds, and whether it finishes; a source with no transcript has nothing to rank, and listen_audio starts one. Each result is one or more spans in source seconds: two spans are two moments that belong together with the middle cut out, and add_clip \`spans\` lays that down. The ranking never saw the picture, so watch_video the spans before committing to them.- note_source(clip_id | asset_id, notes: [{from, to, text}]): writes what you saw onto the source, in source seconds. The sheets are conversation-local and the oldest media drops out as the chat grows; a note is permanent — it saves with the project, comes back as \`observed\` on the media entry in every later turn and chat, and is returned to you by get_state and by watch_video's \`recorded\`. This is how a source longer than one look becomes usable.@@ -329,6 +329,7 @@ Your eyes and ears:- detect_silence(clip_id | asset_id, threshold_db?, min_silence?): silent stretches in source seconds; with clip_id each also carries its timeline times.- detect_beats(clip_id | asset_id, regenerate?): the music's beat grid — bpm and beat times in source seconds, plus timelineBeats ready for split_at with clip_id. The grid persists on the asset, draws as yellow dots on its clips, and every drag or trim snaps to it. "Cut to the beat" / "sync to the music" means detect, then split or place at the timeline beat times; the stored grid carries the user's hand edits, so re-detect only when they ask.- capture_frame(t?): one composited frame, at the given time in seconds or the playhead — for checking the final look. Surveying footage is watch_video's job.+- detect_cuts(asset_id, from?, to?), sample_color(asset_id, samples), track_element(asset_id, from, to?, seed): a source in numbers — every hard cut on its frame, the hex of a region at a second, an element's box per frame with the ease of each move. Replicating a video runs on them.The flow for "edit this for me":1. get_state — every clip, its trim (in/out), speed, gaps, and the other tracks.@@ -352,11 +353,13 @@ Someone points at a video — a file they imported, a link you brought in, a cli**Clipping a long source.** A talk, an interview or a stream is mostly material that only makes sense in place. find_highlights ranks its stretches from the transcript, so an hour is read for its handful of moments without watching all of it; watch_video the spans it returns before committing, since the ranking never saw the picture. add_clip \`spans\` lays a pick down — several stretches of one source as one run of clips, the material between them left out — and a two-span result is one clip whose halves were never next to each other.**Watch the whole thing first.** Every decision below comes off the record, so the record has to be complete: watch from 0 and keep going until \`unwatchedSeconds\` is 0, writing each pass into note_source before the next one. Anything over a couple of minutes gets a coarse pass end to end first (\`interval_seconds\` 15-30) to learn its shape, then a dense pass (0.5) over the stretches that carry text or cut fast. Read every card word for word — its text, its face, where it sits in the frame, when it comes and goes — because a turn later the sheets are gone and the notes are all you have. Reading type is what \`detail\` is for: the dense pass over a stretch carrying words runs at \`detail\` read, and a face or treatment you are about to name runs at original, where the frame arrives at the source's own resolution. A scan cell is a quarter of the frame's width; a small caption in one is a smudge, and a smudge read as a word is how a card ends up saying something the video never said. When the cards ARE the narration — a caption track of what the speaker is saying — the transcript already holds every word, and the close passes are for how they are set: face, weight, size, colour, placement, how many words ride a card, how it enters and leaves. Read a few of those at original and take the rest from the transcript.-**Block out the cuts.** \`sceneChanges\` are the cut times: shot 1 runs from 0 to the first, shot n from one to the next, the last to the end. add_clip with \`blocks\` lays the whole skeleton down in one call — one block per shot on track 0, each the length that shot runs, \`color\` set to what the reference has behind that shot and \`label\` naming what the person drops there. The label rides the timeline chip and the editor state; the frame itself shows the colour alone, so a shot never plays a sentence describing itself. A block is filled by dropping footage on it — it takes the block's place and its length — or with replace_item, so everything you build on top keeps its timing. Cuts timed to a reference are timed to what it sounds like, so that sound belongs on the soundtrack under them: detach_audio with the reference's \`assetId\` puts its whole track there and places none of its picture. When the ask is the reference's own picture playing, add_clip \`spans\` lays its shots down instead, one span per shot at those same times.+**Measure it.** A contact sheet tells you what happens; the numbers come from the source's pixels. detect_cuts reads every frame and returns each cut on its frame and the \`shots\` between them with their lengths — run it over the whole source, continuing from \`coveredTo\`. sample_color reads the hex of any region at any second. track_element gives an element's box frame by frame and the ease of each move; with match ink it gives a title's ink box.-**Rebuild the text as elements.** One add_title per card — the words exactly as they read, the face from the graphics skill's font ids, size in px at 1080 wide, and x/y as the fraction of the frame it sits at (a lower-third card is around y 0.75, a centered statement 0.5). A run of lines timed one after another is add_text_sequence in one call. Match the case the reference uses; match its weight and its color against the frame behind it.+**Block out the cuts.** add_clip with \`blocks\` lays the skeleton from detect_cuts' \`shots\` in one call — one block per shot on track 0, each that shot's length, \`color\` what sample_color reads behind it and \`label\` naming what the person drops there. The label rides the timeline chip and the editor state; the frame itself shows the colour alone, so a shot never plays a sentence describing itself. A block is filled by dropping footage on it — it takes the block's place and its length — or with replace_item, so everything you build on top keeps its timing. Cuts timed to a reference are timed to what it sounds like, so that sound belongs on the soundtrack under them: detach_audio with the reference's \`assetId\` puts its whole track there and places none of its picture. When the ask is the reference's own picture playing, add_clip \`spans\` lays its shots down instead, one span per shot at those same times.-**Then the motion.** set_overlay_keyframes gives each card its own pose track — where it sits, at what scale, at what second — which is how the timing lands on the reference's own frames. set_overlay_animation composes on top for the entrance, the exit, a loop, and what a line does while it holds; 0.3-0.6s reads as an entrance, and a per-letter style is what makes a card read as typed. add_shape is the rest of the graphics: bars, cards, rules, a block of color behind a line.+**Rebuild the text as elements.** One add_title per card — the words exactly as they read, the face from the graphics skill's font ids, size in px at 1080 wide, and x/y the centre of its ink box (track_element, match ink, the text's colour); capitals stand about 0.7 of the size tall, so size is the ink height in px at 1080 wide ÷ 0.7 (h × 1920 ÷ 0.7 on 9:16). A run of lines timed one after another is add_text_sequence in one call. Match the case and weight the reference uses, and its colour from sample_color.++**Then the motion.** set_overlay_keyframes gives each card its own pose track — where it sits, at what scale, at what second. track_element over a move hands you the keys: each move's from/to, start/end and ease, with scale as w over the resting w. set_overlay_animation composes on top for the entrance, the exit, a loop, and what a line does while it holds; 0.3-0.6s reads as an entrance, and a per-letter style is what makes a card read as typed. add_shape is the rest of the graphics: bars, cards, rules, a block of color behind a line.**Look at what you made.** Watching told you what the reference does; it does not tell you whether what you made does it too, and a build that reads right from your own notes is still a guess until you see it. compare_to_source puts the source's frame and yours side by side at the same moment, a few at a time. Run it as you build, not once at the end: the first few shots, then each stretch as you finish it — an error in the first card is in every card after it. Spread the moments over what you laid down, including the middle of a move, and read each pair for colour, the words and their face, where a thing sits, its size, and how far along a move is. A title the same colour as the block behind it, or a shape covering the thing it was meant to sit beside, reads as correct in the document and is only visible here. Fix what differs, check the moments you changed, and keep going until a pass turns up nothing.@@ -662,7 +665,7 @@ Rules:- Transcription tools write tracks the user sees — run subtitles_generate / captions_generate only when captions were asked for, never just to read the words: existing cues are in editor_state, sources you have watched or listened to carry their own transcript (media entries marked speech: "transcribed"; get_state includes the timed segments), and audio plays to you via listen_audio. Don't transcribe a video with no speech (subtitles_from_visuals narrates silent footage), and never invent a spoken transcript.- Voiceovers duck other audio by default so they stay audible. Steer a voiceover's delivery with \`direction\` and inline tags like [whispers] rather than rewriting the script.- generate_image / generate_video / generate_character_video / voiceover_generate / read_subtitles_aloud / generate_music make media through hosted models (spends the user's Donkey credits, needs sign-in); call them when the user asked for the media itself — a request for the prompt or script gets text in chat. generate_music makes a music track — a vocal-free background bed by default ("add music to this video"), or a full sung song (instrumental:false); it's sung music, so a spoken narration is voiceover_generate. Bundled stock media (stock_search / stock_add) is free — use it when it fits. Media the user attached is in \`media\`; pass those asset ids as generation references when they say "use this", and place project assets in the cut with add_clip when they ask for them there ("make a movie from these photos"). Generated media previews on a chat card the user can expand, drag in, or file away; add it to the timeline (add_to_timeline:true or an index) only when they asked for it in the cut. When it lands on a card, close with a statement of where it is ("The voiceover is on the card — drag it in or tell me to place it."). Write a rich, specific prompt from their shorthand. Video renders take a minute or two. "Generate a video of/about X" means one generate_video clip; only when they name a narrated multi-shot production (a story, episode, narrated short) does generate_scene plan it and stop at the storyboard — the user reviews the frames (regenerate_shot redraws any before approval, no credits), then call approve_scene, since it renders many paid shots (regenerate_shot / recut_scene / restyle_scene revise it after). Close a plan message with the approval path as a statement: "Approve it — say so or hit Approve on the card — and I'll render the shots." Tweaks before approval are theirs to name.-- You can see and hear: audio the user attaches to their message plays right in it — answer "what does this say" from what you hear, no tool call. For project footage, watch_video samples a source's frames into timestamped contact sheets (scene changes included) — watch before cutting footage you haven't seen, and write each look down with note_source — sheets are conversation-local, notes stay on the source; listen_audio plays a source's audio — an audio asset or a video/clip's own track, one stretch per call, with \`coveredTo\` and \`unheardSeconds\` saying how far it reached and what nobody has played (an ask about how a whole source sounds listens on from \`unheardFrom\` until that is 0); detect_silence finds dead air; find_highlights ranks the stretches of a long source worth clipping, from what is said over them; detect_beats reads the music's beat grid (persisted on the asset, drawn as dots on its clips, snapped to by every drag) for cutting to the music; capture_frame shows one rendered frame, at the time you pass or the playhead; compare_to_source puts that frame beside the source's own at the same moment, which is how work copied from something is checked against it. The watching-and-cutting skill has the flow.+- You can see and hear: audio the user attaches to their message plays right in it — answer "what does this say" from what you hear, no tool call. For project footage, watch_video samples a source's frames into timestamped contact sheets (scene changes included) — watch before cutting footage you haven't seen, and write each look down with note_source — sheets are conversation-local, notes stay on the source; listen_audio plays a source's audio — an audio asset or a video/clip's own track, one stretch per call, with \`coveredTo\` and \`unheardSeconds\` saying how far it reached and what nobody has played (an ask about how a whole source sounds listens on from \`unheardFrom\` until that is 0); detect_silence finds dead air; find_highlights ranks the stretches of a long source worth clipping, from what is said over them; detect_beats reads the music's beat grid (persisted on the asset, drawn as dots on its clips, snapped to by every drag) for cutting to the music; capture_frame shows one rendered frame, at the time you pass or the playhead; compare_to_source puts that frame beside the source's own at the same moment, which is how work copied from something is checked against it; detect_cuts, sample_color and track_element measure a source's cuts, colours and moving elements as numbers. The watching-and-cutting skill has the flow.- A figure the video states — a number, date, price, name or release in the narration, a title or a caption — goes through check_facts before it goes on screen. Report every claim it does not support, with what the sources say and where, and leave the user's words as they wrote them until they ask.- End on a statement. The final sentence of every reply states what changed, what failed, or the one suggestion you're making; "let me know if", "would you like", "do you want me to", and every other trailing offer or question are banned closers. The only question that may end a reply is the single clarifying one a genuinely ambiguous or blocked ask earns.`;}diff --git a/site/src/cut/server/ai/headlessTools.ts b/site/src/cut/server/ai/headlessTools.tsindex 09bd4b96..fabc4e68 100644--- a/site/src/cut/server/ai/headlessTools.ts+++ b/site/src/cut/server/ai/headlessTools.ts@@ -39,6 +39,9 @@ const PAGE_MEDIA_TOOLS: ReadonlySet<string> = new Set(["freeze_frame","capture_frame","compare_to_source",+ "detect_cuts",+ "sample_color",+ "track_element","render_preview","create_sticker","subtitles_generate",