Open source CapCut alternative.
TypeScript
feat(site): the AI finds and changes a whole row in one call [rebuild]
David Led2027524e814
diff --git a/docs/guides/cut/ai-assistant.md b/docs/guides/cut/ai-assistant.mdindex abe57065..c96ad639 100644--- a/docs/guides/cut/ai-assistant.md+++ b/docs/guides/cut/ai-assistant.md@@ -87,7 +87,7 @@ Deciding what the user wants is prompt text, executed by the model. The prompt oWith a tab attached, tool calls execute in it against the same store the user is editing — same state, same selection, same undo history.- **Readable failure, then retry.** Tools validate and clamp their inputs and throw human-readable errors ("No clip with that id — call get_state for current ids"), which return to the model as tool errors it can recover from. A dropped bridge is separate from a bad request: the idempotent tool listing retries on a momentary engine hiccup so the model never lands with an empty tool set, and an "unreachable" call is one the model quietly re-issues, keeping talk of reconnecting out of the reply.-- **One undo step per turn.** History batches while the assistant is busy, so ⌘Z reverts the whole turn in one step.+- **One undo step per turn.** History batches while the assistant is busy, so ⌘Z reverts the whole turn in one step. A sweep rides the same batch: `select_items` judges every item on a row against one description in a single fan-out, and the mutating tools take `ids`, so finding and changing fifty clips is one call each.- **Small results.** Tools return ids and rounded numbers, plus a short note when behavior surprised ("that spot was taken — slid right").- **Frames become images.** A tool result's `image`/`images` data URLs leave the JSON and reach the model as real images on every path — the engine bridge emits MCP image blocks after the data text, and the Gemini loop rides the first on the function response and the rest as image parts. Watching happens in whatever model the user is chatting with; there is no side model.- **Watching leaves a map.** A watch records which moments and cut candidates a source holds, persisted on the asset so it survives the thread, and the first watch of a source starts a quiet background sweep that maps the rest. The map aims later watches; only sheets returned in the conversation count as footage the model has seen.diff --git a/site/src/cut/components/Inspector.tools.ts b/site/src/cut/components/Inspector.tools.tsindex 1194c821..da06e37c 100644--- a/site/src/cut/components/Inspector.tools.ts+++ b/site/src/cut/components/Inspector.tools.ts@@ -27,7 +27,7 @@ import {MASK_RADIUS_MAX,PEN_MIN_POINTS,} from "@donkeycut/effects-kit";-import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";+import { bool, ids, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";/** Per-field slider hints; ranges interpolate the exported constants so the* schema can never drift from the model the renderer clamps to. */@@ -180,6 +180,7 @@ export const INSPECTOR_TOOLS = ["Update a soundtrack clip: volume (0..3), fadeIn/fadeOut seconds, start position, lane, in/out trim, speed, reverse, hidden, or duck. Audio overlaps across lanes; clips on the same lane slide to its next free slot. To put music under narration, move music to lane 1 and start 0 in the same call, with narration on lane 0. `duck` is voiceover ducking — while this clip plays, ALL other audio drops to that gain (0..1); pass 1 to clear ducking.",inputSchema: obj({id: str("Soundtrack clip id"),+ ids: ids("id"),lane: { type: "integer", minimum: 0, description: "Audio lane, 0-based. Separate lanes play together." },volume: num("0..3 (1 = unchanged, above 1 boosts)"),fadeIn: num("Fade-in seconds"),@@ -196,7 +197,7 @@ export const INSPECTOR_TOOLS = [{name: "set_clip_volume",description: "Set the gain on a video clip's own audio (soundtrack clips use update_audio).",- inputSchema: obj({ clipId: str("Video clip id"), volume: num("0..3 (1 = unchanged, up to 3 boosts the clip's own sound)") }, ["clipId", "volume"]),+ inputSchema: obj({ clipId: str("Video clip id"), ids: ids("clipId"), volume: num("0..3 (1 = unchanged, up to 3 boosts the clip's own sound)") }, ["volume"]),},{name: "set_clip_sound",@@ -256,6 +257,7 @@ export const INSPECTOR_TOOLS = ["Set how a video clip meets its box (the project frame, or its region): 'fit' letterboxes the whole picture (default), 'fill' scales it to cover the box and crops the overflow. zoom pushes further in from there, 1..4. Whenever the picture overflows, panX/panY (-1..1, 0=centered) choose what stays visible — e.g. panY=-1 keeps the top. flipH mirrors the picture left for right (a front-camera take that reads backward, text and all, wants flipH true), flipV top for bottom. Only the fields you pass change; the rest of the clip's framing stays. Landscape footage in a vertical project usually wants fill plus a pan that holds the subject.",inputSchema: obj({clipId: str("Video clip id"),+ ids: ids("clipId"),mode: { type: "string", enum: ["fit", "fill"], description: "Framing mode" },zoom: num("Zoom past the fitted size, 1 (none) .. 4"),panX: num("Crop pan -1 (left) .. 1 (right), when the picture overflows"),@@ -285,7 +287,7 @@ export const INSPECTOR_TOOLS = [name: "set_speed",description:"Set a video clip's playback speed, one rate across the whole clip, and/or play it backward. Faster shortens the clip on the timeline; slower stretches it. Later clips, titles, captions, and soundtrack shift to stay in sync. A clip carrying a speed curve loses it when speed is passed: the rate becomes uniform again (set_speed_curve is the tool for a rate that changes through the footage). `reverse: true` plays the clip's trim backward, picture and sound, at its rate — the clip keeps its length and its curve, and its head now shows source `out`; `reverse: false` turns it forward again. `smooth: true` smooths the clip's slow motion: wherever its rate is under 1×, the frames between source frames are synthesized (the export estimates the motion between them) so the slow stretch moves without stepping; at 1× and above nothing changes. Turn it on for real slow motion from ordinary footage; `smooth: false` shows the source frames as they are.",- inputSchema: obj({ clipId: str("Video clip id"), speed: num("Playback rate (1 = normal, no upper limit)"), reverse: bool("Play the footage backward (true) or forward (false)"), smooth: bool("Smooth slow motion: synthesize the frames between source frames wherever the rate is under 1×") }, ["clipId"]),+ inputSchema: obj({ clipId: str("Video clip id"), ids: ids("clipId"), speed: num("Playback rate (1 = normal, no upper limit)"), reverse: bool("Play the footage backward (true) or forward (false)"), smooth: bool("Smooth slow motion: synthesize the frames between source frames wherever the rate is under 1×") }),},{name: "set_speed_curve",@@ -334,6 +336,7 @@ export const INSPECTOR_TOOLS = [`Apply a named color preset to a video clip, layered under its manual grade. Presets by category — ${gradePresetCatalogText()}. amount 0..1 scales the preset toward neutral (default 1); protect_skin keeps its color shifts off skin tones. Pass preset "none" to clear. Preview, tiles, and export all render the same result.`,inputSchema: obj({clipId: str("Video clip id"),+ ids: ids("clipId"),preset: {type: "string",enum: [...GRADE_PRESET_IDS, "none"],diff --git a/site/src/cut/components/SubtitlesPanel.tools.ts b/site/src/cut/components/SubtitlesPanel.tools.tsindex da05316c..e58e586c 100644--- a/site/src/cut/components/SubtitlesPanel.tools.ts+++ b/site/src/cut/components/SubtitlesPanel.tools.ts@@ -6,7 +6,7 @@* handlers on `SubtitlesToolName`.*/-import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";+import { bool, ids, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";import {DEFAULT_WORDS_PER_CUE,MAX_WORDS_PER_CUE,@@ -146,12 +146,12 @@ export const SUBTITLES_TOOLS = [{name: "update_cue",description: "Edit a subtitle cue's text or retime it (start/end seconds).",- inputSchema: obj({ id: str("Cue id"), text: str("New text"), start: num("Start s"), end: num("End s") }, ["id"]),+ inputSchema: obj({ id: str("Cue id"), ids: ids("id"), text: str("New text"), start: num("Start s"), end: num("End s") }),},{name: "delete_cue",description: "Delete a subtitle cue.",- inputSchema: obj({ id: str("Cue id") }, ["id"]),+ inputSchema: obj({ id: str("Cue id"), ids: ids("id") }),},{name: "merge_cue",diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.tsindex 13810cf4..f00744e5 100644--- a/site/src/cut/components/Timeline.tools.ts+++ b/site/src/cut/components/Timeline.tools.ts@@ -6,7 +6,7 @@* and `aiTools.ts` keys its handlers on `TimelineToolName`.*/-import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";+import { bool, ids, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";import { ITEM_KIND_IDS, canSplitItem } from "@/cut/lib/itemKinds";import { TIMELINE_ITEM_KINDS } from "@/cut/lib/timelineGroups";import {@@ -103,7 +103,7 @@ export const TIMELINE_TOOLS = [{name: "set_clip_muted",description: "Mute or unmute a video clip's own audio.",- inputSchema: obj({ clipId: str("Video clip id"), muted: bool("true to mute") }, ["clipId", "muted"]),+ inputSchema: obj({ clipId: str("Video clip id"), ids: ids("clipId"), muted: bool("true to mute") }, ["muted"]),},{name: "rename_item",@@ -115,7 +115,7 @@ export const TIMELINE_TOOLS = [name: "set_clip_hidden",description:"Hide or show a video clip on any track. A hidden clip stays on the timeline (grayed) but is excluded from playback and export — its span plays black and silent on track 0; an overlay layer just disappears.",- inputSchema: obj({ clipId: str("Video clip id"), hidden: bool("true to hide") }, ["clipId", "hidden"]),+ inputSchema: obj({ clipId: str("Video clip id"), ids: ids("clipId"), hidden: bool("true to hide") }, ["hidden"]),},{name: "set_track_hidden",@@ -154,7 +154,8 @@ export const TIMELINE_TOOLS = [inputSchema: obj({kind: { type: "string", enum: TIMELINE_ITEM_KINDS, description: "Item kind — 'clip' is any video clip, whatever track; 'overlay' is any title-lane element" },id: str("Item id"),- }, ["kind", "id"]),+ ids: ids("id"),+ }, ["kind"]),},{name: "remove_gap",@@ -310,6 +311,23 @@ export const TIMELINE_TOOLS = [),}),},+ {+ name: "select_items",+ description:+ 'Find every item a description covers, in one call: "the clips with no one talking", "the titles that are all caps", "the short ones at the end". Each item on the track is judged against the description at once and the matches come back as ids — this is how a sweep is done, instead of reading the state and deciding one item at a time. Then land the change on all of them with a single call: set_clip_muted, set_clip_hidden, set_clip_volume, set_color_preset, set_speed, set_framing, set_transition, delete_item, update_cue, delete_cue, update_audio and update_overlay all take `ids` in place of their one-item argument, and the whole write is one undo step. Describe what the items have in common, not what to do with them. It judges what the timeline already says about each item — its name, timing, text, and settings — so anything that needs the footage watched or the audio heard is measured first (watch_video, detect_silence, measure_level) and the finding is done here.',+ inputSchema: obj(+ {+ kind: {+ type: "string",+ enum: [...TIMELINE_ITEM_KINDS],+ description: "Which row of the timeline to look through",+ },+ describe: str("What the wanted items have in common, in plain words"),+ limit: num("At most this many ids, best fit first (default: every match)"),+ },+ ["kind", "describe"]+ ),+ },{name: "set_view",description:diff --git a/site/src/cut/components/TransitionsPanel.tools.ts b/site/src/cut/components/TransitionsPanel.tools.tsindex cdf9545b..849a91f1 100644--- a/site/src/cut/components/TransitionsPanel.tools.ts+++ b/site/src/cut/components/TransitionsPanel.tools.ts@@ -7,7 +7,7 @@* renamed tool breaks the build until every side catches up.*/-import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";+import { bool, ids, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";import { ANIM_STYLE_IDS, TRANSITION_STYLE_IDS } from "@/cut/lib/types";export const TRANSITIONS_TOOLS = [@@ -17,13 +17,14 @@ export const TRANSITIONS_TOOLS = ["Set the transition from this clip into the next clip on its track (any video track), in seconds (0 clears it, max 2). 0.5s suits a blend on the picture; the cross dissolve (audiocross) wants about 0.3s, since it is centered on the cut and both clips sound through it. A transition is a blend across the cut — it never moves, trims or overlaps clips, and it plays only while the pair touches. On upper tracks every style blends as an alpha dissolve (the tracks beneath show through). Only valid when a next same-track clip exists. Each edge holds one effect: setting a transition clears the animations adjacent to its joint (and set_animation on that edge replaces the transition). Read the transitions-and-fades skill before styling cuts.",inputSchema: obj({clipId: str("Video clip id (the clip the transition starts from)"),+ ids: ids("clipId"),seconds: num("Transition length in seconds, 0–2 (0 = hard cut)"),style: {type: "string",enum: [...TRANSITION_STYLE_IDS],description: "Transition look (default crossfade)",},- }, ["clipId", "seconds"]),+ }, ["seconds"]),},{name: "remove_transition",diff --git a/site/src/cut/lib/aiToolDef.ts b/site/src/cut/lib/aiToolDef.tsindex b87ec083..b08b941d 100644--- a/site/src/cut/lib/aiToolDef.ts+++ b/site/src/cut/lib/aiToolDef.ts@@ -24,6 +24,15 @@ export const obj = (additionalProperties: false,});+/** The list form of a one-item tool: the same write over every id, in one+ * call and one undo step. What a sweep applies after `select_items` finds+ * the items. */+export const ids = (one: string) => ({+ type: "array",+ items: { type: "string" },+ description: `Several ids to change the same way, in place of ${one} — one call, one undo step (select_items finds them)`,+});+export const num = (description: string) => ({ type: "number", description });export const str = (description: string) => ({ type: "string", description });export const bool = (description: string) => ({ type: "boolean", description });diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.tsindex 8693d909..6b25e0fd 100644--- a/site/src/cut/lib/aiTools.ts+++ b/site/src/cut/lib/aiTools.ts@@ -187,6 +187,7 @@ import { resolveVoiceAsk, synthesizeSpeech, SPEECH_VOICES } from "./tts";import { hostedPost } from "./hosted";import { cutJudge } from "./chatRuntime";import { askJudgeChunked, noul } from "./judge";+import { SWEEP_CHUNK, sweepPicks, sweepQuestions, sweepState, type SweepCandidate } from "./sweepSelect";import { clampVideoDuration, defaultVideoAspects, videoResolutionOf } from "./videoModels";import { DUCK_DEFAULT, generateSubtitlesReadout } from "./voiceover";import {@@ -1387,6 +1388,41 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {return { panel };},+ select_items: async (s, input) => {+ const kind = String(input.kind) as ItemKind;+ if (!ITEM_KIND_IDS.includes(kind)) throw new ToolError(`Unknown kind: ${kind}`);+ const describe = String(input.describe ?? "").trim();+ if (!describe) throw new ToolError("describe is required — say what the wanted items have in common.");+ const items = sweepCandidates(s, kind);+ if (items.length === 0) throw new ToolError(`Nothing of kind "${kind}" on the timeline.`);+ const post = (payload: Record<string, unknown>, signal?: AbortSignal) =>+ hostedPost("/api/inference/judge", payload, signal);+ let answers: Record<string, { type: string; noul?: number }>;+ try {+ answers = (await askJudgeChunked(+ post,+ (keys) => sweepState(describe, items, keys),+ sweepQuestions(describe, items),+ SWEEP_CHUNK+ )) as Record<string, { type: string; noul?: number }>;+ } catch (e) {+ throw new ToolError(`Finding items is unavailable: ${e instanceof Error ? e.message : String(e)}`);+ }+ const hits = sweepPicks(+ answers as Record<string, { type: "noul"; noul: number } | undefined>,+ items,+ cutJudge().sweepFit+ ).sort((a, b) => b.fit - a.fit);+ const kept = isNum(input.limit) ? hits.slice(0, Math.max(1, Math.floor(input.limit))) : hits;+ return {+ kind,+ searched: items.length,+ // The tools that take `ids` land one change on all of them at once.+ ids: kept.map((c) => c.id),+ matched: kept.map((c) => ({ id: c.id, is: c.label })),+ };+ },+group_items: (s) => {const groupId = s.groupSelection();if (!groupId) throw new ToolError("Select at least two timeline items to group.");@@ -4419,7 +4455,51 @@ async function dispatchTool(s: Editor, name: string, input: Record<string, unknoconst key = name === "update_title" ? "update_overlay" : name;const run = (toolRuns as Partial<Record<string, ToolRun>>)[key];if (!run) throw new ToolError(`Unknown tool: ${name}`);- return run(s, input, operation);+ const field = SWEEP_TARGETS[key];+ const ids = field ? sweepIds(key, field, input) : null;+ if (!ids) return run(s, input, operation);++ // A sweep's apply: the same write over every id, as one undo step, with+ // the live state re-read each time because a write can move or remove the+ // items after it. A failure on one item is reported and the rest still land.+ const rest = Object.fromEntries(Object.entries(input).filter(([k]) => k !== "ids"));+ s.beginHistoryBatch();+ const landed: string[] = [];+ const ran: unknown[] = [];+ const failed: { id: string; error: string }[] = [];+ try {+ for (const id of ids) {+ try {+ ran.push(await run(useEditor.getState(), { ...rest, [field]: id }, operation));+ landed.push(id);+ } catch (e) {+ failed.push({ id, error: e instanceof Error ? e.message : String(e) });+ }+ }+ } finally {+ s.endHistoryBatch();+ }+ if (ran.length === 0)+ throw new ToolError(`None of those ${ids.length} landed: ${failed.map((f) => f.error).join("; ")}`);+ // `ids` and `failed` sit at the top level: the turn ledger reads both, so+ // the closing reply is grounded in what actually changed.+ return { ran: ran.length, ids: landed, results: ran, ...(failed.length > 0 ? { failed } : {}) };+}++/** The ids a fan-out writes, or null for a one-item call. Every sweepable+ * tool takes one target or a list, and a call carrying neither reaches no+ * item at all, so it is refused here rather than in each handler. */+function sweepIds(name: string, field: string, input: Record<string, unknown>): string[] | null {+ const list = Array.isArray(input.ids)+ ? input.ids.filter((x): x is string => typeof x === "string" && x.length > 0)+ : null;+ const one = typeof input[field] === "string" && (input[field] as string).length > 0;+ if (list && list.length > 0) {+ if (one) throw new ToolError(`${name} takes ${field} or ids, not both.`);+ return list;+ }+ if (!one) throw new ToolError(`${name} needs ${field}, or ids for several at once (select_items finds them).`);+ return null;}/** Used inside the existing serialized engine session or isolated cloud worker.@@ -4876,6 +4956,82 @@ async function listenToSource(return blobToInlineAudio(await res.blob());}+/** One line per timeline item for a sweep to judge: what it is, when it+ * plays, and the settings already on it. This is everything the finding+ * reads, so an ask about what the footage shows or the audio holds is+ * measured first and the measurement goes in the description. */+function sweepCandidates(s: Editor, kind: ItemKind): SweepCandidate[] {+ const assetName = (id: string) => s.assets.find((a) => a.id === id)?.name ?? "media";+ const span = (start: number, len: number) => `starts ${round2(start)}s, ${round2(len)}s long`;+ if (kind === "clip")+ return s.clips.map((c) => ({+ id: c.id,+ label: [+ `"${c.name || assetName(c.assetId)}"`,+ `video track ${c.track}`,+ span(c.start, retimeOf(c).len),+ c.muted ? "muted" : "",+ c.hidden ? "hidden" : "",+ c.speed && c.speed !== 1 ? `${round2(c.speed)}× speed` : "",+ c.reverse ? "reversed" : "",+ c.grade?.preset ? `${c.grade.preset.id} look` : "",+ c.removal && !c.removal.off ? "cut out of its background" : "",+ ]+ .filter(Boolean)+ .join(", "),+ }));+ if (kind === "audio")+ return s.audioClips.map((a) => ({+ id: a.id,+ label: [+ `"${a.name || assetName(a.assetId)}"`,+ span(a.start, retimeOf(a).len),+ a.volume !== undefined && a.volume !== 1 ? `${Math.round(a.volume * 100)}% volume` : "",+ a.duck !== undefined ? "ducks under speech" : "",+ a.hidden ? "hidden" : "",+ ]+ .filter(Boolean)+ .join(", "),+ }));+ if (kind === "overlay")+ return s.overlays.map((o) => ({+ id: o.id,+ label: [+ isTextOverlay(o) ? `text "${o.text}"` : `${o.kind ?? "element"} "${o.name ?? o.kind ?? "element"}"`,+ `${round2(o.start)}s to ${round2(o.end)}s`,+ o.hidden ? "hidden" : "",+ ]+ .filter(Boolean)+ .join(", "),+ }));+ if (kind === "cue")+ return s.subtitles.cues.map((c) => ({+ id: c.id,+ label: `caption "${c.text}", ${round2(c.start)}s to ${round2(c.end)}s`,+ }));+ return s.transitions.map((t) => ({+ id: t.id,+ label: `${t.style} transition, ${round2(t.seconds)}s at ${round2(t.start)}s${t.hidden ? ", switched off" : ""}`,+ }));+}++/** The one-item argument each tool a sweep lands on takes, so `ids` runs the+ * same write over a list of them. One entry point: the dispatcher fans the+ * call out, and every handler stays a single-item write. */+const SWEEP_TARGETS: Readonly<Record<string, string>> = {+ set_clip_muted: "clipId",+ set_clip_hidden: "clipId",+ set_clip_volume: "clipId",+ set_color_preset: "clipId",+ set_speed: "clipId",+ set_framing: "clipId",+ set_transition: "clipId",+ delete_item: "id",+ update_cue: "id",+ delete_cue: "id",+ update_audio: "id",+};+/** The clip a sound tool names — a video clip or a soundtrack clip, whichever* carries the id — with its current treatment and the store write that* lands a new one. */diff --git a/site/src/cut/lib/judge.ts b/site/src/cut/lib/judge.tsindex 52e801e2..3a0fb42e 100644--- a/site/src/cut/lib/judge.ts+++ b/site/src/cut/lib/judge.ts@@ -2,7 +2,15 @@ import type { PostFn } from "./pi/donkeyStream";import type { Entry, JudgeQuestion, JudgeResult } from "@/lib/inference/judge";export { choice, noul, score } from "@/lib/inference/judge";-export type { ChoiceAnswer, Entry, JudgeQuestion, JudgeResult, NoulAnswer, ScoreAnswer } from "@/lib/inference/judge";+export type {+ ChoiceAnswer,+ Entry,+ JudgeQuestion,+ JudgeResult,+ NoulAnswer,+ NoulQuestion,+ ScoreAnswer,+} from "@/lib/inference/judge";// The page's and the worker's side of a judgment: post the state and the// questions to the judge route through the bound transport (session anddiff --git a/site/src/cut/lib/pi/mutationLedger.test.ts b/site/src/cut/lib/pi/mutationLedger.test.tsnew file mode 100644index 00000000..fb3dd966--- /dev/null+++ b/site/src/cut/lib/pi/mutationLedger.test.ts@@ -0,0 +1,47 @@+import { describe, expect, test } from "bun:test";+import { harvestIds, ledgerText, recordCall, type LedgerRecord } from "./mutationLedger";++describe("recordCall", () => {+ test("a read leaves no record and a mutation keeps its ids", () => {+ const records: LedgerRecord[] = [];+ recordCall(records, "get_state", { videoTrack: [] }, undefined);+ recordCall(records, "set_clip_muted", { id: "c1", muted: true }, undefined);+ expect(records).toEqual([{ name: "set_clip_muted", ids: ["c1"] }]);+ });++ // A sweep returns a value when some of its ids land and others do not, so+ // the ledger reads the shortfall out of the result: the reply has to own+ // the ones that failed.+ test("a sweep that half landed records what changed and what did not", () => {+ const records: LedgerRecord[] = [];+ recordCall(+ records,+ "set_clip_muted",+ {+ ran: 2,+ ids: ["c1", "c3"],+ results: [{ id: "c1" }, { id: "c3" }],+ failed: [{ id: "c2", error: "No video clip with id c2." }],+ },+ undefined,+ );+ expect(records[0]).toEqual({ name: "set_clip_muted", ids: ["c1", "c3"] });+ expect(records[1].error).toContain("c2: No video clip with id c2.");+ const text = ledgerText(records, [])!;+ expect(text).toContain("ran: set_clip_muted (c1, c3)");+ expect(text).toContain("FAILED: set_clip_muted");+ });++ test("a sweep with nothing left over records no failure", () => {+ const records: LedgerRecord[] = [];+ recordCall(records, "delete_item", { ran: 2, ids: ["ov1", "ov2"], results: [] }, undefined);+ expect(records).toHaveLength(1);+ expect(records[0].error).toBeUndefined();+ });+});++describe("harvestIds", () => {+ test("a sweep's landed ids come off the top level", () => {+ expect(harvestIds({ ran: 2, ids: ["c1", "c2"], results: [{ id: "c9" }] })).toEqual(["c1", "c2"]);+ });+});diff --git a/site/src/cut/lib/pi/mutationLedger.ts b/site/src/cut/lib/pi/mutationLedger.tsindex 775d5a58..cdbf1d43 100644--- a/site/src/cut/lib/pi/mutationLedger.ts+++ b/site/src/cut/lib/pi/mutationLedger.ts@@ -28,6 +28,7 @@ const READ_ONLY = new Set(["wait_for_renders","seek","select",+ "select_items","set_playing","set_view",]);@@ -61,8 +62,27 @@ const ID_FIELDS = ["transitionIds","cueIds","assetIds",+ // The ids a sweep's fan-out landed, as it reports them.+ "ids",];+/** What a fan-out could not land, read off its aggregate result. A sweep+ * that half worked returns a value rather than throwing, so the ledger reads+ * the shortfall out of the result to keep the reply honest about it. */+function partialFailure(response: unknown): string | undefined {+ if (!response || typeof response !== "object" || Array.isArray(response)) return undefined;+ const failed = (response as { failed?: unknown }).failed;+ if (!Array.isArray(failed) || failed.length === 0) return undefined;+ const say = failed+ .slice(0, 4)+ .map((f) => {+ const r = (f ?? {}) as { id?: unknown; error?: unknown };+ return `${typeof r.id === "string" ? r.id : "an item"}: ${typeof r.error === "string" ? r.error : "failed"}`;+ })+ .join("; ");+ return `${failed.length} of the ids did not land — ${say}${failed.length > 4 ? "; …" : ""}`;+}+export function harvestIds(response: unknown): string[] {if (!response || typeof response !== "object" || Array.isArray(response)) return [];const r = response as Record<string, unknown>;@@ -82,7 +102,15 @@ export function recordCall(error: string | undefined): void {if (!isMutatingTool(name)) return;- records.push({ name, ids: error ? [] : harvestIds(response), error });+ if (error) {+ records.push({ name, ids: [], error });+ return;+ }+ records.push({ name, ids: harvestIds(response) });+ // A sweep that landed some ids and lost others records both: the ones it+ // changed, and a failure line naming what it did not.+ const partial = partialFailure(response);+ if (partial) records.push({ name, ids: [], error: partial });}/** The ledger as one context message body, or null when the turn has run nodiff --git a/site/src/cut/lib/sweepApply.test.ts b/site/src/cut/lib/sweepApply.test.tsnew file mode 100644index 00000000..9418273e--- /dev/null+++ b/site/src/cut/lib/sweepApply.test.ts@@ -0,0 +1,63 @@+import { beforeEach, expect, test } from "bun:test";+import { runAiTool } from "./aiTools";+import { useEditor } from "./store";++// The apply half of a sweep: one call over a list of ids, landing as one+// undo step, with the shortfall reported rather than swallowed.++beforeEach(async () => {+ useEditor.setState({+ clips: [],+ overlays: [],+ transitions: [],+ audioClips: [],+ assets: [+ { id: "a1", fileName: "one.mp4", name: "One", type: "video", duration: 5, url: "" },+ { id: "a2", fileName: "two.mp4", name: "Two", type: "video", duration: 5, url: "" },+ { id: "a3", fileName: "three.mp4", name: "Three", type: "video", duration: 5, url: "" },+ ],+ });+ for (const id of ["a1", "a2", "a3"]) await runAiTool("add_clip", { asset_id: id });+});++const clipIds = () => useEditor.getState().clips.map((c) => c.id);++test("one call mutes every id and reports them at the top level", async () => {+ const ids = clipIds();+ const out = (await runAiTool("set_clip_muted", { ids, muted: true })) as { ran: number; ids: string[] };+ expect(out.ran).toBe(3);+ expect(out.ids).toEqual(ids);+ expect(useEditor.getState().clips.every((c) => c.muted)).toBe(true);+});++test("the whole sweep is one undo step", async () => {+ await runAiTool("set_clip_muted", { ids: clipIds(), muted: true });+ useEditor.getState().undo();+ expect(useEditor.getState().clips.some((c) => c.muted)).toBe(false);+});++test("an id that does not land is named beside the ones that did", async () => {+ const ids = clipIds();+ const out = (await runAiTool("set_clip_muted", { ids: [...ids, "gone"], muted: true })) as {+ ran: number;+ ids: string[];+ failed: { id: string }[];+ };+ expect(out.ran).toBe(3);+ expect(out.ids).toEqual(ids);+ expect(out.failed.map((f) => f.id)).toEqual(["gone"]);+});++test("a sweep where nothing lands throws", async () => {+ await expect(runAiTool("set_clip_muted", { ids: ["nope", "also-nope"], muted: true })).rejects.toThrow(+ /None of those 2/,+ );+});++test("a call naming no target is refused, and naming both is too", async () => {+ await expect(runAiTool("set_clip_muted", { muted: true })).rejects.toThrow(/needs clipId/);+ await expect(runAiTool("delete_cue", {})).rejects.toThrow(/needs id/);+ await expect(runAiTool("set_clip_muted", { clipId: clipIds()[0], ids: clipIds(), muted: true })).rejects.toThrow(+ /not both/,+ );+});diff --git a/site/src/cut/lib/sweepSelect.test.ts b/site/src/cut/lib/sweepSelect.test.tsnew file mode 100644index 00000000..52db9b89--- /dev/null+++ b/site/src/cut/lib/sweepSelect.test.ts@@ -0,0 +1,52 @@+import { describe, expect, test } from "bun:test";+import { sweepPicks, sweepQuestions, sweepState, type SweepCandidate } from "./sweepSelect";++const items: SweepCandidate[] = [+ { id: "clip-a", label: '"intro.mp4", starts 0s, 4s long' },+ { id: "clip-b", label: '"beach.mp4", starts 4s, 6s long, muted' },+ { id: "clip-c", label: '"outro.mp4", starts 10s, 2s long' },+];++const noul = (n: number) => ({ type: "noul" as const, noul: n });++describe("sweepQuestions", () => {+ test("one yes/no per candidate", () => {+ const qs = sweepQuestions("the muted ones", items);+ expect(Object.keys(qs)).toEqual(["s0", "s1", "s2"]);+ expect(qs.s0.type).toBe("noul");+ });+});++describe("sweepState", () => {+ test("carries the ask, the chunk's own items, and how many there are", () => {+ const state = sweepState("the muted ones", items, ["s1"]) as {+ asked_for: string;+ items: { key: string; is: string }[];+ total: number;+ };+ expect(state.asked_for).toBe("the muted ones");+ expect(state.items).toEqual([{ key: "s1", is: '"beach.mp4", starts 4s, 6s long, muted' }]);+ expect(state.total).toBe(3);+ });++ test("ids stay off the wire — the keys carry them back", () => {+ expect(JSON.stringify(sweepState("x", items, ["s0", "s1", "s2"]))).not.toContain("clip-a");+ });+});++describe("sweepPicks", () => {+ test("every candidate above the floor, with its fit", () => {+ const picks = sweepPicks({ s0: noul(0.9), s1: noul(0.2), s2: noul(0.6) }, items, 0.5);+ expect(picks.map((p) => p.id)).toEqual(["clip-a", "clip-c"]);+ expect(picks[0].fit).toBe(0.9);+ });++ test("a chunk that failed leaves its items out instead of guessing", () => {+ expect(sweepPicks({ s0: noul(0.9) }, items, 0.5).map((p) => p.id)).toEqual(["clip-a"]);+ expect(sweepPicks(null, items, 0.5)).toEqual([]);+ });++ test("nothing matching is an empty list, not everything", () => {+ expect(sweepPicks({ s0: noul(0.1), s1: noul(0.1), s2: noul(0.1) }, items, 0.5)).toEqual([]);+ });+});diff --git a/site/src/cut/lib/sweepSelect.ts b/site/src/cut/lib/sweepSelect.tsnew file mode 100644index 00000000..302b3d8b--- /dev/null+++ b/site/src/cut/lib/sweepSelect.ts@@ -0,0 +1,66 @@+import { noul, type Entry, type NoulAnswer, type NoulQuestion } from "./judge";++// The sweep primitive: which items a description covers.+//+// "Mute the ones with room tone", "delete the short clips", "put the warm+// look on every interview shot" — the finding used to cost one model round+// per item, because the model had to read the state, decide, call a tool,+// read the result, and go again. Here the deciding is one typed judgment per+// candidate, and every one of them is asked in the same fan-out, so the whole+// find settles in the time of one request. What comes back is a list of ids,+// and the tools that take `ids` apply the change to all of them in one call.++export interface SweepCandidate {+ id: string;+ /** How the item reads to the judge: what it is, when it plays, what it holds. */+ label: string;+}++/** Candidates per request, the size find_filler uses. Chunks go out together+ * and any one of them failing fails the sweep, so this sizes the requests+ * rather than isolating them. */+export const SWEEP_CHUNK = 50;++const key = (i: number) => `s${i}`;++/** One yes/no per candidate: does this item match what was asked for? */+export function sweepQuestions(describe: string, items: SweepCandidate[]): Record<string, NoulQuestion> {+ return Object.fromEntries(+ items.map((c, i) => [+ key(i),+ noul(+ `Item \`${key(i)}\` in \`items\` — is it one of the items the editor described as \`asked_for\`?`,+ {+ true: "This item is one of the ones described.",+ false: "This item is not one of them, or the description says nothing about it.",+ },+ ),+ ]),+ );+}++/** The state one chunk reads: what was asked for, and the items this request+ * judges. Ids stay off the wire — the keys carry the identity back. */+export function sweepState(describe: string, items: SweepCandidate[], keys: string[]): Entry {+ const wanted = new Set(keys);+ return {+ asked_for: describe,+ items: items+ .map((c, i) => ({ key: key(i), is: c.label }))+ .filter((c) => wanted.has(c.key)),+ total: items.length,+ };+}++/** The ids above the floor, in the order the timeline holds them. An item+ * the judgment left unanswered reads as no match. */+export function sweepPicks(+ answers: Record<string, NoulAnswer | undefined> | null,+ items: SweepCandidate[],+ floor: number,+): { id: string; label: string; fit: number }[] {+ if (!answers) return [];+ return items+ .map((c, i) => ({ id: c.id, label: c.label, fit: answers[key(i)]?.noul ?? 0 }))+ .filter((c) => c.fit >= floor);+}diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.tsindex 5e7f464a..7c483404 100644--- a/site/src/cut/server/ai/catalog.ts+++ b/site/src/cut/server/ai/catalog.ts@@ -239,6 +239,7 @@ Times are in seconds on the shared timeline. The playhead is currentTime; a skim- set_speed sets a clip's playback rate; it changes the clip's timeline length and pushes overlapping neighbors on its own row. set_speed with \`reverse: true\` plays the clip backward (a rewind, a drop that rises, a pour that un-pours); it changes nothing about length or the curve. set_speed_curve makes the rate change through the footage — nodes of {at: source seconds, speed} or a named preset ramp — for speed ramps, whips, holds, and hits on the beat: detect_beats first, then place nodes at the beats it returns (same unit, no arithmetic). set_speed on a curved clip makes it uniform again. Either tool takes \`smooth: true\` — smooth slow motion — which synthesizes the frames between source frames wherever the clip's rate is under 1× (the export estimates the motion between them) and leaves the rest of the clip alone; editor_state shows it as \`smooth\`. Ordinary footage slowed below about 0.5× wants it on.- set_transition joins a clip into the next one (0–2s, ${TRANSITION_STYLE_IDS.length} styles); set_animation animates one clip's own entrance/exit — read the transitions-and-fades skill before styling cuts. A transition is a bar on its own row at an absolute time, so an edit can leave one lining up with nothing: every timeline mutation reports those as \`parkedTransitions\`, and clearing them (remove_transition) is part of finishing the edit. Grading a clip's picture is the color tools (set_color_preset, set_color_grade, set_color_curves, set_color_wheels, set_color_hsl — read the color-grading skill); a stylized post treatment (grain, VHS, halation…) is add_effect, dropped over the stretch of timeline it covers — as is a treatment on the sound (echo, reverb, telephone…).- Cutting a clip's background away — the AI person matte, ink around the silhouette, a fill behind the subject — is the cutout suite (set_removal, set_removal_stroke, set_removal_background); read the background-removal skill.+- A sweep is find, then apply. select_items judges every item on a row against a description at once and hands back the ids that match ("the clips with no one talking", "the titles that are all caps"); set_clip_muted, set_clip_hidden, set_clip_volume, set_color_preset, set_speed, set_framing, set_transition, delete_item, update_cue, delete_cue, update_audio and update_overlay then take \`ids\` in place of their one-item argument and land the change on all of them in one call, as one undo step. Reading the state and deciding item by item is the slow way to do the same thing. select_items reads what the timeline already says about each item, so anything that depends on the footage or the audio is measured first (watch_video, detect_silence, measure_level) and named in the description.- split_at cuts selected video, audio, elements and captions at the requested time. Transitions stay whole. With no selection, it cuts the track-0 clip at that time.- The Media and Library panels take the same gestures: ⌘/⇧-click a tile, or sweep a rubber-band over the grid, and dragging any picked tile onto the timeline lays the whole set down one after another from the drop point.- The user can multi-select (⌘/⇧-click, or sweep a rubber-band over empty track space) across every kind at once — clips, audio, elements, cues and transitions. Deleting or copying acts on the whole selection, and dragging any member moves the set as one: every item shifts by the same time, relative spacing kept, while unselected items slide out of the way and flow back. A selection of one kind on the multi-row tracks (audio, titles) also retracks together on a vertical drag, one row past the end opening a new track. A hover chip on each video clip toggles its own audio.diff --git a/site/src/lib/config/registry.ts b/site/src/lib/config/registry.tsindex dea1e792..065d64f7 100644--- a/site/src/lib/config/registry.ts+++ b/site/src/lib/config/registry.ts@@ -118,6 +118,8 @@ export const SETTINGS = defineSettings({fillerCut: z.number().min(0).max(1),// The confidence below which a described voice falls back to the default.voicePick: z.number().min(0).max(1),+ // A candidate's fit below which a described sweep leaves it out.+ sweepFit: z.number().min(0).max(1),}).strict(),default: {@@ -130,11 +132,12 @@ export const SETTINGS = defineSettings({stockFit: 0.5,fillerCut: 0.5,voicePick: 0.4,+ sweepFit: 0.5,},public: true,title: "Chat judgments",description:- "Thresholds for the typed judgments that route a chat turn (skill, tool areas), rank stock, find filler words, and resolve described voices.",+ "Thresholds for the typed judgments that route a chat turn (skill, tool areas), pick the items a sweep touches, rank stock, find filler words, and resolve described voices.",},experimentResults: {schema: z