Skip to content

Open source CapCut alternative.

TypeScript

refactor(site): every AI tool definition lives beside its component [rebuild]

David Lecd06498b66d9
diff --git a/site/packages/effects-kit/src/effects.ts b/site/packages/effects-kit/src/effects.ts
index 7d0c9184..1fdd665f 100644
--- a/site/packages/effects-kit/src/effects.ts
+++ b/site/packages/effects-kit/src/effects.ts
@@ -74,7 +74,7 @@ export const EFFECT_IDS: EffectId[] = [
* `vhs` and `grain` are not here — they are effects in their own right above,
* with tearing and moving grain the grades never had.
*/
-const LOOK_EFFECTS: LookStyle[] = [
+export const LOOK_EFFECTS: LookStyle[] = [
"vintage",
"horror",
"halation",
diff --git a/site/src/cut/components/AiPanel.tools.ts b/site/src/cut/components/AiPanel.tools.ts
new file mode 100644
index 00000000..eeee5ebb
--- /dev/null
+++ b/site/src/cut/components/AiPanel.tools.ts
@@ -0,0 +1,82 @@
+/**
+ * The assistant's own tools — its senses on the project (state snapshot,
+ * watching footage, listening, silence detection), the chat-driven fetch and
+ * wait flows, and the server-handled skills library — kept beside the chat
+ * panel that exposes the assistant. The catalog spreads this list into the
+ * model's toolset and `aiTools.ts` keys its handlers on `AiPanelToolName`
+ * (the `server: true` tools run in the engine and take no browser handler).
+ */
+
+import { num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const AI_PANEL_TOOLS = [
+ {
+ name: "get_state",
+ description:
+ "Read the full current editor state: clips, soundtrack, overlay elements (titles, shapes, stickers), subtitles, selection, playhead, view settings, publish metadata. Use this whenever the context snapshot is not enough or might be stale.",
+ inputSchema: obj({}),
+ },
+ {
+ name: "watch_video",
+ description:
+ "Watch a video source with your own eyes: samples its frames at scene changes plus a steady floor into timestamped contact-sheet images, and returns the detected scene-change times (natural cut candidates). Pass clip_id to watch a timeline clip's source (the result includes that clip's source↔timeline time math) or asset_id for any project video or image. The stamp burned into each cell is SOURCE seconds — what trim_clip's in/out use — not timeline seconds. Coverage is capped per call: survey the whole range first, then call again with a narrow from/to and a small interval_seconds where the cut needs care; the result says where coverage stopped. Read the watching-and-cutting skill before editing footage by content.",
+ inputSchema: obj({
+ clip_id: str("Video clip id, track 0 or overlay (defaults from/to to its trimmed in/out)"),
+ asset_id: str("Project asset id (video or image) — watch the source itself"),
+ from: num("Source start s (default: the clip's in, else 0)"),
+ to: num("Source end s (default: the clip's out, else the source's end; spans at most 600s per call)"),
+ interval_seconds: num("Target seconds between sampled frames, 0.5–30 (default spreads ~32 frames across the range)"),
+ }),
+ },
+ {
+ name: "detect_silence",
+ description:
+ "Find silent stretches in a source's audio — dead air, long pauses, gaps between takes. Returns [{start,end,duration}] in SOURCE seconds, plus each one's timeline times when clip_id is passed. Cheap and image-free; pair it with the transcript's cue timings to find filler, then cut with split_at / trim_clip / delete_item — place speech cuts inside these spans (cue timings drift from the audio), and read the watching-and-cutting skill for the pacing rules.",
+ inputSchema: obj({
+ clip_id: str("Clip id — video, overlay, or soundtrack; scopes to its trimmed range and maps results to timeline seconds"),
+ asset_id: str("Project asset id (video or audio)"),
+ from: num("Source start s (default: the clip's in, else 0)"),
+ to: num("Source end s (default: the clip's out, else the source's end)"),
+ threshold_db: num("Loudness below this counts as silence, dBFS (default -30)"),
+ min_silence: num("Shortest silent stretch to report, seconds (default 0.35)"),
+ }),
+ },
+ {
+ name: "listen_audio",
+ description:
+ "Listen with your own ears to a project asset's sound — an audio asset, or the audio track of a video (its speech, music, burned-in narration) — so you can answer what it says or how it sounds. The sound rides back inline (≈12MB cap). Pass clip_id for a timeline clip's source (scopes to its trim) or asset_id for the whole source; add from/to (source seconds) to hear one stretch of a long file. Audio the user attached to their message already plays in it. To WRITE a caption track, use subtitles_generate.",
+ inputSchema: obj({
+ clip_id: str("Clip id — video or soundtrack (defaults from/to to its trimmed in/out)"),
+ asset_id: str("Project asset id (audio or video) — listen to the whole source"),
+ from: num("Source start s (default: the clip's in, else 0)"),
+ to: num("Source end s (default: the clip's out, else the source's end)"),
+ }),
+ },
+ {
+ name: "wait_for_renders",
+ description:
+ "Block until this project's in-flight video renders settle (up to ~100s), then report each one: landed (with its asset id, ready to place) or failed (with the error). Call it whenever the user's ask depends on a render that `renders` in the state shows as running — \"add it when it's done\", \"assemble the clips\" — and then finish the job in the same turn; never tell the user to come back and report when a card appears. If some renders are still running when it returns, say how long they've been going and call it again on the user's go-ahead.",
+ inputSchema: obj({}),
+ },
+ {
+ name: "import_url",
+ description:
+ "Read any URL — TikTok, YouTube, Instagram Reels, an X/Twitter post or Article, an ordinary web page, or a direct video/audio/image link — with the bundled downloader and import what it holds into the project. Free and local. A web page comes back as its article text plus the pictures on it; a post as its video or photos; and the source's own words (returned as sourceText) are quoted for the user beside the media automatically — don't retype them in your reply. A source that is only words returns sourceText with no assets, which is a success: read it and answer from it. This is how you look something up: point it at the page and read what comes back. Media lands on a card in this chat and the user drags it to the timeline, Media, or the Library; place it yourself (add_clip) only when they asked for it in the cut. A short clip downloads in seconds; a long video can take a couple of minutes.",
+ inputSchema: obj({ url: str("The page or media URL to download") }, ["url"]),
+ },
+ {
+ name: "list_skills",
+ description: "List the available skill documents about how this editor works.",
+ inputSchema: obj({}),
+ server: true,
+ },
+ {
+ name: "read_skill",
+ description:
+ "Read a skill document (detailed docs for a part of the editor: every setting, where it lives, and how it behaves). Use before working in an unfamiliar area.",
+ inputSchema: obj({ name: str("Skill name from list_skills") }, ["name"]),
+ server: true,
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type AiPanelToolName = (typeof AI_PANEL_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/AnimationTiles.tools.ts b/site/src/cut/components/AnimationTiles.tools.ts
new file mode 100644
index 00000000..d6754177
--- /dev/null
+++ b/site/src/cut/components/AnimationTiles.tools.ts
@@ -0,0 +1,77 @@
+/**
+ * The assistant's overlay-animation tools — the preset In/Out/Loop ramps the
+ * `AnimationTiles` grid picks from, and the keyframed pose track that lives
+ * under those presets in the inspector. The catalog spreads this list into
+ * the model's toolset and `aiTools.ts` keys its handlers on
+ * `OverlayAnimationToolName`, so the style ids the model can pass are exactly
+ * the ones the tiles render and a removed or renamed tool breaks the build
+ * until every side catches up.
+ */
+
+import {
+ OVERLAY_ANIM_DEFAULT_SECONDS,
+ OVERLAY_ANIM_MAX_SECONDS,
+ OVERLAY_ANIM_MIN_SECONDS,
+ OVERLAY_ANIM_STYLE_IDS,
+ OVERLAY_LOOP_STYLE_IDS,
+} from "@donkeycut/effects-kit";
+import { num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+const RAMP_SECONDS = `${OVERLAY_ANIM_MIN_SECONDS}..${OVERLAY_ANIM_MAX_SECONDS} (default ${OVERLAY_ANIM_DEFAULT_SECONDS})`;
+
+export const OVERLAY_ANIMATION_TOOLS = [
+ {
+ name: "set_overlay_animation",
+ description:
+ `Animate an overlay element (title, shape, or sticker): preset In/Out ramps plus a Loop that runs its whole duration. Omitted slots keep their setting; pass "none" to clear one. In/Out styles: ${OVERLAY_ANIM_STYLE_IDS.join(", ")} — slide names are the motion direction, typewriter animates titles only. Loop styles: ${OVERLAY_LOOP_STYLE_IDS.join(", ")}.`,
+ inputSchema: obj({
+ id: str("Overlay element id"),
+ in_style: {
+ type: "string",
+ enum: [...OVERLAY_ANIM_STYLE_IDS, "none"],
+ description: 'Entrance style, or "none" to clear',
+ },
+ in_seconds: num(`Entrance ramp seconds ${RAMP_SECONDS}`),
+ out_style: {
+ type: "string",
+ enum: [...OVERLAY_ANIM_STYLE_IDS, "none"],
+ description: 'Exit style, or "none" to clear',
+ },
+ out_seconds: num(`Exit ramp seconds ${RAMP_SECONDS}`),
+ loop_style: {
+ type: "string",
+ enum: [...OVERLAY_LOOP_STYLE_IDS, "none"],
+ description: 'Loop style, or "none" to clear',
+ },
+ loop_speed: num("Loop rate multiplier 0.25..4 (default 1)"),
+ }, ["id"]),
+ },
+ {
+ name: "set_overlay_keyframes",
+ description:
+ "Give an overlay element a keyframed pose track, for motion the presets cannot express (a path across the frame, a slow push in, a hold-then-drift). Each key is a whole pose at a time measured in seconds from the element's own start; the pose moves linearly between keys and holds outside them. Omitted fields on a key take the element's current value. Pass an empty list to clear the track and return the element to its resting pose. Preset In/Out/Loop animation still composes on top, so a keyframed title can also fade in.",
+ inputSchema: obj(
+ {
+ id: str("Overlay element id"),
+ keys: {
+ type: "array",
+ description: "Keys in any order; two at the same time collapse to one",
+ items: obj(
+ {
+ t: num("Seconds from the element's start"),
+ x: num("Center x, fraction of frame width 0..1"),
+ y: num("Center y, fraction of frame height 0..1"),
+ scale: num("Size multiplier, 1 = the element's own size (0.1..4)"),
+ rotation: num("Degrees clockwise, -180..180"),
+ opacity: num("0..1"),
+ },
+ ["t"]
+ ),
+ },
+ },
+ ["id", "keys"]
+ ),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type OverlayAnimationToolName = (typeof OVERLAY_ANIMATION_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/AudioPanel.tools.ts b/site/src/cut/components/AudioPanel.tools.ts
new file mode 100644
index 00000000..5026426b
--- /dev/null
+++ b/site/src/cut/components/AudioPanel.tools.ts
@@ -0,0 +1,64 @@
+/**
+ * The assistant's audio-generation tools — AI voiceover, the voice list,
+ * music generation, and speaking a subtitle track aloud — kept beside the
+ * Audio panel whose Voice and Music sub-tabs expose the same generators.
+ * The catalog spreads this list into the model's toolset and `aiTools.ts`
+ * keys its handlers on `AudioToolName`.
+ */
+
+import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const AUDIO_TOOLS = [
+ {
+ name: "list_voices",
+ description:
+ "List the AI voices available for voiceover (Gemini's prebuilt set; each has a one-word character like Warm, Upbeat, Gravelly). Call this when the user asks for a specific kind of voice so you can pass the right voice id to voiceover_generate or read_subtitles_aloud.",
+ inputSchema: obj({}),
+ },
+ {
+ name: "voiceover_generate",
+ description:
+ "Generate a spoken AI voiceover from a script (Donkey's hosted speech model). The audio previews as a playable card in this chat; pass add_to_timeline:true (or a `start`) to also drop it on the soundtrack when the user asked for it in the cut ('add a voiceover', 'narrate this'). When the user asks you to write or rework the script itself, put the text in chat and wait for them to ask for the voiceover. Pick a `voice` id from list_voices, or omit for a good default. `direction` steers delivery in natural language and can ask for another language — 'say it in Spanish' translates the script before synthesis; the script itself may carry inline tags like [whispers] or [excited]. `duck` lowers all other audio to that gain while the voice plays (0..1; ~0.3–0.5 is typical, 1 = don't duck). Needs the user signed in to Donkey (spends their credits).",
+ inputSchema: obj({
+ script: str("What the voice should say"),
+ voice: str("Voice id from list_voices (optional; a sensible default is chosen)"),
+ direction: str("Delivery instruction, e.g. 'Say warmly, like an old friend'; may include a language ask, which translates the script (optional)"),
+ language: str("Pronunciation language as BCP-47, e.g. es-US, ja-JP (optional; default auto-detects — this reads the script as written, it does not translate)"),
+ duck: num("Lower other audio to this gain while the voice plays, 0..1 (default 0.4; 1 = no ducking)"),
+ add_to_timeline: bool("Place it on the soundtrack (default false — it stays on its chat card until the user asks)"),
+ start: num("Timeline start in seconds (passing it implies add_to_timeline; default when placed: the playhead)"),
+ }, ["script"]),
+ },
+ {
+ name: "generate_music",
+ description:
+ "Generate a music track from a text prompt (Donkey's hosted music model, Gemini/Lyria) — a full song with sung vocals, or an instrumental bed. Describe mood, genre, instruments, and tempo. By default it renders an instrumental bed (instrumental:true); pass instrumental:false for a full song, where the model writes and sings its own lyrics — or include your own with [Verse]/[Chorus] tags and a singer description in the prompt. length \"clip\" is ~30s (default), \"song\" a longer ~2-minute track. reference_asset_ids matches a track to the user's own media (\"a song that matches this audio\", \"score the tone of this video\"): pass the project asset ids — an audio track to emulate, or video/images whose mood to carry — and a multimodal pass folds a description of that sound into the prompt (the model itself never hears the audio, so always pass the reference rather than describing it yourself). When you match an audio reference that has singing, pass instrumental:false so the new track sings too. When the user asks you to write or refine the lyrics or brief itself, put it in chat and wait for them to ask for the music. It RETURNS once the track has landed and previews as a playable card in this chat; pass add_to_timeline:true (or a `start`) to drop it on the soundtrack when the user asked for it in the cut. For \"background music for this video\", read the timeline (videoTrack, project duration) to fit the mood and length, keep it instrumental so any speech stays on top, and set those clips' `duck` so the bed dips under them. This is SUNG music — a spoken narration is voiceover_generate. Needs the user signed in to Donkey (spends their credits).",
+ inputSchema: obj({
+ prompt: str("The music to generate — mood, genre, instruments, tempo; for a song, the theme to sing about, or your own [Verse]/[Chorus] lyrics"),
+ instrumental: bool("Vocal-free background bed (default true). Pass false to let the model write and sing a full song"),
+ length: { type: "string", enum: ["clip", "song"], description: "clip = ~30s (default), song = ~2-minute track" },
+ reference_asset_ids: {
+ type: "array",
+ items: { type: "string" },
+ description: "Project asset ids to match the sound of — an audio track to emulate, or video/images whose mood the score should carry. Append @seconds to match the passage around a pinned moment (\"<id>@62\")",
+ },
+ volume: num("Bed volume 0..1.5 when placed (default 0.4 so it sits under speech)"),
+ add_to_timeline: bool("Place it on the soundtrack (default false — it stays on its chat card until the user asks)"),
+ start: num("Timeline start in seconds (passing it implies add_to_timeline; default when placed: the playhead)"),
+ }, ["prompt"]),
+ },
+ {
+ name: "read_subtitles_aloud",
+ description:
+ "Speak one subtitle track's cues as an AI voiceover (Donkey's hosted speech model) — each line placed at its own cue time — and add it to the soundtrack. Turns captions into narration. Requires subtitles to exist first (generate them if needed). `duck` lowers other audio under the voice (0..1). Needs the user signed in to Donkey (spends their credits).",
+ inputSchema: obj({
+ voice: str("Voice id from list_voices (optional)"),
+ direction: str("Delivery instruction, e.g. 'Narrate briskly, documentary style'; may include a language ask, which translates the lines (optional)"),
+ language: str("Pronunciation language as BCP-47, e.g. es-US, ja-JP (optional; default auto-detects — reads cues as written; for another language use a translated subtitle track)"),
+ duck: num("Lower other audio to this gain while the voice plays, 0..1 (default 0.4)"),
+ track: num("Subtitle track to read, 0-based (default: the active track)"),
+ }),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type AudioToolName = (typeof AUDIO_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/Editor.tools.ts b/site/src/cut/components/Editor.tools.ts
new file mode 100644
index 00000000..056b43cc
--- /dev/null
+++ b/site/src/cut/components/Editor.tools.ts
@@ -0,0 +1,23 @@
+/**
+ * The assistant's editor-shell tools — the unlimited undo/redo history the
+ * shell binds to ⌘Z / ⇧⌘Z — kept beside the editor component. The catalog
+ * spreads this list into the model's toolset and `aiTools.ts` keys its
+ * handlers on `EditorToolName`.
+ */
+
+import { obj, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const EDITOR_TOOLS = [
+ {
+ name: "undo",
+ description: "Undo the last edit (unlimited).",
+ inputSchema: obj({}),
+ },
+ {
+ name: "redo",
+ description: "Redo the last undone edit.",
+ inputSchema: obj({}),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type EditorToolName = (typeof EDITOR_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/EffectsPanel.tools.ts b/site/src/cut/components/EffectsPanel.tools.ts
new file mode 100644
index 00000000..055a63e0
--- /dev/null
+++ b/site/src/cut/components/EffectsPanel.tools.ts
@@ -0,0 +1,34 @@
+/**
+ * The assistant's Effects tools, kept beside the panel that gives users the
+ * same controls. The catalog spreads this list into the model's toolset and
+ * `aiTools.ts` keys its handlers on `EffectsToolName`, so an effect added to
+ * the effects-kit registry is advertised here on the next compile and a
+ * removed or renamed tool breaks the build until every side catches up.
+ */
+
+import { EFFECT_IDS, LOOK_EFFECTS } from "@donkeycut/effects-kit";
+import { num, obj, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+/** The effects with a treatment recipe of their own; the rest of the list is
+ * a zoom plus the graded looks. */
+export const EFFECT_TREATMENTS = EFFECT_IDS.filter(
+ (id) => id !== "zoom" && !(LOOK_EFFECTS as string[]).includes(id)
+);
+
+export const EFFECTS_TOOLS = [
+ {
+ name: "add_effect",
+ description:
+ `Add a time-ranged visual effect element over the video: a zoom into part of the frame, the treatments (${EFFECT_TREATMENTS.join(", ")}) or a graded look (${LOOK_EFFECTS.join(", ")}). It filters everything under it for its window (start..end) and stacks with other effects. Tasteful default: one effect at a time, amount under ~0.5.`,
+ inputSchema: obj({
+ effect: { type: "string", enum: [...EFFECT_IDS], description: "Effect id" },
+ start: num("Start time s (default: playhead)"),
+ end: num("End time s (default: start+3)"),
+ amount: num("Strength 0.05..1 (default 0.5). For zoom it is the depth: 0.25 shallow, 0.5 moderate, 1 deep"),
+ focus_x: num("Zoom only: the x of the point to zoom into, 0..1 (default 0.5)"),
+ focus_y: num("Zoom only: the y of the point to zoom into, 0..1 (default 0.5)"),
+ }, ["effect"]),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type EffectsToolName = (typeof EFFECTS_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/ElementsPanel.tools.ts b/site/src/cut/components/ElementsPanel.tools.ts
new file mode 100644
index 00000000..561e831b
--- /dev/null
+++ b/site/src/cut/components/ElementsPanel.tools.ts
@@ -0,0 +1,68 @@
+/**
+ * The assistant's Elements tools — shapes and stickers — kept beside the
+ * panel that gives users the same tiles. The catalog spreads this list into
+ * the model's toolset and `aiTools.ts` keys its handlers on
+ * `ElementsToolName`, so the panel and the model always offer the same
+ * element kinds and a removed or renamed tool breaks the build until every
+ * side catches up.
+ */
+
+import { num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+import { SHAPE_LABELS, type ShapeKind } from "@/cut/lib/types";
+
+/** Every shape the panel's grid shows, in `SHAPE_LABELS` order. */
+export const SHAPE_KINDS = Object.keys(SHAPE_LABELS) as ShapeKind[];
+
+export const ELEMENTS_TOOLS = [
+ {
+ name: "add_shape",
+ description:
+ `Add a vector shape overlay (${SHAPE_KINDS.join(", ")}). Position is the shape center as frame fractions; w/h are frame fractions too (a line/arrow's h is its stroke thickness). Rotation gives lines and arrows their direction.`,
+ inputSchema: obj({
+ shape: { type: "string", enum: [...SHAPE_KINDS], description: "Shape kind" },
+ start: num("Start time s (default: playhead)"),
+ end: num("End time s (default: start+3)"),
+ x: num("Center x 0..1 (default 0.5)"),
+ y: num("Center y 0..1 (default 0.5)"),
+ w: num("Width, fraction of frame width"),
+ h: num("Height, fraction of frame height (line/arrow: thickness)"),
+ fill: str("CSS color (default #FFFFFF)"),
+ fill_opacity: num("Fill opacity 0..1 (rect/ellipse)"),
+ radius: num("Rect corner radius, px at 1080 short side"),
+ stroke_color: str("Outline color (rect/ellipse)"),
+ stroke_width: num("Outline width px at 1080 short side (0 removes it)"),
+ rotation: num("Degrees clockwise, -180..180"),
+ opacity: num("Whole-element opacity 0..1"),
+ }, ["shape"]),
+ },
+ {
+ name: "add_sticker",
+ description:
+ "Add a sticker overlay from a project image asset (asset ids come from `media`; sticker uploads carry origin \"sticker\"). Width is a frame-width fraction; height follows the source's own aspect.",
+ inputSchema: obj({
+ asset_id: str("Project image asset id"),
+ start: num("Start time s (default: playhead)"),
+ end: num("End time s (default: start+3)"),
+ x: num("Center x 0..1 (default 0.5)"),
+ y: num("Center y 0..1 (default 0.5)"),
+ w: num("Width, fraction of frame width (default 0.25)"),
+ rotation: num("Degrees clockwise, -180..180"),
+ opacity: num("Whole-element opacity 0..1"),
+ }),
+ },
+ {
+ name: "create_sticker",
+ description:
+ "Create a custom sticker from an idea: the hosted image model draws it (signed in, spends credits), the background is removed (people on-device, other subjects via hosted matting), a white die-cut outline is added, and it lands as an origin-\"sticker\" asset placed on the timeline at the playhead. Takes ~10-20s; the tool returns when the sticker is placed.",
+ inputSchema: obj({
+ idea: str("What the sticker shows, e.g. \"a corgi in sunglasses\""),
+ start: num("Start time s (default: playhead)"),
+ end: num("End time s (default: start+3)"),
+ x: num("Center x 0..1 (default 0.5)"),
+ y: num("Center y 0..1 (default 0.5)"),
+ w: num("Width, fraction of frame width (default 0.25)"),
+ }, ["idea"]),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type ElementsToolName = (typeof ELEMENTS_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/ElementsPanel.tsx b/site/src/cut/components/ElementsPanel.tsx
index 33132f24..cc3d1ac5 100644
--- a/site/src/cut/components/ElementsPanel.tsx
+++ b/site/src/cut/components/ElementsPanel.tsx
@@ -6,6 +6,7 @@ import { clearElementDrag, setElementDragData } from "@/cut/lib/assetDrag";
import { PICKED_RING, pickGridNav, useAssetPick } from "@/cut/lib/assetPick";
import { useEditor } from "@/cut/lib/store";
import { SHAPE_LABELS, type ShapeKind } from "@/cut/lib/types";
+import { SHAPE_KINDS } from "@/cut/components/ElementsPanel.tools";
import { useLocalPref } from "@/cut/lib/uiState";
import { cn } from "@/lib/utils";
import { CreateSticker, StickerTile, useProjectStickers } from "./Stickers";
@@ -25,8 +26,6 @@ const CATEGORIES = [
{ id: "shapes", label: "Shapes" },
] as const;
-const SHAPE_KINDS = Object.keys(SHAPE_LABELS) as ShapeKind[];
-
export function ElementsPanel({ projectId }: { projectId: string }) {
const readOnly = useEditor((s) => s.readOnly);
const [view, setView] = useLocalPref<Category>("cut-elements-view", "stickers", (v) =>
diff --git a/site/src/cut/components/GeneratePanel.tools.ts b/site/src/cut/components/GeneratePanel.tools.ts
new file mode 100644
index 00000000..64f35d16
--- /dev/null
+++ b/site/src/cut/components/GeneratePanel.tools.ts
@@ -0,0 +1,37 @@
+/**
+ * The assistant's video-generation tools — a single AI clip and the
+ * UGC-style talking-character clip — kept beside the Video tab's generator
+ * panel. The catalog spreads this list into the model's toolset and
+ * `aiTools.ts` keys its handlers on `VideoGenToolName`.
+ */
+
+import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const VIDEO_GEN_TOOLS = [
+ {
+ name: "generate_video",
+ description:
+ "Generate an AI video clip from a text prompt (Donkey's hosted video model: one pass, audio included, 720p, up to ~10s — the model picks the length) — the default for any video ask: \"generate/make a video of/about X\" means ONE clip from this tool, even though it sounds whole. Reach for generate_scene only when they name a narrated multi-shot production (a story, episode, or narrated short). When the user asks you to write or improve the prompt itself, put it in chat and wait for them to ask for the video. reference_asset_id anchors the render on one project asset: for an image or video, by default the tool first designs the opening frame from it with the image model (that still previews in this chat), then animates that exact frame — so tell the user it designs the frame, then renders; for audio, what it carries (speech transcribed, sound described) folds into the prompt instead. When the user wants the referenced image itself brought to life (\"animate this image/picture\"), pass animate_reference:true — the image becomes the literal first frame, unchanged, and the prompt describes only the motion. A safety-blocked or rejected anchor degrades the render on its own (identity reference, then text-only — the same ladder scene shots walk), so never resubmit a failed render with weaker inputs yourself. The render takes a minute or two and the tool RETURNS as soon as it's started — the clip previews as a live card in this chat when it finishes. Pass add_to_timeline:true (or an index) only when they asked for it in the cut; otherwise the user drags it in from the card. Needs the user signed in to Donkey (spends their credits).",
+ inputSchema: obj({
+ prompt: str("The shot to generate — describe motion, subject, and mood"),
+ aspect: { type: "string", enum: ["16:9", "9:16"], description: "Clip shape (default: the supported shape closest to the project aspect)" },
+ reference_asset_id: str("One project asset id to reference — an image or video anchors the clip's opening frame; audio folds what it carries into the prompt. Append @seconds to read a video/audio reference at a pinned moment (\"<id>@16.3\")"),
+ animate_reference: bool("The referenced image IS the opening frame: animate it directly, unchanged, skipping the frame-design step (use when the user says to animate that image)"),
+ add_to_timeline: bool("Insert the clip on the video track when it lands (default false — it stays on its chat card until the user asks)"),
+ index: num("Insert position on the video track (passing it implies add_to_timeline)"),
+ }, ["prompt"]),
+ },
+ {
+ name: "generate_character_video",
+ description:
+ "Generate a UGC-style selfie clip of a stock talking character speaking a line you write (Donkey's hosted video model). Pick a character id from stock_search kind:\"character\" — each has a persona and look; the same person then delivers the line to camera. Like generate_video this RETURNS IMMEDIATELY and the clip previews in this chat a minute or two later; add_to_timeline:true places it when it lands. Needs the user signed in to Donkey (spends their credits).",
+ inputSchema: obj({
+ character_id: str("Talking-character id from stock_search"),
+ line: str("What the character says, spoken to camera"),
+ add_to_timeline: bool("Insert the clip on the video track when it lands (default false — it stays on its chat card until the user asks)"),
+ index: num("Insert position on the video track (passing it implies add_to_timeline)"),
+ }, ["character_id", "line"]),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type VideoGenToolName = (typeof VIDEO_GEN_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/ImageGenPanel.tools.ts b/site/src/cut/components/ImageGenPanel.tools.ts
new file mode 100644
index 00000000..49e9cc4e
--- /dev/null
+++ b/site/src/cut/components/ImageGenPanel.tools.ts
@@ -0,0 +1,29 @@
+/**
+ * The assistant's image-generation tool, kept beside the Image tab's
+ * generator panel. The catalog spreads this list into the model's toolset
+ * and `aiTools.ts` keys its handlers on `ImageGenToolName`.
+ */
+
+import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const IMAGE_GEN_TOOLS = [
+ {
+ name: "generate_image",
+ description:
+ "Generate an AI image from a text prompt (Donkey's hosted image model). Use for B-roll, cover frames, or backgrounds the user doesn't have footage for; when the user asks you to write or improve the prompt itself, put it in chat and wait for them to ask for the image. reference_asset_ids attaches project media the render should draw from (the user's attached references, a clip to restyle) — the prompt is recomposed around them; an audio reference contributes what it carries (its speech transcribed, its sound described) to that recomposed prompt. Returns when the image has landed. The image previews as a card in this chat, where the user can expand it, drag it onto the timeline, or file it into Media or the Library from its \"…\" menu; pass add_to_timeline:true (or an index) only when they asked for it in the cut. Needs the user signed in to Donkey (spends their credits).",
+ inputSchema: obj({
+ prompt: str("What to depict — be specific about subject, style, and lighting"),
+ aspect: { type: "string", enum: ["16:9", "9:16", "1:1"], description: "Image shape (default: the supported shape closest to the project aspect)" },
+ resolution: { type: "string", enum: ["1K", "2K", "4K"], description: "Output detail (default 1K)" },
+ reference_asset_ids: {
+ type: "array",
+ items: { type: "string" },
+ description: "Project asset ids to reference — images and videos ride as pictures; audio folds what it carries into the prompt. Append @seconds to read a video/audio reference at a pinned moment (\"<id>@16.3\")",
+ },
+ add_to_timeline: bool("Insert the still on the video track (default false — it stays on its chat card until the user asks)"),
+ index: num("Insert position on the video track (passing it implies add_to_timeline; 0 = first/cover)"),
+ }, ["prompt"]),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type ImageGenToolName = (typeof IMAGE_GEN_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/Inspector.tools.ts b/site/src/cut/components/Inspector.tools.ts
new file mode 100644
index 00000000..e0ef9198
--- /dev/null
+++ b/site/src/cut/components/Inspector.tools.ts
@@ -0,0 +1,109 @@
+/**
+ * The assistant's inspector tools — the per-item edits the inspector exposes
+ * for whatever is selected: overlay elements, soundtrack clips, and a video
+ * clip's volume, detached audio, framing, speed, and color grade — kept
+ * beside the inspector component. The catalog spreads this list into the
+ * model's toolset and `aiTools.ts` keys its handlers on `InspectorToolName`.
+ */
+
+import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const INSPECTOR_TOOLS = [
+ {
+ name: "update_overlay",
+ description:
+ "Update any overlay element — title, shape, or sticker — by id (from the selection or state). Titles take text/size/font/weight/color/shadow/plate; shapes take w/h/fill/fill_opacity/radius/stroke; stickers take w. Every kind takes timing, position, rotation, opacity, hidden. This is the tool for 'make this text better' requests too.",
+ inputSchema: obj({
+ id: str("Overlay element id"),
+ text: str("New text (titles)"),
+ start: num("Start s"),
+ end: num("End s"),
+ x: num("Center x 0..1"),
+ y: num("Center y 0..1"),
+ size: num("Font size px at 1080w (titles)"),
+ color: str("CSS text color (titles)"),
+ font: str("Font id (titles; see the graphics skill)"),
+ weight: { type: "number", enum: [400, 700], description: "Font weight (titles)" },
+ italic: bool("Italic (titles)"),
+ align: { type: "string", enum: ["left", "center", "right"], description: "Multi-line alignment (titles)" },
+ letter_spacing: num("Tracking in em (titles; 0 = normal)"),
+ line_height: num("Line height multiplier (titles; default 1.25)"),
+ shadow: bool("Drop shadow (titles)"),
+ plate: bool("Backdrop plate (titles)"),
+ behind_subject: bool("Sit the text behind the person in the shot (titles)"),
+ w: num("Width, fraction of frame width (shapes/stickers)"),
+ h: num("Height, fraction of frame height (shapes)"),
+ fill: str("Fill color (shapes)"),
+ fill_opacity: num("Fill opacity 0..1 (rect/ellipse)"),
+ radius: num("Rect corner radius, px at 1080 short side"),
+ stroke_color: str("Outline color — text or shape"),
+ stroke_width: num("Outline width: em for titles (0..0.15), px at 1080 for shapes; 0 removes it"),
+ rotation: num("Degrees clockwise, -180..180 (0 clears)"),
+ opacity: num("Whole-element opacity 0..1 (1 clears)"),
+ hidden: bool("Hide the element without deleting it"),
+ }, ["id"]),
+ },
+ {
+ name: "update_audio",
+ description:
+ "Update a soundtrack clip: volume (0..1.5), fadeIn/fadeOut seconds, start position, in/out trim, speed, hidden, or duck. `duck` is voiceover ducking — while this clip plays, ALL other audio (video-clip sound and other music) drops to that gain (0..1); pass 1 to clear ducking. Use it to make a voiceover sit over quieter music.",
+ inputSchema: obj({
+ id: str("Soundtrack clip id"),
+ volume: num("0..1.5"),
+ fadeIn: num("Fade-in seconds"),
+ fadeOut: num("Fade-out seconds"),
+ start: num("Timeline start s"),
+ in: num("Source in s"),
+ out: num("Source out s"),
+ speed: num("Playback rate (1 = normal, no upper limit)"),
+ duck: num("Duck other audio to this gain while this clip plays, 0..1 (1 clears ducking)"),
+ hidden: bool("Silence the clip without removing it (grayed on the timeline)"),
+ }, ["id"]),
+ },
+ {
+ name: "set_clip_volume",
+ description: "Set the gain on a video clip's own audio (soundtrack clips use update_audio).",
+ inputSchema: obj({ clipId: str("Video clip id"), volume: num("0..1.5 (1 = unchanged)") }, ["clipId", "volume"]),
+ },
+ {
+ name: "detach_audio",
+ description:
+ "Detach Audio: lift a clip's sound onto the soundtrack track (mutes the clip) so it can be edited independently. Select the clip first or pass its id.",
+ inputSchema: obj({ clipId: str("Video clip id (optional if one is selected)") }),
+ },
+ {
+ name: "set_framing",
+ description:
+ "Set how a video clip meets the 9:16 frame: 'fit' letterboxes the whole picture (default), 'fill' scales it to cover the frame and crops the overflow. In fill mode panX/panY (-1..1, 0=centered) choose which part stays visible — e.g. panY=-1 keeps the top.",
+ inputSchema: obj({
+ clipId: str("Video clip id"),
+ mode: { type: "string", enum: ["fit", "fill"], description: "Framing mode" },
+ panX: num("Crop pan -1 (left) .. 1 (right), fill mode only"),
+ panY: num("Crop pan -1 (top) .. 1 (bottom), fill mode only"),
+ }, ["clipId", "mode"]),
+ },
+ {
+ name: "set_speed",
+ description:
+ "Set a video clip's playback speed. Faster shortens the clip on the timeline; slower stretches it. Later titles and captions shift to stay in sync.",
+ inputSchema: obj({ clipId: str("Video clip id"), speed: num("Playback rate (1 = normal, no upper limit)") }, ["clipId", "speed"]),
+ },
+ {
+ name: "set_color_grade",
+ description:
+ "Color-grade a video clip (any track; stills too). Fields patch the clip's current grade: only the ones you pass change, and every value 0 is neutral. reset:true clears the whole grade first; auto:true fits a starting grade from the clip's decoded frame (auto-tone: exposure to middle gray, contrast stretch, gray-world white balance — needs the clip's frame decoded, so seek into it first if this errors), and explicit fields then override it. Preview, timeline thumbnails, and export all render the same result.",
+ inputSchema: obj({
+ clipId: str("Video clip id"),
+ brightness: num("-50..50, 0 neutral"),
+ contrast: num("-50..50, 0 neutral"),
+ saturation: num("-50..50 (-50 = grayscale), 0 neutral"),
+ exposure: num("-50..50 (±1 stop), 0 neutral"),
+ temperature: num("-50..50, positive = warmer, 0 neutral"),
+ hue: num("Hue rotation in degrees, -180..180"),
+ auto: bool("Fit a starting grade from the clip's current frame"),
+ reset: bool("Clear the existing grade before applying fields"),
+ }, ["clipId"]),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type InspectorToolName = (typeof INSPECTOR_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/LibraryView.tools.ts b/site/src/cut/components/LibraryView.tools.ts
new file mode 100644
index 00000000..1995dab7
--- /dev/null
+++ b/site/src/cut/components/LibraryView.tools.ts
@@ -0,0 +1,65 @@
+/**
+ * The assistant's Library tools — browsing the shared shelf, importing its
+ * assets and templates into the project, saving new templates, and
+ * organizing folders — kept beside the Library view that exposes the same
+ * shelf inside the Media tab. The catalog spreads this list into the model's
+ * toolset and `aiTools.ts` keys its handlers on `LibraryToolName`.
+ */
+
+import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const LIBRARY_TOOLS = [
+ {
+ name: "library_list",
+ description:
+ "List the shared Library — reusable media saved across projects: folders, assets (video/audio/image), and templates (saved arrangements of clips, overlays, titles, and captions). Library items live outside the project: library_add imports an asset, template_add re-materializes a template.",
+ inputSchema: obj({}),
+ },
+ {
+ name: "library_add",
+ description:
+ "Copy a Library asset into the project (it appears in `media` and previews as a card in this chat). This is the import step \"library\"-scope attachments need before editor tools can touch them. Pass add_to_timeline:true (or start/index) only when the user asked for it in the cut: video/image land on track 0, audio on the soundtrack.",
+ inputSchema: obj({
+ id: str("Library asset id (from library_list or an attachment's metadata)"),
+ add_to_timeline: bool("Also place it on the timeline (default false — it stays a project asset until the user asks)"),
+ start: num("Timeline start s (implies add_to_timeline)"),
+ index: num("Insert position on video track 0 (video/image; implies add_to_timeline)"),
+ }, ["id"]),
+ },
+ {
+ name: "template_add",
+ description:
+ "Re-materialize a Library template into the project: its media import as assets and its clips, overlays, titles, and captions land editable, exactly as saved (clip layers append to track 0; free-positioned parts line up at the playhead). Call it only when the user asked for the template in the cut.",
+ inputSchema: obj({ id: str("Template id from library_list") }, ["id"]),
+ },
+ {
+ name: "save_template",
+ description:
+ "Save timeline items as a reusable template in this project's Media, kept by reference — the source media plus the edit arranging it, re-editable when added back. The user can push it to the shared Library from the Media panel. Pass the ids of the items to include: video clips (any track), soundtrack clips, titles, and subtitle cues.",
+ inputSchema: obj({
+ name: str("Template name"),
+ item_ids: {
+ type: "array",
+ items: { type: "string" },
+ description: "Timeline item ids to include",
+ },
+ }, ["name", "item_ids"]),
+ },
+ {
+ name: "library_organize",
+ description:
+ "Organize the shared Library: create_folder / rename_folder / delete_folder (a deleted folder's items drop to the root), move_asset files an asset or template into a folder (omit folder_id for the root), delete_asset / delete_template remove an item. Deletes are permanent — projects keep their own copies, but delete only what the user explicitly asked to remove.",
+ inputSchema: obj({
+ action: {
+ type: "string",
+ enum: ["create_folder", "rename_folder", "delete_folder", "move_asset", "delete_asset", "delete_template"],
+ description: "The organize operation",
+ },
+ name: str("Folder name (create_folder, rename_folder)"),
+ folder_id: str("Folder id (rename_folder, delete_folder, move_asset destination — omit for root)"),
+ id: str("Library asset or template id (move_asset, delete_asset, delete_template)"),
+ }, ["action"]),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type LibraryToolName = (typeof LIBRARY_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/Preview.tools.ts b/site/src/cut/components/Preview.tools.ts
new file mode 100644
index 00000000..91b8c05e
--- /dev/null
+++ b/site/src/cut/components/Preview.tools.ts
@@ -0,0 +1,29 @@
+/**
+ * The assistant's preview tools — the playhead, playback, and the composited
+ * frame the preview shows — kept beside the player component that exposes
+ * the same transport. The catalog spreads this list into the model's toolset
+ * and `aiTools.ts` keys its handlers on `PreviewToolName`.
+ */
+
+import { bool, num, obj, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const PREVIEW_TOOLS = [
+ {
+ name: "capture_frame",
+ description:
+ "Capture the composited video frame the preview shows at the playhead as an image. Titles and captions are drawn over the canvas in the UI, so they don't appear here — this checks the footage, not the text.",
+ inputSchema: obj({}),
+ },
+ {
+ name: "seek",
+ description: "Move the playhead to a time (seconds, clamped to the cut).",
+ inputSchema: obj({ t: num("Timeline time in seconds") }, ["t"]),
+ },
+ {
+ name: "set_playing",
+ description: "Start or stop playback.",
+ inputSchema: obj({ playing: bool("true to play, false to pause") }, ["playing"]),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type PreviewToolName = (typeof PREVIEW_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/SceneCard.tools.ts b/site/src/cut/components/SceneCard.tools.ts
new file mode 100644
index 00000000..4baac260
--- /dev/null
+++ b/site/src/cut/components/SceneCard.tools.ts
@@ -0,0 +1,65 @@
+/**
+ * The assistant's scene-production tools — planning a narrated multi-shot
+ * cut, the approval gate, and the shot-level revisions — kept beside the
+ * scene plan card that exposes the storyboard in chat. The catalog spreads
+ * this list into the model's toolset and `aiTools.ts` keys its handlers on
+ * `SceneToolName`.
+ */
+
+import { num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const SCENE_TOOLS = [
+ {
+ name: "generate_scene",
+ description:
+ "Generate a narrated, multi-shot cut assembled on the timeline — script, voiceover, and several shots — from a brief, or animate audio already in the project. Use this only when the user names that kind of production (\"tell a story about…\", \"turn this into a narrated 20-second short\", \"an episode\"); a plain \"generate/make a video of/about X\" is a single clip — generate_video, not this. It writes a script, breaks it into shots, and draws each shot's opening frame, then RETURNS that storyboard and STOPS: the user reviews the frames (and can ask to redraw any of them with regenerate_shot — cheap, no credits at this stage) — get their go-ahead and call approve_scene before the shots render, because each shot spends credits. Pass from_audio_asset_id to animate an audio spine the project already has (a voiceover, recording, or song → shots tiled over it) instead of writing a new script. Read the scene-productions skill before your first call — it covers the brief and look, references, aspect, and the from-audio flow. Needs the user signed in to Donkey (spends credits).",
+ inputSchema: obj({
+ brief: str("What the video is about — the story or subject. Omit only when animating existing audio."),
+ from_audio_asset_id: str("Animate this project audio asset (id from media) — shots tile over it instead of a new script"),
+ audio_language: str("BCP-47 language of the from_audio asset's speech (e.g. ko-KR). REQUIRED with from_audio_asset_id whenever the speech is not English — the on-device transcriber picks its recognizer from it, and the wrong recognizer garbles every shot planned over the audio"),
+ target_seconds: num("Rough total length in seconds, 6–90 (default ~24)"),
+ style: str("Optional look to steer the whole video (e.g. 'moody neon, handheld'); usually left to the brief"),
+ reference_asset_ids: {
+ type: "array",
+ items: { type: "string" },
+ description: "Project image/video asset ids that anchor the look (the user's attached references)",
+ },
+ }),
+ },
+ {
+ name: "approve_scene",
+ description:
+ "Approve the pending generate_scene shot plan and start rendering. The shots render in the background and land on the timeline as they finish. Call this only when the user confirms (\"go\", \"do it\", \"looks good\") — approving spends credits per shot.",
+ inputSchema: obj({}),
+ },
+ {
+ name: "cancel_scene",
+ description:
+ "Stop the scene generation running in this project — planning or rendering halts immediately and the plan is discarded. Clips already placed stay on the timeline; credits already spent are not refunded. Call it when the user wants the run stopped, or to clear the way for a new generate_scene.",
+ inputSchema: obj({}),
+ },
+ {
+ name: "regenerate_shot",
+ description:
+ "Redo one shot of the generated scene by its number (1-based), optionally nudging it (\"wider\", \"at night\", \"more energetic\"). At the storyboard gate (before approval) it redraws just that shot's opening frame — cheap, no credits, the run stays waiting for approval. After the scene has rendered it re-renders the shot and swaps its clip on the timeline by itself — never delete the clip first.",
+ inputSchema: obj({ n: num("Shot number, 1-based"), note: str("Optional change to apply to that shot") }, ["n"]),
+ },
+ {
+ name: "recut_scene",
+ description:
+ "Re-cut a contiguous span of the generated scene's shots: replans just that span against the same audio — it can split into more shots or merge into fewer — keeping the style, cast, and every other shot's clip, then renders only the new shots (credits per new shot). Use it when the user wants a section restructured (\"split the last shot\", \"shots 2-3 drag, tighten them\", \"that part is missing the soccer scene\"); one shot redone as-is is regenerate_shot, a whole-look change is restyle_scene. Only valid after a scene has finished.",
+ inputSchema: obj({
+ from_shot: num("First shot of the span, 1-based"),
+ to_shot: num("Last shot of the span, 1-based (equal to from_shot for one shot)"),
+ instruction: str("What the span should become — the content to cover and how to pace it"),
+ }, ["from_shot", "to_shot", "instruction"]),
+ },
+ {
+ name: "restyle_scene",
+ description:
+ "Restyle the whole generated scene and redo every shot with a new look (\"make it black-and-white film noir\", \"turn it into anime\"). Only valid after a scene has been generated. Spends credits (every shot re-renders), so confirm first.",
+ inputSchema: obj({ style: str("The new look for the whole video") }, ["style"]),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type SceneToolName = (typeof SCENE_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/SidePanel.tools.ts b/site/src/cut/components/SidePanel.tools.ts
new file mode 100644
index 00000000..e07c48c9
--- /dev/null
+++ b/site/src/cut/components/SidePanel.tools.ts
@@ -0,0 +1,60 @@
+/**
+ * The assistant's side-panel tools — opening and collapsing the tabs, the
+ * Media panel's filing and trash, and the Details tab's publish metadata and
+ * project fade — kept beside the side panel that hosts those views
+ * (`MediaPanel` and `PublishPanel` live inside it). The catalog spreads this
+ * list into the model's toolset and `aiTools.ts` keys its handlers on
+ * `SidePanelToolName`.
+ */
+
+import { num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+import { SIDE_PANEL_TABS } from "@/cut/lib/types";
+
+export const SIDE_PANEL_TOOLS = [
+ {
+ name: "set_side_panel",
+ description:
+ "Open a side-panel tab, or collapse the panel to the icon rail with 'none' so the preview canvas takes the freed width — use 'none' when the user asks to clean up or maximize the workspace. 'publish' is the Details tab; 'media' holds the project's files and the shared Library.",
+ inputSchema: obj({
+ panel: {
+ type: "string",
+ enum: [...SIDE_PANEL_TABS, "none"],
+ description: "Tab to open, or 'none' to collapse the panel",
+ },
+ }, ["panel"]),
+ },
+ {
+ name: "file_asset",
+ description:
+ "File a project asset where the user keeps things: to \"media\" moves created media (generated, chat, voiceover…) into the Media panel by clearing its origin tag; to \"library\" copies any project asset into the shared Library for reuse. The same moves as each card's \"…\" menu — make them when the user asks.",
+ inputSchema: obj({
+ asset_id: str("Project asset id from `media`"),
+ to: { type: "string", enum: ["media", "library"], description: "Destination" },
+ }, ["asset_id", "to"]),
+ },
+ {
+ name: "delete_asset",
+ description:
+ "Remove a project asset and every timeline clip that uses it — the Media panel's trash. Removed media does not come back with undo, so call it only when the user explicitly asked to remove that media, and report how many clips went with it.",
+ inputSchema: obj({ asset_id: str("Project asset id from `media`") }, ["asset_id"]),
+ },
+ {
+ name: "set_publish",
+ description:
+ "Set the TikTok publish metadata: caption (4,000 char limit incl. tags), tags (space-separated words, # added automatically), soundTitle, handle.",
+ inputSchema: obj({
+ caption: str("Caption text"),
+ tags: str("Space or comma separated tags"),
+ soundTitle: str("Sound title"),
+ handle: str("Creator handle without @"),
+ }),
+ },
+ {
+ name: "set_project_fade",
+ description:
+ "Set the whole video's fade in from black and/or fade out to black, in seconds (0 clears, max 2). Applied to the final picture and mix at the start/end of the cut, independent of which clip sits there.",
+ inputSchema: obj({ fadeIn: num("Fade-in seconds (omit to keep)"), fadeOut: num("Fade-out seconds (omit to keep)") }),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type SidePanelToolName = (typeof SIDE_PANEL_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/StockPanels.tools.ts b/site/src/cut/components/StockPanels.tools.ts
new file mode 100644
index 00000000..70861a6f
--- /dev/null
+++ b/site/src/cut/components/StockPanels.tools.ts
@@ -0,0 +1,32 @@
+/**
+ * The assistant's stock-media tools, kept beside the stock browsers
+ * (`StockVideosPanel`, `StockImagesPanel`) that expose the same bundled
+ * catalogs. The catalog spreads this list into the model's toolset and
+ * `aiTools.ts` keys its handlers on `StockToolName`.
+ */
+
+import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const STOCK_TOOLS = [
+ {
+ name: "stock_search",
+ description:
+ "Search Cut's bundled stock catalogs: footage clips and stock images across Business/Nature/Travel/City/Technology/Anime/Animal/Food categories, plus talking characters (personas for generate_character_video). Stock is local and free — check it before spending generation credits when existing media could serve. Add a match to the project with stock_add.",
+ inputSchema: obj({
+ query: str("Words to match against prompts, categories, and tags (omit to browse)"),
+ kind: { type: "string", enum: ["video", "image", "character"], description: "Limit to one catalog (default: all)" },
+ }),
+ },
+ {
+ name: "stock_add",
+ description:
+ "Import a stock video or image (by stock_search id) into the project. It previews as a card in this chat; pass add_to_timeline:true (or a `start`) to also drop it on the video track when the user asked for it in the cut. Free — the media ships with Cut.",
+ inputSchema: obj({
+ id: str("Stock item id from stock_search"),
+ add_to_timeline: bool("Place it on video track 0 (default false — it stays on its chat card until the user asks)"),
+ start: num("Timeline start s (passing it implies add_to_timeline; default when placed: appended at the end)"),
+ }, ["id"]),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type StockToolName = (typeof STOCK_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/SubtitlesPanel.tools.ts b/site/src/cut/components/SubtitlesPanel.tools.ts
new file mode 100644
index 00000000..83d216dc
--- /dev/null
+++ b/site/src/cut/components/SubtitlesPanel.tools.ts
@@ -0,0 +1,83 @@
+/**
+ * The assistant's subtitle tools — transcription, caption rewriting, track
+ * management, per-cue edits, and the caption view toggles — kept beside the
+ * subtitles panel that exposes the same transcript editor. The catalog
+ * spreads this list into the model's toolset and `aiTools.ts` keys its
+ * handlers on `SubtitlesToolName`.
+ */
+
+import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const SUBTITLES_TOOLS = [
+ {
+ name: "subtitles_generate",
+ description:
+ "Transcribe the cut on-device (Apple speech) and create subtitle captions on a subtitle track (the active one unless `track` says otherwise; other tracks keep their captions) — it WRITES captions the user sees, so call it only when they asked for subtitles; to just hear what an audio asset says, listen_audio. Cue times come back snapped to the audio, so treat them as in sync — retime a cue only when the user points at one. Runs in the background; returns when finished. If no speech is found, no subtitles are added.",
+ inputSchema: obj({
+ locale: str("Speech language as BCP-47 like en-US (default: the track's language)"),
+ track: num("Subtitle track to write, 0-based (default: the active track)"),
+ }),
+ },
+ {
+ name: "captions_generate",
+ description:
+ "Transcribe (if needed) then REWRITE one subtitle track's captions into punchy social-video captions — short lines that fit inside the video frame (they may wrap onto two lines but never overflow), a few emoji, a curiosity-hook opener. style: clean | hook | punchy (default hook). Cue timings are preserved. It replaces the track's existing text wholesale — for a targeted edit (removing filler words, fixing one line), edit the cues with update_cue instead.",
+ inputSchema: obj({
+ style: str("Caption style: clean, hook, or punchy"),
+ track: num("Subtitle track to rewrite, 0-based (default: the active track)"),
+ }),
+ },
+ {
+ name: "subtitles_from_visuals",
+ description:
+ "Caption a cut that has NO usable speech by watching sampled frames and writing timed narration captions (uses the user's Claude login to look at the frames). Use this instead of subtitles_generate when the video is silent b-roll, music-only, or otherwise has nothing to transcribe. Runs in the background; returns when finished.",
+ inputSchema: obj({
+ locale: str("Caption language as BCP-47 like en-US (default: the track's language)"),
+ track: num("Subtitle track to write, 0-based (default: the active track)"),
+ }),
+ },
+ {
+ name: "subtitles_add_track",
+ description:
+ "Add a subtitle track (up to 3, one language each — e.g. English on track 0, Korean on track 1; each shows as its own caption line, draggable to its own spot). The new track becomes active and starts empty: fill it with subtitles_translate_track, or subtitles_generate in its language.",
+ inputSchema: obj({ language: str("The track's language as BCP-47, e.g. ko-KR (optional)") }),
+ },
+ {
+ name: "subtitles_remove_track",
+ description:
+ "Remove a subtitle track and its captions; higher tracks shift down. Removing the only track empties it — so this also answers \"delete the subtitles\".",
+ inputSchema: obj({ track: num("Track to remove, 0-based") }, ["track"]),
+ },
+ {
+ name: "subtitles_translate_track",
+ description:
+ "Translate existing captions into another language on their own subtitle track — the way to answer \"add Korean subtitles\". Reuses the track already set to that language or adds one (max 3), then translates the source track's cues one-to-one, timings kept. The captions must exist first (subtitles_generate).",
+ inputSchema: obj({
+ language: str("Target language as BCP-47, e.g. ko-KR"),
+ from_track: num("Source track, 0-based (default: the first track with captions)"),
+ }, ["language"]),
+ },
+ {
+ name: "subtitles_set_view",
+ description: "Toggle subtitles on the video (preview + export burn-in) and/or the timeline cue track.",
+ inputSchema: obj({ showOnVideo: bool("Captions on the video"), showOnTimeline: bool("Cue track on the timeline") }),
+ },
+ {
+ name: "update_cue",
+ description: "Edit a subtitle cue's text or retime it (start/end seconds).",
+ inputSchema: obj({ id: str("Cue id"), text: str("New text"), start: num("Start s"), end: num("End s") }, ["id"]),
+ },
+ {
+ name: "delete_cue",
+ description: "Delete a subtitle cue.",
+ inputSchema: obj({ id: str("Cue id") }, ["id"]),
+ },
+ {
+ name: "merge_cue",
+ description:
+ "Merge a subtitle cue into the previous cue on its own track (joins their text and timing). Not valid for a track's first cue.",
+ inputSchema: obj({ id: str("Cue id to merge into its predecessor") }, ["id"]),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type SubtitlesToolName = (typeof SUBTITLES_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.ts
new file mode 100644
index 00000000..d1048994
--- /dev/null
+++ b/site/src/cut/components/Timeline.tools.ts
@@ -0,0 +1,206 @@
+/**
+ * The assistant's timeline tools — selection, cutting, placement, tracks,
+ * overlay video layers, the toolbar's Text button, freeze frames, and the
+ * timeline view — kept beside the timeline component that exposes the same
+ * editing surface. The catalog spreads this list into the model's toolset
+ * and `aiTools.ts` keys its handlers on `TimelineToolName`.
+ */
+
+import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const TIMELINE_TOOLS = [
+ {
+ name: "select",
+ description:
+ "Select a video clip (on any track), soundtrack clip, or overlay element — title, shape, or sticker (or clear the selection). Selection drives the inspector panel.",
+ inputSchema: obj({
+ kind: { type: "string", enum: ["clip", "audio", "overlay", "none"], description: "What to select — 'clip' is any video clip, whatever track; 'overlay' is any title-lane element" },
+ id: str("The item id (omit for kind=none)"),
+ }, ["kind"]),
+ },
+ {
+ name: "split_at",
+ description:
+ "Split the video (or a selected soundtrack/overlay clip) at a time, like pressing S. Omit t to split at the playhead.",
+ inputSchema: obj({ t: num("Timeline seconds to cut at (optional)") }),
+ },
+ {
+ name: "move_clip",
+ description:
+ "Reorder a track-0 video clip to a new index: it lifts out (its old spot becomes a gap) and clips from the landing index shift right to make room — nothing else moves, so sound and titles stay synced. To move one clip in time, use place_clip.",
+ inputSchema: obj({ clipId: str("Video clip id"), toIndex: num("Target index, 0-based") }, ["clipId", "toIndex"]),
+ },
+ {
+ name: "place_clip",
+ description:
+ "Move a track-0 video clip to a timeline start time (seconds). The track is free-positioned: gaps are allowed and play black. If another clip occupies that spot, the clip slides right to the next free one.",
+ inputSchema: obj({ clipId: str("Video clip id"), start: num("Target timeline start s") }, ["clipId", "start"]),
+ },
+ {
+ name: "add_clip",
+ description:
+ "Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at `start`, inserted at `index`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at `start`, default the playhead). Asset ids come from `media` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut (\"add my beach photo\", \"stitch these into a movie\"); otherwise media stays on its card or panel for them to drag.",
+ inputSchema: obj({
+ asset_id: str("Project asset id from `media` in editor_state"),
+ start: num("Timeline start s"),
+ index: num("Insert position on video track 0 (video/image only; 0 = first)"),
+ }, ["asset_id"]),
+ },
+ {
+ name: "trim_clip",
+ description:
+ "Set a video clip's trim points inside its source media (seconds). Changing `in` hides the leading part; `out` the trailing part.",
+ inputSchema: obj({ clipId: str("Video clip id"), in: num("New in point (optional)"), out: num("New out point (optional)") }, ["clipId"]),
+ },
+ {
+ name: "refine_speech_cuts",
+ description:
+ "True up speech-cut edges mechanically: re-scans the audio around each clip's in/out and re-trims so every edge sits a ~0.25s breath inside a real pause — an edge that clips a word moves outward to the next pause, stray dead air trims off. Clip spacing survives: a tightened clip closes up, butted joints stay butted, and deliberate gaps keep their width. An edge with no pause within ~2s stays put and comes back flagged. Run it on every clip you recut for speech, after cutting and before listening; leave out clips whose edge you placed mid-speech on purpose.",
+ inputSchema: obj(
+ {
+ clip_ids: {
+ type: "array",
+ items: { type: "string" },
+ description: "Video clip ids whose in/out edges are speech cuts",
+ },
+ },
+ ["clip_ids"]
+ ),
+ },
+ {
+ name: "set_clip_muted",
+ description: "Mute or unmute a video clip's own audio.",
+ inputSchema: obj({ clipId: str("Video clip id"), muted: bool("true to mute") }, ["clipId", "muted"]),
+ },
+ {
+ name: "set_clip_hidden",
+ description:
+ "Hide or show a video clip on any track. A hidden clip stays on the timeline (grayed) but is excluded from playback and export — its span plays black and silent on track 0; an overlay layer just disappears.",
+ inputSchema: obj({ clipId: str("Video clip id"), hidden: bool("true to hide") }, ["clipId", "hidden"]),
+ },
+ {
+ name: "set_track_hidden",
+ description:
+ "Hide or show a whole timeline row at once — the track header's eye toggle. Applies to every item on that row: a hidden video track plays black and silent, a hidden soundtrack lane is silent, hidden text (title) lanes and subtitle tracks drop out of the picture. Items stay on the timeline, grayed.",
+ inputSchema: obj({
+ kind: { type: "string", enum: ["video", "soundtrack", "text", "subtitles"], description: "Which row kind" },
+ track: num("Track/lane index, 0-based"),
+ hidden: bool("true to hide"),
+ }, ["kind", "track", "hidden"]),
+ },
+ {
+ name: "set_track_muted",
+ description:
+ "Mute or unmute every clip on one video track at once — the track header's speaker toggle. The picture keeps playing.",
+ inputSchema: obj({ track: num("Video track index, 0-based"), muted: bool("true to mute") }, ["track", "muted"]),
+ },
+ {
+ name: "delete_item",
+ description:
+ "Delete a video clip (any track), soundtrack clip, or overlay element (title/shape/sticker) by id. While track 0 is the only video track, deleting a track-0 clip ripples: its footprint closes and everything after it — clips, titles, captions, soundtrack — slides left in sync; items wholly inside that span are removed with it, straddlers are trimmed. With overlay video tracks present the delete leaves its gap in place (remove_gap closes it). Deletes on other tracks remove just that item.",
+ inputSchema: obj({
+ kind: { type: "string", enum: ["clip", "audio", "overlay"], description: "Item kind — 'clip' is any video clip, whatever track; 'overlay' is any title-lane element" },
+ id: str("Item id"),
+ }, ["kind", "id"]),
+ },
+ {
+ name: "remove_gap",
+ description:
+ "Close the empty span on one video track containing `at` seconds: that track's later clips slide left to shut the gap; every other track, titles, captions, and the soundtrack stay put. Errors unless `at` falls inside a gap on that track.",
+ inputSchema: obj({
+ at: num("A time (seconds) inside the gap to close"),
+ track: num("Video track the gap is on (default 0)"),
+ }, ["at"]),
+ },
+ {
+ name: "add_overlay_video",
+ description:
+ "Put a project video or image asset on an overlay video track, composited over track 0: tracks stack bottom-up, and the topmost full-frame clip covers everything below. Pick a layout to share the frame — halves for a split screen, pip for picture-in-picture. Asset ids come from `media` in the editor state.",
+ inputSchema: obj({
+ asset_id: str("Project asset id (video or image)"),
+ start: num("Timeline start s (default: the playhead)"),
+ track: num("Video track, 1 or higher; 1 = first layer above track 0, higher = further in front (default 1)"),
+ layout: {
+ type: "string",
+ enum: ["full", "top", "bottom", "left", "right", "pip"],
+ description: "Frame region (default full = covers the frame)",
+ },
+ }, ["asset_id"]),
+ },
+ {
+ name: "update_overlay_video",
+ description:
+ "Update an overlay video clip: move it (start, track), trim (in/out), mute, hide, change its frame region (layout preset, or a custom region rect in frame fractions), fit, or speed.",
+ inputSchema: obj({
+ id: str("Overlay video clip id"),
+ start: num("Timeline start s"),
+ in: num("Source in s"),
+ out: num("Source out s"),
+ track: num("Video track, 1 or higher; higher = further in front"),
+ muted: bool("Mute the clip's own audio"),
+ hidden: bool("Hide the layer without deleting it"),
+ layout: {
+ type: "string",
+ enum: ["full", "top", "bottom", "left", "right", "pip"],
+ description: "Frame region preset",
+ },
+ region: obj({
+ x: num("Left edge 0..1"),
+ y: num("Top edge 0..1"),
+ w: num("Width 0..1"),
+ h: num("Height 0..1"),
+ }, ["x", "y", "w", "h"]),
+ fit: { type: "string", enum: ["fit", "fill"], description: "How the video meets its region" },
+ speed: num("Playback rate (1 = normal, no upper limit)"),
+ }, ["id"]),
+ },
+ {
+ name: "add_title",
+ description:
+ "Add a text title overlay. Position is the text center as a fraction of the frame (x,y in 0..1; y=0.42 is the default band). Size is px at 1080-wide. Font ids come from the graphics skill (system set + the bundled Google families).",
+ inputSchema: obj({
+ text: str("The title text (\\n for line breaks)"),
+ start: num("Start time s (default: playhead)"),
+ end: num("End time s (default: start+3)"),
+ x: num("Center x 0..1 (default 0.5)"),
+ y: num("Center y 0..1 (default 0.42)"),
+ size: num("Font size px at 1080w (default 88)"),
+ color: str("CSS color (default #FFFFFF)"),
+ font: str("Font id (see the graphics skill; default sf)"),
+ weight: { type: "number", enum: [400, 700], description: "Font weight" },
+ italic: bool("Italic"),
+ align: { type: "string", enum: ["left", "center", "right"], description: "Multi-line alignment (default center)" },
+ letter_spacing: num("Tracking in em (0 = normal, 0.1 = airy)"),
+ line_height: num("Line height multiplier (default 1.25)"),
+ stroke_color: str("Text outline color"),
+ stroke_width: num("Text outline width in em (0..0.15; 0 removes it)"),
+ shadow: bool("Drop shadow (default true)"),
+ plate: bool("Translucent plate behind text (default false)"),
+ behind_subject: bool("Sit the text behind the person in the shot (needs a clearly separated speaker)"),
+ rotation: num("Degrees clockwise, -180..180"),
+ opacity: num("Whole-element opacity 0..1"),
+ }, ["text"]),
+ },
+ {
+ name: "freeze_frame",
+ description:
+ "Extract the video frame at a time (default: the playhead — what the user is looking at) as a still clip and insert it into the timeline. Default insert position is index 0, making it the first thing viewers see (a cover/hook frame).",
+ inputSchema: obj({
+ t: num("Timeline time of the frame to grab (default: playhead)"),
+ duration: num("Still clip length in seconds, 0.5–10 (default 1)"),
+ index: num("Insert position on the video track (default 0 = first)"),
+ }),
+ },
+ {
+ name: "set_view",
+ description:
+ "Adjust the timeline view: zoom (pxPerSec 12..800), fit the whole cut, or panel height (170..600).",
+ inputSchema: obj({
+ pxPerSec: num("Zoom in px per second"),
+ fit: bool("Fit the whole cut to the window"),
+ timelineH: num("Timeline panel height px"),
+ }),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type TimelineToolName = (typeof TIMELINE_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/TopBar.tools.ts b/site/src/cut/components/TopBar.tools.ts
new file mode 100644
index 00000000..b76bd992
--- /dev/null
+++ b/site/src/cut/components/TopBar.tools.ts
@@ -0,0 +1,30 @@
+/**
+ * The assistant's top-bar tools — the aspect pill, the project name, and the
+ * export dialog — kept beside the top bar that exposes the same controls.
+ * The catalog spreads this list into the model's toolset and `aiTools.ts`
+ * keys its handlers on `TopBarToolName`.
+ */
+
+import { obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+
+export const TOP_BAR_TOOLS = [
+ {
+ name: "set_aspect",
+ description:
+ "Set the project's output frame ratio as \"W:H\". The frame is the user's own setting, so call this only when they asked for a shape or a format (\"make it vertical\", \"for TikTok\", \"square\") — never inferred from a reference image's shape, a platform named in passing, or what would suit a generation you are about to run. Presets: 16:9 (YouTube), 9:16 (TikTok/Reels/Shorts), 1:1, 4:3, 3:4, 2:1 — but any ratio up to an 8:1 shape works, with whole or decimal sides (\"9:5\", \"2.39:1\" — stored reduced, so 2.39:1 becomes 239:100). The frame renders with its short side at 1080px: 9:16 → 1080×1920, 16:9 → 1920×1080, 9:5 → 1944×1080.",
+ inputSchema: obj({ aspect: { type: "string", description: "Output ratio as \"W:H\", e.g. \"9:16\", \"1:1\", \"9:5\", \"2.39:1\"" } }, ["aspect"]),
+ },
+ {
+ name: "set_project_name",
+ description: "Rename the current project.",
+ inputSchema: obj({ name: str("New project name") }, ["name"]),
+ },
+ {
+ name: "open_export",
+ description:
+ "Open the export dialog so the user can render the cut (presets from Original quality down to Draft 720p). Exporting itself stays a user action.",
+ inputSchema: obj({}),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type TopBarToolName = (typeof TOP_BAR_TOOLS)[number]["name"];
diff --git a/site/src/cut/components/TransitionsPanel.tools.ts b/site/src/cut/components/TransitionsPanel.tools.ts
new file mode 100644
index 00000000..08d6a4d2
--- /dev/null
+++ b/site/src/cut/components/TransitionsPanel.tools.ts
@@ -0,0 +1,45 @@
+/**
+ * The assistant's Transitions tools — the styled join between two clips and
+ * a clip's own entrance/exit — kept beside the panel that gives users the
+ * same tiles. The catalog spreads this list into the model's toolset and
+ * `aiTools.ts` keys its handlers on `TransitionsToolName`, so the style ids
+ * the model can pass are exactly the ones the panel renders and a removed or
+ * renamed tool breaks the build until every side catches up.
+ */
+
+import { num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+import { ANIM_STYLE_IDS, TRANSITION_STYLE_IDS } from "@/cut/lib/types";
+
+export const TRANSITIONS_TOOLS = [
+ {
+ name: "set_transition",
+ description:
+ "Set the transition from this clip into the next clip on its track (any video track), in seconds (0 clears it, max 2). A transition is a blend across the cut — it never moves, trims or overlaps clips, and it plays only while the pair touches. On upper tracks every style blends as an alpha dissolve (the tracks beneath show through). Only valid when a next same-track clip exists. Each edge holds one effect: setting a transition clears the animations adjacent to its joint (and set_animation on that edge replaces the transition). Read the transitions-and-fades skill before styling cuts.",
+ inputSchema: obj({
+ clipId: str("Video clip id (the clip the transition starts from)"),
+ seconds: num("Transition length in seconds, 0–2 (0 = hard cut)"),
+ style: {
+ type: "string",
+ enum: [...TRANSITION_STYLE_IDS],
+ description: "Transition look (default crossfade)",
+ },
+ }, ["clipId", "seconds"]),
+ },
+ {
+ name: "set_animation",
+ description:
+ "Animate one clip's own entrance (which:'in') or exit (which:'out'): fade, zoom, pop, or a slide named by its motion direction. style 'none' clears. Track-0 clips take every style; upper-track clips only fade and zoom. Each edge holds one effect, last pick wins: animating an edge a transition owns replaces that transition, and setting a transition clears the animations adjacent to its joint. At an abutting cut the animation plays over the neighbor's held frame; at the timeline's ends and across gaps it plays against black.",
+ inputSchema: obj({
+ clipId: str("Video clip id"),
+ which: { type: "string", enum: ["in", "out"], description: "Entrance or exit" },
+ style: {
+ type: "string",
+ enum: [...ANIM_STYLE_IDS, "none"],
+ description: "Animation style, or 'none' to clear",
+ },
+ seconds: num("Ramp length in seconds, 0.1–2 (default 0.5)"),
+ }, ["clipId", "which", "style"]),
+ },
+] as const satisfies readonly AiToolDef[];
+
+export type TransitionsToolName = (typeof TRANSITIONS_TOOLS)[number]["name"];
diff --git a/site/src/cut/lib/aiToolDef.ts b/site/src/cut/lib/aiToolDef.ts
new file mode 100644
index 00000000..b87ec083
--- /dev/null
+++ b/site/src/cut/lib/aiToolDef.ts
@@ -0,0 +1,29 @@
+/**
+ * The shape of one assistant tool and the JSON Schema helpers for writing
+ * them. Panel-owned tools live beside their panel components (the
+ * `*.tools.ts` siblings under `components/`); the catalog imports those
+ * lists, so this module stays dependency-free enough for the engine binary,
+ * the browser, and the panels to share.
+ */
+
+export interface AiToolDef {
+ name: string;
+ description: string;
+ inputSchema: Record<string, unknown>;
+ /** Handled directly by the server (no browser round-trip). */
+ server?: boolean;
+}
+
+export const obj = (
+ properties: Record<string, unknown>,
+ required: string[] = []
+): Record<string, unknown> => ({
+ type: "object",
+ properties,
+ required,
+ additionalProperties: false,
+});
+
+export const num = (description: string) => ({ type: "number", description });
+export const str = (description: string) => ({ type: "string", description });
+export const bool = (description: string) => ({ type: "boolean", description });
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index 3a2c0327..0c6da6f5 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -14,6 +14,24 @@ import {
type OverlayAnimStyle,
type OverlayLoopStyle,
} from "@donkeycut/effects-kit";
+import type { AiPanelToolName } from "@/cut/components/AiPanel.tools";
+import type { OverlayAnimationToolName } from "@/cut/components/AnimationTiles.tools";
+import type { AudioToolName } from "@/cut/components/AudioPanel.tools";
+import type { EditorToolName } from "@/cut/components/Editor.tools";
+import type { EffectsToolName } from "@/cut/components/EffectsPanel.tools";
+import { SHAPE_KINDS, type ElementsToolName } from "@/cut/components/ElementsPanel.tools";
+import type { VideoGenToolName } from "@/cut/components/GeneratePanel.tools";
+import type { ImageGenToolName } from "@/cut/components/ImageGenPanel.tools";
+import type { InspectorToolName } from "@/cut/components/Inspector.tools";
+import type { LibraryToolName } from "@/cut/components/LibraryView.tools";
+import type { PreviewToolName } from "@/cut/components/Preview.tools";
+import type { SceneToolName } from "@/cut/components/SceneCard.tools";
+import type { SidePanelToolName } from "@/cut/components/SidePanel.tools";
+import type { StockToolName } from "@/cut/components/StockPanels.tools";
+import type { SubtitlesToolName } from "@/cut/components/SubtitlesPanel.tools";
+import type { TimelineToolName } from "@/cut/components/Timeline.tools";
+import type { TopBarToolName } from "@/cut/components/TopBar.tools";
+import type { TransitionsToolName } from "@/cut/components/TransitionsPanel.tools";
import { apiFetch, apiJson, getBackend } from "./backend";
import { refFromAsset, refFromStockVideo, type AssetRef } from "./assetRef";
import { chatOwner, tagChatAsset } from "./chatAssets";
@@ -46,7 +64,7 @@ import {
} from "./media";
import { isLottieAsset } from "./lottieAssets";
import { BREATH, REACH, refineEdge, type SilenceSpan } from "./cutRefine";
-import { requestSidePanel, SIDE_PANEL_TABS } from "./panelRequest";
+import { requestSidePanel } from "./panelRequest";
import { blobToInlineAudio, refToInlineAudio, visualRefs, type InlineImage } from "./refMedia";
import { characterPrompt, stockAspectDims, stockTitle } from "./stock";
import { STOCK_IMAGES } from "./stockManifest";
@@ -77,6 +95,7 @@ import {
rectOf,
regionLabel,
SHAPE_LABELS,
+ SIDE_PANEL_TABS,
SPEED_FLOOR,
TRANSITION_STYLE_IDS,
type AnimStyle,
@@ -135,25 +154,224 @@ const GRADE_KEYS = ["brightness", "contrast", "saturation", "exposure", "tempera
const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v));
const isNum = (v: unknown): v is number => typeof v === "number" && Number.isFinite(v);
-class ToolError extends Error {}
-
-/**
- * Execute an assistant tool call against the live editor store.
- * Returns a small JSON-safe result; throws ToolError with a readable message.
- */
-export async function runAiTool(
- name: string,
- input: Record<string, unknown>
-): Promise<unknown> {
- const s = useEditor.getState();
-
- switch (name) {
- case "get_state":
+/** Every tool that runs in the browser: the full catalog minus the
+ * `server: true` skills tools the engine answers itself. */
+type BrowserToolName = Exclude<
+ | AiPanelToolName
+ | AudioToolName
+ | EditorToolName
+ | EffectsToolName
+ | ElementsToolName
+ | ImageGenToolName
+ | InspectorToolName
+ | LibraryToolName
+ | OverlayAnimationToolName
+ | PreviewToolName
+ | SceneToolName
+ | SidePanelToolName
+ | StockToolName
+ | SubtitlesToolName
+ | TimelineToolName
+ | TopBarToolName
+ | TransitionsToolName
+ | VideoGenToolName,
+ "list_skills" | "read_skill"
+>;
+
+type Editor = ReturnType<typeof useEditor.getState>;
+
+type ToolRun = (s: Editor, input: Record<string, unknown>) => Promise<unknown> | unknown;
+
+/** The browser handler for every tool, keyed by the names the `*.tools.ts`
+ * modules beside the components export, so a tool added, removed, or renamed
+ * there refuses to compile until this map matches. */
+const toolRuns: Record<BrowserToolName, ToolRun> = {
+ add_shape: (s, input) => {
+ const shape = String(input.shape) as ShapeKind;
+ if (!(shape in SHAPE_LABELS))
+ throw new ToolError(`shape must be one of ${SHAPE_KINDS.join(", ")}.`);
+ if (isNum(input.start)) s.seek(input.start);
+ s.addShape(shape);
+ const sel = useEditor.getState().selection;
+ if (sel?.kind !== "overlay") throw new ToolError("Could not create the shape.");
+ applyOverlayPatchSettled(sel.id, overlayPatch({ ...input, id: sel.id }, "shape"));
+ const o = useEditor.getState().overlays.find((x) => x.id === sel.id)!;
+ return { id: o.id, shape, start: round2(o.start), end: round2(o.end) };
+ },
+
+ add_sticker: (s, input) => {
+ const assetId = typeof input.asset_id === "string" && input.asset_id ? input.asset_id : undefined;
+ if (!assetId) throw new ToolError("Pass asset_id.");
+ const stickerAsset = s.assets.find((a) => a.id === assetId);
+ if (stickerAsset?.type !== "image") throw new ToolError(`No image asset with id ${assetId}.`);
+ if (isNum(input.start)) s.seek(input.start);
+ s.addSticker({ assetId, ...(isLottieAsset(stickerAsset) ? { lottie: true } : {}) });
+ const sel = useEditor.getState().selection;
+ if (sel?.kind !== "overlay") throw new ToolError("Could not create the sticker.");
+ applyOverlayPatchSettled(sel.id, overlayPatch({ ...input, id: sel.id }, "sticker"));
+ const o = useEditor.getState().overlays.find((x) => x.id === sel.id)!;
+ return { id: o.id, assetId, start: round2(o.start), end: round2(o.end) };
+ },
+
+ create_sticker: async (s, input) => {
+ const idea = String(input.idea ?? "").trim();
+ if (!idea) throw new ToolError("idea is required.");
+ if (!s.projectId) throw new ToolError("No project open.");
+ const { createCustomSticker } = await import("./stickerCreate");
+ const asset = await createCustomSticker(s.projectId, idea);
+ if (isNum(input.start)) s.seek(input.start);
+ useEditor.getState().addSticker({ assetId: asset.id });
+ const sel = useEditor.getState().selection;
+ if (sel?.kind !== "overlay") throw new ToolError("Could not place the sticker.");
+ applyOverlayPatchSettled(sel.id, overlayPatch({ ...input, id: sel.id }, "sticker"));
+ const o = useEditor.getState().overlays.find((x) => x.id === sel.id)!;
+ return { id: o.id, assetId: asset.id, start: round2(o.start), end: round2(o.end) };
+ },
+
+ add_effect: (s, input) => {
+ const effect = String(input.effect ?? "");
+ if (!(EFFECT_IDS as string[]).includes(effect))
+ throw new ToolError(`effect must be one of ${EFFECT_IDS.join(", ")}.`);
+ if (isNum(input.start)) s.seek(input.start);
+ s.addEffect(effect as EffectId);
+ const sel = useEditor.getState().selection;
+ if (sel?.kind !== "overlay") throw new ToolError("Could not create the effect.");
+ applyOverlayPatchSettled(sel.id, overlayPatch({ ...input, id: sel.id }, "effect"));
+ // Where a zoom holds: the rest of the effects fill the frame, so the
+ // point only means anything for this one.
+ if (effect === "zoom" && (isNum(input.focus_x) || isNum(input.focus_y))) {
+ useEditor.getState().updateOverlay(sel.id, {
+ focus: {
+ x: clamp(isNum(input.focus_x) ? input.focus_x : 0.5, 0, 1),
+ y: clamp(isNum(input.focus_y) ? input.focus_y : 0.5, 0, 1),
+ },
+ });
+ }
+ const o = useEditor.getState().overlays.find((x) => x.id === sel.id)!;
+ return { id: o.id, effect, start: round2(o.start), end: round2(o.end) };
+ },
+
+ set_overlay_animation: (s, input) => {
+ const o = requireItem(s.overlays, input.id, "overlay element");
+ const anim: OverlayAnim = { ...(o.anim ?? {}) };
+ const edge = (slot: "in" | "out", styleKey: string, secondsKey: string) => {
+ const raw = input[styleKey];
+ const secs = isNum(input[secondsKey])
+ ? clamp(input[secondsKey], OVERLAY_ANIM_MIN_SECONDS, OVERLAY_ANIM_MAX_SECONDS)
+ : undefined;
+ if (typeof raw === "string") {
+ if (raw === "none") {
+ delete anim[slot];
+ return;
+ }
+ if (!(OVERLAY_ANIM_STYLE_IDS as string[]).includes(raw))
+ throw new ToolError(`Unknown ${slot} style: ${raw}`);
+ if (raw === "typewriter" && (o.kind ?? "text") !== "text")
+ throw new ToolError("typewriter animates titles only.");
+ anim[slot] = {
+ style: raw as OverlayAnimStyle,
+ seconds: secs ?? anim[slot]?.seconds ?? OVERLAY_ANIM_DEFAULT_SECONDS,
+ };
+ } else if (secs !== undefined && anim[slot]) {
+ anim[slot] = { ...anim[slot]!, seconds: secs };
+ }
+ };
+ edge("in", "in_style", "in_seconds");
+ edge("out", "out_style", "out_seconds");
+ const rawLoop = input.loop_style;
+ const speed = isNum(input.loop_speed) ? clamp(input.loop_speed, 0.25, 4) : undefined;
+ if (typeof rawLoop === "string") {
+ if (rawLoop === "none") delete anim.loop;
+ else if (!(OVERLAY_LOOP_STYLE_IDS as string[]).includes(rawLoop))
+ throw new ToolError(`Unknown loop style: ${rawLoop}`);
+ else anim.loop = { style: rawLoop as OverlayLoopStyle, speed: speed ?? anim.loop?.speed ?? 1 };
+ } else if (speed !== undefined && anim.loop) {
+ anim.loop = { ...anim.loop, speed };
+ }
+ s.updateOverlay(o.id, { anim: anim.in || anim.out || anim.loop ? anim : undefined });
+ const next = useEditor.getState().overlays.find((x) => x.id === o.id)!;
+ return { id: next.id, anim: next.anim ?? null };
+ },
+
+ set_overlay_keyframes: (s, input) => {
+ const o = requireItem(s.overlays, input.id, "overlay element");
+ const raw = input.keys;
+ if (!Array.isArray(raw)) throw new ToolError("keys must be a list.");
+ const dur = Math.max(0.1, o.end - o.start);
+ let kf = o.kf;
+ if (raw.length === 0) {
+ s.clearOverlayKeys(o.id);
+ kf = undefined;
+ } else {
+ // Each key builds on the pose already at its time, so a key that only
+ // names `x` keeps whatever the element was doing otherwise.
+ s.pushHistory();
+ for (const k of raw as Record<string, unknown>[]) {
+ if (!isNum(k.t)) throw new ToolError("Every key needs a time `t`.");
+ s.setOverlayKey(
+ o.id,
+ clamp(k.t, 0, dur),
+ {
+ ...(isNum(k.x) ? { x: clamp(k.x, 0.02, 0.98) } : {}),
+ ...(isNum(k.y) ? { y: clamp(k.y, 0.02, 0.98) } : {}),
+ ...(isNum(k.scale) ? { scale: clamp(k.scale, 0.1, 4) } : {}),
+ ...(isNum(k.rotation) ? { rotation: clamp(Math.round(k.rotation), -180, 180) } : {}),
+ ...(isNum(k.opacity) ? { opacity: clamp(k.opacity, 0, 1) } : {}),
+ },
+ { transient: true }
+ );
+ }
+ kf = useEditor.getState().overlays.find((x) => x.id === o.id)?.kf;
+ }
+ return { id: o.id, keys: (kf ?? []).map((k) => ({ ...k, t: round2(k.t) })) };
+ },
+
+ set_transition: (s, input) => {
+ const clip = requireItem(s.clips, input.clipId, "video clip");
+ // A transition joins a clip to the next one on its own track.
+ const row = s.clips.filter((c) => c.track === clip.track);
+ if (row.findIndex((c) => c.id === clip.id) === row.length - 1)
+ throw new ToolError("That is its track's last clip — nothing after it to transition into.");
+ if (!isNum(input.seconds)) throw new ToolError("seconds is required (0 clears the transition).");
+ let style: TransitionStyle | undefined;
+ if (input.style !== undefined) {
+ const requested = TRANSITION_STYLE_SYNONYMS[input.style as string] ?? input.style;
+ style = TRANSITION_STYLE_IDS.find((x) => x === requested);
+ if (!style) throw new ToolError(`Unknown style. Use one of: ${TRANSITION_STYLE_IDS.join(", ")}.`);
+ }
+ s.setClipTransition(clip.id, input.seconds, style);
+ const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
+ return { id: next.id, transition: next.transition ?? 0, style: next.transitionStyle ?? "crossfade" };
+ },
+
+ set_animation: (s, input) => {
+ const clip = requireItem(s.clips, input.clipId, "video clip");
+ const which = input.which === "in" || input.which === "out" ? input.which : null;
+ if (!which) throw new ToolError('which must be "in" or "out".');
+ if (input.style === "none") {
+ s.setClipAnim(clip.id, which, null);
+ return { id: clip.id, which, style: "none" };
+ }
+ const requested = ANIM_STYLE_SYNONYMS[input.style as string] ?? input.style;
+ const style = ANIM_STYLE_IDS.find((x) => x === requested);
+ if (!style)
+ throw new ToolError(`Unknown style. Use one of: ${ANIM_STYLE_IDS.join(", ")}, none.`);
+ if (clip.track > 0 && overlayAnimStyle(style) !== style)
+ throw new ToolError("Upper-track clips animate with fade or zoom only.");
+ const seconds = isNum(input.seconds) ? input.seconds : ANIM_DEFAULT_SECONDS;
+ s.setClipAnim(clip.id, which, { style, seconds });
+ const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
+ const anim = which === "in" ? next.animIn : next.animOut;
+ return { id: next.id, which, style: anim?.style ?? "none", seconds: anim?.seconds ?? 0 };
+ },
+
+ get_state: () => {
// The tool result carries the whole transcript; the per-message context
// snapshot trims it, so this is how the model pulls every cue when needed.
return buildAiContext({ fullCues: true });
+ },
- case "capture_frame": {
+ capture_frame: () => {
const canvas = document.querySelector<HTMLCanvasElement>(".stage canvas");
if (!canvas) throw new ToolError("No preview canvas on screen.");
const scaled = document.createElement("canvas");
@@ -165,9 +383,9 @@ export async function runAiTool(
ctx.drawImage(canvas, 0, 0, scaled.width, scaled.height);
// Overlays/captions are DOM, not canvas — note that for the model.
return { image: scaled.toDataURL("image/jpeg", 0.75), note: "Video frame only; titles and captions overlay this in the UI." };
- }
+ },
- case "watch_video": {
+ watch_video: async (s, input) => {
const { projectId, asset, clip, speed, from, to } = resolveWatchRange(s, input);
if (asset.type === "audio")
throw new ToolError(`"${asset.name}" is audio — listen_audio hears it, detect_silence finds its dead air.`);
@@ -263,9 +481,9 @@ export async function runAiTool(
: "") +
(stamped ? "" : " (Stamps unavailable — sheetFrames lists each cell's time.)"),
};
- }
+ },
- case "detect_silence": {
+ detect_silence: async (s, input) => {
const { projectId, asset, clip, speed, from, to } = resolveWatchRange(s, input);
if (asset.type === "image") throw new ToolError(`"${asset.name}" is an image — it has no audio.`);
const silences = await fetchSilences(projectId, asset, {
@@ -303,9 +521,9 @@ export async function runAiTool(
? { note: "No silence at these settings — a higher threshold_db or shorter min_silence hears more." }
: {}),
};
- }
+ },
- case "listen_audio": {
+ listen_audio: async (s, input) => {
const { projectId, asset, clip, from, to } = resolveWatchRange(s, input);
if (asset.type === "image")
throw new ToolError(`"${asset.name}" is an image — it has no sound. watch_video shows it.`);
@@ -330,19 +548,20 @@ export async function runAiTool(
...(to !== undefined ? { to: round2(to) } : {}),
audio: `data:${inline.mimeType};base64,${inline.data}`,
};
- }
+ },
- case "seek": {
+ seek: (s, input) => {
if (!isNum(input.t)) throw new ToolError("t (seconds) is required.");
s.seek(input.t);
return { playhead: useEditor.getState().currentTime };
- }
+ },
- case "set_playing":
+ set_playing: (s, input) => {
s.setPlaying(Boolean(input.playing));
return { playing: Boolean(input.playing) };
+ },
- case "select": {
+ select: (s, input) => {
const kind = String(input.kind);
if (kind === "none") {
s.select(null);
@@ -365,26 +584,26 @@ export async function runAiTool(
kind === "overlayClip" ? "clip" : kind === "text" ? "overlay" : (kind as "clip" | "audio" | "overlay");
s.select({ kind: selKind, id });
return { selection: { kind: selKind, id } };
- }
+ },
- case "set_side_panel": {
+ set_side_panel: (_s, input) => {
const panel = String(input.panel);
if (panel !== "none" && !SIDE_PANEL_TABS.some((t) => t === panel))
throw new ToolError(`Unknown panel: ${panel}`);
requestSidePanel(panel === "none" ? null : (panel as SidePanelTab));
return { panel };
- }
+ },
- case "split_at": {
+ split_at: (s, input) => {
const before = s.clips.length + s.audioClips.length;
s.splitAtPlayhead(isNum(input.t) ? input.t : undefined);
const after = useEditor.getState();
const made = after.clips.length + after.audioClips.length - before;
if (made === 0) throw new ToolError("Nothing to split at that time.");
return { split: true, videoClips: after.clips.length };
- }
+ },
- case "move_clip": {
+ move_clip: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
// Reordering by index is a base-sequence operation; a layer clip is
// free-positioned — move it with update_overlay_video (start/track).
@@ -396,9 +615,9 @@ export async function runAiTool(
const row = track0Clips(s.clips);
s.moveClip(clip.id, clamp(Math.round(input.toIndex), 0, row.length - 1));
return { order: track0Clips(useEditor.getState().clips).map((c) => c.id) };
- }
+ },
- case "place_clip": {
+ place_clip: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
if (!isNum(input.start)) throw new ToolError("start (seconds) is required.");
const len = (clip.out - clip.in) / (clip.speed && clip.speed > 0 ? clip.speed : 1);
@@ -418,14 +637,14 @@ export async function runAiTool(
? { note: "That spot was taken — slid right to the next free one." }
: {}),
};
- }
+ },
- case "add_clip": {
+ add_clip: (s, input) => {
const asset = requireItem(s.assets, input.asset_id, "project asset");
return placeAssetOnTimeline(asset, input);
- }
+ },
- case "add_overlay_video": {
+ add_overlay_video: (s, input) => {
const asset = requireItem(s.assets, input.asset_id, "project asset");
if (asset.type !== "video" && asset.type !== "image")
throw new ToolError("Only video or image assets can sit on a video track.");
@@ -451,9 +670,9 @@ export async function runAiTool(
len: round2((c.out - c.in) / (c.speed && c.speed > 0 ? c.speed : 1)),
layout: regionLabel(rectOf(c)),
};
- }
+ },
- case "update_overlay_video": {
+ update_overlay_video: (s, input) => {
const c = requireItem(overlayLayers(s.clips), input.id, "overlay video clip");
const asset = s.assets.find((a) => a.id === c.assetId);
// A still has no source bound, so its clip can stretch to any length.
@@ -497,9 +716,9 @@ export async function runAiTool(
muted: next.muted,
...(next.hidden ? { hidden: true } : {}),
};
- }
+ },
- case "trim_clip": {
+ trim_clip: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
const asset = s.assets.find((a) => a.id === clip.assetId);
// A still has no source bound, so its clip can stretch to any length.
@@ -511,9 +730,9 @@ export async function runAiTool(
// invariant: extending a clip pushes the following run right.
s.setClipTrim(clip.id, nextIn, nextOut);
return { in: nextIn, out: nextOut, len: Math.round((nextOut - nextIn) * 100) / 100 };
- }
+ },
- case "refine_speech_cuts": {
+ refine_speech_cuts: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const ids = Array.isArray(input.clip_ids) ? input.clip_ids.map(String) : [];
@@ -644,30 +863,30 @@ export async function runAiTool(
}),
note: `Each moved edge sits ~${BREATH}s inside a real pause; clip spacing is preserved. Listen at any flagged edge.`,
};
- }
+ },
- case "set_clip_muted": {
+ set_clip_muted: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
s.updateClip(clip.id, { muted: Boolean(input.muted) });
return { muted: Boolean(input.muted) };
- }
+ },
- case "set_clip_volume": {
+ set_clip_volume: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
if (!isNum(input.volume)) throw new ToolError("volume is required (0..1.5).");
const volume = clamp(input.volume, 0, 1.5);
s.updateClip(clip.id, { volume: Math.abs(volume - 1) < 1e-4 ? undefined : volume });
const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
return { id: next.id, volume: next.volume ?? 1 };
- }
+ },
- case "set_clip_hidden": {
+ set_clip_hidden: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
s.updateClip(clip.id, { hidden: input.hidden ? true : undefined });
return { id: clip.id, hidden: Boolean(input.hidden) };
- }
+ },
- case "set_track_hidden": {
+ set_track_hidden: (s, input) => {
if (!isNum(input.track) || input.track < 0)
throw new ToolError("track must be 0 or higher.");
const track = Math.round(input.track);
@@ -701,9 +920,9 @@ export async function runAiTool(
default:
throw new ToolError('`kind` must be "video", "soundtrack", "text", or "subtitles".');
}
- }
+ },
- case "set_track_muted": {
+ set_track_muted: (s, input) => {
if (!isNum(input.track) || input.track < 0)
throw new ToolError("track must be 0 or higher.");
const track = Math.round(input.track);
@@ -711,9 +930,9 @@ export async function runAiTool(
if (!n) throw new ToolError(`No clips on video track ${track}.`);
s.setTrackMuted(track, Boolean(input.muted));
return { track, muted: Boolean(input.muted), clips: n };
- }
+ },
- case "detach_audio": {
+ detach_audio: (s, input) => {
const id = input.clipId ? String(input.clipId) : s.selection?.kind === "clip" ? s.selection.id : null;
if (!id) throw new ToolError("Pass clipId or select a video clip first.");
const clip = requireItem(s.clips, id, "video clip");
@@ -724,9 +943,9 @@ export async function runAiTool(
if (asset) void ensurePeaks(asset);
const sel = useEditor.getState().selection;
return { audioClipId: sel?.kind === "audio" ? sel.id : null };
- }
+ },
- case "delete_item": {
+ delete_item: (s, input) => {
const kind = String(input.kind) as "clip" | "overlayClip" | "audio" | "overlay" | "text";
const id = String(input.id ?? "");
const pool =
@@ -741,9 +960,9 @@ export async function runAiTool(
s.select({ kind: selKind, id });
s.deleteSelection();
return { deleted: { kind: selKind, id } };
- }
+ },
- case "remove_gap": {
+ remove_gap: (s, input) => {
if (!isNum(input.at)) throw new ToolError("at (seconds) is required.");
const track = isNum(input.track) ? input.track : 0;
const lane = { kind: "video", index: track } as const;
@@ -751,9 +970,9 @@ export async function runAiTool(
if (!gap) throw new ToolError(`No track-${track} gap at ${input.at}s — pass a time inside the empty span.`);
s.removeLaneGap(lane, input.at);
return { closed: { track, ...gap } };
- }
+ },
- case "add_title": {
+ add_title: (s, input) => {
if (typeof input.text !== "string" || !input.text.trim())
throw new ToolError("text is required.");
if (isNum(input.start)) s.seek(input.start);
@@ -763,150 +982,9 @@ export async function runAiTool(
applyOverlayPatchSettled(sel.id, overlayPatch({ ...input, id: sel.id }, "text"));
const o = useEditor.getState().overlays.find((x) => x.id === sel.id)!;
return { id: o.id, ...(isTextOverlay(o) ? { text: o.text } : {}), start: o.start, end: o.end };
- }
-
- case "add_shape": {
- const shape = String(input.shape) as ShapeKind;
- if (!(shape in SHAPE_LABELS)) throw new ToolError("shape must be rect, ellipse, line, or arrow.");
- if (isNum(input.start)) s.seek(input.start);
- s.addShape(shape);
- const sel = useEditor.getState().selection;
- if (sel?.kind !== "overlay") throw new ToolError("Could not create the shape.");
- applyOverlayPatchSettled(sel.id, overlayPatch({ ...input, id: sel.id }, "shape"));
- const o = useEditor.getState().overlays.find((x) => x.id === sel.id)!;
- return { id: o.id, shape, start: round2(o.start), end: round2(o.end) };
- }
-
- case "add_sticker": {
- const assetId = typeof input.asset_id === "string" && input.asset_id ? input.asset_id : undefined;
- if (!assetId) throw new ToolError("Pass asset_id.");
- const stickerAsset = s.assets.find((a) => a.id === assetId);
- if (stickerAsset?.type !== "image") throw new ToolError(`No image asset with id ${assetId}.`);
- if (isNum(input.start)) s.seek(input.start);
- s.addSticker({ assetId, ...(isLottieAsset(stickerAsset) ? { lottie: true } : {}) });
- const sel = useEditor.getState().selection;
- if (sel?.kind !== "overlay") throw new ToolError("Could not create the sticker.");
- applyOverlayPatchSettled(sel.id, overlayPatch({ ...input, id: sel.id }, "sticker"));
- const o = useEditor.getState().overlays.find((x) => x.id === sel.id)!;
- return { id: o.id, assetId, start: round2(o.start), end: round2(o.end) };
- }
-
- case "add_effect": {
- const effect = String(input.effect ?? "");
- if (!(EFFECT_IDS as string[]).includes(effect))
- throw new ToolError(`effect must be one of ${EFFECT_IDS.join(", ")}.`);
- if (isNum(input.start)) s.seek(input.start);
- s.addEffect(effect as EffectId);
- const sel = useEditor.getState().selection;
- if (sel?.kind !== "overlay") throw new ToolError("Could not create the effect.");
- applyOverlayPatchSettled(sel.id, overlayPatch({ ...input, id: sel.id }, "effect"));
- // Where a zoom holds: the rest of the effects fill the frame, so the
- // point only means anything for this one.
- if (effect === "zoom" && (isNum(input.focus_x) || isNum(input.focus_y))) {
- useEditor.getState().updateOverlay(sel.id, {
- focus: {
- x: clamp(isNum(input.focus_x) ? input.focus_x : 0.5, 0, 1),
- y: clamp(isNum(input.focus_y) ? input.focus_y : 0.5, 0, 1),
- },
- });
- }
- const o = useEditor.getState().overlays.find((x) => x.id === sel.id)!;
- return { id: o.id, effect, start: round2(o.start), end: round2(o.end) };
- }
+ },
- case "create_sticker": {
- const idea = String(input.idea ?? "").trim();
- if (!idea) throw new ToolError("idea is required.");
- if (!s.projectId) throw new ToolError("No project open.");
- const { createCustomSticker } = await import("./stickerCreate");
- const asset = await createCustomSticker(s.projectId, idea);
- if (isNum(input.start)) s.seek(input.start);
- useEditor.getState().addSticker({ assetId: asset.id });
- const sel = useEditor.getState().selection;
- if (sel?.kind !== "overlay") throw new ToolError("Could not place the sticker.");
- applyOverlayPatchSettled(sel.id, overlayPatch({ ...input, id: sel.id }, "sticker"));
- const o = useEditor.getState().overlays.find((x) => x.id === sel.id)!;
- return { id: o.id, assetId: asset.id, start: round2(o.start), end: round2(o.end) };
- }
-
- case "set_overlay_animation": {
- const o = requireItem(s.overlays, input.id, "overlay element");
- const anim: OverlayAnim = { ...(o.anim ?? {}) };
- const edge = (slot: "in" | "out", styleKey: string, secondsKey: string) => {
- const raw = input[styleKey];
- const secs = isNum(input[secondsKey])
- ? clamp(input[secondsKey], OVERLAY_ANIM_MIN_SECONDS, OVERLAY_ANIM_MAX_SECONDS)
- : undefined;
- if (typeof raw === "string") {
- if (raw === "none") {
- delete anim[slot];
- return;
- }
- if (!(OVERLAY_ANIM_STYLE_IDS as string[]).includes(raw))
- throw new ToolError(`Unknown ${slot} style: ${raw}`);
- if (raw === "typewriter" && (o.kind ?? "text") !== "text")
- throw new ToolError("typewriter animates titles only.");
- anim[slot] = {
- style: raw as OverlayAnimStyle,
- seconds: secs ?? anim[slot]?.seconds ?? OVERLAY_ANIM_DEFAULT_SECONDS,
- };
- } else if (secs !== undefined && anim[slot]) {
- anim[slot] = { ...anim[slot]!, seconds: secs };
- }
- };
- edge("in", "in_style", "in_seconds");
- edge("out", "out_style", "out_seconds");
- const rawLoop = input.loop_style;
- const speed = isNum(input.loop_speed) ? clamp(input.loop_speed, 0.25, 4) : undefined;
- if (typeof rawLoop === "string") {
- if (rawLoop === "none") delete anim.loop;
- else if (!(OVERLAY_LOOP_STYLE_IDS as string[]).includes(rawLoop))
- throw new ToolError(`Unknown loop style: ${rawLoop}`);
- else anim.loop = { style: rawLoop as OverlayLoopStyle, speed: speed ?? anim.loop?.speed ?? 1 };
- } else if (speed !== undefined && anim.loop) {
- anim.loop = { ...anim.loop, speed };
- }
- s.updateOverlay(o.id, { anim: anim.in || anim.out || anim.loop ? anim : undefined });
- const next = useEditor.getState().overlays.find((x) => x.id === o.id)!;
- return { id: next.id, anim: next.anim ?? null };
- }
-
- case "set_overlay_keyframes": {
- const o = requireItem(s.overlays, input.id, "overlay element");
- const raw = input.keys;
- if (!Array.isArray(raw)) throw new ToolError("keys must be a list.");
- const dur = Math.max(0.1, o.end - o.start);
- let kf = o.kf;
- if (raw.length === 0) {
- s.clearOverlayKeys(o.id);
- kf = undefined;
- } else {
- // Each key builds on the pose already at its time, so a key that only
- // names `x` keeps whatever the element was doing otherwise.
- s.pushHistory();
- for (const k of raw as Record<string, unknown>[]) {
- if (!isNum(k.t)) throw new ToolError("Every key needs a time `t`.");
- s.setOverlayKey(
- o.id,
- clamp(k.t, 0, dur),
- {
- ...(isNum(k.x) ? { x: clamp(k.x, 0.02, 0.98) } : {}),
- ...(isNum(k.y) ? { y: clamp(k.y, 0.02, 0.98) } : {}),
- ...(isNum(k.scale) ? { scale: clamp(k.scale, 0.1, 4) } : {}),
- ...(isNum(k.rotation) ? { rotation: clamp(Math.round(k.rotation), -180, 180) } : {}),
- ...(isNum(k.opacity) ? { opacity: clamp(k.opacity, 0, 1) } : {}),
- },
- { transient: true }
- );
- }
- kf = useEditor.getState().overlays.find((x) => x.id === o.id)?.kf;
- }
- return { id: o.id, keys: (kf ?? []).map((k) => ({ ...k, t: round2(k.t) })) };
- }
-
- // "update_title" stays as an alias so older threads keep working.
- case "update_title":
- case "update_overlay": {
+ update_overlay: (s, input) => {
const o = requireItem(s.overlays, input.id, "overlay element");
s.updateOverlay(o.id, overlayPatch(input, o.kind ?? "text"));
const next = useEditor.getState().overlays.find((x) => x.id === o.id)!;
@@ -917,9 +995,9 @@ export async function runAiTool(
start: round2(next.start),
end: round2(next.end),
};
- }
+ },
- case "update_audio": {
+ update_audio: (s, input) => {
const a = requireItem(s.audioClips, input.id, "soundtrack clip");
const aSpeed = a.speed && a.speed > 0 ? a.speed : 1;
const len = (a.out - a.in) / aSpeed;
@@ -952,9 +1030,9 @@ export async function runAiTool(
...patch,
...("start" in patch ? { start: round2(landed.start) } : {}),
};
- }
+ },
- case "set_framing": {
+ set_framing: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
const mode = input.mode === "fill" ? "fill" : "fit";
s.updateClip(clip.id, {
@@ -964,9 +1042,9 @@ export async function runAiTool(
});
const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
return { id: next.id, fit: next.fit, panX: next.panX ?? 0, panY: next.panY ?? 0 };
- }
+ },
- case "freeze_frame": {
+ freeze_frame: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const spans = getClipSpans(s.clips, s.assets);
@@ -1033,9 +1111,9 @@ export async function runAiTool(
duration: body.duration,
from: { time: Math.round(t * 100) / 100, source: span.asset.name },
};
- }
+ },
- case "generate_image": {
+ generate_image: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const prompt = String(input.prompt ?? "").trim();
@@ -1077,9 +1155,9 @@ export async function runAiTool(
clipId: placed.clipId,
index: placed.index,
};
- }
+ },
- case "generate_video": {
+ generate_video: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const prompt = String(input.prompt ?? "").trim();
@@ -1185,9 +1263,9 @@ export async function runAiTool(
}
// No reference: text is the whole request, so the single rung runs ungated.
return launchVideoJob(projectId, input, [{ prompt, opts: { ...baseOpts } }]);
- }
+ },
- case "wait_for_renders": {
+ wait_for_renders: async (s) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const watched = useGenerate
@@ -1228,9 +1306,9 @@ export async function runAiTool(
? { note: `${stillRunning} still rendering — call wait_for_renders again to keep waiting.` }
: {}),
};
- }
+ },
- case "generate_character_video": {
+ generate_character_video: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const id = String(input.character_id ?? "");
@@ -1260,9 +1338,9 @@ export async function runAiTool(
],
{ extra: { character: character.id } }
);
- }
+ },
- case "generate_scene": {
+ generate_scene: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const gen = useGenerate.getState();
@@ -1309,29 +1387,29 @@ export async function runAiTool(
shots: res.shotCount,
note: `${res.message} A storyboard card below shows each shot's opening frame for the user, so keep your reply to one short line — don't re-describe the shots or the timing. Ask them to review it; if they want a frame changed, call regenerate_shot (at this stage it just redraws that frame, no credits). When they approve, call approve_scene — that starts the paid video renders, so don't approve on your own.`,
};
- }
+ },
- case "approve_scene": {
+ approve_scene: () => {
const res = useGenScene.getState().approve();
if (!res.ok) throw new ToolError(res.message);
return { rendering: true, note: res.message };
- }
+ },
- case "cancel_scene": {
+ cancel_scene: () => {
const res = useGenScene.getState().cancel();
if (!res.ok) throw new ToolError(res.message);
return { stopped: true, note: res.message };
- }
+ },
- case "regenerate_shot": {
+ regenerate_shot: (_s, input) => {
if (!isNum(input.n)) throw new ToolError("n (the 1-based shot number) is required.");
const note = typeof input.note === "string" && input.note.trim() ? input.note.trim() : undefined;
const res = useGenScene.getState().regenerateShot(Math.round(input.n), note);
if (!res.ok) throw new ToolError(res.message);
return { note: res.message };
- }
+ },
- case "recut_scene": {
+ recut_scene: (_s, input) => {
if (!isNum(input.from_shot) || !isNum(input.to_shot))
throw new ToolError("from_shot and to_shot (1-based, inclusive) are required.");
const instruction = String(input.instruction ?? "").trim();
@@ -1341,17 +1419,17 @@ export async function runAiTool(
.recutShots(Math.round(input.from_shot), Math.round(input.to_shot), instruction);
if (!res.ok) throw new ToolError(res.message);
return { note: res.message };
- }
+ },
- case "restyle_scene": {
+ restyle_scene: (_s, input) => {
const style = String(input.style ?? "").trim();
if (!style) throw new ToolError("style is required — the new look for the whole video.");
const res = useGenScene.getState().restyle(style);
if (!res.ok) throw new ToolError(res.message);
return { note: res.message };
- }
+ },
- case "stock_search": {
+ stock_search: (s, input) => {
const q = String(input.query ?? "").trim().toLowerCase();
const kindIn =
input.kind === "video" || input.kind === "image" || input.kind === "character"
@@ -1412,9 +1490,9 @@ export async function runAiTool(
total: matches.length,
...(matches.length > 12 ? { truncated: true } : {}),
};
- }
+ },
- case "stock_add": {
+ stock_add: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const id = String(input.id ?? "");
@@ -1451,9 +1529,9 @@ export async function runAiTool(
addedToTimeline: addToTimeline,
clipId,
};
- }
+ },
- case "import_url": {
+ import_url: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const url = String(input.url ?? "").trim();
@@ -1472,9 +1550,9 @@ export async function runAiTool(
})),
...(text ? { sourceText: text } : {}),
};
- }
+ },
- case "library_list": {
+ library_list: async () => {
const lib = await fetchLibrary();
return {
folders: lib.folders.map((f) => ({ id: f.id, name: f.name })),
@@ -1492,9 +1570,9 @@ export async function runAiTool(
parts: t.layers.length + t.audio.length + t.texts.length + t.cues.length,
})),
};
- }
+ },
- case "library_add": {
+ library_add: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const lib = (await fetchLibrary()).assets.find((a) => a.id === String(input.id ?? ""));
@@ -1516,9 +1594,9 @@ export async function runAiTool(
? { addedToTimeline: true, clip: placeAssetOnTimeline(asset, input) }
: { addedToTimeline: false }),
};
- }
+ },
- case "template_add": {
+ template_add: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const t = (await fetchLibrary()).templates.find((x) => x.id === String(input.id ?? ""));
@@ -1537,9 +1615,9 @@ export async function runAiTool(
duration: round2(t.duration),
parts: t.layers.length + t.audio.length + t.texts.length + t.cues.length,
};
- }
+ },
- case "save_template": {
+ save_template: (s, input) => {
if (!s.projectId) throw new ToolError("No project open.");
const name = String(input.name ?? "").trim();
if (!name) throw new ToolError("A template name is required.");
@@ -1560,9 +1638,9 @@ export async function runAiTool(
if (!built) throw new ToolError("Could not build a template from those items.");
const saved = useEditor.getState().addTemplate({ ...built, name });
return { id: saved.id, name: saved.name, duration: round2(saved.duration) };
- }
+ },
- case "library_organize": {
+ library_organize: async (_s, input) => {
// The library spans both shelves, so every id is resolved against the
// current listing to find the backend that owns it.
const lib = await fetchLibrary();
@@ -1622,9 +1700,9 @@ export async function runAiTool(
default:
throw new ToolError(`Unknown action "${String(input.action)}".`);
}
- }
+ },
- case "file_asset": {
+ file_asset: async (s, input) => {
const asset = requireItem(s.assets, input.asset_id, "project asset");
if (input.to === "media") {
s.updateAsset(asset.id, { origin: undefined, chatId: undefined });
@@ -1637,18 +1715,18 @@ export async function runAiTool(
return { filed: "library", libraryId: saved.id, name: saved.name };
}
throw new ToolError('`to` must be "media" or "library".');
- }
+ },
- case "delete_asset": {
+ delete_asset: (s, input) => {
const asset = requireItem(s.assets, input.asset_id, "project asset");
const uses =
s.clips.filter((c) => c.assetId === asset.id).length +
s.audioClips.filter((c) => c.assetId === asset.id).length;
s.removeAsset(asset.id);
return { removed: asset.name, clipsRemoved: uses };
- }
+ },
- case "subtitles_generate": {
+ subtitles_generate: async (_s, input) => {
const lane = targetSubtitleTrack(input);
if (typeof input.locale === "string" && input.locale)
useEditor.getState().setSubtitleTrackMeta(lane, { locale: input.locale });
@@ -1667,9 +1745,9 @@ export async function runAiTool(
? "No speech found — no subtitles were added to the video."
: undefined,
};
- }
+ },
- case "captions_generate": {
+ captions_generate: async (_s, input) => {
const raw = typeof input.style === "string" ? input.style : "hook";
const style = (["clean", "hook", "punchy"].includes(raw) ? raw : "hook") as
| "clean"
@@ -1692,9 +1770,9 @@ export async function runAiTool(
? "No speech found — no captions were added to the video."
: undefined,
};
- }
+ },
- case "subtitles_from_visuals": {
+ subtitles_from_visuals: async (_s, input) => {
const lane = targetSubtitleTrack(input);
if (typeof input.locale === "string" && input.locale)
useEditor.getState().setSubtitleTrackMeta(lane, { locale: input.locale });
@@ -1705,9 +1783,9 @@ export async function runAiTool(
const made = laneCues(cur.subtitles, lane).length;
if (made > 0 && cur.timelineH < 276) cur.setTimelineH(276);
return { status: cur.subtitleStatus, track: lane, cues: made };
- }
+ },
- case "subtitles_add_track": {
+ subtitles_add_track: (s, input) => {
const count = subtitleLaneCount(s.subtitles);
if (count >= MAX_SUBTITLE_LANES)
throw new ToolError(`Already at ${MAX_SUBTITLE_LANES} subtitle tracks.`);
@@ -1717,9 +1795,9 @@ export async function runAiTool(
: undefined
);
return { track: count, tracks: count + 1 };
- }
+ },
- case "subtitles_remove_track": {
+ subtitles_remove_track: (s, input) => {
if (!isNum(input.track)) throw new ToolError("track is required.");
const lane = Math.round(input.track);
const count = subtitleLaneCount(s.subtitles);
@@ -1733,9 +1811,9 @@ export async function runAiTool(
}
s.removeSubtitleTrack(lane);
return { removed: lane, tracks: count - 1 };
- }
+ },
- case "subtitles_translate_track": {
+ subtitles_translate_track: async (s, input) => {
const language = typeof input.language === "string" ? input.language.trim() : "";
if (!language) throw new ToolError("language (BCP-47, e.g. ko-KR) is required.");
const subs = s.subtitles;
@@ -1773,17 +1851,17 @@ export async function runAiTool(
from,
cues: laneCues(cur.subtitles, target).length,
};
- }
+ },
- case "list_voices": {
+ list_voices: () => {
// Gemini's prebuilt voice catalog is fixed and ships hardcoded.
return {
voices: SPEECH_VOICES.map((v) => ({ name: v.name, style: v.style })),
total: SPEECH_VOICES.length,
};
- }
+ },
- case "voiceover_generate": {
+ voiceover_generate: (s, input) => {
if (typeof input.script !== "string" || !input.script.trim())
throw new ToolError("script is required.");
const start = isNum(input.start) ? Math.max(0, input.start) : s.currentTime;
@@ -1797,9 +1875,9 @@ export async function runAiTool(
input,
place
);
- }
+ },
- case "generate_music": {
+ generate_music: async (s, input) => {
const projectId = s.projectId;
if (!projectId) throw new ToolError("No project open.");
const prompt = String(input.prompt ?? "").trim();
@@ -1883,9 +1961,9 @@ export async function runAiTool(
volume,
addedToTimeline: true,
};
- }
+ },
- case "read_subtitles_aloud": {
+ read_subtitles_aloud: async (s, input) => {
const lane = targetSubtitleTrack(input);
if (!laneCues(useEditor.getState().subtitles, lane).some((c) => c.text.trim()))
throw new ToolError("No subtitles on that track — generate subtitles first.");
@@ -1911,18 +1989,18 @@ export async function runAiTool(
duck: out.duck,
lines: out.lines,
};
- }
+ },
- case "subtitles_set_view": {
+ subtitles_set_view: (s, input) => {
const patch: { showOnVideo?: boolean; showOnTimeline?: boolean } = {};
if (typeof input.showOnVideo === "boolean") patch.showOnVideo = input.showOnVideo;
if (typeof input.showOnTimeline === "boolean") patch.showOnTimeline = input.showOnTimeline;
s.setSubtitlesView(patch);
if (patch.showOnTimeline && s.subtitles.cues.length > 0 && s.timelineH < 276) s.setTimelineH(276);
return patch;
- }
+ },
- case "update_cue": {
+ update_cue: (s, input) => {
const cue = requireItem(s.subtitles.cues, input.id, "subtitle cue");
if (typeof input.text === "string") s.setCueText(cue.id, input.text);
if (isNum(input.start) || isNum(input.end)) {
@@ -1933,15 +2011,15 @@ export async function runAiTool(
}
const next = useEditor.getState().subtitles.cues.find((c) => c.id === cue.id);
return next ? { id: next.id, start: next.start, end: next.end, text: next.text } : { deleted: cue.id };
- }
+ },
- case "delete_cue": {
+ delete_cue: (s, input) => {
const cue = requireItem(s.subtitles.cues, input.id, "subtitle cue");
s.deleteCue(cue.id);
return { deleted: cue.id };
- }
+ },
- case "set_publish": {
+ set_publish: (s, input) => {
const patch: Record<string, string> = {};
for (const k of ["caption", "tags", "soundTitle", "handle"] as const) {
if (typeof input[k] === "string") patch[k] = input[k] as string;
@@ -1949,9 +2027,9 @@ export async function runAiTool(
if (Object.keys(patch).length === 0) throw new ToolError("Nothing to change.");
s.setPublish(patch);
return patch;
- }
+ },
- case "set_view": {
+ set_view: (s, input) => {
const out: Record<string, number> = {};
if (input.fit) {
const el = document.querySelector<HTMLElement>(".tl-scroll");
@@ -1970,31 +2048,34 @@ export async function runAiTool(
out.timelineH = useEditor.getState().timelineH;
}
return out;
- }
+ },
- case "undo":
+ undo: (s) => {
s.undo();
return { ok: true };
- case "redo":
+ },
+
+ redo: (s) => {
s.redo();
return { ok: true };
+ },
- case "open_export": {
+ open_export: (s) => {
if (getClipSpans(s.clips, s.assets).length === 0)
throw new ToolError("Add a video to the timeline first.");
s.setExportOpen(true);
return { open: true };
- }
+ },
- case "set_speed": {
+ set_speed: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
if (!isNum(input.speed)) throw new ToolError("speed is required (e.g. 1.5).");
s.setClipSpeed(clip.id, input.speed);
const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
return { id: next.id, speed: next.speed ?? 1 };
- }
+ },
- case "set_color_grade": {
+ set_color_grade: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
let grade: ColorGrade = input.reset === true ? {} : { ...clip.grade };
if (input.auto === true) {
@@ -2009,57 +2090,18 @@ export async function runAiTool(
s.updateClip(clip.id, { grade: normalizeGrade(grade) });
const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
return { id: next.id, grade: next.grade ?? "neutral" };
- }
-
- case "set_transition": {
- const clip = requireItem(s.clips, input.clipId, "video clip");
- // A transition joins a clip to the next one on its own track.
- const row = s.clips.filter((c) => c.track === clip.track);
- if (row.findIndex((c) => c.id === clip.id) === row.length - 1)
- throw new ToolError("That is its track's last clip — nothing after it to transition into.");
- if (!isNum(input.seconds)) throw new ToolError("seconds is required (0 clears the transition).");
- let style: TransitionStyle | undefined;
- if (input.style !== undefined) {
- const requested = TRANSITION_STYLE_SYNONYMS[input.style as string] ?? input.style;
- style = TRANSITION_STYLE_IDS.find((x) => x === requested);
- if (!style) throw new ToolError(`Unknown style. Use one of: ${TRANSITION_STYLE_IDS.join(", ")}.`);
- }
- s.setClipTransition(clip.id, input.seconds, style);
- const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
- return { id: next.id, transition: next.transition ?? 0, style: next.transitionStyle ?? "crossfade" };
- }
+ },
- case "set_animation": {
- const clip = requireItem(s.clips, input.clipId, "video clip");
- const which = input.which === "in" || input.which === "out" ? input.which : null;
- if (!which) throw new ToolError('which must be "in" or "out".');
- if (input.style === "none") {
- s.setClipAnim(clip.id, which, null);
- return { id: clip.id, which, style: "none" };
- }
- const requested = ANIM_STYLE_SYNONYMS[input.style as string] ?? input.style;
- const style = ANIM_STYLE_IDS.find((x) => x === requested);
- if (!style)
- throw new ToolError(`Unknown style. Use one of: ${ANIM_STYLE_IDS.join(", ")}, none.`);
- if (clip.track > 0 && overlayAnimStyle(style) !== style)
- throw new ToolError("Upper-track clips animate with fade or zoom only.");
- const seconds = isNum(input.seconds) ? input.seconds : ANIM_DEFAULT_SECONDS;
- s.setClipAnim(clip.id, which, { style, seconds });
- const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
- const anim = which === "in" ? next.animIn : next.animOut;
- return { id: next.id, which, style: anim?.style ?? "none", seconds: anim?.seconds ?? 0 };
- }
-
- case "merge_cue": {
+ merge_cue: (s, input) => {
const cue = requireItem(s.subtitles.cues, input.id, "subtitle cue");
const mates = laneCues(s.subtitles, cue.lane ?? 0);
if (mates.findIndex((c) => c.id === cue.id) <= 0)
throw new ToolError("That is its track's first cue — nothing before it to merge into.");
s.mergeCueIntoPrev(cue.id);
return { mergedInto: "previous cue" };
- }
+ },
- case "set_aspect": {
+ set_aspect: (s, input) => {
const a = typeof input.aspect === "string" ? normalizeAspect(input.aspect) : null;
if (!a)
throw new ToolError(
@@ -2068,9 +2110,9 @@ export async function runAiTool(
s.setAspect(a);
const f = frameOf(a);
return { aspect: a, frame: `${f.w}x${f.h}` };
- }
+ },
- case "set_project_fade": {
+ set_project_fade: (s, input) => {
if (!isNum(input.fadeIn) && !isNum(input.fadeOut))
throw new ToolError("Pass fadeIn and/or fadeOut seconds (0 clears).");
s.setProjectFade({
@@ -2079,18 +2121,33 @@ export async function runAiTool(
});
const after = useEditor.getState();
return { fadeIn: after.fadeIn, fadeOut: after.fadeOut };
- }
+ },
- case "set_project_name": {
+ set_project_name: (s, input) => {
if (typeof input.name !== "string" || !input.name.trim())
throw new ToolError("name is required.");
s.setProjectName(input.name.trim());
return { name: input.name.trim() };
- }
+ },
+};
- default:
- throw new ToolError(`Unknown tool: ${name}`);
- }
+class ToolError extends Error {}
+
+/**
+ * Execute an assistant tool call against the live editor store.
+ * Returns a small JSON-safe result; throws ToolError with a readable message.
+ */
+export async function runAiTool(
+ name: string,
+ input: Record<string, unknown>
+): Promise<unknown> {
+ const s = useEditor.getState();
+
+ // "update_title" stays as an alias so older threads keep working.
+ const key = name === "update_title" ? "update_overlay" : name;
+ const run = (toolRuns as Partial<Record<string, ToolRun>>)[key];
+ if (!run) throw new ToolError(`Unknown tool: ${name}`);
+ return run(s, input);
}
/** Synthesize speech segments with hosted Gemini voices (the user's Donkey
diff --git a/site/src/cut/lib/panelRequest.ts b/site/src/cut/lib/panelRequest.ts
index 16e31178..e9db3b3f 100644
--- a/site/src/cut/lib/panelRequest.ts
+++ b/site/src/cut/lib/panelRequest.ts
@@ -9,18 +9,6 @@ import { create } from "zustand";
import type { SidePanelTab } from "./types";
-export const SIDE_PANEL_TABS: SidePanelTab[] = [
- "media",
- "elements",
- "effects",
- "transitions",
- "video",
- "image",
- "audio",
- "subtitles",
- "publish",
-];
-
interface PanelRequestState {
target: SidePanelTab | null;
/** Bumps per request so repeating the same target still fires the effect. */
diff --git a/site/src/cut/lib/types.ts b/site/src/cut/lib/types.ts
index 8328070e..1bf370d7 100644
--- a/site/src/cut/lib/types.ts
+++ b/site/src/cut/lib/types.ts
@@ -865,6 +865,18 @@ export type SidePanelTab =
| "subtitles"
| "publish";
+export const SIDE_PANEL_TABS: SidePanelTab[] = [
+ "media",
+ "elements",
+ "effects",
+ "transitions",
+ "video",
+ "image",
+ "audio",
+ "subtitles",
+ "publish",
+];
+
export interface ProjectDoc {
version: 1;
name: string;
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index b99e8698..b90b8c55 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -1,890 +1,55 @@
/**
* The assistant's tool catalog and skills library.
*
- * Tools are defined once here (JSON Schema) and exposed to both providers
- * through the stdio MCP proxy. Every tool except the `server: true` ones is
- * executed in the browser against the editor store, so the assistant edits
- * the exact same state the user sees.
+ * Tools are defined once (JSON Schema) and exposed to both providers through
+ * the stdio MCP proxy. Every tool except the `server: true` ones is executed
+ * in the browser against the editor store, so the assistant edits the exact
+ * same state the user sees. Every tool is defined in a `*.tools.ts` sibling
+ * of the component that exposes the same feature to the user and aggregated
+ * here, so the UI and the model always offer the same catalog.
*/
-import { ANIM_STYLE_IDS, TRANSITION_STYLE_IDS } from "../../lib/types";
-
-export interface AiToolDef {
- name: string;
- description: string;
- inputSchema: Record<string, unknown>;
- /** Handled directly by the server (no browser round-trip). */
- server?: boolean;
-}
-
-const obj = (
- properties: Record<string, unknown>,
- required: string[] = []
-): Record<string, unknown> => ({
- type: "object",
- properties,
- required,
- additionalProperties: false,
-});
-
-const num = (description: string) => ({ type: "number", description });
-const str = (description: string) => ({ type: "string", description });
-const bool = (description: string) => ({ type: "boolean", description });
+import { AI_PANEL_TOOLS } from "@/cut/components/AiPanel.tools";
+import { OVERLAY_ANIMATION_TOOLS } from "@/cut/components/AnimationTiles.tools";
+import { AUDIO_TOOLS } from "@/cut/components/AudioPanel.tools";
+import { EDITOR_TOOLS } from "@/cut/components/Editor.tools";
+import { EFFECT_TREATMENTS, EFFECTS_TOOLS } from "@/cut/components/EffectsPanel.tools";
+import { ELEMENTS_TOOLS } from "@/cut/components/ElementsPanel.tools";
+import { VIDEO_GEN_TOOLS } from "@/cut/components/GeneratePanel.tools";
+import { IMAGE_GEN_TOOLS } from "@/cut/components/ImageGenPanel.tools";
+import { INSPECTOR_TOOLS } from "@/cut/components/Inspector.tools";
+import { LIBRARY_TOOLS } from "@/cut/components/LibraryView.tools";
+import { PREVIEW_TOOLS } from "@/cut/components/Preview.tools";
+import { SCENE_TOOLS } from "@/cut/components/SceneCard.tools";
+import { SIDE_PANEL_TOOLS } from "@/cut/components/SidePanel.tools";
+import { STOCK_TOOLS } from "@/cut/components/StockPanels.tools";
+import { SUBTITLES_TOOLS } from "@/cut/components/SubtitlesPanel.tools";
+import { TIMELINE_TOOLS } from "@/cut/components/Timeline.tools";
+import { TOP_BAR_TOOLS } from "@/cut/components/TopBar.tools";
+import { TRANSITIONS_TOOLS } from "@/cut/components/TransitionsPanel.tools";
+import { LOOK_EFFECTS, OVERLAY_ANIM_STYLE_IDS, OVERLAY_LOOP_STYLE_IDS } from "@donkeycut/effects-kit";
+import type { AiToolDef } from "../../lib/aiToolDef";
+import { TRANSITION_STYLE_IDS } from "../../lib/types";
export const AI_TOOLS: AiToolDef[] = [
- {
- name: "get_state",
- description:
- "Read the full current editor state: clips, soundtrack, overlay elements (titles, shapes, stickers), subtitles, selection, playhead, view settings, publish metadata. Use this whenever the context snapshot is not enough or might be stale.",
- inputSchema: obj({}),
- },
- {
- name: "capture_frame",
- description:
- "Capture the composited video frame the preview shows at the playhead as an image. Titles and captions are drawn over the canvas in the UI, so they don't appear here — this checks the footage, not the text.",
- inputSchema: obj({}),
- },
- {
- name: "watch_video",
- description:
- "Watch a video source with your own eyes: samples its frames at scene changes plus a steady floor into timestamped contact-sheet images, and returns the detected scene-change times (natural cut candidates). Pass clip_id to watch a timeline clip's source (the result includes that clip's source↔timeline time math) or asset_id for any project video or image. The stamp burned into each cell is SOURCE seconds — what trim_clip's in/out use — not timeline seconds. Coverage is capped per call: survey the whole range first, then call again with a narrow from/to and a small interval_seconds where the cut needs care; the result says where coverage stopped. Read the watching-and-cutting skill before editing footage by content.",
- inputSchema: obj({
- clip_id: str("Video clip id, track 0 or overlay (defaults from/to to its trimmed in/out)"),
- asset_id: str("Project asset id (video or image) — watch the source itself"),
- from: num("Source start s (default: the clip's in, else 0)"),
- to: num("Source end s (default: the clip's out, else the source's end; spans at most 600s per call)"),
- interval_seconds: num("Target seconds between sampled frames, 0.5–30 (default spreads ~32 frames across the range)"),
- }),
- },
- {
- name: "detect_silence",
- description:
- "Find silent stretches in a source's audio — dead air, long pauses, gaps between takes. Returns [{start,end,duration}] in SOURCE seconds, plus each one's timeline times when clip_id is passed. Cheap and image-free; pair it with the transcript's cue timings to find filler, then cut with split_at / trim_clip / delete_item — place speech cuts inside these spans (cue timings drift from the audio), and read the watching-and-cutting skill for the pacing rules.",
- inputSchema: obj({
- clip_id: str("Clip id — video, overlay, or soundtrack; scopes to its trimmed range and maps results to timeline seconds"),
- asset_id: str("Project asset id (video or audio)"),
- from: num("Source start s (default: the clip's in, else 0)"),
- to: num("Source end s (default: the clip's out, else the source's end)"),
- threshold_db: num("Loudness below this counts as silence, dBFS (default -30)"),
- min_silence: num("Shortest silent stretch to report, seconds (default 0.35)"),
- }),
- },
- {
- name: "listen_audio",
- description:
- "Listen with your own ears to a project asset's sound — an audio asset, or the audio track of a video (its speech, music, burned-in narration) — so you can answer what it says or how it sounds. The sound rides back inline (≈12MB cap). Pass clip_id for a timeline clip's source (scopes to its trim) or asset_id for the whole source; add from/to (source seconds) to hear one stretch of a long file. Audio the user attached to their message already plays in it. To WRITE a caption track, use subtitles_generate.",
- inputSchema: obj({
- clip_id: str("Clip id — video or soundtrack (defaults from/to to its trimmed in/out)"),
- asset_id: str("Project asset id (audio or video) — listen to the whole source"),
- from: num("Source start s (default: the clip's in, else 0)"),
- to: num("Source end s (default: the clip's out, else the source's end)"),
- }),
- },
- {
- name: "seek",
- description: "Move the playhead to a time (seconds, clamped to the cut).",
- inputSchema: obj({ t: num("Timeline time in seconds") }, ["t"]),
- },
- {
- name: "set_playing",
- description: "Start or stop playback.",
- inputSchema: obj({ playing: bool("true to play, false to pause") }, ["playing"]),
- },
- {
- name: "select",
- description:
- "Select a video clip (on any track), soundtrack clip, or overlay element — title, shape, or sticker (or clear the selection). Selection drives the inspector panel.",
- inputSchema: obj({
- kind: { type: "string", enum: ["clip", "audio", "overlay", "none"], description: "What to select — 'clip' is any video clip, whatever track; 'overlay' is any title-lane element" },
- id: str("The item id (omit for kind=none)"),
- }, ["kind"]),
- },
- {
- name: "set_side_panel",
- description:
- "Open a side-panel tab, or collapse the panel to the icon rail with 'none' so the preview canvas takes the freed width — use 'none' when the user asks to clean up or maximize the workspace. 'publish' is the Details tab; 'media' holds the project's files and the shared Library.",
- inputSchema: obj({
- panel: {
- type: "string",
- enum: ["media", "elements", "effects", "transitions", "video", "image", "audio", "subtitles", "publish", "none"],
- description: "Tab to open, or 'none' to collapse the panel",
- },
- }, ["panel"]),
- },
- {
- name: "split_at",
- description:
- "Split the video (or a selected soundtrack/overlay clip) at a time, like pressing S. Omit t to split at the playhead.",
- inputSchema: obj({ t: num("Timeline seconds to cut at (optional)") }),
- },
- {
- name: "move_clip",
- description:
- "Reorder a track-0 video clip to a new index: it lifts out (its old spot becomes a gap) and clips from the landing index shift right to make room — nothing else moves, so sound and titles stay synced. To move one clip in time, use place_clip.",
- inputSchema: obj({ clipId: str("Video clip id"), toIndex: num("Target index, 0-based") }, ["clipId", "toIndex"]),
- },
- {
- name: "place_clip",
- description:
- "Move a track-0 video clip to a timeline start time (seconds). The track is free-positioned: gaps are allowed and play black. If another clip occupies that spot, the clip slides right to the next free one.",
- inputSchema: obj({ clipId: str("Video clip id"), start: num("Target timeline start s") }, ["clipId", "start"]),
- },
- {
- name: "add_clip",
- description:
- "Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at `start`, inserted at `index`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at `start`, default the playhead). Asset ids come from `media` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut (\"add my beach photo\", \"stitch these into a movie\"); otherwise media stays on its card or panel for them to drag.",
- inputSchema: obj({
- asset_id: str("Project asset id from `media` in editor_state"),
- start: num("Timeline start s"),
- index: num("Insert position on video track 0 (video/image only; 0 = first)"),
- }, ["asset_id"]),
- },
- {
- name: "trim_clip",
- description:
- "Set a video clip's trim points inside its source media (seconds). Changing `in` hides the leading part; `out` the trailing part.",
- inputSchema: obj({ clipId: str("Video clip id"), in: num("New in point (optional)"), out: num("New out point (optional)") }, ["clipId"]),
- },
- {
- name: "refine_speech_cuts",
- description:
- "True up speech-cut edges mechanically: re-scans the audio around each clip's in/out and re-trims so every edge sits a ~0.25s breath inside a real pause — an edge that clips a word moves outward to the next pause, stray dead air trims off. Clip spacing survives: a tightened clip closes up, butted joints stay butted, and deliberate gaps keep their width. An edge with no pause within ~2s stays put and comes back flagged. Run it on every clip you recut for speech, after cutting and before listening; leave out clips whose edge you placed mid-speech on purpose.",
- inputSchema: obj(
- {
- clip_ids: {
- type: "array",
- items: { type: "string" },
- description: "Video clip ids whose in/out edges are speech cuts",
- },
- },
- ["clip_ids"]
- ),
- },
- {
- name: "set_clip_muted",
- description: "Mute or unmute a video clip's own audio.",
- inputSchema: obj({ clipId: str("Video clip id"), muted: bool("true to mute") }, ["clipId", "muted"]),
- },
- {
- name: "set_clip_volume",
- description: "Set the gain on a video clip's own audio (soundtrack clips use update_audio).",
- inputSchema: obj({ clipId: str("Video clip id"), volume: num("0..1.5 (1 = unchanged)") }, ["clipId", "volume"]),
- },
- {
- name: "set_clip_hidden",
- description:
- "Hide or show a video clip on any track. A hidden clip stays on the timeline (grayed) but is excluded from playback and export — its span plays black and silent on track 0; an overlay layer just disappears.",
- inputSchema: obj({ clipId: str("Video clip id"), hidden: bool("true to hide") }, ["clipId", "hidden"]),
- },
- {
- name: "set_track_hidden",
- description:
- "Hide or show a whole timeline row at once — the track header's eye toggle. Applies to every item on that row: a hidden video track plays black and silent, a hidden soundtrack lane is silent, hidden text (title) lanes and subtitle tracks drop out of the picture. Items stay on the timeline, grayed.",
- inputSchema: obj({
- kind: { type: "string", enum: ["video", "soundtrack", "text", "subtitles"], description: "Which row kind" },
- track: num("Track/lane index, 0-based"),
- hidden: bool("true to hide"),
- }, ["kind", "track", "hidden"]),
- },
- {
- name: "set_track_muted",
- description:
- "Mute or unmute every clip on one video track at once — the track header's speaker toggle. The picture keeps playing.",
- inputSchema: obj({ track: num("Video track index, 0-based"), muted: bool("true to mute") }, ["track", "muted"]),
- },
- {
- name: "detach_audio",
- description:
- "Detach Audio: lift a clip's sound onto the soundtrack track (mutes the clip) so it can be edited independently. Select the clip first or pass its id.",
- inputSchema: obj({ clipId: str("Video clip id (optional if one is selected)") }),
- },
- {
- name: "delete_item",
- description:
- "Delete a video clip (any track), soundtrack clip, or overlay element (title/shape/sticker) by id. While track 0 is the only video track, deleting a track-0 clip ripples: its footprint closes and everything after it — clips, titles, captions, soundtrack — slides left in sync; items wholly inside that span are removed with it, straddlers are trimmed. With overlay video tracks present the delete leaves its gap in place (remove_gap closes it). Deletes on other tracks remove just that item.",
- inputSchema: obj({
- kind: { type: "string", enum: ["clip", "audio", "overlay"], description: "Item kind — 'clip' is any video clip, whatever track; 'overlay' is any title-lane element" },
- id: str("Item id"),
- }, ["kind", "id"]),
- },
- {
- name: "remove_gap",
- description:
- "Close the empty span on one video track containing `at` seconds: that track's later clips slide left to shut the gap; every other track, titles, captions, and the soundtrack stay put. Errors unless `at` falls inside a gap on that track.",
- inputSchema: obj({
- at: num("A time (seconds) inside the gap to close"),
- track: num("Video track the gap is on (default 0)"),
- }, ["at"]),
- },
- {
- name: "add_overlay_video",
- description:
- "Put a project video or image asset on an overlay video track, composited over track 0: tracks stack bottom-up, and the topmost full-frame clip covers everything below. Pick a layout to share the frame — halves for a split screen, pip for picture-in-picture. Asset ids come from `media` in the editor state.",
- inputSchema: obj({
- asset_id: str("Project asset id (video or image)"),
- start: num("Timeline start s (default: the playhead)"),
- track: num("Video track, 1 or higher; 1 = first layer above track 0, higher = further in front (default 1)"),
- layout: {
- type: "string",
- enum: ["full", "top", "bottom", "left", "right", "pip"],
- description: "Frame region (default full = covers the frame)",
- },
- }, ["asset_id"]),
- },
- {
- name: "update_overlay_video",
- description:
- "Update an overlay video clip: move it (start, track), trim (in/out), mute, hide, change its frame region (layout preset, or a custom region rect in frame fractions), fit, or speed.",
- inputSchema: obj({
- id: str("Overlay video clip id"),
- start: num("Timeline start s"),
- in: num("Source in s"),
- out: num("Source out s"),
- track: num("Video track, 1 or higher; higher = further in front"),
- muted: bool("Mute the clip's own audio"),
- hidden: bool("Hide the layer without deleting it"),
- layout: {
- type: "string",
- enum: ["full", "top", "bottom", "left", "right", "pip"],
- description: "Frame region preset",
- },
- region: obj({
- x: num("Left edge 0..1"),
- y: num("Top edge 0..1"),
- w: num("Width 0..1"),
- h: num("Height 0..1"),
- }, ["x", "y", "w", "h"]),
- fit: { type: "string", enum: ["fit", "fill"], description: "How the video meets its region" },
- speed: num("Playback rate (1 = normal, no upper limit)"),
- }, ["id"]),
- },
- {
- name: "add_title",
- description:
- "Add a text title overlay. Position is the text center as a fraction of the frame (x,y in 0..1; y=0.42 is the default band). Size is px at 1080-wide. Font ids come from the graphics skill (system set + the bundled Google families).",
- inputSchema: obj({
- text: str("The title text (\\n for line breaks)"),
- start: num("Start time s (default: playhead)"),
- end: num("End time s (default: start+3)"),
- x: num("Center x 0..1 (default 0.5)"),
- y: num("Center y 0..1 (default 0.42)"),
- size: num("Font size px at 1080w (default 88)"),
- color: str("CSS color (default #FFFFFF)"),
- font: str("Font id (see the graphics skill; default sf)"),
- weight: { type: "number", enum: [400, 700], description: "Font weight" },
- italic: bool("Italic"),
- align: { type: "string", enum: ["left", "center", "right"], description: "Multi-line alignment (default center)" },
- letter_spacing: num("Tracking in em (0 = normal, 0.1 = airy)"),
- line_height: num("Line height multiplier (default 1.25)"),
- stroke_color: str("Text outline color"),
- stroke_width: num("Text outline width in em (0..0.15; 0 removes it)"),
- shadow: bool("Drop shadow (default true)"),
- plate: bool("Translucent plate behind text (default false)"),
- behind_subject: bool("Sit the text behind the person in the shot (needs a clearly separated speaker)"),
- rotation: num("Degrees clockwise, -180..180"),
- opacity: num("Whole-element opacity 0..1"),
- }, ["text"]),
- },
- {
- name: "add_shape",
- description:
- "Add a vector shape overlay: rect, ellipse, line, or arrow. Position is the shape center as frame fractions; w/h are frame fractions too (a line/arrow's h is its stroke thickness). Rotation gives lines and arrows their direction.",
- inputSchema: obj({
- shape: { type: "string", enum: ["rect", "ellipse", "line", "arrow"], description: "Shape kind" },
- start: num("Start time s (default: playhead)"),
- end: num("End time s (default: start+3)"),
- x: num("Center x 0..1 (default 0.5)"),
- y: num("Center y 0..1 (default 0.5)"),
- w: num("Width, fraction of frame width"),
- h: num("Height, fraction of frame height (line/arrow: thickness)"),
- fill: str("CSS color (default #FFFFFF)"),
- fill_opacity: num("Fill opacity 0..1 (rect/ellipse)"),
- radius: num("Rect corner radius, px at 1080 short side"),
- stroke_color: str("Outline color (rect/ellipse)"),
- stroke_width: num("Outline width px at 1080 short side (0 removes it)"),
- rotation: num("Degrees clockwise, -180..180"),
- opacity: num("Whole-element opacity 0..1"),
- }, ["shape"]),
- },
- {
- name: "add_sticker",
- description:
- "Add a sticker overlay from a project image asset (asset ids come from `media`; sticker uploads carry origin \"sticker\"). Width is a frame-width fraction; height follows the source's own aspect.",
- inputSchema: obj({
- asset_id: str("Project image asset id"),
- start: num("Start time s (default: playhead)"),
- end: num("End time s (default: start+3)"),
- x: num("Center x 0..1 (default 0.5)"),
- y: num("Center y 0..1 (default 0.5)"),
- w: num("Width, fraction of frame width (default 0.25)"),
- rotation: num("Degrees clockwise, -180..180"),
- opacity: num("Whole-element opacity 0..1"),
- }),
- },
- {
- name: "update_overlay",
- description:
- "Update any overlay element — title, shape, or sticker — by id (from the selection or state). Titles take text/size/font/weight/color/shadow/plate; shapes take w/h/fill/fill_opacity/radius/stroke; stickers take w. Every kind takes timing, position, rotation, opacity, hidden. This is the tool for 'make this text better' requests too.",
- inputSchema: obj({
- id: str("Overlay element id"),
- text: str("New text (titles)"),
- start: num("Start s"),
- end: num("End s"),
- x: num("Center x 0..1"),
- y: num("Center y 0..1"),
- size: num("Font size px at 1080w (titles)"),
- color: str("CSS text color (titles)"),
- font: str("Font id (titles; see the graphics skill)"),
- weight: { type: "number", enum: [400, 700], description: "Font weight (titles)" },
- italic: bool("Italic (titles)"),
- align: { type: "string", enum: ["left", "center", "right"], description: "Multi-line alignment (titles)" },
- letter_spacing: num("Tracking in em (titles; 0 = normal)"),
- line_height: num("Line height multiplier (titles; default 1.25)"),
- shadow: bool("Drop shadow (titles)"),
- plate: bool("Backdrop plate (titles)"),
- behind_subject: bool("Sit the text behind the person in the shot (titles)"),
- w: num("Width, fraction of frame width (shapes/stickers)"),
- h: num("Height, fraction of frame height (shapes)"),
- fill: str("Fill color (shapes)"),
- fill_opacity: num("Fill opacity 0..1 (rect/ellipse)"),
- radius: num("Rect corner radius, px at 1080 short side"),
- stroke_color: str("Outline color — text or shape"),
- stroke_width: num("Outline width: em for titles (0..0.15), px at 1080 for shapes; 0 removes it"),
- rotation: num("Degrees clockwise, -180..180 (0 clears)"),
- opacity: num("Whole-element opacity 0..1 (1 clears)"),
- hidden: bool("Hide the element without deleting it"),
- }, ["id"]),
- },
- {
- name: "create_sticker",
- description:
- "Create a custom sticker from an idea: the hosted image model draws it (signed in, spends credits), the background is removed (people on-device, other subjects via hosted matting), a white die-cut outline is added, and it lands as an origin-\"sticker\" asset placed on the timeline at the playhead. Takes ~10-20s; the tool returns when the sticker is placed.",
- inputSchema: obj({
- idea: str("What the sticker shows, e.g. \"a corgi in sunglasses\""),
- start: num("Start time s (default: playhead)"),
- end: num("End time s (default: start+3)"),
- x: num("Center x 0..1 (default 0.5)"),
- y: num("Center y 0..1 (default 0.5)"),
- w: num("Width, fraction of frame width (default 0.25)"),
- }, ["idea"]),
- },
- {
- name: "add_effect",
- description:
- "Add a time-ranged visual effect element over the video: a zoom into part of the frame, the treatments (grain, vhs, glitch, blur, vignette, lightleak, flash, shake) or a graded look (vintage, horror, halation, tech, noir, pastel, blockbuster, dreamy). It filters everything under it for its window (start..end) and stacks with other effects. Tasteful default: one effect at a time, amount under ~0.5.",
- inputSchema: obj({
- effect: { type: "string", enum: ["zoom", "grain", "vhs", "glitch", "blur", "vignette", "lightleak", "flash", "shake", "vintage", "horror", "halation", "tech", "noir", "pastel", "blockbuster", "dreamy"], description: "Effect id" },
- start: num("Start time s (default: playhead)"),
- end: num("End time s (default: start+3)"),
- amount: num("Strength 0.05..1 (default 0.5). For zoom it is the depth: 0.25 shallow, 0.5 moderate, 1 deep"),
- focus_x: num("Zoom only: the x of the point to zoom into, 0..1 (default 0.5)"),
- focus_y: num("Zoom only: the y of the point to zoom into, 0..1 (default 0.5)"),
- }, ["effect"]),
- },
- {
- name: "set_overlay_animation",
- description:
- "Animate an overlay element (title, shape, or sticker): preset In/Out ramps plus a Loop that runs its whole duration. Omitted slots keep their setting; pass \"none\" to clear one. In/Out styles: fade, pop, zoom, slideleft/slideright/slideup/slidedown (names are the motion direction), typewriter (titles only). Loop styles: pulse, wiggle, spin, float.",
- inputSchema: obj({
- id: str("Overlay element id"),
- in_style: str("Entrance style, or \"none\" to clear"),
- in_seconds: num("Entrance ramp seconds 0.1..2 (default 0.5)"),
- out_style: str("Exit style, or \"none\" to clear"),
- out_seconds: num("Exit ramp seconds 0.1..2 (default 0.5)"),
- loop_style: str("Loop style, or \"none\" to clear"),
- loop_speed: num("Loop rate multiplier 0.25..4 (default 1)"),
- }, ["id"]),
- },
- {
- name: "set_overlay_keyframes",
- description:
- "Give an overlay element a keyframed pose track, for motion the presets cannot express (a path across the frame, a slow push in, a hold-then-drift). Each key is a whole pose at a time measured in seconds from the element's own start; the pose moves linearly between keys and holds outside them. Omitted fields on a key take the element's current value. Pass an empty list to clear the track and return the element to its resting pose. Preset In/Out/Loop animation still composes on top, so a keyframed title can also fade in.",
- inputSchema: obj(
- {
- id: str("Overlay element id"),
- keys: {
- type: "array",
- description: "Keys in any order; two at the same time collapse to one",
- items: obj(
- {
- t: num("Seconds from the element's start"),
- x: num("Center x, fraction of frame width 0..1"),
- y: num("Center y, fraction of frame height 0..1"),
- scale: num("Size multiplier, 1 = the element's own size (0.1..4)"),
- rotation: num("Degrees clockwise, -180..180"),
- opacity: num("0..1"),
- },
- ["t"]
- ),
- },
- },
- ["id", "keys"]
- ),
- },
- {
- name: "update_audio",
- description:
- "Update a soundtrack clip: volume (0..1.5), fadeIn/fadeOut seconds, start position, in/out trim, speed, hidden, or duck. `duck` is voiceover ducking — while this clip plays, ALL other audio (video-clip sound and other music) drops to that gain (0..1); pass 1 to clear ducking. Use it to make a voiceover sit over quieter music.",
- inputSchema: obj({
- id: str("Soundtrack clip id"),
- volume: num("0..1.5"),
- fadeIn: num("Fade-in seconds"),
- fadeOut: num("Fade-out seconds"),
- start: num("Timeline start s"),
- in: num("Source in s"),
- out: num("Source out s"),
- speed: num("Playback rate (1 = normal, no upper limit)"),
- duck: num("Duck other audio to this gain while this clip plays, 0..1 (1 clears ducking)"),
- hidden: bool("Silence the clip without removing it (grayed on the timeline)"),
- }, ["id"]),
- },
- {
- name: "set_framing",
- description:
- "Set how a video clip meets the 9:16 frame: 'fit' letterboxes the whole picture (default), 'fill' scales it to cover the frame and crops the overflow. In fill mode panX/panY (-1..1, 0=centered) choose which part stays visible — e.g. panY=-1 keeps the top.",
- inputSchema: obj({
- clipId: str("Video clip id"),
- mode: { type: "string", enum: ["fit", "fill"], description: "Framing mode" },
- panX: num("Crop pan -1 (left) .. 1 (right), fill mode only"),
- panY: num("Crop pan -1 (top) .. 1 (bottom), fill mode only"),
- }, ["clipId", "mode"]),
- },
- {
- name: "freeze_frame",
- description:
- "Extract the video frame at a time (default: the playhead — what the user is looking at) as a still clip and insert it into the timeline. Default insert position is index 0, making it the first thing viewers see (a cover/hook frame).",
- inputSchema: obj({
- t: num("Timeline time of the frame to grab (default: playhead)"),
- duration: num("Still clip length in seconds, 0.5–10 (default 1)"),
- index: num("Insert position on the video track (default 0 = first)"),
- }),
- },
- {
- name: "generate_image",
- description:
- "Generate an AI image from a text prompt (Donkey's hosted image model). Use for B-roll, cover frames, or backgrounds the user doesn't have footage for; when the user asks you to write or improve the prompt itself, put it in chat and wait for them to ask for the image. reference_asset_ids attaches project media the render should draw from (the user's attached references, a clip to restyle) — the prompt is recomposed around them; an audio reference contributes what it carries (its speech transcribed, its sound described) to that recomposed prompt. Returns when the image has landed. The image previews as a card in this chat, where the user can expand it, drag it onto the timeline, or file it into Media or the Library from its \"…\" menu; pass add_to_timeline:true (or an index) only when they asked for it in the cut. Needs the user signed in to Donkey (spends their credits).",
- inputSchema: obj({
- prompt: str("What to depict — be specific about subject, style, and lighting"),
- aspect: { type: "string", enum: ["16:9", "9:16", "1:1"], description: "Image shape (default: the supported shape closest to the project aspect)" },
- resolution: { type: "string", enum: ["1K", "2K", "4K"], description: "Output detail (default 1K)" },
- reference_asset_ids: {
- type: "array",
- items: { type: "string" },
- description: "Project asset ids to reference — images and videos ride as pictures; audio folds what it carries into the prompt. Append @seconds to read a video/audio reference at a pinned moment (\"<id>@16.3\")",
- },
- add_to_timeline: bool("Insert the still on the video track (default false — it stays on its chat card until the user asks)"),
- index: num("Insert position on the video track (passing it implies add_to_timeline; 0 = first/cover)"),
- }, ["prompt"]),
- },
- {
- name: "generate_video",
- description:
- "Generate an AI video clip from a text prompt (Donkey's hosted video model: one pass, audio included, 720p, up to ~10s — the model picks the length) — the default for any video ask: \"generate/make a video of/about X\" means ONE clip from this tool, even though it sounds whole. Reach for generate_scene only when they name a narrated multi-shot production (a story, episode, or narrated short). When the user asks you to write or improve the prompt itself, put it in chat and wait for them to ask for the video. reference_asset_id anchors the render on one project asset: for an image or video, by default the tool first designs the opening frame from it with the image model (that still previews in this chat), then animates that exact frame — so tell the user it designs the frame, then renders; for audio, what it carries (speech transcribed, sound described) folds into the prompt instead. When the user wants the referenced image itself brought to life (\"animate this image/picture\"), pass animate_reference:true — the image becomes the literal first frame, unchanged, and the prompt describes only the motion. A safety-blocked or rejected anchor degrades the render on its own (identity reference, then text-only — the same ladder scene shots walk), so never resubmit a failed render with weaker inputs yourself. The render takes a minute or two and the tool RETURNS as soon as it's started — the clip previews as a live card in this chat when it finishes. Pass add_to_timeline:true (or an index) only when they asked for it in the cut; otherwise the user drags it in from the card. Needs the user signed in to Donkey (spends their credits).",
- inputSchema: obj({
- prompt: str("The shot to generate — describe motion, subject, and mood"),
- aspect: { type: "string", enum: ["16:9", "9:16"], description: "Clip shape (default: the supported shape closest to the project aspect)" },
- reference_asset_id: str("One project asset id to reference — an image or video anchors the clip's opening frame; audio folds what it carries into the prompt. Append @seconds to read a video/audio reference at a pinned moment (\"<id>@16.3\")"),
- animate_reference: bool("The referenced image IS the opening frame: animate it directly, unchanged, skipping the frame-design step (use when the user says to animate that image)"),
- add_to_timeline: bool("Insert the clip on the video track when it lands (default false — it stays on its chat card until the user asks)"),
- index: num("Insert position on the video track (passing it implies add_to_timeline)"),
- }, ["prompt"]),
- },
- {
- name: "generate_character_video",
- description:
- "Generate a UGC-style selfie clip of a stock talking character speaking a line you write (Donkey's hosted video model). Pick a character id from stock_search kind:\"character\" — each has a persona and look; the same person then delivers the line to camera. Like generate_video this RETURNS IMMEDIATELY and the clip previews in this chat a minute or two later; add_to_timeline:true places it when it lands. Needs the user signed in to Donkey (spends their credits).",
- inputSchema: obj({
- character_id: str("Talking-character id from stock_search"),
- line: str("What the character says, spoken to camera"),
- add_to_timeline: bool("Insert the clip on the video track when it lands (default false — it stays on its chat card until the user asks)"),
- index: num("Insert position on the video track (passing it implies add_to_timeline)"),
- }, ["character_id", "line"]),
- },
- {
- name: "wait_for_renders",
- description:
- "Block until this project's in-flight video renders settle (up to ~100s), then report each one: landed (with its asset id, ready to place) or failed (with the error). Call it whenever the user's ask depends on a render that `renders` in the state shows as running — \"add it when it's done\", \"assemble the clips\" — and then finish the job in the same turn; never tell the user to come back and report when a card appears. If some renders are still running when it returns, say how long they've been going and call it again on the user's go-ahead.",
- inputSchema: obj({}),
- },
- {
- name: "generate_scene",
- description:
- "Generate a narrated, multi-shot cut assembled on the timeline — script, voiceover, and several shots — from a brief, or animate audio already in the project. Use this only when the user names that kind of production (\"tell a story about…\", \"turn this into a narrated 20-second short\", \"an episode\"); a plain \"generate/make a video of/about X\" is a single clip — generate_video, not this. It writes a script, breaks it into shots, and draws each shot's opening frame, then RETURNS that storyboard and STOPS: the user reviews the frames (and can ask to redraw any of them with regenerate_shot — cheap, no credits at this stage) — get their go-ahead and call approve_scene before the shots render, because each shot spends credits. Pass from_audio_asset_id to animate an audio spine the project already has (a voiceover, recording, or song → shots tiled over it) instead of writing a new script. Read the scene-productions skill before your first call — it covers the brief and look, references, aspect, and the from-audio flow. Needs the user signed in to Donkey (spends credits).",
- inputSchema: obj({
- brief: str("What the video is about — the story or subject. Omit only when animating existing audio."),
- from_audio_asset_id: str("Animate this project audio asset (id from media) — shots tile over it instead of a new script"),
- audio_language: str("BCP-47 language of the from_audio asset's speech (e.g. ko-KR). REQUIRED with from_audio_asset_id whenever the speech is not English — the on-device transcriber picks its recognizer from it, and the wrong recognizer garbles every shot planned over the audio"),
- target_seconds: num("Rough total length in seconds, 6–90 (default ~24)"),
- style: str("Optional look to steer the whole video (e.g. 'moody neon, handheld'); usually left to the brief"),
- reference_asset_ids: {
- type: "array",
- items: { type: "string" },
- description: "Project image/video asset ids that anchor the look (the user's attached references)",
- },
- }),
- },
- {
- name: "approve_scene",
- description:
- "Approve the pending generate_scene shot plan and start rendering. The shots render in the background and land on the timeline as they finish. Call this only when the user confirms (\"go\", \"do it\", \"looks good\") — approving spends credits per shot.",
- inputSchema: obj({}),
- },
- {
- name: "cancel_scene",
- description:
- "Stop the scene generation running in this project — planning or rendering halts immediately and the plan is discarded. Clips already placed stay on the timeline; credits already spent are not refunded. Call it when the user wants the run stopped, or to clear the way for a new generate_scene.",
- inputSchema: obj({}),
- },
- {
- name: "regenerate_shot",
- description:
- "Redo one shot of the generated scene by its number (1-based), optionally nudging it (\"wider\", \"at night\", \"more energetic\"). At the storyboard gate (before approval) it redraws just that shot's opening frame — cheap, no credits, the run stays waiting for approval. After the scene has rendered it re-renders the shot and swaps its clip on the timeline by itself — never delete the clip first.",
- inputSchema: obj({ n: num("Shot number, 1-based"), note: str("Optional change to apply to that shot") }, ["n"]),
- },
- {
- name: "recut_scene",
- description:
- "Re-cut a contiguous span of the generated scene's shots: replans just that span against the same audio — it can split into more shots or merge into fewer — keeping the style, cast, and every other shot's clip, then renders only the new shots (credits per new shot). Use it when the user wants a section restructured (\"split the last shot\", \"shots 2-3 drag, tighten them\", \"that part is missing the soccer scene\"); one shot redone as-is is regenerate_shot, a whole-look change is restyle_scene. Only valid after a scene has finished.",
- inputSchema: obj({
- from_shot: num("First shot of the span, 1-based"),
- to_shot: num("Last shot of the span, 1-based (equal to from_shot for one shot)"),
- instruction: str("What the span should become — the content to cover and how to pace it"),
- }, ["from_shot", "to_shot", "instruction"]),
- },
- {
- name: "restyle_scene",
- description:
- "Restyle the whole generated scene and redo every shot with a new look (\"make it black-and-white film noir\", \"turn it into anime\"). Only valid after a scene has been generated. Spends credits (every shot re-renders), so confirm first.",
- inputSchema: obj({ style: str("The new look for the whole video") }, ["style"]),
- },
- {
- name: "stock_search",
- description:
- "Search Cut's bundled stock catalogs: footage clips and stock images across Business/Nature/Travel/City/Technology/Anime/Animal/Food categories, plus talking characters (personas for generate_character_video). Stock is local and free — check it before spending generation credits when existing media could serve. Add a match to the project with stock_add.",
- inputSchema: obj({
- query: str("Words to match against prompts, categories, and tags (omit to browse)"),
- kind: { type: "string", enum: ["video", "image", "character"], description: "Limit to one catalog (default: all)" },
- }),
- },
- {
- name: "stock_add",
- description:
- "Import a stock video or image (by stock_search id) into the project. It previews as a card in this chat; pass add_to_timeline:true (or a `start`) to also drop it on the video track when the user asked for it in the cut. Free — the media ships with Cut.",
- inputSchema: obj({
- id: str("Stock item id from stock_search"),
- add_to_timeline: bool("Place it on video track 0 (default false — it stays on its chat card until the user asks)"),
- start: num("Timeline start s (passing it implies add_to_timeline; default when placed: appended at the end)"),
- }, ["id"]),
- },
- {
- name: "import_url",
- description:
- "Read any URL — TikTok, YouTube, Instagram Reels, an X/Twitter post or Article, an ordinary web page, or a direct video/audio/image link — with the bundled downloader and import what it holds into the project. Free and local. A web page comes back as its article text plus the pictures on it; a post as its video or photos; and the source's own words (returned as sourceText) are quoted for the user beside the media automatically — don't retype them in your reply. A source that is only words returns sourceText with no assets, which is a success: read it and answer from it. This is how you look something up: point it at the page and read what comes back. Media lands on a card in this chat and the user drags it to the timeline, Media, or the Library; place it yourself (add_clip) only when they asked for it in the cut. A short clip downloads in seconds; a long video can take a couple of minutes.",
- inputSchema: obj({ url: str("The page or media URL to download") }, ["url"]),
- },
- {
- name: "library_list",
- description:
- "List the shared Library — reusable media saved across projects: folders, assets (video/audio/image), and templates (saved arrangements of clips, overlays, titles, and captions). Library items live outside the project: library_add imports an asset, template_add re-materializes a template.",
- inputSchema: obj({}),
- },
- {
- name: "library_add",
- description:
- "Copy a Library asset into the project (it appears in `media` and previews as a card in this chat). This is the import step \"library\"-scope attachments need before editor tools can touch them. Pass add_to_timeline:true (or start/index) only when the user asked for it in the cut: video/image land on track 0, audio on the soundtrack.",
- inputSchema: obj({
- id: str("Library asset id (from library_list or an attachment's metadata)"),
- add_to_timeline: bool("Also place it on the timeline (default false — it stays a project asset until the user asks)"),
- start: num("Timeline start s (implies add_to_timeline)"),
- index: num("Insert position on video track 0 (video/image; implies add_to_timeline)"),
- }, ["id"]),
- },
- {
- name: "template_add",
- description:
- "Re-materialize a Library template into the project: its media import as assets and its clips, overlays, titles, and captions land editable, exactly as saved (clip layers append to track 0; free-positioned parts line up at the playhead). Call it only when the user asked for the template in the cut.",
- inputSchema: obj({ id: str("Template id from library_list") }, ["id"]),
- },
- {
- name: "save_template",
- description:
- "Save timeline items as a reusable template in this project's Media, kept by reference — the source media plus the edit arranging it, re-editable when added back. The user can push it to the shared Library from the Media panel. Pass the ids of the items to include: video clips (any track), soundtrack clips, titles, and subtitle cues.",
- inputSchema: obj({
- name: str("Template name"),
- item_ids: {
- type: "array",
- items: { type: "string" },
- description: "Timeline item ids to include",
- },
- }, ["name", "item_ids"]),
- },
- {
- name: "library_organize",
- description:
- "Organize the shared Library: create_folder / rename_folder / delete_folder (a deleted folder's items drop to the root), move_asset files an asset or template into a folder (omit folder_id for the root), delete_asset / delete_template remove an item. Deletes are permanent — projects keep their own copies, but delete only what the user explicitly asked to remove.",
- inputSchema: obj({
- action: {
- type: "string",
- enum: ["create_folder", "rename_folder", "delete_folder", "move_asset", "delete_asset", "delete_template"],
- description: "The organize operation",
- },
- name: str("Folder name (create_folder, rename_folder)"),
- folder_id: str("Folder id (rename_folder, delete_folder, move_asset destination — omit for root)"),
- id: str("Library asset or template id (move_asset, delete_asset, delete_template)"),
- }, ["action"]),
- },
- {
- name: "file_asset",
- description:
- "File a project asset where the user keeps things: to \"media\" moves created media (generated, chat, voiceover…) into the Media panel by clearing its origin tag; to \"library\" copies any project asset into the shared Library for reuse. The same moves as each card's \"…\" menu — make them when the user asks.",
- inputSchema: obj({
- asset_id: str("Project asset id from `media`"),
- to: { type: "string", enum: ["media", "library"], description: "Destination" },
- }, ["asset_id", "to"]),
- },
- {
- name: "delete_asset",
- description:
- "Remove a project asset and every timeline clip that uses it — the Media panel's trash. Removed media does not come back with undo, so call it only when the user explicitly asked to remove that media, and report how many clips went with it.",
- inputSchema: obj({ asset_id: str("Project asset id from `media`") }, ["asset_id"]),
- },
- {
- name: "subtitles_generate",
- description:
- "Transcribe the cut on-device (Apple speech) and create subtitle captions on a subtitle track (the active one unless `track` says otherwise; other tracks keep their captions) — it WRITES captions the user sees, so call it only when they asked for subtitles; to just hear what an audio asset says, listen_audio. Cue times come back snapped to the audio, so treat them as in sync — retime a cue only when the user points at one. Runs in the background; returns when finished. If no speech is found, no subtitles are added.",
- inputSchema: obj({
- locale: str("Speech language as BCP-47 like en-US (default: the track's language)"),
- track: num("Subtitle track to write, 0-based (default: the active track)"),
- }),
- },
- {
- name: "captions_generate",
- description:
- "Transcribe (if needed) then REWRITE one subtitle track's captions into punchy social-video captions — short lines that fit inside the video frame (they may wrap onto two lines but never overflow), a few emoji, a curiosity-hook opener. style: clean | hook | punchy (default hook). Cue timings are preserved. It replaces the track's existing text wholesale — for a targeted edit (removing filler words, fixing one line), edit the cues with update_cue instead.",
- inputSchema: obj({
- style: str("Caption style: clean, hook, or punchy"),
- track: num("Subtitle track to rewrite, 0-based (default: the active track)"),
- }),
- },
- {
- name: "subtitles_from_visuals",
- description:
- "Caption a cut that has NO usable speech by watching sampled frames and writing timed narration captions (uses the user's Claude login to look at the frames). Use this instead of subtitles_generate when the video is silent b-roll, music-only, or otherwise has nothing to transcribe. Runs in the background; returns when finished.",
- inputSchema: obj({
- locale: str("Caption language as BCP-47 like en-US (default: the track's language)"),
- track: num("Subtitle track to write, 0-based (default: the active track)"),
- }),
- },
- {
- name: "subtitles_add_track",
- description:
- "Add a subtitle track (up to 3, one language each — e.g. English on track 0, Korean on track 1; each shows as its own caption line, draggable to its own spot). The new track becomes active and starts empty: fill it with subtitles_translate_track, or subtitles_generate in its language.",
- inputSchema: obj({ language: str("The track's language as BCP-47, e.g. ko-KR (optional)") }),
- },
- {
- name: "subtitles_remove_track",
- description:
- "Remove a subtitle track and its captions; higher tracks shift down. Removing the only track empties it — so this also answers \"delete the subtitles\".",
- inputSchema: obj({ track: num("Track to remove, 0-based") }, ["track"]),
- },
- {
- name: "subtitles_translate_track",
- description:
- "Translate existing captions into another language on their own subtitle track — the way to answer \"add Korean subtitles\". Reuses the track already set to that language or adds one (max 3), then translates the source track's cues one-to-one, timings kept. The captions must exist first (subtitles_generate).",
- inputSchema: obj({
- language: str("Target language as BCP-47, e.g. ko-KR"),
- from_track: num("Source track, 0-based (default: the first track with captions)"),
- }, ["language"]),
- },
- {
- name: "list_voices",
- description:
- "List the AI voices available for voiceover (Gemini's prebuilt set; each has a one-word character like Warm, Upbeat, Gravelly). Call this when the user asks for a specific kind of voice so you can pass the right voice id to voiceover_generate or read_subtitles_aloud.",
- inputSchema: obj({}),
- },
- {
- name: "voiceover_generate",
- description:
- "Generate a spoken AI voiceover from a script (Donkey's hosted speech model). The audio previews as a playable card in this chat; pass add_to_timeline:true (or a `start`) to also drop it on the soundtrack when the user asked for it in the cut ('add a voiceover', 'narrate this'). When the user asks you to write or rework the script itself, put the text in chat and wait for them to ask for the voiceover. Pick a `voice` id from list_voices, or omit for a good default. `direction` steers delivery in natural language and can ask for another language — 'say it in Spanish' translates the script before synthesis; the script itself may carry inline tags like [whispers] or [excited]. `duck` lowers all other audio to that gain while the voice plays (0..1; ~0.3–0.5 is typical, 1 = don't duck). Needs the user signed in to Donkey (spends their credits).",
- inputSchema: obj({
- script: str("What the voice should say"),
- voice: str("Voice id from list_voices (optional; a sensible default is chosen)"),
- direction: str("Delivery instruction, e.g. 'Say warmly, like an old friend'; may include a language ask, which translates the script (optional)"),
- language: str("Pronunciation language as BCP-47, e.g. es-US, ja-JP (optional; default auto-detects — this reads the script as written, it does not translate)"),
- duck: num("Lower other audio to this gain while the voice plays, 0..1 (default 0.4; 1 = no ducking)"),
- add_to_timeline: bool("Place it on the soundtrack (default false — it stays on its chat card until the user asks)"),
- start: num("Timeline start in seconds (passing it implies add_to_timeline; default when placed: the playhead)"),
- }, ["script"]),
- },
- {
- name: "generate_music",
- description:
- "Generate a music track from a text prompt (Donkey's hosted music model, Gemini/Lyria) — a full song with sung vocals, or an instrumental bed. Describe mood, genre, instruments, and tempo. By default it renders an instrumental bed (instrumental:true); pass instrumental:false for a full song, where the model writes and sings its own lyrics — or include your own with [Verse]/[Chorus] tags and a singer description in the prompt. length \"clip\" is ~30s (default), \"song\" a longer ~2-minute track. reference_asset_ids matches a track to the user's own media (\"a song that matches this audio\", \"score the tone of this video\"): pass the project asset ids — an audio track to emulate, or video/images whose mood to carry — and a multimodal pass folds a description of that sound into the prompt (the model itself never hears the audio, so always pass the reference rather than describing it yourself). When you match an audio reference that has singing, pass instrumental:false so the new track sings too. When the user asks you to write or refine the lyrics or brief itself, put it in chat and wait for them to ask for the music. It RETURNS once the track has landed and previews as a playable card in this chat; pass add_to_timeline:true (or a `start`) to drop it on the soundtrack when the user asked for it in the cut. For \"background music for this video\", read the timeline (videoTrack, project duration) to fit the mood and length, keep it instrumental so any speech stays on top, and set those clips' `duck` so the bed dips under them. This is SUNG music — a spoken narration is voiceover_generate. Needs the user signed in to Donkey (spends their credits).",
- inputSchema: obj({
- prompt: str("The music to generate — mood, genre, instruments, tempo; for a song, the theme to sing about, or your own [Verse]/[Chorus] lyrics"),
- instrumental: bool("Vocal-free background bed (default true). Pass false to let the model write and sing a full song"),
- length: { type: "string", enum: ["clip", "song"], description: "clip = ~30s (default), song = ~2-minute track" },
- reference_asset_ids: {
- type: "array",
- items: { type: "string" },
- description: "Project asset ids to match the sound of — an audio track to emulate, or video/images whose mood the score should carry. Append @seconds to match the passage around a pinned moment (\"<id>@62\")",
- },
- volume: num("Bed volume 0..1.5 when placed (default 0.4 so it sits under speech)"),
- add_to_timeline: bool("Place it on the soundtrack (default false — it stays on its chat card until the user asks)"),
- start: num("Timeline start in seconds (passing it implies add_to_timeline; default when placed: the playhead)"),
- }, ["prompt"]),
- },
- {
- name: "read_subtitles_aloud",
- description:
- "Speak one subtitle track's cues as an AI voiceover (Donkey's hosted speech model) — each line placed at its own cue time — and add it to the soundtrack. Turns captions into narration. Requires subtitles to exist first (generate them if needed). `duck` lowers other audio under the voice (0..1). Needs the user signed in to Donkey (spends their credits).",
- inputSchema: obj({
- voice: str("Voice id from list_voices (optional)"),
- direction: str("Delivery instruction, e.g. 'Narrate briskly, documentary style'; may include a language ask, which translates the lines (optional)"),
- language: str("Pronunciation language as BCP-47, e.g. es-US, ja-JP (optional; default auto-detects — reads cues as written; for another language use a translated subtitle track)"),
- duck: num("Lower other audio to this gain while the voice plays, 0..1 (default 0.4)"),
- track: num("Subtitle track to read, 0-based (default: the active track)"),
- }),
- },
- {
- name: "subtitles_set_view",
- description: "Toggle subtitles on the video (preview + export burn-in) and/or the timeline cue track.",
- inputSchema: obj({ showOnVideo: bool("Captions on the video"), showOnTimeline: bool("Cue track on the timeline") }),
- },
- {
- name: "update_cue",
- description: "Edit a subtitle cue's text or retime it (start/end seconds).",
- inputSchema: obj({ id: str("Cue id"), text: str("New text"), start: num("Start s"), end: num("End s") }, ["id"]),
- },
- {
- name: "delete_cue",
- description: "Delete a subtitle cue.",
- inputSchema: obj({ id: str("Cue id") }, ["id"]),
- },
- {
- name: "set_publish",
- description:
- "Set the TikTok publish metadata: caption (4,000 char limit incl. tags), tags (space-separated words, # added automatically), soundTitle, handle.",
- inputSchema: obj({
- caption: str("Caption text"),
- tags: str("Space or comma separated tags"),
- soundTitle: str("Sound title"),
- handle: str("Creator handle without @"),
- }),
- },
- {
- name: "set_view",
- description:
- "Adjust the timeline view: zoom (pxPerSec 12..800), fit the whole cut, or panel height (170..600).",
- inputSchema: obj({
- pxPerSec: num("Zoom in px per second"),
- fit: bool("Fit the whole cut to the window"),
- timelineH: num("Timeline panel height px"),
- }),
- },
- {
- name: "undo",
- description: "Undo the last edit (unlimited).",
- inputSchema: obj({}),
- },
- {
- name: "redo",
- description: "Redo the last undone edit.",
- inputSchema: obj({}),
- },
- {
- name: "open_export",
- description:
- "Open the export dialog so the user can render the cut (presets from Original quality down to Draft 720p). Exporting itself stays a user action.",
- inputSchema: obj({}),
- },
- {
- name: "set_speed",
- description:
- "Set a video clip's playback speed. Faster shortens the clip on the timeline; slower stretches it. Later titles and captions shift to stay in sync.",
- inputSchema: obj({ clipId: str("Video clip id"), speed: num("Playback rate (1 = normal, no upper limit)") }, ["clipId", "speed"]),
- },
- {
- name: "set_color_grade",
- description:
- "Color-grade a video clip (any track; stills too). Fields patch the clip's current grade: only the ones you pass change, and every value 0 is neutral. reset:true clears the whole grade first; auto:true fits a starting grade from the clip's decoded frame (auto-tone: exposure to middle gray, contrast stretch, gray-world white balance — needs the clip's frame decoded, so seek into it first if this errors), and explicit fields then override it. Preview, timeline thumbnails, and export all render the same result.",
- inputSchema: obj({
- clipId: str("Video clip id"),
- brightness: num("-50..50, 0 neutral"),
- contrast: num("-50..50, 0 neutral"),
- saturation: num("-50..50 (-50 = grayscale), 0 neutral"),
- exposure: num("-50..50 (±1 stop), 0 neutral"),
- temperature: num("-50..50, positive = warmer, 0 neutral"),
- hue: num("Hue rotation in degrees, -180..180"),
- auto: bool("Fit a starting grade from the clip's current frame"),
- reset: bool("Clear the existing grade before applying fields"),
- }, ["clipId"]),
- },
- {
- name: "set_transition",
- description:
- "Set the transition from this clip into the next clip on its track (any video track), in seconds (0 clears it, max 2). A transition is a blend across the cut — it never moves, trims or overlaps clips, and it plays only while the pair touches. On upper tracks every style blends as an alpha dissolve (the tracks beneath show through). Only valid when a next same-track clip exists. Each edge holds one effect: setting a transition clears the animations adjacent to its joint (and set_animation on that edge replaces the transition). Read the transitions-and-fades skill before styling cuts.",
- inputSchema: obj({
- clipId: str("Video clip id (the clip the transition starts from)"),
- seconds: num("Transition length in seconds, 0–2 (0 = hard cut)"),
- style: {
- type: "string",
- enum: [...TRANSITION_STYLE_IDS],
- description: "Transition look (default crossfade)",
- },
- }, ["clipId", "seconds"]),
- },
- {
- name: "set_animation",
- description:
- "Animate one clip's own entrance (which:'in') or exit (which:'out'): fade, zoom, pop, or a slide named by its motion direction. style 'none' clears. Track-0 clips take every style; upper-track clips only fade and zoom. Each edge holds one effect, last pick wins: animating an edge a transition owns replaces that transition, and setting a transition clears the animations adjacent to its joint. At an abutting cut the animation plays over the neighbor's held frame; at the timeline's ends and across gaps it plays against black.",
- inputSchema: obj({
- clipId: str("Video clip id"),
- which: { type: "string", enum: ["in", "out"], description: "Entrance or exit" },
- style: {
- type: "string",
- enum: [...ANIM_STYLE_IDS, "none"],
- description: "Animation style, or 'none' to clear",
- },
- seconds: num("Ramp length in seconds, 0.1–2 (default 0.5)"),
- }, ["clipId", "which", "style"]),
- },
- {
- name: "merge_cue",
- description:
- "Merge a subtitle cue into the previous cue on its own track (joins their text and timing). Not valid for a track's first cue.",
- inputSchema: obj({ id: str("Cue id to merge into its predecessor") }, ["id"]),
- },
- {
- name: "set_aspect",
- description:
- "Set the project's output frame ratio as \"W:H\". The frame is the user's own setting, so call this only when they asked for a shape or a format (\"make it vertical\", \"for TikTok\", \"square\") — never inferred from a reference image's shape, a platform named in passing, or what would suit a generation you are about to run. Presets: 16:9 (YouTube), 9:16 (TikTok/Reels/Shorts), 1:1, 4:3, 3:4, 2:1 — but any ratio up to an 8:1 shape works, with whole or decimal sides (\"9:5\", \"2.39:1\" — stored reduced, so 2.39:1 becomes 239:100). The frame renders with its short side at 1080px: 9:16 → 1080×1920, 16:9 → 1920×1080, 9:5 → 1944×1080.",
- inputSchema: obj({ aspect: { type: "string", description: "Output ratio as \"W:H\", e.g. \"9:16\", \"1:1\", \"9:5\", \"2.39:1\"" } }, ["aspect"]),
- },
- {
- name: "set_project_fade",
- description:
- "Set the whole video's fade in from black and/or fade out to black, in seconds (0 clears, max 2). Applied to the final picture and mix at the start/end of the cut, independent of which clip sits there.",
- inputSchema: obj({ fadeIn: num("Fade-in seconds (omit to keep)"), fadeOut: num("Fade-out seconds (omit to keep)") }),
- },
- {
- name: "set_project_name",
- description: "Rename the current project.",
- inputSchema: obj({ name: str("New project name") }, ["name"]),
- },
- {
- name: "list_skills",
- description: "List the available skill documents about how this editor works.",
- inputSchema: obj({}),
- server: true,
- },
- {
- name: "read_skill",
- description:
- "Read a skill document (detailed docs for a part of the editor: every setting, where it lives, and how it behaves). Use before working in an unfamiliar area.",
- inputSchema: obj({ name: str("Skill name from list_skills") }, ["name"]),
- server: true,
- },
+ ...AI_PANEL_TOOLS,
+ ...PREVIEW_TOOLS,
+ ...TIMELINE_TOOLS,
+ ...INSPECTOR_TOOLS,
+ ...ELEMENTS_TOOLS,
+ ...EFFECTS_TOOLS,
+ ...OVERLAY_ANIMATION_TOOLS,
+ ...TRANSITIONS_TOOLS,
+ ...SUBTITLES_TOOLS,
+ ...AUDIO_TOOLS,
+ ...IMAGE_GEN_TOOLS,
+ ...VIDEO_GEN_TOOLS,
+ ...SCENE_TOOLS,
+ ...STOCK_TOOLS,
+ ...LIBRARY_TOOLS,
+ ...SIDE_PANEL_TOOLS,
+ ...TOP_BAR_TOOLS,
+ ...EDITOR_TOOLS,
];
/** Deep documentation the model can pull in on demand. */
@@ -908,7 +73,7 @@ Times are in seconds on the shared timeline. The playhead is currentTime; a skim
- A clip's timeline length is (out-in)/speed; total duration runs to the last clip's end, gaps included. Transitions blend at cuts and never change layout or length.
- trim_clip changes in/out inside the source media. in >= 0, out <= source duration, out-in >= 0.1.
- set_speed sets a clip's playback rate; it changes the clip's timeline length, and later titles/captions ripple to stay in sync.
-- set_transition joins a clip into the next one (0–2s, 17 styles); set_animation animates one clip's own entrance/exit — read the transitions-and-fades skill before styling cuts. Splitting or deleting clears the affected transition. Grading a clip's picture is add_effect: an effect dropped over a clip opens covering it.
+- set_transition joins a clip into the next one (0–2s, ${TRANSITION_STYLE_IDS.length} styles); set_animation animates one clip's own entrance/exit — read the transitions-and-fades skill before styling cuts. Splitting or deleting clears the affected transition. Grading a clip's picture is add_effect: an effect dropped over a clip opens covering it.
- split_at cuts the track-0 clip under that time into two clips at the exact frame. With a soundtrack or overlay clip selected it splits that instead.
- The user can multi-select (⌘/⇧-click) and delete several items at once; a hover chip on each video clip toggles its own audio.
- Anything can be hidden instead of deleted: it stays on the timeline (grayed) but drops out of playback and export — a hidden video clip plays black and silent. Per item: set_clip_hidden, or the \`hidden\` field on update_overlay_video/update_overlay/update_audio. Whole rows: set_track_hidden (video/soundtrack/text/subtitles) and set_track_muted (video) — the eye/speaker toggles on the track headers.
@@ -960,9 +125,9 @@ Titles (add_title / update_overlay): text, size (frame px; the design short side
Shapes (add_shape): rect, ellipse, line, arrow. w/h are frame fractions; rect/ellipse take fill + fill_opacity, corner radius (rect), and an outline (stroke_color/stroke_width); line/arrow draw in \`fill\` with h as their thickness, and rotation gives them their direction (0° points right, 90° points down).
Stickers (add_sticker): a project image asset (generated cutouts in the Elements panel carry origin "sticker"); w is a frame-width fraction and height follows the source's aspect. Lottie JSON uploads become animated stickers that loop for the element's duration.
Custom stickers (create_sticker): one call generates the image, removes the background, adds the die-cut outline, and places it — use it when the user asks for a sticker of something they don't have (it spends credits like image generation).
-Effects (add_effect): time-ranged treatments over the finished picture — the footage and everything laid over it. Two families, one list: the treatments (grain, vhs, glitch, blur, vignette, lightleak, flash, shake) and the graded looks (vintage, horror, halation, tech, noir, pastel, blockbuster, dreamy). Placed like any element and tuned with \`amount\` (update_overlay); an effect added over a clip opens covering that clip, and trims like anything else. One effect at a time reads best, amounts under ~0.5.
+Effects (add_effect): time-ranged treatments over the finished picture — the footage and everything laid over it. Two families, one list: the treatments (${EFFECT_TREATMENTS.join(", ")}) and the graded looks (${LOOK_EFFECTS.join(", ")}). Placed like any element and tuned with \`amount\` (update_overlay); an effect added over a clip opens covering that clip, and trims like anything else. One effect at a time reads best, amounts under ~0.5.
Advanced text (update_overlay): italic, align (multi-line), letter_spacing (em), line_height, stroke_color/stroke_width (em outline behind the fill), and richer shadows. Bundled Google families join the system set — inter, montserrat, poppins, oswald, space-grotesk, playfair, caveat, bebas, anton, archivo-black, bangers, lobster, pacifico, permanent-marker, dm-serif — and the user can upload .ttf/.otf fonts (they appear as font ids like "asset:<id>").
-Animation (set_overlay_animation): every element takes preset In/Out ramps (fade, pop, zoom, slides, typewriter for titles) and a Loop (pulse, wiggle, spin, float) that runs its whole duration. Entrances 0.3–0.6s read well; one loop per scene, sparingly.
+Animation (set_overlay_animation): every element takes preset In/Out ramps (${OVERLAY_ANIM_STYLE_IDS.join(", ")} — typewriter for titles only) and a Loop (${OVERLAY_LOOP_STYLE_IDS.join(", ")}) that runs its whole duration. Entrances 0.3–0.6s read well; one loop per scene, sparingly.
Keyframes (set_overlay_keyframes): for motion no preset covers — a path across the frame, a slow push in, a drift that holds first. A key is a whole pose (position, scale, rotation, opacity) at a time in seconds from the element's start; the pose moves linearly between keys and holds outside them, and presets still compose on top. Two or three keys carry almost everything; reach for a preset first and keyframes when the user describes a specific path or timing. In the UI the diamond in the inspector adds a key at the playhead, and dragging the element then records into it.
Behind the speaker: a title with behind_subject sits behind the person in the shot (on-device person segmentation; preview and export match). It reads well when the speaker is clearly separated from the background; with no detectable person it degrades to a normal front title. Groups: the user can group elements (select several, Group in the inspector) — selecting one selects all, and moves/resize/rotate/timing ride together.
Good TikTok titles: short punchy lines, high contrast (white/yellow + shadow or plate), size 72–110, keep inside the middle 80% of the frame (x 0.1..0.9, y 0.1..0.9), avoid the caption band (y≈0.8) when subtitles are on.