Skip to content

Open source CapCut alternative.

TypeScript

feat(cut): copy items from another timeline [rebuild]

David Le69bb730495d9
diff --git a/site/src/cut/components/AiPanel.tools.ts b/site/src/cut/components/AiPanel.tools.ts
index e6488525..5cc99723 100644
--- a/site/src/cut/components/AiPanel.tools.ts
+++ b/site/src/cut/components/AiPanel.tools.ts
@@ -8,7 +8,7 @@
*/
import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
-import { WATCH_DETAIL_NOTES, WATCH_DETAILS } from "@/cut/lib/types";
+import { TIMELINE_IDS, WATCH_DETAIL_NOTES, WATCH_DETAILS } from "@/cut/lib/types";
import { FACT_CLAIM_MAX_CHARS, FACT_CLAIMS_MAX } from "@/lib/inference/factCheck";
import { URL_BATCH_MAX } from "@/cut/lib/urlBatch";
import { FIND_MEDIA_DEFAULT, FIND_MEDIA_MAX, FIND_MEDIA_SNAPSHOT_CAP } from "@/cut/lib/findMedia";
@@ -20,6 +20,7 @@ export const AI_PANEL_TOOLS = [
"Read the full current editor state: clips, soundtrack, overlay elements (titles, shapes, stickers), subtitles, selection, playhead, view settings, publish metadata. Use this whenever the context snapshot is not enough or might be stale. A big cut read whole can run past what a result carries: name the sections you need (soundtrack, overlays, videoTrack, overlayVideo, transitions, subtitles, media, project, …) to check an edit.",
inputSchema: obj({
sections: { type: "array", items: { type: "string" }, description: "Top-level sections to return (the playhead always rides along); omit for everything" },
+ timeline: { type: "string", enum: [...TIMELINE_IDS], description: "Read another timeline's items (videoTrack, soundtrack, overlays, …) without opening it; omit for the open one" },
}),
},
{
diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.ts
index 2ed319dd..e7c9518a 100644
--- a/site/src/cut/components/Timeline.tools.ts
+++ b/site/src/cut/components/Timeline.tools.ts
@@ -88,7 +88,7 @@ export const TIMELINE_TOOLS = [
{
name: "add_clip",
description:
- "Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at `start`, inserted at `index`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at `start`, default the playhead). Asset ids come from `media` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut (\"add my beach photo\", \"stitch these into a movie\"); otherwise media stays on its card or panel for them to drag. Pass `blocks` instead of an asset to lay out a cut whose footage does not exist yet: one block per shot on track 0, each the length that shot runs, labelled with what belongs there. That is how a video you watched becomes a timeline — the cuts land at their real times, titles and sound go on top, and the person fills each block later. Blocks own no file and store nothing; footage dropped on one takes its place and its length, and replace_item does the same from here. Pass `spans` to cut one source into a run of clips — the moments of a talk worth keeping, with everything between them left out, or a video cut at its own shot boundaries when its picture is what plays.",
+ "Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at `start`, inserted at `index`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at `start`, default the playhead). Asset ids come from `media` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut (\"add my beach photo\", \"stitch these into a movie\"); otherwise media stays on its card or panel for them to drag. Pass `blocks` instead of an asset to lay out a cut whose footage does not exist yet: one block per shot on track 0, each the length that shot runs, labelled with what belongs there. That is how a video you watched becomes a timeline — the cuts land at their real times, titles and sound go on top, and the person fills each block later. Blocks own no file and store nothing; footage dropped on one takes its place and its length, and replace_item does the same from here. Pass `spans` to cut one source into a run of clips — the moments of a talk worth keeping, with everything between them left out, or a video cut at its own shot boundaries when its picture is what plays. add_clip lays down the whole file; an item another timeline already has (\"Main's music\", \"the same intro as Main\") is copied with copy_from_timeline, which keeps its trim and mix.",
inputSchema: obj({
asset_id: str("Project asset id from `media` in editor_state"),
blocks: {
@@ -397,6 +397,16 @@ export const TIMELINE_TOOLS = [
timeline: { type: "string", enum: [...TIMELINE_IDS], description: "The timeline to open" },
}, ["timeline"]),
},
+ {
+ name: "copy_from_timeline",
+ description:
+ "Copy items another timeline holds onto the open one, exactly as that timeline has them: a soundtrack clip keeps its in/out, volume, fades, speed and sound treatment; clips, elements, captions and transitions keep every property, and several items keep their spacing. This is how \"Main's music\" or \"the same title as on Main\" lands on this timeline. A parked timeline's soundtrack is in editor_state project.timelines; get_state with `timeline` lists all of its items. The source timeline is left as it was.",
+ inputSchema: obj({
+ timeline: { type: "string", enum: [...TIMELINE_IDS], description: "The timeline to copy from" },
+ ids: { type: "array", items: { type: "string" }, description: "Ids of the items on that timeline" },
+ at: num("Where the earliest copy starts on the open timeline, seconds; omit to keep the source's times"),
+ }, ["timeline", "ids"]),
+ },
{
name: "set_view",
description:
diff --git a/site/src/cut/lib/aiContext.ts b/site/src/cut/lib/aiContext.ts
index 5ac5ceaa..6cb1a9cb 100644
--- a/site/src/cut/lib/aiContext.ts
+++ b/site/src/cut/lib/aiContext.ts
@@ -1,7 +1,7 @@
"use client";
import { formatBytes } from "@/lib/bytes";
-import { isLinkedAssetType, isTimelineId, sanitizeTimelines, timelineAspect, TIMELINE_IDS, TIMELINE_LABELS, type TimelineBody, type TimelineId } from "./types";
+import { isLinkedAssetType, isTimelineId, parkedTimeline, sanitizeTimelines, timelineAspect, TIMELINE_IDS, TIMELINE_LABELS, type TimelineBody, type TimelineId } from "./types";
import { FIND_MEDIA_SNAPSHOT_CAP } from "./findMedia";
import { GUIDE_PRESETS, guideFits, guideGeometry, safeAreaOf, sanitizeGuideLines, sanitizeGuides, type GuideId, type GuideLines } from "./guides";
import { hasOverlayAnim, retimeOf, speedCurveOf, WHEEL_LABELS, WHEEL_ZONES, type ClipSound, type OutputSpace, type SpeedNode } from "@donkeycut/effects-kit";
@@ -355,6 +355,27 @@ export function buildAiContext(opts?: { fullCues?: boolean; chatId?: string | nu
};
}
+/** A parked timeline's items, described the way the open one's are: what
+ * get_state reads with `timeline`, so the chat can copy an item another
+ * timeline holds without opening it. */
+export function describeParkedTimeline(id: TimelineId) {
+ const s = useEditor.getState();
+ const body = parkedTimeline(s, id);
+ const core = describeState({ ...s, ...body, timeline: id }, { fullCues: true, chatId: chatOwner(), live: true });
+ return {
+ timeline: id,
+ aspect: body.aspect,
+ duration: roundT(projectDuration(body)),
+ videoTrack: core.videoTrack,
+ overlayVideo: core.overlayVideo,
+ transitions: core.transitions,
+ soundtrack: core.soundtrack,
+ overlays: core.overlays,
+ ...("groupCameras" in core ? { groupCameras: core.groupCameras } : {}),
+ subtitles: core.subtitles,
+ };
+}
+
/** A stored document described the way the open project is: the same
* shape the model reads in <editor_state>, built from the document alone.
* This is how another project is read as a reference — its edit, its
@@ -489,6 +510,11 @@ function describeState(
duration: roundT(body ? projectDuration(body) : 0),
clips: body?.clips.length ?? 0,
aspect: timelineAspect(s, id),
+ // A parked timeline's sound as it is mixed there, so "Main's music"
+ // is that clip with its trim and levels (copy_from_timeline).
+ ...(id !== s.timeline && body?.audioClips.length
+ ? { soundtrack: body.audioClips.map((a) => ({ id: a.id, ...describeAudio(a, assetById) })) }
+ : {}),
};
}),
aspect: s.aspect,
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index 4b863d46..46f4d637 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -16,7 +16,7 @@ import { submitPreviewSnapshot } from "./exportClient";
import { ITEM_KIND_IDS, ITEM_KINDS, ROW_ITEM_KINDS, type ItemKind } from "./itemKinds";
import { insertItemRow, packRows } from "./laneTracks";
-import { timelineItem } from "./timelineItems";
+import { timelineCopies, timelineItem } from "./timelineItems";
import { movePreviewSelection, previewSelectionSnapshot } from "@/cut/lib/previewSelection";
import { libraryShareTargetSchema, shareSettingsSchema, librarySharePath } from "@/cut/lib/librarySharing";
@@ -239,7 +239,7 @@ import { measureLoudnessSliced, roundedLoudness, type LoudnessMeasure } from "./
import { framesAt, readMediaFileSize } from "./mediaRead";
import { renderStageFrame, storeStageStill } from "./stageFrame";
import { createRasterCanvas, decodeRasterImageUrl, rasterCanvasToDataUrl } from "./raster";
-import { buildAiContext, describeCameraCard, describeDoc, hiddenFromChat, OTHER_CHATS_MEDIA_NOTE, roundT } from "./aiContext";
+import { buildAiContext, describeCameraCard, describeDoc, describeParkedTimeline, hiddenFromChat, OTHER_CHATS_MEDIA_NOTE, roundT } from "./aiContext";
import { CARD_SIDES, newCard, normalizeCard, type CardSide } from "./cameraCard";
import { CAPTION_STYLES, captionEmphasis, cueSpot, EMPHASIS_SCALE_MAX, EMPHASIS_SCALE_MIN, laneCues, subtitleLaneCount, trackLocale } from "./subtitles";
import { cueEmphasis, emphasisIndicesOf, emphasizedWords } from "./captionEmphasis";
@@ -324,7 +324,9 @@ import {
SIDE_PANEL_TABS,
SPEED_FLOOR,
isTimelineId,
+ parkedTimeline,
TIMELINE_IDS,
+ TIMELINE_LABELS,
TRANSITION_STYLE_IDS,
type AnimStyle,
type AssetWatch,
@@ -1178,10 +1180,17 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
return { id: next.id, which, style: anim?.style ?? "none", seconds: anim?.seconds ?? 0 };
},
- get_state: (_s, input) => {
+ get_state: (s, input) => {
+ if (input.timeline !== undefined && !isTimelineId(input.timeline))
+ throw new ToolError(`timeline must be one of ${TIMELINE_IDS.join(", ")}.`);
// The tool result carries the whole transcript; the per-message context
// snapshot trims it, so this is how the model pulls every cue when needed.
- const full = buildAiContext({ fullCues: true }) as Record<string, unknown>;
+ // A parked timeline reads as its items alone.
+ const full = (
+ isTimelineId(input.timeline) && input.timeline !== s.timeline
+ ? describeParkedTimeline(input.timeline)
+ : buildAiContext({ fullCues: true })
+ ) as Record<string, unknown>;
const want: unknown[] = Array.isArray(input.sections) ? input.sections : [];
if (want.length === 0) return full;
// A big cut read whole runs past what a tool result can carry; a read
@@ -1190,7 +1199,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
const bad = want.filter((k) => typeof k !== "string" || !names.includes(k));
if (bad.length > 0) throw new ToolError(`Unknown section ${bad.join(", ")}; sections are ${names.join(", ")}.`);
return Object.fromEntries(
- names.filter((k) => k === "playhead" || want.includes(k)).map((k) => [k, full[k]])
+ names.filter((k) => k === "playhead" || k === "timeline" || want.includes(k)).map((k) => [k, full[k]])
);
},
@@ -5263,6 +5272,34 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
return useEditor.getState().notes;
},
+ copy_from_timeline: (s, input) => {
+ if (!isTimelineId(input.timeline))
+ throw new ToolError(`timeline must be one of ${TIMELINE_IDS.join(", ")}.`);
+ const ids = Array.isArray(input.ids) ? input.ids.filter((id): id is string => typeof id === "string") : [];
+ if (ids.length === 0) throw new ToolError("Name the items to copy in ids.");
+ const source = input.timeline === s.timeline ? s : parkedTimeline(s, input.timeline);
+ // Each id is found in whichever item list holds it on the source.
+ const found = ids.map((id) => {
+ const kind = ITEM_KIND_IDS.find((k) => timelineItem(source, { kind: k, id } as NonNullable<Selection>));
+ return kind ? ({ kind, id } as NonNullable<Selection>) : null;
+ });
+ const missing = ids.filter((_, i) => !found[i]);
+ if (missing.length > 0)
+ throw new ToolError(`${TIMELINE_LABELS[input.timeline]} has no item ${missing.join(", ")}; get_state with timeline "${input.timeline}" lists its items.`);
+ const items = timelineCopies(source, found);
+ const at = isNum(input.at) ? Math.max(0, input.at) : Math.min(...items.map((cb) => cb.item.start));
+ const landed = s.pasteCopies(items, at);
+ if (!landed) throw new ToolError("Nothing could be copied onto this timeline.");
+ const now = useEditor.getState();
+ return {
+ timeline: now.timeline,
+ copied: landed.map((sel) => {
+ const item = timelineItem(now, sel)!;
+ return { kind: sel.kind, id: sel.id, start: roundT(item.start) };
+ }),
+ };
+ },
+
switch_timeline: async (_s, input) => {
if (!isTimelineId(input.timeline))
throw new ToolError(`timeline must be one of ${TIMELINE_IDS.join(", ")}.`);
diff --git a/site/src/cut/lib/store.ts b/site/src/cut/lib/store.ts
index 4a23253a..f6588521 100644
--- a/site/src/cut/lib/store.ts
+++ b/site/src/cut/lib/store.ts
@@ -1027,6 +1027,10 @@ export interface EditorState {
* the playhead otherwise) — sliding past anything already on the target
* lane — and select the pasted item(s). */
paste: (at?: number) => boolean;
+ /** Paste `items` at `at` the way `paste` lands the clipboard, and answer
+ * what landed, or null when nothing could. Copies from another timeline
+ * come in this way, leaving the clipboard to the person. */
+ pasteCopies: (items: TimelineClipboardItem[], at: number) => NonNullable<Selection>[] | null;
}
// Per-project undo/redo stacks; both reset when a project loads. Capped so a
@@ -5287,15 +5291,17 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {
clipboard = [];
},
- paste: (at = previewAt()) => {
+ paste: (at = previewAt()) => get().pasteCopies(clipboard, at) !== null,
+
+ pasteCopies: (items, at) => {
const s = get();
- if (s.readOnly || !clipboard.length || !Number.isFinite(at)) return false;
- if (clipboard.some((cb) => clipboardItemAssetIds(cb).some((id) => !s.assets.some((a) => a.id === id)))) return false;
+ if (s.readOnly || !items.length || !Number.isFinite(at)) return null;
+ if (items.some((cb) => clipboardItemAssetIds(cb).some((id) => !s.assets.some((a) => a.id === id)))) return null;
push();
- const { next, selection } = pasteTimelineItems(s, clipboard, Math.max(0, at), uid);
+ const { next, selection } = pasteTimelineItems(s, items, Math.max(0, at), uid);
set({ ...next, subtitles: { ...s.subtitles, cues: next.subtitles.cues },
selection: selection[selection.length - 1] ?? null, multiSelection: selection, selectedKey: null });
- return true;
+ return selection;
},
undo: () => {
diff --git a/site/src/cut/lib/timelineCopyTool.test.ts b/site/src/cut/lib/timelineCopyTool.test.ts
new file mode 100644
index 00000000..7a2a6f0c
--- /dev/null
+++ b/site/src/cut/lib/timelineCopyTool.test.ts
@@ -0,0 +1,58 @@
+import { beforeEach, expect, test } from "bun:test";
+import { buildAiContext } from "./aiContext";
+import { runAiTool } from "./aiTools";
+import { useEditor } from "./store";
+import type { AudioClip, MediaAsset } from "./types";
+
+// Asked to put Main's music on Second, the chat laid the music file down
+// whole: Second had no view of Main's soundtrack clip, and nothing it could
+// call copied one. Main's clip is trimmed and mixed, and the copy is that clip.
+
+const music: MediaAsset = { id: "music", fileName: "music.m4a", name: "Music", type: "audio", duration: 23.47, url: "" };
+const bed: AudioClip = { id: "bed", assetId: "music", start: 0, in: 0, out: 23.4, volume: 0.4, fadeIn: 0.5, fadeOut: 2 };
+
+beforeEach(async () => {
+ await useEditor.getState().openProjectDoc("copy", { name: "Copy" }, [music]);
+ useEditor.setState({ assets: [music], audioClips: [bed] });
+ useEditor.getState().switchTimeline("second", { ownAspect: true });
+});
+
+test("the snapshot carries a parked timeline's soundtrack with its trims", () => {
+ const { project } = buildAiContext() as { project: { timelines: { id: string; soundtrack?: unknown[] }[] } };
+ const main = project.timelines.find((t) => t.id === "main")!;
+ expect(main.soundtrack).toHaveLength(1);
+ expect(main.soundtrack![0]).toMatchObject({ id: "bed", asset: "Music", start: 0, in: 0, out: 23.4, volume: 0.4, fadeIn: 0.5, fadeOut: 2 });
+});
+
+test("get_state reads a parked timeline's items", async () => {
+ const read = (await runAiTool("get_state", { timeline: "main", sections: ["soundtrack"] })) as {
+ timeline: string;
+ soundtrack: { id: string; out: number }[];
+ };
+ expect(read.timeline).toBe("main");
+ expect(read.soundtrack.map((a) => [a.id, a.out])).toEqual([["bed", 23.4]]);
+});
+
+test("copy_from_timeline lays Main's soundtrack clip on Second as Main has it", async () => {
+ await runAiTool("copy_from_timeline", { timeline: "main", ids: ["bed"], at: 1.5 });
+ const s = useEditor.getState();
+ expect(s.audioClips).toHaveLength(1);
+ const [copy] = s.audioClips;
+ expect(copy).toMatchObject({ assetId: "music", start: 1.5, in: 0, out: 23.4, volume: 0.4, fadeIn: 0.5, fadeOut: 2 });
+ expect(copy.id).not.toBe("bed");
+ // Main keeps its own clip.
+ expect(s.timelines.main?.audioClips.map((a) => a.id)).toEqual(["bed"]);
+});
+
+test("copy_from_timeline keeps the source's timing when no time is given", async () => {
+ useEditor.getState().switchTimeline("main", { ownAspect: true });
+ useEditor.getState().updateAudio("bed", { start: 3 });
+ useEditor.getState().switchTimeline("second", { ownAspect: true });
+ await runAiTool("copy_from_timeline", { timeline: "main", ids: ["bed"] });
+ expect(useEditor.getState().audioClips[0].start).toBe(3);
+});
+
+test("copy_from_timeline names the ids it could not find", async () => {
+ const error = await runAiTool("copy_from_timeline", { timeline: "main", ids: ["nope"] }).catch((e: unknown) => e);
+ expect(error instanceof Error && error.message.includes("nope")).toBe(true);
+});
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index c2d8e4c4..1d63635a 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -245,7 +245,7 @@ Cut is a local, project-based short-video editor. Each timeline has its own outp
- Center: the video preview canvas (composited at the project's frame size) with draggable text overlays and subtitle captions.
- Right: the Inspector — its content follows the selection (video clip, overlay video, soundtrack clip, title, or cue).
- Bottom: the timeline (resizable by dragging its top border). Rows top-to-bottom: the video tracks in z-order (positive tracks, then track 0, then negative tracks behind it), soundtrack lanes (green), titles (purple), subtitle tracks (amber, when enabled). Every track is free-positioned in time.
-- A project has ${TIMELINE_IDS.length} timelines (${TIMELINE_IDS.map((id) => TIMELINE_LABELS[id]).join(", ")}), picked from the menu at the left of the timeline toolbar. Each holds its own clips, soundtrack, transitions, elements, subtitles and aspect ratio over the shared media. editor_state project.timeline names the open one; the preview, the export and every edit work on it, and switch_timeline opens another.
+- A project has ${TIMELINE_IDS.length} timelines (${TIMELINE_IDS.map((id) => TIMELINE_LABELS[id]).join(", ")}), picked from the menu at the left of the timeline toolbar. Each holds its own clips, soundtrack, transitions, elements, subtitles and aspect ratio over the shared media. editor_state project.timeline names the open one; the preview, the export and every edit work on it, switch_timeline opens another, and copy_from_timeline brings over an item another one holds, trimmed and mixed as it is there.
Everything autosaves to the project folder. Undo/redo is unlimited (⌘Z / ⇧⌘Z).
Times are in seconds on the shared timeline. The playhead is currentTime; a skimmer previews under the mouse without moving the playhead.`,