Open source CapCut alternative.
TypeScript
feat(site): a cut can be blocked out before its footage exists [rebuild]
David Le87128023e26c
diff --git a/docs/guides/cut/ai-assistant.md b/docs/guides/cut/ai-assistant.mdindex 9951947b..ce8eb4c2 100644--- a/docs/guides/cut/ai-assistant.md+++ b/docs/guides/cut/ai-assistant.md@@ -94,6 +94,7 @@ With a tab attached, tool calls execute in it against the same store the user is- **Frames become images.** A tool result's `image`/`images` data URLs leave the JSON and reach the model as real images on every path — the engine bridge emits MCP image blocks after the data text, and the Gemini loop rides the first on the function response and the rest as image parts. Watching happens in whatever model the user is chatting with; there is no side model.- **Watching leaves a map.** A watch records which moments and cut candidates a source holds, persisted on the asset so it survives the thread, and the first watch of a source starts a quiet background sweep that maps the rest. The map aims later watches; only sheets returned in the conversation count as footage the model has seen.- **Watching leaves a record.** Contact sheets live in one conversation and the media trim drops the oldest of them, so the model writes each look down: a note-source call stores what a span showed — quoted on-screen text, what happens, where the cuts land — against its source seconds, on the asset, saved with the project. Notes come back in the editor snapshot as `observed`, in full from get-state, and on every watch result beside the spans still undescribed. A close note supersedes the coarse one over the same ground, and a watch of a source that still owes a note is refused, which is what makes an hour-long source workable: it is surveyed coarsely, described, then re-watched densely only where the work is.+- **A cut can be blocked out before its footage exists.** A block is a shot with nothing in it yet: a source that owns no file and draws itself, holding its slot on track 0 for the length that shot runs. Laying a run of them is the ordinary clip placement, so "make one like this" becomes a timeline carrying the reference's own cut times, with titles and sound built on top. Footage dropped on a block takes its place and its length; from chat that is the same item replace any clip uses.- **Speech cuts land on the words.** A recut edge is not left where the model's arithmetic put it. The audio around it is re-read, that recording's own room tone and voice level are measured, and the edge is placed a beat clear of where a word truly starts or finishes — tail and soft attack included, so a quiet take and a noisy one are cut the same way. Pace belongs to that step: fast, natural and relaxed set how big the beat is, and the fastest of them still lets every word finish. So "make it punchy" shortens the air rather than cutting into the speech, and where a pause is too short for both sides' air the two edges meet inside it instead of one clipping a word.- **Media rides a byte budget.** Contact sheets, audio, and attachment payloads all replay inline, so before each call the session is trimmed to a 16MB media budget — oldest parts leave first, with a note naming the tools that fetch them again. Text, tool calls, tool results, and thought signatures always survive, and the trim runs on a copy so the stored session keeps everything.- **A Donkey Cut link is a document.** A project or share link pasted into chat is read as its document, and its edit is rebuilt in the open project on the user's own footage through the Library's template rail, whole or item by item, with the media the project lacks copied across; the reference's document is the recipe, so nothing is reconstructed from pixels, and a project held only in another browser's storage is readable only from that browser.@@ -126,6 +127,7 @@ A thread saves one session slot per provider, so switching models mid-thread kee| Inline media per model call | 16MB, trimmed oldest-first || Browser tool execution | 120s || One watch-video call | 600s of source, 4 sheets (36 frames), 150 decoded candidates, a 90s decode budget; the result says where coverage stopped |+| One blockout call | 40 shots, each 0.2-600s || Snapshot caps | 60 media items, 60 cues || One queued turn job | 1MB of messages, one live turn per project || Saved threads per project | 30 |diff --git a/site/scripts/lib/cut-eval/cases.ts b/site/scripts/lib/cut-eval/cases.tsindex 3fc54bc0..4559a42e 100644--- a/site/scripts/lib/cut-eval/cases.ts+++ b/site/scripts/lib/cut-eval/cases.ts@@ -36,6 +36,8 @@ import {assistantTurn,plainUserTurn,userTurn,+ REFERENCE_ASSET,+ REFERENCE_STATE,} from "./fixtures";export type Bucket = "instant" | "chat" | "single-tool" | "multi-tool";@@ -69,6 +71,10 @@ export interface EvalCase {limits?: { roundBudget?: number; maxExtensions?: number };/** Auto-continue extensions the turn must land in. */extensions?: { min?: number; max?: number };+ /** How often the quality gate must send this turn back to work: a turn that+ * half-watches a source and describes the whole of it has a floor, and an+ * ordinary finished turn has a ceiling of 0. */+ qualityGate?: { min?: number; max?: number };/** Debris lines the turn ledger reports (the live editor computes these from* the store; the eval injects them). */debris?: string[];@@ -814,6 +820,106 @@ export function cases(audio: { dataBase64: string; mimeType: string }): EvalCasereturn undefined;},},+ {+ // The live case: someone imports a 71.6s reference and asks for one like+ // it. The turn that shipped watched 20s of it, built titles over an+ // EMPTY track 0, and signed off — so the person had text floating over+ // nothing. Seeing all of it and blocking its cuts out as placeholders is+ // the job; the quality gate is what stops the turn closing short.+ name: "replicate-a-video-blocks-out-the-cuts",+ bucket: "multi-tool",+ input: () => [+ userTurn("remake this video for me — I'll put my own footage in after", {+ state: REFERENCE_STATE,+ }),+ ],+ reply: /shot|block|placeholder|drop|footage/i,+ requiredTools: ["watch_video", "note_source", "add_clip"],+ state: REFERENCE_STATE,+ simulate: () => {+ const duration = REFERENCE_ASSET.duration;+ // Each pass covers the next stretch from where the caller asked, so+ // the coverage the model reads is real and moves.+ let coveredTo = 0;+ let blockedOut = 0;+ let blockedEarly = false;+ let placedReference = false;+ const sim = (name: string, args: Record<string, unknown>) => {+ if (name === "watch_video") {+ const from = typeof args.from === "number" ? args.from : 0;+ const to = Math.min(duration, Math.max(from, coveredTo) + 24);+ coveredTo = Math.max(coveredTo, to);+ const cuts: number[] = [];+ for (let t = Math.ceil(from / 2) * 2; t < to; t += 2) cuts.push(Number(t.toFixed(2)));+ return {+ images: [],+ sheetFrames: [],+ distinctFrames: cuts.length,+ sceneChanges: cuts,+ coveredTo: to,+ truncated: to < duration,+ unwatchedSeconds: Number(Math.max(0, duration - coveredTo).toFixed(2)),+ recorded: [],+ unnoted: coveredTo < duration ? [{ from: coveredTo, to: duration }] : [],+ source: { assetId: REFERENCE_ASSET.id, name: REFERENCE_ASSET.name, duration },+ note: "Eval: yellow Archivo-Black hook titles over a talking head, cuts every ~2s.",+ };+ }+ if (name === "note_source") return { source: { assetId: REFERENCE_ASSET.id }, unnoted: [] };+ if (name === "add_clip") {+ // Placing the reference itself is the mistake this case watches+ // for; a run of blocks is the shape it wants.+ if (args.asset_id === REFERENCE_ASSET.id) {+ placedReference = true;+ throw new Error(+ "the reference is not footage for this cut — its shots are blocks until the person supplies their own"+ );+ }+ const shots = Array.isArray(args.blocks) ? args.blocks : [];+ if (shots.length === 0) return { id: "c-new" };+ if (coveredTo < duration - 0.5) blockedEarly = true;+ blockedOut += shots.reduce(+ (sum: number, shot: unknown) =>+ sum + (typeof (shot as { seconds?: unknown })?.seconds === "number" ? (shot as { seconds: number }).seconds : 0),+ 0+ );+ return {+ placed: shots.map((_: unknown, i: number) => ({+ clipId: `ph${i + 1}`,+ assetId: `a-ph${i + 1}`,+ })),+ note: "Eval: the blocks are on track 0.",+ };+ }+ return undefined;+ };+ sim.verify = () => {+ const out: string[] = [];+ if (coveredTo < duration - 0.5)+ out.push(`watched only ${coveredTo.toFixed(1)}s of a ${duration}s source`);+ if (blockedEarly) out.push("blocked the cut out before the source was watched through");+ if (blockedOut > 0 && blockedOut < duration * 0.6)+ out.push(`blocked out ${blockedOut.toFixed(1)}s of a ${duration}s reference`);+ // The same shots laid twice make a cut twice as long as the thing+ // it copies — what a turn sent back to look does if it starts over.+ if (blockedOut > duration * 1.25)+ out.push(`blocked out ${blockedOut.toFixed(1)}s over a ${duration}s reference`);+ if (placedReference) out.push("put the reference itself on the timeline");+ return out;+ };+ return sim;+ },+ // Rebuilding the reference's graphics is the job, so the element tools+ // serve; only the shape of the rebuild is under test.+ stubs: {+ add_title: { id: "o1" },+ add_text_sequence: { made: [{ id: "o2" }] },+ add_shape: { id: "o3" },+ set_overlay_animation: { id: "o1" },+ update_overlay: { id: "o1" },+ set_background: { background: { kind: "color", color: "#000000" } },+ },+ },{// Scope: "the grey ones" are the parked bars alone. The attached pair// keeps playing; clearing one of them is the over-rotation regression.diff --git a/site/scripts/lib/cut-eval/fixtures.ts b/site/scripts/lib/cut-eval/fixtures.tsindex 3783afe4..4efdbbd2 100644--- a/site/scripts/lib/cut-eval/fixtures.ts+++ b/site/scripts/lib/cut-eval/fixtures.ts@@ -276,6 +276,26 @@ export const TWEET_ASSET = {origin: "chat",};+/** A reference someone brought into an empty project and asked for "one like+ * this": one long imported video, nothing on the timeline yet. The live case+ * this mirrors ended with 25 titles floating over an empty track 0 and+ * two-thirds of the source never watched. */+export const REFERENCE_ASSET = {+ id: "a-ref1",+ name: "4.6 Years Until I Never Work Again",+ type: "video",+ duration: 71.6,+ origin: "chat",+};+export const REFERENCE_STATE = {+ ...EDITOR_STATE,+ project: { ...EDITOR_STATE.project, name: "Marina", duration: 0 },+ media: [REFERENCE_ASSET],+ videoTrack: [],+ soundtrack: [],+ overlays: [],+};+export const TWEET_STATE = {...EDITOR_STATE,media: [...EDITOR_STATE.media, TWEET_ASSET],diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.tsindex f00744e5..fc78f096 100644--- a/site/src/cut/components/Timeline.tools.ts+++ b/site/src/cut/components/Timeline.tools.ts@@ -65,13 +65,25 @@ export const TIMELINE_TOOLS = [{name: "add_clip",description:- "Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at `start`, inserted at `index`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at `start`, default the playhead). Asset ids come from `media` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut (\"add my beach photo\", \"stitch these into a movie\"); otherwise media stays on its card or panel for them to drag.",+ "Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at `start`, inserted at `index`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at `start`, default the playhead). Asset ids come from `media` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut (\"add my beach photo\", \"stitch these into a movie\"); otherwise media stays on its card or panel for them to drag. Pass `blocks` instead of an asset to lay out a cut whose footage does not exist yet: one block per shot on track 0, each the length that shot runs, labelled with what belongs there. That is how a video you watched becomes a timeline — the cuts land at their real times, titles and sound go on top, and the person fills each block later. Blocks own no file and store nothing; footage dropped on one takes its place and its length, and replace_item does the same from here. `reference_asset_id` names the source the shots were copied from: what is being copied does not play in the copy, so the tools stop placing it once it is named.",inputSchema: obj({asset_id: str("Project asset id from `media` in editor_state"),+ blocks: {+ type: "array",+ description: "Shots with no footage yet, in order — one block each on track 0",+ items: obj(+ {+ seconds: num("How long this shot runs on the timeline"),+ label: str("What belongs in this shot — shown on the block and on its timeline chip"),+ },+ ["seconds"]+ ),+ },+ reference_asset_id: str("Blocks only: the source these shots were copied from"),lane: { type: "integer", minimum: 0, description: "Audio assets only: soundtrack lane, default 0. Put music on lane 1 to overlap narration on lane 0." },start: num("Timeline start s"),index: num("Insert position on video track 0 (video/image only; 0 = first)"),- }, ["asset_id"]),+ }),},{name: "trim_clip",@@ -298,6 +310,18 @@ export const TIMELINE_TOOLS = [background: bool("Set the project background to the look's frame color (default true)"),}),},+ {+ name: "replace_item",+ description:+ "Replace what an item plays, keeping the item: a video clip (including a blocked-out shot) keeps its place and its length and plays the asset you name instead, trimmed from that source's head; a soundtrack clip keeps its slot; a sticker shows the new image. This is how a block becomes the person's own footage (\"put the kitchen clip in shot 3\"), and how any clip swaps its source without moving what comes after. A block left playing nothing goes with it. The source a cut was copied from is not a fill for its own blocks.",+ inputSchema: obj(+ {+ id: str("The item to refill — a video clip, a soundtrack clip, or a sticker"),+ asset_id: str("Project asset that plays there instead"),+ },+ ["id", "asset_id"]+ ),+ },{name: "freeze_frame",description:diff --git a/site/src/cut/components/Timeline.tsx b/site/src/cut/components/Timeline.tsxindex 9a717efc..72078e4b 100644--- a/site/src/cut/components/Timeline.tsx+++ b/site/src/cut/components/Timeline.tsx@@ -55,7 +55,7 @@ import { downloadMedia, dropEdgeFrames, ensurePeaks, importImage, importStockAudimport { planFilmstrip, type FilmTile } from "@/cut/lib/filmstrip";import { waveGain } from "@/cut/lib/waveform";import { canSplitItem } from "@/cut/lib/itemKinds";-import { track0Clips, laneGapAt, sameLane, type LaneRef, clipLen, clipSpeed, getClipSpans, maxClipFade, overlayLaneOrder, overlayLayers, projectDuration, resolveTransitions, rippleInsert, useEditor } from "@/cut/lib/store";+import { track0Clips, laneGapAt, blockClipAt, sameLane, type LaneRef, clipLen, clipSpeed, getClipSpans, maxClipFade, overlayLaneOrder, overlayLayers, projectDuration, resolveTransitions, rippleInsert, useEditor } from "@/cut/lib/store";import type { VideoTrackPlacement } from "@/cut/lib/store";import { playheadAt, setSkim, skimAt, subscribePlayhead, usePlayhead, useSkim } from "@/cut/lib/playhead";import { useBrushUi } from "@/cut/lib/removal/brushUi";@@ -63,6 +63,7 @@ import { useMatteBakes } from "@/cut/lib/removal/bakeJobs";import { laneHidden, subtitleLaneCount } from "@/cut/lib/subtitles";import { formatTime, formatTimecode } from "@/cut/lib/time";import { EFFECT_LABELS, hasSpeedCurve, headSrc, retimeOf, SPEED_CURVE_MAX, SPEED_CURVE_MIN, tailSrc, type EffectId, type Retime, type SpeedNode } from "@donkeycut/effects-kit";+import { BLOCK_COLOR } from "@/cut/lib/blockSource";import { assetIsSilent, emptySubtitles, fontStack, IMAGE_CLIP_SECONDS, isAudioTransition, clipName, isTextOverlay, overlayName, SHAPE_LABELS, TRANSITION_STYLE_LABELS, transitionBarStart, transitionDefaultSeconds, XBAR_MAGNET_PX, type ShapeKind } from "@/cut/lib/types";import type { AudioClip, ClipSpan, ColorGrade, MediaAsset, Overlay, Selection, StickerOverlay, SubtitleCue, TimelineTransition, TransitionBoundaryKind, TransitionStyle, VideoClip } from "@/cut/lib/types";import { isLottieAsset } from "@/cut/lib/lottieAssets";@@ -810,13 +811,27 @@ export function Timeline() {// takes the drop as an insert at the pointer — its later clips ripple right —// so the slot previews the ripple landing the drop will actually take.const previewCross = useCallback(- (target: TrackTarget | null, start = 0, len = 0, ghost?: DropGhost) => {+ (target: TrackTarget | null, start = 0, len = 0, ghost?: DropGhost, carried = 0) => {// The drag let go of the placement system (it is home, or it ended): the// next resolve starts from the geometry alone.if (target === null) {heldRow.current = null;return setOverlayDrop(null);}+ // One piece of media over a block fills it, so the ghost takes the+ // block's own span.+ if (carried === 1 && target.kind === "track" && target.track === 0) {+ const cur = useEditor.getState();+ const block = blockClipAt(cur.clips, cur.assets, Math.max(0, start));+ if (block)+ return setOverlayDrop({+ target,+ t: block.start,+ len: clipLen(block),+ shifts: [],+ ghost,+ });+ }const landing =target.kind === "track"? rippleInsert(@@ -1681,6 +1696,9 @@ export function Timeline() {only?: (asset: MediaAsset) => boolean) => {const next = runCursor(t, atElement);+ // One clip released on a block fills it: the footage takes the block's+ // place and its length, and the block goes with it.+ const block = blockUnder(items, t, place);for (const item of items) {try {const asset = await item.land(projectId);@@ -1690,6 +1708,10 @@ export function Timeline() {if (asset.type === "font") continue;if (only && !only(asset)) continue;const sticker = !!stickerOf(asset);+ if (block && !sticker && isClipMedia(asset.type)) {+ useEditor.getState().replaceClipAsset(block.id, asset.id);+ continue;+ }placeAssetAt(asset.id, asset.type, next(asset, sticker), audioRow, place, elementRow);} catch (err) {reportSwallowed(`[cut] drop failed for ${item.name}`, err);@@ -1701,6 +1723,18 @@ export function Timeline() {* mixed selection may also be carrying. */const clipsOnly = (asset: MediaAsset) => isClipMedia(asset.type) && !stickerOf(asset);+ /** The block a single carried clip would fill at this point.+ * A run of several clips inserts as a run — one block holds one shot. */+ const blockUnder = (items: { kind?: string }[], t: number, place: TrackTarget) => {+ // Only the clips count: one video dragged alongside its music is still one+ // shot, and both drop handlers have to agree on that or the same gesture+ // fills the block from one row and inserts beside it from another.+ if (items.filter((i) => isClipMedia(i.kind)).length !== 1 || place.kind !== "track" || place.track !== 0)+ return null;+ const cur = useEditor.getState();+ return blockClipAt(cur.clips, cur.assets, t);+ };+// The video the drag carries — project media, a library clip, a stock clip// or an image (which lands as a still) — with the ghost its landing// segment paints. A sound or a sticker is no video.@@ -1723,10 +1757,13 @@ export function Timeline() {e.dataTransfer.dropEffect = "copy";const x = e.clientX;const y = e.clientY;+ // Read off the event now: the preview runs a frame later, with the+ // event long gone.+ const clips = carriedMedia(e).filter((item) => isClipMedia(item.kind)).length;schedulePreview(() => {setAssetDrop(null);setDropType("video"); // keep the insertion zones lit however the drag entered- previewCross(resolveDropTrack(y), Math.max(0, timeAt(x)), vid.duration, vid.ghost);+ previewCross(resolveDropTrack(y), Math.max(0, timeAt(x)), vid.duration, vid.ghost, clips);});},onDragLeave: (e: React.DragEvent) => {@@ -1987,10 +2024,17 @@ export function Timeline() {return;}setOverlayDrop(null);+ // Over a block the drop fills it, so the ghost takes the block's+ // own span — its start, its length, nothing rippling.+ const cur = useEditor.getState();+ const block = blockUnder(carried, Math.max(0, timeAt(x)), place);+ if (block) {+ setAssetDrop({ t: block.start, len: clipLen(block), ghost, shifts: [] });+ return;+ }// Preview the true landing spot: a drop at the pointer inserts here,// rippling later clips right, so the ghost sits where the segment will// actually land — a box under the pointer that lands minutes away lies.- const cur = useEditor.getState();const { start, shifts } = rippleInsert(track0Clips(cur.clips), dropTimeAt(x), duration);setAssetDrop({ t: start, len: duration, ghost, shifts });});@@ -3754,7 +3798,7 @@ function ClipView({}}><div className="pointer-events-none absolute inset-x-0 top-0" style={{ height: FILM_H }}>- <Filmstrip frames={filmstrip} grade={clip.grade} failed={stripFailed} hidden={offscreen} />+ <Filmstrip frames={filmstrip} grade={clip.grade} failed={stripFailed} hidden={offscreen} block={asset?.block} /></div>{hasWave && (<div@@ -4602,9 +4646,14 @@ function Filmstrip({grade,failed,hidden,+ block,}: {frames: { src: string; left: number; width: number }[];grade?: ColorGrade;+ /** The clip is a shot with nothing in it yet: the strip is a flat slab in+ * the block's own color, and the clip's label above it says what belongs+ * there. Nothing is loading, so nothing pulses. */+ block?: { label: string; color?: string };/** The strip's last generation attempt failed: hold a still placeholder.* An endless pulse reads as loading forever, and animating a clip-width* element burns paint time for the life of the timeline. */@@ -4620,6 +4669,14 @@ function Filmstrip({// No thumbs yet — the media is still streaming into the browser store. A// placeholder slab fills the box until the strip can draw real frames.if (!frames.length) {+ if (block) {+ return (+ <div+ className="tl-filmstrip-block pointer-events-none absolute inset-0"+ style={{ background: block.color || BLOCK_COLOR }}+ />+ );+ }if (failed) {return (<div className="tl-filmstrip-failed pointer-events-none absolute inset-0 bg-[var(--skeleton)]" />diff --git a/site/src/cut/lib/aiContext.ts b/site/src/cut/lib/aiContext.tsindex 5bc54576..0a374178 100644--- a/site/src/cut/lib/aiContext.ts+++ b/site/src/cut/lib/aiContext.ts@@ -412,7 +412,7 @@ function describeState(s.mediaFolders.length > 0 ? s.mediaFolders.map((f) => ({ id: f.id, name: f.name })) : null,// Every project asset visible to this chat, timeline-placed or not (media// another chat still owns is filtered out above). `origin` marks Cut-made- // media (generated/voiceover/recording/stock/freeze); no origin = a user+ // media (generated/voiceover/recording/stock/freeze/block); no origin = a user// import shown in the Media panel.media: shownAssets.map((a) => ({id: a.id,diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.tsindex a9d6a855..c47369e9 100644--- a/site/src/cut/lib/aiTools.ts+++ b/site/src/cut/lib/aiTools.ts@@ -185,6 +185,7 @@ import { composeMusicPrompt } from "./composeGen";import { stockAssetInDoc } from "./genvideo/docWriter";import { resolveVoiceAsk, synthesizeSpeech, SPEECH_VOICES } from "./tts";import { hostedPost } from "./hosted";+import { blockAsset } from "./blockSource";import { cutJudge } from "./chatRuntime";import { askJudgeChunked, noul } from "./judge";import { SWEEP_CHUNK, sweepPicks, sweepQuestions, sweepState, type SweepCandidate } from "./sweepSelect";@@ -380,6 +381,11 @@ function statsReport(stats: ColorStats) {* enough to ride a turn. */const CAPTURE_LONG_SIDE = 640;+/** The most shots one blockout lays down. Every block in a call lands as one+ * undo step, and a whole feature's worth of them in a single step is not+ * something anyone can step back through; a caller with more comes back. */+const MAX_BLOCKS = 40;+const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v));const isNum = (v: unknown): v is number => typeof v === "number" && Number.isFinite(v);@@ -440,6 +446,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {if (!assetId) throw new ToolError("Pass asset_id.");const stickerAsset = s.assets.find((a) => a.id === assetId);if (stickerAsset?.type !== "image") throw new ToolError(`No image asset with id ${assetId}.`);+ refuseReference(stickerAsset);if (isNum(input.start)) s.seek(input.start);s.addSticker({ assetId, ...(isLottieAsset(stickerAsset) ? { lottie: true } : {}), ...aimedLane(input) });const sel = useEditor.getState().selection;@@ -1477,6 +1484,9 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {},add_clip: (s, input) => {+ // A run of blocks is the same placement with nothing to play yet: the+ // shots of a cut that has been laid out before its footage exists.+ if (Array.isArray(input.blocks)) return placeBlocks(s, input.blocks, input);const asset = requireItem(s.assets, input.asset_id, "project asset");return placeAssetOnTimeline(asset, input);},@@ -1485,6 +1495,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {const asset = requireItem(s.assets, input.asset_id, "project asset");if (asset.type !== "video" && asset.type !== "image")throw new ToolError("Only video or image assets can sit on a video track.");+ refuseReference(asset);const start = isNum(input.start) ? Math.max(0, input.start) : playheadAt();// Tracks stack bottom-up from track 0; overlays live on 1+. A stale// negative (the old behind-track model) clamps to the first layer.@@ -2590,6 +2601,61 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {return { id: clip.id, background: { kind, assetId } };},+ replace_item: (s, input) => {+ const id = typeof input.id === "string" ? input.id : "";+ const asset = requireItem(s.assets, input.asset_id, "project asset");+ refuseReference(asset);+ const clip = s.clips.find((c) => c.id === id);+ if (clip) {+ if (asset.type !== "video" && asset.type !== "image")+ throw new ToolError(`"${asset.name}" is ${asset.type} — a video clip plays video or a still.`);+ const before = clipLen(clip);+ s.replaceClipAsset(clip.id, asset.id);+ const next = useEditor.getState().clips.find((c) => c.id === clip.id);+ if (!next || next.assetId !== asset.id) throw new ToolError("That clip could not take this source.");+ return {+ id: next.id,+ kind: "clip",+ assetId: asset.id,+ name: asset.name,+ start: round2(next.start),+ lenBefore: round2(before),+ lenAfter: round2(clipLen(next)),+ ...tracksAfter(),+ };+ }+ const audio = s.audioClips.find((a) => a.id === id);+ if (audio) {+ if (asset.type !== "audio")+ throw new ToolError(`"${asset.name}" is ${asset.type} — a soundtrack clip plays audio.`);+ const before = audio.out - audio.in;+ s.replaceAudioAsset(audio.id, asset.id);+ const next = useEditor.getState().audioClips.find((a) => a.id === id);+ if (!next || next.assetId !== asset.id) throw new ToolError("That clip could not take this source.");+ return {+ id: next.id,+ kind: "audio",+ assetId: asset.id,+ name: asset.name,+ start: round2(next.start),+ lenBefore: round2(before),+ lenAfter: round2(next.out - next.in),+ };+ }+ const overlay = s.overlays.find((o) => o.id === id);+ if (overlay && overlay.kind === "sticker") {+ if (asset.type !== "image")+ throw new ToolError(`"${asset.name}" is ${asset.type} — a sticker shows a still.`);+ // A Lottie document animates and a still does not; the flag is read off+ // the source, so it moves with it or the export plays the wrong one.+ s.updateOverlay(overlay.id, { assetId: asset.id, lottie: isLottieAsset(asset) || undefined });+ return { id: overlay.id, kind: "sticker", assetId: asset.id, name: asset.name };+ }+ throw new ToolError(+ `Nothing with id "${id}" carries a source. Clips, soundtrack clips and stickers do; text and shapes are edited with update_overlay.`+ );+ },+freeze_frame: async (s, input) => {const projectId = s.projectId;if (!projectId) throw new ToolError("No project open.");@@ -5149,7 +5215,77 @@ function audioLaneInput(value: unknown): number {return value;}+/** Lay out shots that have no footage yet: one block per shot on track 0,+ * each the length that shot runs, in one undo step. The blocks own no files,+ * so this writes nothing to storage — and each one is an ordinary clip, so it+ * trims, moves, takes a transition, and is replaced when the footage lands. */+function placeBlocks(s: Editor, raw: unknown[], input: Record<string, unknown>) {+ const shots = raw+ .filter((x): x is Record<string, unknown> => typeof x === "object" && x !== null)+ .map((x) => ({+ seconds: isNum(x.seconds) ? clamp(x.seconds, 0.2, 600) : 0,+ label: typeof x.label === "string" ? x.label.trim() : "",+ }))+ .filter((x) => x.seconds > 0);+ if (shots.length === 0)+ throw new ToolError("Pass blocks: [{seconds, label}] — one entry per shot, in order.");+ if (shots.length > MAX_BLOCKS)+ throw new ToolError(+ `That is ${shots.length} shots — ${MAX_BLOCKS} is the most one call lays down. Lay the first stretch and call again for the rest.`+ );+ const reference =+ typeof input.reference_asset_id === "string" && input.reference_asset_id+ ? requireItem(s.assets, input.reference_asset_id, "project asset")+ : null;+ const frame = frameOf(s.aspect);+ const made: { clipId: string; assetId: string; start: number; seconds: number; label: string }[] = [];+ let at = isNum(input.start) ? Math.max(0, input.start) : undefined;+ useEditor.getState().beginHistoryBatch();+ try {+ // What the shots are copied from does not play in the copy.+ if (reference) useEditor.getState().updateAsset(reference.id, { reference: true });+ for (const [i, shot] of shots.entries()) {+ const asset = blockAsset(shot, i, frame);+ const cur = useEditor.getState();+ cur.addAsset(asset);+ // One at a time, each sized before the next goes down, so every block+ // appends against a row whose end is already true and no two intersect.+ const clipId = cur.addClipFromAsset(asset.id, at);+ if (!clipId) continue;+ useEditor.getState().updateClip(clipId, { out: shot.seconds });+ const placed = useEditor.getState().clips.find((c) => c.id === clipId);+ if (!placed) continue;+ at = undefined; // the rest follow the row+ made.push({+ clipId: placed.id,+ assetId: asset.id,+ start: round2(placed.start),+ seconds: round2(shot.seconds),+ label: asset.block?.label ?? "",+ });+ }+ } finally {+ useEditor.getState().endHistoryBatch();+ }+ return {+ placed: made,+ ...tracksAfter(),+ note: "Each block holds its shot's place on track 0. A block is filled by dropping footage on it — the footage takes its place and its length — or with replace_item; titles, sound and transitions go on top of the skeleton now.",+ };+}++/** A reference is what the cut is copied from, so it never plays in it: the+ * blocks standing in for its shots are waiting on the person's own footage.+ * Their hands can still place it — this holds the assistant's tools. */+function refuseReference(asset: MediaAsset): void {+ if (!asset.reference) return;+ throw new ToolError(+ `"${asset.name}" is the reference this cut is blocked out from, so it does not go in the cut. Fill a block with the person's own footage (replace_item), or leave it standing.`+ );+}+function placeAssetOnTimeline(asset: MediaAsset, input: Record<string, unknown>) {+ refuseReference(asset);const s = useEditor.getState();const start = isNum(input.start) ? Math.max(0, input.start) : undefined;if (asset.type === "audio") {diff --git a/site/src/cut/lib/blockSource.ts b/site/src/cut/lib/blockSource.tsnew file mode 100644index 00000000..8a645d56--- /dev/null+++ b/site/src/cut/lib/blockSource.ts@@ -0,0 +1,76 @@+import { wrapTextToRoom } from "@donkeycut/effects-kit";+import { createRasterCanvas, type RasterSurface } from "./raster";+import type { MediaAsset, StoredAsset } from "./types";++// A block is a shot with nothing in it yet: a source that holds a slot on the+// timeline and draws itself — a filled rect carrying the label of whatever+// belongs there. It owns no file, so blocking out a cut writes nothing to+// storage and decodes nothing; the picture is painted once, at whatever size+// asks for it, and cached by the reader that asked. A clip playing a block+// cuts, trims, moves and takes a transition like any other clip, and the day+// the footage arrives it replaces the block in place.++export const BLOCK_COLOR = "#161A22";+const LABEL_COLOR = "#E9EDF5";+const STROKE_COLOR = "#2E3644";++/** The source a shot stands on until its footage exists. It takes the+ * project's own frame, so a block fills the picture the way the footage that+ * replaces it will. */+export function blockAsset(+ shot: { seconds: number; label?: string; color?: string },+ i: number,+ frame: { w: number; h: number }+): MediaAsset {+ const label = shot.label?.trim() || `Shot ${i + 1}`;+ return {+ id: `block-${Date.now().toString(36)}-${i}-${Math.random().toString(36).slice(2, 6)}`,+ // No file: nothing is stored, so nothing is named on disk either.+ fileName: "",+ name: label,+ type: "image",+ duration: shot.seconds,+ width: frame.w,+ height: frame.h,+ url: "",+ block: { label, ...(shot.color ? { color: shot.color } : {}) },+ // Cut made this, so it belongs where it was made and not in the Media+ // panel with the person's own imports.+ origin: "block",+ };+}++/** Paint a block at this size. Both frame readers — the preview's and the+ * export's — call this in place of fetching and decoding a still. */+export function drawBlock(+ asset: Pick<StoredAsset, "block" | "name">,+ width: number,+ height: number+): RasterSurface {+ const w = Math.max(16, Math.round(width));+ const h = Math.max(16, Math.round(height));+ const canvas = createRasterCanvas(w, h);+ const ctx = canvas.getContext("2d") as CanvasRenderingContext2D | null;+ if (!ctx) return canvas;+ const short = Math.min(w, h);+ ctx.fillStyle = asset.block?.color || BLOCK_COLOR;+ ctx.fillRect(0, 0, w, h);++ const inset = Math.round(short * 0.05);+ ctx.strokeStyle = STROKE_COLOR;+ ctx.lineWidth = Math.max(2, Math.round(short * 0.006));+ ctx.strokeRect(inset, inset, w - inset * 2, h - inset * 2);++ const label = asset.block?.label?.trim() || asset.name || "";+ if (!label) return canvas;+ const size = Math.round(short * 0.062);+ ctx.font = `600 ${size}px system-ui, -apple-system, "Helvetica Neue", Arial, sans-serif`;+ ctx.fillStyle = LABEL_COLOR;+ ctx.textAlign = "center";+ ctx.textBaseline = "middle";+ const lines = wrapTextToRoom(label, w - inset * 4, (line) => ctx.measureText(line).width).split("\n").slice(0, 4);+ const step = size * 1.3;+ const top = h / 2 - ((lines.length - 1) * step) / 2;+ lines.forEach((line, i) => ctx.fillText(line, w / 2, top + i * step));+ return canvas;+}diff --git a/site/src/cut/lib/blockoutReference.test.ts b/site/src/cut/lib/blockoutReference.test.tsnew file mode 100644index 00000000..eae637aa--- /dev/null+++ b/site/src/cut/lib/blockoutReference.test.ts@@ -0,0 +1,172 @@+import { beforeEach, expect, test } from "bun:test";+import { runAiTool } from "./aiTools";+import { storedAssets, useEditor } from "./store";++// A cut blocked out from a reference must not end up playing that reference.+// The mark rides the asset, so the refusal holds for every tool that places+// media, whatever the model was reaching for when it tried.++beforeEach(() => {+ useEditor.setState({+ clips: [],+ overlays: [],+ transitions: [],+ audioClips: [],+ assets: [+ { id: "ref", fileName: "ref.mp4", name: "Reference", type: "video", duration: 60, url: "", reference: true },+ { id: "mine", fileName: "mine.mp4", name: "My shot", type: "video", duration: 12, url: "" },+ { id: "still", fileName: "ref.png", name: "Reference still", type: "image", duration: 0, url: "", reference: true },+ ],+ });+});++const message = async (p: Promise<unknown>) =>+ p.then(() => "", (e: unknown) => (e instanceof Error ? e.message : String(e)));++test("the assistant cannot put the reference in the cut it is copied from", async () => {+ const error = await message(runAiTool("add_clip", { asset_id: "ref" }));+ expect(error).toContain("reference this cut is blocked out from");+ expect(useEditor.getState().clips).toHaveLength(0);+});++test("the reference cannot fill a block either", async () => {+ await runAiTool("add_clip", { asset_id: "mine", start: 0 });+ const clip = useEditor.getState().clips[0]!;+ const error = await message(runAiTool("replace_item", { id: clip.id, asset_id: "ref" }));+ expect(error).toContain("reference this cut is blocked out from");+ expect(useEditor.getState().clips[0]!.assetId).toBe("mine");+});++test("the person's own footage still goes down", async () => {+ await runAiTool("add_clip", { asset_id: "mine", start: 0 });+ expect(useEditor.getState().clips.map((c) => c.assetId)).toEqual(["mine"]);+});++test("an unmarked source is nobody's reference", async () => {+ useEditor.getState().updateAsset("ref", { reference: undefined });+ await runAiTool("add_clip", { asset_id: "ref", start: 0 });+ expect(useEditor.getState().clips.map((c) => c.assetId)).toEqual(["ref"]);+});++test("a run of blocks lays a cut out end to end, storing nothing", async () => {+ useEditor.getState().updateAsset("ref", { reference: undefined });+ const out = (await runAiTool("add_clip", {+ blocks: [+ { seconds: 4, label: "wide of the kitchen" },+ { seconds: 2.5, label: "hands only" },+ { seconds: 6 },+ ],+ reference_asset_id: "ref",+ })) as { placed: { start: number; seconds: number; label: string }[] };++ expect(out.placed.map((p) => [p.start, p.seconds])).toEqual([[0, 4], [4, 2.5], [6.5, 6]]);+ expect(out.placed[2]!.label).toBe("Shot 3");+ const state = useEditor.getState();+ // Each block is an ordinary clip on track 0, and no two intersect.+ const blocks = state.clips.filter((c) => c.track === 0).sort((a, b) => a.start - b.start);+ expect(blocks).toHaveLength(3);+ for (let i = 1; i < blocks.length; i++)+ expect(blocks[i]!.start).toBeGreaterThanOrEqual(blocks[i - 1]!.start + (blocks[i - 1]!.out - blocks[i - 1]!.in));+ // The sources draw themselves: no file, no url, nothing uploaded.+ const made = state.assets.filter((a) => a.block !== undefined);+ expect(made).toHaveLength(3);+ expect(made.every((a) => a.url === "" && a.fileName === "")).toBe(true);+ // Naming the source makes it the reference, so the tools stop placing it.+ expect(state.assets.find((a) => a.id === "ref")!.reference).toBe(true);+});++test("the whole blockout is one undo step", async () => {+ useEditor.getState().updateAsset("ref", { reference: undefined });+ await runAiTool("add_clip", {+ blocks: [{ seconds: 3 }, { seconds: 3 }, { seconds: 3 }],+ });+ expect(useEditor.getState().clips).toHaveLength(3);+ useEditor.getState().undo();+ expect(useEditor.getState().clips).toHaveLength(0);+});++test("footage takes a block's place and its length", async () => {+ await runAiTool("add_clip", { blocks: [{ seconds: 5, label: "the shot" }] });+ const block = useEditor.getState().clips[0]!;+ await runAiTool("replace_item", { id: block.id, asset_id: "mine" });+ const filled = useEditor.getState().clips[0]!;+ expect(filled.assetId).toBe("mine");+ expect(filled.start).toBe(block.start);+ expect(filled.out - filled.in).toBe(5);+ // The block it replaced is gone with it.+ expect(useEditor.getState().assets.some((a) => a.block !== undefined)).toBe(false);+});++test("a blockout survives the save", () => {+ useEditor.getState().updateAsset("ref", { reference: true });+ useEditor.setState({+ assets: [+ ...useEditor.getState().assets,+ {+ id: "b1",+ fileName: "",+ name: "wide of the kitchen",+ type: "image",+ duration: 4,+ url: "",+ block: { label: "wide of the kitchen" },+ origin: "block",+ },+ ],+ });+ const saved = storedAssets(useEditor.getState().assets);+ // Without these two the blockout comes back as an image with no file: the+ // shot draws nothing, footage dropped on it inserts beside it instead of+ // filling it, and the reference is placeable again.+ expect(saved.find((a) => a.id === "b1")?.block).toEqual({ label: "wide of the kitchen" });+ expect(saved.find((a) => a.id === "ref")?.reference).toBe(true);+});++test("the reference is refused on every door, not just track 0", async () => {+ for (const call of [+ runAiTool("add_overlay_video", { asset_id: "ref", track: 1 }),+ runAiTool("add_sticker", { asset_id: "still" }),+ ]) {+ expect(await message(call)).toContain("reference this cut is blocked out from");+ }+ expect(useEditor.getState().clips).toHaveLength(0);+ expect(useEditor.getState().overlays).toHaveLength(0);+});++test("a swap drops coverage painted for the source it replaces", async () => {+ await runAiTool("add_clip", { asset_id: "mine", start: 0 });+ const clip = useEditor.getState().clips[0]!;+ useEditor.getState().updateClip(clip.id, { mask: { kind: "rect", x: 0, y: 0, w: 0.5, h: 0.5 } });+ useEditor.getState().updateAsset("ref", { reference: undefined });+ await runAiTool("replace_item", { id: clip.id, asset_id: "ref" });+ expect(useEditor.getState().clips[0]!.mask).toBeUndefined();+});++test("a clip never plays past the end of a shorter source", async () => {+ await runAiTool("add_clip", { blocks: [{ seconds: 20, label: "long shot" }] });+ const block = useEditor.getState().clips[0]!;+ await runAiTool("replace_item", { id: block.id, asset_id: "mine" });+ const filled = useEditor.getState().clips[0]!;+ expect(filled.out).toBeLessThanOrEqual(12);+});++test("a sticker's swap carries how it plays", async () => {+ useEditor.setState({+ assets: [+ { id: "anim", fileName: "wave.json", name: "Wave", type: "image", duration: 0, url: "" },+ { id: "flat", fileName: "star.png", name: "Star", type: "image", duration: 0, url: "" },+ ],+ overlays: [],+ });+ await runAiTool("add_sticker", { asset_id: "anim" });+ const sticker = useEditor.getState().overlays[0]!;+ expect((sticker as { lottie?: boolean }).lottie).toBe(true);++ // A still does not animate: left set, the export plays a PNG through the+ // animation renderer.+ await runAiTool("replace_item", { id: sticker.id, asset_id: "flat" });+ expect((useEditor.getState().overlays[0] as { lottie?: boolean }).lottie).toBeUndefined();++ await runAiTool("replace_item", { id: sticker.id, asset_id: "anim" });+ expect((useEditor.getState().overlays[0] as { lottie?: boolean }).lottie).toBe(true);+});diff --git a/site/src/cut/lib/exportClient.ts b/site/src/cut/lib/exportClient.tsindex b2a382ad..1a5f53d5 100644--- a/site/src/cut/lib/exportClient.ts+++ b/site/src/cut/lib/exportClient.ts@@ -20,6 +20,7 @@ import { downloadFile, downloadFromUrl } from "./download";import { bitrateFor, canRenderInBrowser, renderProjectToMp4 } from "./exportRender";import { putSigned } from "./media";import { renderRemovalPieces } from "./removalVideo";+import { drawBlock } from "./blockSource";import { createRasterCanvas, rasterCanvasToPng } from "./raster";import { clipLen, clipSpeed, getClipSpans, overlayLayers, projectDuration, spanSequence, useEditor } from "./store";import { captionStyle, cueOverlay, cueWordFrames, laneCues, laneHidden, subtitleLaneCount, trackPos } from "./subtitles";@@ -40,6 +41,7 @@ import type {import {deliveryContainer,+ deliverySpan,EXPORT_CONTAINERS,exportBaseName,specMediaFiles,@@ -873,7 +875,20 @@ export async function buildExportPayload(kf: posed(sp.clip).kf,border: undefined as string | undefined,removal: undefined as { rgb: string; alpha: string } | undefined,+ staged: undefined as boolean | undefined,}));+ // A block owns no file: its card is painted here at the output size and+ // travels with the job, so this render draws what the preview draws.+ for (let i = 0; i < spans.length; i++) {+ if (!spans[i].asset.block) continue;+ const name = `block_${i}.png`;+ pngs.push({+ name,+ blob: await rasterCanvasToPng(drawBlock(spans[i].asset, settings.width, settings.height)),+ });+ clipEntries[i].file = name;+ clipEntries[i].staged = true;+ }// Track-0 segments render at the full output frame (regioned clips pad out// to it), so their masks paint full-frame. A subject mask rides the shared// matte; painted pictures still travel beside it when the pose track keys@@ -1518,7 +1533,16 @@ export async function runBrowserExport(const claim = await backend.fetch("/api/cut/export/client/presign", {method: "POST",headers: { "Content-Type": "application/json" },- body: JSON.stringify({ projectId, outName: exportOutName(settings) }),+ // The size this render is heading for — the number the dialog showed —+ // travels with the claim, so an account with no room for the file is told+ // now rather than after the encode.+ body: JSON.stringify({+ projectId,+ outName: exportOutName(settings),+ bytes: Math.round(+ estimateExportBytes(settings, deliverySpan(settings.range, projectDuration(doc)))+ ),+ }),});const claimed = await apiJson<{ jobId?: string; url?: string; outName?: string; type?: string }>(claim);if (!claim.ok || !claimed.jobId || !claimed.url) {@@ -1559,7 +1583,9 @@ export async function runBrowserExport(body: JSON.stringify({ jobId }),});const body = await apiJson<{ id?: string }>(done);- if (!done.ok || !body.id) throw new Error(body.error ?? "Could not save the export.");+ if (!done.ok || !body.id) {+ throw cloudRefusal(done, body) ?? new Error(body.error ?? "Could not save the export.");+ }return body.id;} catch (err) {// A render that stopped — cancelled, failed, or refused — gives back thediff --git a/site/src/cut/lib/exportDelivery.test.ts b/site/src/cut/lib/exportDelivery.test.tsnew file mode 100644index 00000000..31b7ea93--- /dev/null+++ b/site/src/cut/lib/exportDelivery.test.ts@@ -0,0 +1,23 @@+import { expect, test } from "bun:test";+import { specMediaFiles } from "./exportDelivery";++// What a render has to fetch from the project's own media folder. A block's+// card is painted by the client and travels with the job, so listing it here+// sends the worker looking for an object that was never stored and stages it+// where the render does not read.+test("a staged picture is not one of the project's media files", () => {+ const files = specMediaFiles({+ clips: [+ { file: "a.mp4" },+ { file: "block_1.png", staged: true },+ { file: "b.mp4" },+ ],+ overlayVideos: [{ file: "pip.mp4" }],+ audio: [{ file: "music.mp3" }],+ });+ expect(files).toEqual(["a.mp4", "b.mp4", "pip.mp4", "music.mp3"]);+});++test("each media file is listed once", () => {+ expect(specMediaFiles({ clips: [{ file: "a.mp4" }, { file: "a.mp4" }] })).toEqual(["a.mp4"]);+});diff --git a/site/src/cut/lib/exportDelivery.ts b/site/src/cut/lib/exportDelivery.tsindex f15ebf0b..10342bef 100644--- a/site/src/cut/lib/exportDelivery.ts+++ b/site/src/cut/lib/exportDelivery.ts@@ -98,13 +98,17 @@ export function videoBitrateFor(s: {/** Every media file a render spec plays — clips, picture-in-picture video,* sound — by name, once each. */export function specMediaFiles(spec: {- clips?: { file?: string }[];+ clips?: { file?: string; staged?: boolean }[];overlayVideos?: { file?: string }[];audio?: { file?: string }[];}): string[] {const names = new Set<string>();for (const c of [...(spec.clips ?? []), ...(spec.overlayVideos ?? []), ...(spec.audio ?? [])]) {- if (c?.file) names.add(c.file);+ // A staged picture is painted by the client and travels with the job, so+ // it is not one of the project's media files: looking for it in the+ // project's folder finds nothing, and staging it there puts it where the+ // render does not read.+ if (c?.file && !("staged" in c && c.staged)) names.add(c.file);}return [...names];}diff --git a/site/src/cut/lib/exportRender.ts b/site/src/cut/lib/exportRender.tsindex 66e664a0..dc6d90fc 100644--- a/site/src/cut/lib/exportRender.ts+++ b/site/src/cut/lib/exportRender.ts@@ -29,6 +29,7 @@ import {type AudioCodec,type VideoCodec,} from "mediabunny";+import { drawBlock } from "./blockSource";import { audioFxSpans } from "./audioEffects";import { renderMix, type MixClip, type MixItem, type MixSpec } from "./audioMix";import { FrameCompositor, MISSING_FRAME, type Frame } from "./composite";@@ -354,6 +355,13 @@ export class ClipReader {* with no decodable video). */private open(): Promise<ClipSink | null> {return (this.opened ??= (async () => {+ if (this.asset.block) {+ // Nothing to fetch: the block paints itself at the source size the+ // render asks for, once, and that painting is the frame.+ const canvas = drawBlock(this.asset, this.asset.width ?? 1080, this.asset.height ?? 1920);+ this.still = { kind: "ready", image: canvas, width: canvas.width, height: canvas.height };+ return null;+ }if (this.asset.type === "image") {const blob = await (await fetch(this.url())).blob();const bitmap = await createImageBitmap(blob);diff --git a/site/src/cut/lib/frameSource.ts b/site/src/cut/lib/frameSource.tsindex 69a1ddb5..a41212dc 100644--- a/site/src/cut/lib/frameSource.ts+++ b/site/src/cut/lib/frameSource.ts@@ -36,6 +36,7 @@ import {} from "./mediaRead";import { allowance, canvasBytes, decodedFrameBytes, holdMemory } from "./memoryBudget";import { meterPull, meterSource, meterWalk } from "./perfTrace";+import { drawBlock } from "./blockSource";import type { MediaAsset } from "./types";/** Dev-only: pool lifecycle events, into the same log the engine writes.@@ -1181,6 +1182,17 @@ export class ClipFrameSource {if (this.opening) return this.opening;this.opening = (async () => {try {+ // A block owns no file: it paints itself at the height this source+ // reads at, and the painting is the still from then on.+ if (this.asset.block) {+ const w = Math.round(this.height * ((this.asset.width ?? 1080) / (this.asset.height ?? 1920)));+ const canvas = drawBlock(this.asset, w, this.height);+ if (this.closed) return;+ this.still = { image: canvas, width: canvas.width, height: canvas.height, timestamp: 0 };+ this.attempts = 0;+ this.onFrame();+ return;+ }const res = await fetch(this.asset.url, { mode: "cors" });const bitmap = await createImageBitmap(await res.blob());if (this.closed) return bitmap.close();diff --git a/site/src/cut/lib/store.ts b/site/src/cut/lib/store.tsindex 45fe032a..48d555aa 100644--- a/site/src/cut/lib/store.ts+++ b/site/src/cut/lib/store.ts@@ -191,6 +191,24 @@ export const track0Clips = (clips: VideoClip[]) => clips.filter((c) => c.track =/** Every video clip not on track 0 — the composited layers. */export const overlayLayers = (clips: VideoClip[]) => clips.filter((c) => c.track !== 0);+/** The block a time falls inside on track 0, if any. Footage released on one+ * takes its place instead of squeezing in beside it, which is how a blocked-out+ * cut gets filled in. */+export function blockClipAt(+ clips: VideoClip[],+ assets: MediaAsset[],+ t: number+): VideoClip | null {+ // A drag calls this once per frame, so it walks the row and looks up one+ // asset — the clip the pointer is over — instead of building a set over+ // every asset in the project.+ for (const c of clips) {+ if (c.track !== 0 || t < c.start || t >= c.start + clipLen(c)) continue;+ return assets.find((a) => a.id === c.assetId)?.block !== undefined ? c : null;+ }+ return null;+}+/** Ground a clip stack: the lowest occupied row becomes track 0. Deleting or* dragging away the last track-0 clip re-grounds the rows above it, so the* spine — the sequence that carries transitions, fades, and ripple — always@@ -526,12 +544,22 @@ export interface EditorState {removeAsset: (id: string) => void;/** Add a video clip from an asset onto video track 0 — at `start` (sliding* to the track's next free slot), or appended at the end when omitted. */- addClipFromAsset: (assetId: string, start?: number) => void;+ addClipFromAsset: (assetId: string, start?: number) => string | null;+ /** Swap what a video clip plays, keeping its place and its length: the+ * slot a block holds becomes the person's own footage,+ * trimmed from the source's head to the block's length (a shorter source+ * makes the clip its own length). The block it replaced is dropped with it+ * when nothing else plays it. */+ replaceClipAsset: (clipId: string, assetId: string) => void;/** Add a soundtrack clip from an audio asset at `start` (default: the* playhead). `opts.duck` marks it a voiceover that lowers everything else* to that gain while it plays; `opts.lane` picks the audio track it lands* on (default: the first one). */addAudioFromAsset: (assetId: string, start?: number, opts?: { duck?: number; lane?: number }) => void;+ /** Swap a soundtrack clip's source, keeping its slot: the clip stays where+ * it starts and plays the new sound from its head, up to the slot's length+ * or all the source has. */+ replaceAudioAsset: (clipId: string, assetId: string) => void;/** The panel “+” add, and every paste: the asset lands at `at` — by default* the preview time (the skimmer while one is live, the playhead otherwise) —* on the lowest video track or audio lane with room for its whole length@@ -2493,7 +2521,7 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {addClipFromAsset: (assetId, start) => {const asset = get().assets.find((a) => a.id === assetId);- if (!asset || (asset.type !== "video" && asset.type !== "image")) return;+ if (!asset || (asset.type !== "video" && asset.type !== "image")) return null;push();const out = asset.type === "image" ? IMAGE_CLIP_SECONDS : asset.duration;const len = Math.max(MIN_LEN, out);@@ -2517,6 +2545,62 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {clips: [...s.clips, clip].sort((a, b) => a.start - b.start),...sole({ kind: "clip", id: clip.id }),}));+ return clip.id;+ },++ replaceClipAsset: (clipId, assetId) => {+ const clip = get().clips.find((c) => c.id === clipId);+ const asset = get().assets.find((a) => a.id === assetId);+ if (!clip || !asset || (asset.type !== "video" && asset.type !== "image")) return;+ if (clip.assetId === assetId) return;+ const was = get().assets.find((a) => a.id === clip.assetId);+ // The slot the clip holds is what the new source fills.+ const slot = clipLen(clip);+ // Never past the end of what now plays there: a source shorter than the+ // slot plays all it has.+ const out =+ asset.type === "image" ? Math.max(MIN_LEN, slot) : Math.min(asset.duration, Math.max(MIN_LEN, slot));+ get().beginHistoryBatch();+ set((s) => ({+ clips: s.clips.map((c) =>+ c.id === clipId+ ? {+ ...c,+ assetId,+ in: 0,+ out,+ speed: undefined,+ speedCurve: undefined,+ reverse: undefined,+ // Coverage painted frame by frame for the old picture, and a+ // key pulled off its pixels, mean nothing to the new one. A+ // subject matte is computed from whatever plays, so it stays.+ ...(clip.mask && clip.mask.kind !== "subject" ? { mask: undefined } : {}),+ removal: undefined,+ }+ : c+ ),+ }));+ // A block exists for the footage that replaces it; once nothing plays+ // it, it goes with it.+ if (was?.block !== undefined && !get().clips.some((c) => c.assetId === was.id))+ get().removeAsset(was.id);+ get().endHistoryBatch();+ },++ replaceAudioAsset: (clipId, assetId) => {+ const clip = get().audioClips.find((a) => a.id === clipId);+ const asset = get().assets.find((a) => a.id === assetId);+ if (!clip || !asset || asset.type !== "audio" || clip.assetId === assetId) return;+ const slot = clip.out - clip.in;+ push();+ set((s) => ({+ audioClips: s.audioClips.map((a) =>+ a.id === clipId+ ? { ...a, assetId, in: 0, out: Math.max(MIN_LEN, Math.min(asset.duration, slot)) }+ : a+ ),+ }));},addAudioFromAsset: (assetId, start, opts) => {@@ -4807,7 +4891,7 @@ export const docOverlays = (() => {export function storedAssets(assets: MediaAsset[]): StoredAsset[] {return assets.filter((a) => !tabOnlyUpload(a))- .map(({ id, fileName, name, type, duration, width, height, origin, chatId, folderId, language, watch, speech, beats, sceneCuts, copiedFrom }) => ({+ .map(({ id, fileName, name, type, duration, width, height, origin, chatId, folderId, language, watch, speech, beats, sceneCuts, copiedFrom, block, reference }) => ({id,fileName,name,@@ -4824,6 +4908,8 @@ export function storedAssets(assets: MediaAsset[]): StoredAsset[] {...(beats !== undefined ? { beats } : {}),...(sceneCuts !== undefined ? { sceneCuts } : {}),...(copiedFrom !== undefined ? { copiedFrom } : {}),+ ...(block !== undefined ? { block } : {}),+ ...(reference !== undefined ? { reference } : {}),}));}diff --git a/site/src/cut/lib/types.ts b/site/src/cut/lib/types.tsindex c15e5906..4ba16e1c 100644--- a/site/src/cut/lib/types.ts+++ b/site/src/cut/lib/types.ts@@ -198,7 +198,18 @@ export interface StoredAsset {* drop, or upload), so it belongs in the Media panel. Any value marks media* Cut created or fetched — it lives where it was made (the timeline, a* generation panel, or an AI chat card) and is kept out of the Media panel. */- origin?: "voiceover" | "generated" | "recording" | "stock" | "freeze" | "chat" | "sticker" | "matte";+ origin?:+ | "voiceover"+ | "generated"+ | "recording"+ | "stock"+ | "freeze"+ | "chat"+ | "sticker"+ | "matte"+ // A shot with nothing in it yet, holding its place on a blocked-out cut+ // until the person's own footage takes it.+ | "block";/** BCP-47 of the audio's spoken language, when known (stamped on voiceovers* at synthesis) — what transcription should run its recognizer in. */language?: string;@@ -216,6 +227,16 @@ export interface StoredAsset {* replicate out of another project. A second copy of the same source finds* this one and lands nothing twice. */copiedFrom?: { projectId: string; assetId: string };+ /** A shot with nothing in it yet: this source owns no file and draws+ * itself — a filled rect carrying the label of what belongs there. See+ * blockSource.ts. Replacing it with real footage is an ordinary item+ * replace; the clip keeps its place and its length. */+ block?: { label: string; color?: string };+ /** The source a blocked-out cut was copied from. Set when a blockout+ * names it, and it keeps the assistant's own tools from putting+ * the thing being copied into the copy: the blocks hold those slots until+ * the person's footage arrives. The person's own drag is never refused. */+ reference?: true;}/** A folder in the Media panel's Project Files view — a flat, project-localdiff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.tsindex 7c483404..7bbfae17 100644--- a/site/src/cut/server/ai/catalog.ts+++ b/site/src/cut/server/ai/catalog.ts@@ -220,7 +220,7 @@ export const AI_SKILLS: Record<string, string> = {"editor-overview": `# Cut editor overviewCut is a local, project-based short-video editor. Each project has an output aspect ratio "W:H" — presets 16:9, 9:16, 1:1, 4:3, 3:4, 2:1, or any custom ratio like 9:5 — rendered with the frame's short side at 1080px and switchable from the pill in the top bar; the current one is in editor_state project.aspect. The Guides button beside the timeline zoom turns on preview guides — thirds, center, safe margins, the short-form keep-out zone shared by TikTok, Reels and Shorts, and custom lines the user drags into place (set_guides) — which editor_state project.guides reports with the safeArea they leave. Layout:- Left icon rail tabs: Media (user-imported files + Exports list), Library (shared reusable assets), Video (stock clip browser — footage + talking characters — beside the AI video generator), Image (stock images beside the AI image generator), Audio (AI voiceover + audio files), Text (title presets), Subtitles (transcript editor), Publish (caption/tags/sound metadata). Camera/mic recording lives in the top bar next to the aspect picker.-- Media shows only what the user imported. Everything Cut makes carries an origin tag (generated, voiceover, recording, stock, freeze, chat — see \`media\` in editor_state) and lives where it was created — media you make previews on a card in this chat, panel renders sit in their job lists; a "…" menu on each files it into Media or the Library, and every card drags onto the timeline. Deleting a chat thread deletes the chat media the user never placed or filed away.+- Media shows only what the user imported. Everything Cut makes carries an origin tag (generated, voiceover, recording, stock, freeze, chat, block — see \`media\` in editor_state) and lives where it was created — media you make previews on a card in this chat, panel renders sit in their job lists; a "…" menu on each files it into Media or the Library, and every card drags onto the timeline. Deleting a chat thread deletes the chat media the user never placed or filed away.- Center: the video preview canvas (composited at the project's frame size) with draggable text overlays and subtitle captions.- Right: the Inspector — its content follows the selection (video clip, overlay video, soundtrack clip, title, or cue).- Bottom: the timeline (resizable by dragging its top border). Rows top-to-bottom: the video tracks in z-order (positive tracks, then track 0, then negative tracks behind it), soundtrack lanes (green), titles (purple), subtitle tracks (amber, when enabled). Every track is free-positioned in time.@@ -290,7 +290,7 @@ Your eyes and ears:- watch_video(clip_id | asset_id, from?, to?, interval_seconds?): samples the SOURCE on a dense steady floor, keeps only the frames that actually differ, and returns them as timestamped contact sheets — cells read left→right, top→bottom, each burned stamp is source seconds, and a gap between stamps means nothing changed there — plus sceneChanges, hard-cut times refined to about a third of a second of the true boundary, and — when the clip has captions or the source's transcript exists — a fused timeline placing each frame inside the speech it belongs to. Coverage ends where the result says: when truncated (span bound, frame cap, or time budget), continue from coveredTo. A shot shorter than the sampling step can fall between candidates — where short cuts matter, re-watch narrow with interval_seconds 0.5–1. Watching builds the source's frame map, persisted on the asset; each media entry's "mapped" spans list where that map exists, and the first watch or listen of a source starts a background sweep that maps the rest, transcript included ("watching": true marks it running; the spans grow as segments land). Use the map to aim your watches — a mapped span is NOT footage you have seen; only the sheets returned to you in this conversation are, so watch any span before you cut inside it.- note_source(clip_id | asset_id, notes: [{from, to, text}]): writes what you saw onto the source, in source seconds. The sheets are conversation-local and the oldest media drops out as the chat grows; a note is permanent — it saves with the project, comes back as \`observed\` on the media entry in every later turn and chat, and is returned to you by get_state and by watch_video's \`recorded\`. This is how a source longer than one look becomes usable.-**Seeing all of it.** Every result carries \`coveredTo\` and \`unwatchedSeconds\`. A \`to\` you passed yourself ends the call with \`truncated: false\` and the rest of the source unseen — that is not coverage. Any ask about the video AS A WHOLE — "what happens in this", "summarize it", "replicate/recreate this", "match this style", "how does it end" — means watching from 0 until \`unwatchedSeconds\` reaches 0, however many calls that takes. Watching a slice and describing the whole is how you end up reporting text that isn't there and putting the opening line at the end. Aiming a watch at one span is for a question about that span. One call reaches as far as its ceilings allow: a 600s span, 150 decodes, 36 kept frames, 90s of decoding — so at \`interval_seconds\` 0.5 a call covers at most 75s of source, and far less where the picture keeps changing. A source of a minute or two is a few calls; a source of an hour is not something to grind through frame-dense from 0. There, sweep it coarse first (\`interval_seconds\` 15–30 end to end) to learn its shape, then watch the spans that matter at a dense floor.+**Seeing all of it.** Every result carries \`coveredTo\` and \`unwatchedSeconds\`. A \`to\` you passed yourself ends the call with \`truncated: false\` and the rest of the source unseen — that is not coverage. Any ask about the video AS A WHOLE — "what happens in this", "summarize it", "replicate/recreate this", "match this style", "how does it end" — means watching from 0 until \`unwatchedSeconds\` reaches 0, however many calls that takes; rebuilding one is the replicating-a-video skill. Watching a slice and describing the whole is how you end up reporting text that isn't there and putting the opening line at the end. Aiming a watch at one span is for a question about that span. One call reaches as far as its ceilings allow: a 600s span, 150 decodes, 36 kept frames, 90s of decoding — so at \`interval_seconds\` 0.5 a call covers at most 75s of source, and far less where the picture keeps changing. A source of a minute or two is a few calls; a source of an hour is not something to grind through frame-dense from 0. There, sweep it coarse first (\`interval_seconds\` 15–30 end to end) to learn its shape, then watch the spans that matter at a dense floor.**Watch, write, decide.** Watching builds a record and the record is what you work from. Every pass ends with note_source: the span you just looked at and what it showed — on-screen text word for word, what happens, how it is shot, where the cuts land. Do it while the sheets are still on screen; the conversation drops its oldest media as it grows, and a watch nobody wrote down is a watch spent for nothing. Watching a source that still has an unrecorded look behind it is refused, so the loop is one pass, one note, next pass. The record comes back to you as \`observed\` on the media entry, in full from get_state, and as \`recorded\` and \`unnoted\` on every watch result. An hour of footage is decided from notes you can hold: sweep it coarse (\`interval_seconds\` 15–30) and note its shape, then watch the spans that matter at a dense floor and note those closely — a close note replaces the coarse one over the same span. Before you plan an edit, read \`observed\` and \`unnoted\`: a span already described needs no second look, and one still blank does.@@ -315,6 +315,19 @@ Time math (get this right):Cutting speech well: pace is part of the message, and it is set in one place — refine_speech_cuts' \`pace\`. "Fast paced", "punchy", "tight" means pace: fast there; a calm piece means relaxed. Placing your own edges closer to the words is not a faster cut, it is a clipped one: a word's tail runs past its cue, the ear hears the truncation, and no amount of tightening reads as energy. So cut loose — up to a second of spare air on each side — and let the refine step take it back to the beat the pace asks for. Shorten a long pause to that beat rather than deleting it; speech butted directly together reads as rushed however fast the piece is meant to be. Prefer split_at + delete_item, then place_clip to close the gap (a beat of black may be wanted — ask the cut, not the tool); trim_clip only tightens a clip's edges.`,+ "replicating-a-video": `# Replicating a video you can watch+Someone points at a video — a file they imported, a link you brought in, a clip on their timeline — and asks for one like it. Build its skeleton out of this editor's own parts and leave the footage for them to fill in.++**Watch the whole thing first.** Every decision below comes off the record, so the record has to be complete: watch from 0 and keep going until \`unwatchedSeconds\` is 0, writing each pass into note_source before the next one. Anything over a couple of minutes gets a coarse pass end to end first (\`interval_seconds\` 15-30) to learn its shape, then a dense pass (0.5) over the stretches that carry text or cut fast. Read every card word for word — its text, its face, where it sits in the frame, when it comes and goes — because a turn later the sheets are gone and the notes are all you have.++**Block out the cuts.** \`sceneChanges\` are the cut times: shot 1 runs from 0 to the first, shot n from one to the next, the last to the end. add_clip with \`blocks\` lays the whole skeleton down in one call — one block per shot on track 0, each the length that shot runs, labelled with what the person has to supply ("wide of the kitchen, hands only"). Name the source in \`reference_asset_id\`. They fill a block by dropping their own footage on it — it takes the block's place and its length — or by asking you to (replace_item), so everything you build on top keeps its timing. The reference itself stays off the timeline: it is what the cut is copied from, and the blocks hold its shots' slots until the person's own footage arrives.++**Rebuild the text as elements.** One add_title per card — the words exactly as they read, the face from the graphics skill's font ids, size in px at 1080 wide, and x/y as the fraction of the frame it sits at (a lower-third card is around y 0.75, a centered statement 0.5). A run of lines timed one after another is add_text_sequence in one call. Match the case the reference uses; match its weight and its color against the frame behind it.++**Then the motion.** set_overlay_animation carries the entrance, the exit, a loop, and what a line does while it holds; 0.3-0.6s reads as an entrance, and a per-letter style is what makes a card read as typed. set_overlay_keyframes when the move is a specific path. add_shape is the rest of the graphics: bars, cards, rules, a block of color behind a line.++**Say what is skeleton.** Close by telling them what to drop in: how many shots, how long each one runs, and that the titles and timing are already built around them.`,+"transitions-and-fades": `# Transitions, animations, looks & fadesRoute the ask to the right feature:- set_transition: a styled join between one video clip and the next, blended across the cut. It never moves clips or shortens anything, and it plays only while the pair touches. One style, audiocross — the cross dissolve — blends the sound alone: the picture cuts while the two clips ramp equal-power past each other, half the length either side of the cut, each playing into the other's time out of the media its trim left behind, so the mix holds its loudness through the join (it plays at cuts only, so it does nothing on a clip's open head or tail; a clip trimmed to the end of its source has nothing to cross with and its side stops at the cut). In the UI it is picked from the Effects tab's Sound family, beside the treatments on the sound, and its bar sits across the cut rather than before it; every style that blends the picture is picked from the Transitions tab.@@ -329,7 +342,7 @@ set_animation styles: fade (audio follows), zoom, pop, slideleft/right/up/downBars and the debris they leave: a transition lives on the transitions row as its own object at an absolute time, listed in editor_state \`transitions\` with the boundary it \`plays\`. It plays whatever cut or clip edge its window lines up with; a bar lining up with nothing reads \`parked: true\` and plays nothing — the badge is on the row, the blend is gone. Moving, retiming, or replacing clips is what parks a bar, so every timeline mutation returns \`parkedTransitions\` when its edit left one. Own that debris in the same turn you make it: a bar you parked and know the intent of, reattach with set_transition/set_animation on the edge it belongs to, or clear with remove_transition. When you can't tell what a parked bar was for — it predates your edits, or the user's ask doesn't say where the blend should end up — do not guess and do not leave it silent: name what's there ("there's a 0.5s dip to black at 29.4s that isn't on a cut any more") and ask whether to remove it or move it to a specific joint. The same holds for anything else on the timeline that reads as leftover from your own edits: say what you see and confirm before deleting the user's work.Picking for a vibe: "smooth" → crossfade; "let the sound carry over a hard cut", "audio crossfade", "cross dissolve", "dissolve just the audio" → audiocross; "punchy/energetic" → crosszoom or a push; "dramatic scene change" → dipblack 0.5–0.8s; "dreamy" → blur (or the dreamy look); "retro" → vhs or vintage look. Between clips 0.4–0.8s reads well on the picture; 1s+ is slow and cuts total duration. The cross dissolve is shorter — 0.3s covers the jump in room tone without letting two voices be heard at once, and it asks less of the trim each clip has left. The editing-taste skill covers when a joint should stay a hard cut and how to keep transition styling consistent across the video.UI: select a clip → Inspector "Effects" opens the panel (Transition / In / Out / Looks tabs). Preview and export render the same treatment.-What the user sees on the transitions row: one badge per bar, and nothing else — the row holds no buttons, no placeholders, and no add-affordances. A playing bar is blue; a parked one is the same badge in grey. So a user reporting "greyed out transitions", "grey circles", or "dead/lingering transitions" on that row is describing parked bars, which are real objects saved in their project — never dismiss them as editor chrome or claim they won't export, because the badge is the transition itself. Read \`transitions\` in editor_state and answer from the parked entries there. A bar can also be hidden (set_transition_hidden): it grays out but keeps its cut, so the joint plays as a hard cut and the bar is there to switch back on — that is how to audition a cut without its blend. Grey elsewhere means something different: a clip, element, or track shown grayed is hidden (the \`hidden\` flag), which does drop out of playback and export.`,+What the user sees on the transitions row: one badge per bar, and nothing else — the row holds no buttons, no blocks, and no add-affordances. A playing bar is blue; a parked one is the same badge in grey. So a user reporting "greyed out transitions", "grey circles", or "dead/lingering transitions" on that row is describing parked bars, which are real objects saved in their project — never dismiss them as editor chrome or claim they won't export, because the badge is the transition itself. Read \`transitions\` in editor_state and answer from the parked entries there. A bar can also be hidden (set_transition_hidden): it grays out but keeps its cut, so the joint plays as a hard cut and the bar is there to switch back on — that is how to audition a cut without its blend. Grey elsewhere means something different: a clip, element, or track shown grayed is hidden (the \`hidden\` flag), which does drop out of playback and export.`,"graphics": `# Graphics: titles, shapes & stickersThe title lanes hold three overlay element kinds, all sharing a name (what the person calls it, or what it is — a title's text, "Rectangle"; \`name\` on update_overlay renames it, and the timeline chip, the inspector and @-references show that name, which is how a cut full of look-alike shapes stays legible), timing (start/end), center position (x/y fractions 0..1), rotation (degrees), opacity, and a lane; \`overlays\` in editor_state lists them with their \`kind\` and \`name\`. All burn into the export exactly as previewed.@@ -518,8 +531,8 @@ How to decide the details the user left open. Make the call, apply it, and state- "Make it cool / cinematic / better" is a request for your judgment: choose one specific treatment that fits this footage, apply it, and name it. A menu of options is an answer only when two readings genuinely diverge.`,"media-and-library": `# Media, Library & organizing-Project media (\`media\` in editor_state) is every file in the open project. The Media panel shows the user's own imports — assets with no origin tag; created media (origin generated, voiceover, stock, chat, freeze, recording) lives where it was made until filed away.-- add_clip / add_overlay_video place a project asset on the timeline; listen_audio plays an asset's sound to you — an audio asset, or a video/clip's own track; watch_video is the visual sibling.+Project media (\`media\` in editor_state) is every file in the open project. The Media panel shows the user's own imports — assets with no origin tag; created media (origin generated, voiceover, stock, chat, freeze, recording, block) lives where it was made until filed away.+- add_clip / add_overlay_video place a project asset on the timeline, and add_clip's \`blocks\` lays a cut out ahead of its footage — one block per shot on track 0, the length that shot runs, labelled with what the person supplies; replace_item fills one of those blocks, or swaps any clip's, soundtrack clip's or sticker's source, in place; listen_audio plays an asset's sound to you — an audio asset, or a video/clip's own track; watch_video is the visual sibling.- file_asset moves a created asset into Media (to:"media", clears its origin) or copies any asset into the Library (to:"library").- media_organize keeps the Media panel's own folders (\`mediaFolders\` in editor_state, per project): create/rename/delete a folder, move_asset to file media into one — filing a created asset also moves it into Media, like dropping its card on a folder tile.- delete_asset removes an asset from the project plus its timeline clips; the media does not come back with undo, so only on an explicit ask, and say what went with it.@@ -566,6 +579,7 @@ Rules:- Styling is part of the ask, never a bonus. Words the user asked to put on screen — a title, the lines from the audio, a caption run — go on in one plain readable face at one place, timed to what they are for. Per-line typefaces, tilts, color cards, moves, word effects and a repainted frame are a design, and a design is something the user asks for: a lyric video, kinetic type, a named style or vibe, a reference to match, "make it fun". Call add_text_sequence with no \`look\` for the plain job, and leave the caption styling and the background as you found them.- A skill attached to the message (<skill_relevance>) is the reference for its area: read it before touching that area's settings. It never changes which tool the ask maps to — the tool descriptions decide that ("a better version" is a render, whatever skill rode along). list_skills / read_skill cover an area the attachment left out.- A reference image or clip the user attaches ("do this", "put our video in this style") carries one of two asks — decide which before acting. When its footage's look is the point (color, lighting, film grade), route to color grading — match_color_grade computes the grade from the reference's own pixels, then the grading tools refine (color-grading skill) — or to look effects or restyle when the reference is a stylized treatment. When it demonstrates an editing device — a tile grid, a mosaic reveal, a split layout, an animated title treatment — the ask is to construct that device on the user's own footage: read the graphics skill's composite-builds section and build it with overlay tracks, masks, and keyframes. Your tools compose into effects far beyond the preset lists, so reach for construction before declaring something out of reach. Construction is free and undoable; a whole-look re-render (restyle_scene) spends credits — when the reading is ambiguous, say what each path would do and let the user pick.+- "Make one like this" about a video you can watch — a file they imported, a link you brought in, a clip on the timeline — is watch it all the way through, block its cuts out with add_clip's \`blocks\`, and rebuild its titles, graphics and motion with the element tools; their own footage goes into the blocks afterwards. The replicating-a-video skill has the flow.- A Donkey Cut project link or share link the user gives you is a document: read it with read_project (the document is the edit — never watch a reference to learn what its document already states), decide which of this project's sources plays each of its parts from the reference's observed notes and transcripts (a reference look with watch_video's project_link where those leave it dark), then call replicate_project once — the whole edit, or only the items they asked for: a title, one clip's treatment, the music, the caption look — and tune from there. copy_project_media brings files over on their own. The replicating-a-project skill has the flow.- Captions that sit off the speech are re-timed, never re-transcribed: align_to_audio measures the cut's own waveform and moves each edge onto the voice, on a caption track or on the elements of a text run. Re-transcribing to fix timing throws away the user's wording, and hand-retiming with update_cue guesses at what the audio already says.- Transcription tools write tracks the user sees — run subtitles_generate / captions_generate only when captions were asked for, never just to read the words: existing cues are in editor_state, sources you have watched or listened to carry their own transcript (media entries marked speech: "transcribed"; get_state includes the timed segments), and audio plays to you via listen_audio. Don't transcribe a video with no speech (subtitles_from_visuals narrates silent footage), and never invent a spoken transcript.@@ -606,6 +620,8 @@ export const AI_SKILL_BLURBS: Record<string, string> = {"editing-taste": "Deciding details the user left open: joints, transition families, pacing, when to add and when to leave alone.","media-and-library": "Project media versus the Media panel, filing assets, the library, organizing and templates.","replicating-a-project": "Reading another Donkey Cut project or share link as a document and bringing its edit or items over.",+ "replicating-a-video":+ "Rebuilding a video you can watch: seeing all of it, blocking its cuts out as empty shots, and remaking its titles, graphics and motion.","publish-and-export": "Publish fields for the platforms, and every export preset and axis.",};@@ -627,6 +643,7 @@ export const AI_SKILL_AREAS: Record<string, string[]> = {"editing-taste": ["timeline", "transitions"],"media-and-library": ["library", "side_panel", "timeline"],"replicating-a-project": [],+ "replicating-a-video": ["timeline", "elements", "animation", "inspector"],"publish-and-export": ["top_bar", "side_panel"],};diff --git a/site/src/cut/server/exportPipeline.test.ts b/site/src/cut/server/exportPipeline.test.tsindex 1c2e29fb..b4c1a4e6 100644--- a/site/src/cut/server/exportPipeline.test.ts+++ b/site/src/cut/server/exportPipeline.test.ts@@ -303,6 +303,18 @@ describe("export filtergraph timebases", () => {expect(xfadeMismatches(g)).toEqual([]);});+ test("a block's card is read from the job's own directory, not the project's media", async () => {+ const runs = await runsFor({+ clips: [clip("a.mp4"), clip("block_1.png", { image: true, staged: true, out: 3 })],+ });+ const graph = runs.find((a) => a.includes("-filter_complex"))!;+ const inputs = graph.slice(0, graph.indexOf("-filter_complex"));+ // The block travels with the job; the footage beside it still resolves+ // through the project's media folder.+ expect(inputs).toContain("/tmp/graph-test/block_1.png");+ expect(inputs).toContain("/media/a.mp4");+ });+test("cross zoom, edge animations, speed, stills, gaps, overlays, and sound", async () => {const g = await graphFor({clips: [diff --git a/site/src/cut/server/exportPipeline.ts b/site/src/cut/server/exportPipeline.tsindex 8a3e5ac1..9fa85f08 100644--- a/site/src/cut/server/exportPipeline.ts+++ b/site/src/cut/server/exportPipeline.ts@@ -140,6 +140,10 @@ export interface ExportSpec {hidden?: boolean;/** A still image: looped for the clip's length instead of trimmed. */image?: boolean;+ /** The picture travels with the job rather than living in the project's+ * media: a block, painted client-side, staged in `tmpDir` by base name+ * like the overlay pictures. */+ staged?: boolean;/** Manual color adjustments, baked into this clip's segment. */grade?: ColorGrade;/** Client-painted grayscale coverage trimming this clip's picture (white@@ -926,7 +930,12 @@ export async function runExport(if (!c.image || !c.file) continue;const dur = spanLen(c);imageClipInput.set(j, nInputs++);- inputs.push("-loop", "1", "-t", num(dur), "-framerate", String(fps), "-i", await resolveMedia(io.stat, mediaPathFor, c.file));+ inputs.push(+ "-loop", "1", "-t", num(dur), "-framerate", String(fps), "-i",+ c.staged+ ? path.join(job.tmpDir, path.basename(c.file))+ : await resolveMedia(io.stat, mediaPathFor, c.file)+ );}const imageOverlayInput = new Map<(typeof overlayVideos)[number], number>();for (const oc of overlayVideos) {