Open source CapCut alternative.
TypeScript
feat(site): pack lanes, sectioned state and sample peaks [rebuild]
David Le736df4fbc956
diff --git a/site/src/cut/components/AiPanel.tools.ts b/site/src/cut/components/AiPanel.tools.tsindex c40c8292..4a6b5036 100644--- a/site/src/cut/components/AiPanel.tools.ts+++ b/site/src/cut/components/AiPanel.tools.ts@@ -17,8 +17,10 @@ export const AI_PANEL_TOOLS = [{name: "get_state",description:- "Read the full current editor state: clips, soundtrack, overlay elements (titles, shapes, stickers), subtitles, selection, playhead, view settings, publish metadata. Use this whenever the context snapshot is not enough or might be stale.",- inputSchema: obj({}),+ "Read the full current editor state: clips, soundtrack, overlay elements (titles, shapes, stickers), subtitles, selection, playhead, view settings, publish metadata. Use this whenever the context snapshot is not enough or might be stale. A big cut read whole can run past what a result carries: name the sections you need (soundtrack, overlays, videoTrack, overlayVideo, transitions, subtitles, media, project, …) to check an edit.",+ inputSchema: obj({+ sections: { type: "array", items: { type: "string" }, description: "Top-level sections to return (the playhead always rides along); omit for everything" },+ }),},{name: "get_asset_info",@@ -29,7 +31,7 @@ export const AI_PANEL_TOOLS = [{name: "find_media",description:- `Find project media by name: every asset whose name or file name contains query (any case), optionally of one kind, with its id. The context snapshot lists at most ${FIND_MEDIA_SNAPSHOT_CAP} media items (mediaTruncated says when it cut the rest), so use this to reach the others — say the reference images a person just uploaded to a long project. Pages: pass the answer's next as offset for more.`,+ `Find project media by name: every asset whose name or file name contains query (any case), optionally of one kind, with its id. The context snapshot lists at most ${FIND_MEDIA_SNAPSHOT_CAP} media items (mediaTruncated says when it cut the rest), so use this to reach the others — say the reference images a person just uploaded to a long project. Pages: pass the answer's next as offset for more. Matches another chat thread made and hasn't placed are hidden from this chat; heldByOtherChats counts them, so never call such a file missing.`,inputSchema: obj({query: str("Text the name or file name contains (omit for all)"),type: { type: "string", enum: ["video", "image", "audio"], description: "Only this kind of media" },@@ -106,7 +108,7 @@ export const AI_PANEL_TOOLS = [{name: "measure_level",description:- "Measure how loud clips actually play — the level of each clip's audible sound over its trimmed range, in dBFS, with its current volume applied — so volumes are set from numbers, never by ear. Returns per clip: sourceDb (the media), volume, levelDb (what plays), loudestFrameDb (the loudest 20 ms frame as RMS — a sample peak sits higher, so it is no clipping margin), and for every clip except the target, volumeToMatch: the volume that lands it at the target's level; a clip whose source has no audio track comes back with noAudio instead of levels. \"Make the voiceover as loud as the clip\", \"match the levels\", \"balance the music under the speech\": call this with every clip involved and target_id = the one to match, then set_clip_volume / update_audio with volumeToMatch (for a bed under speech, a fraction of it). A shortfallDb means the volume ceiling (3) cannot reach the target; lower the target instead. A clip marked muted plays nothing until set_clip_muted / update_audio unmutes it; its level is what it would play unmuted. Each clip also reports integratedLufs (BS.1770 loudness over its range) and truePeakDbtp, both as it plays with its volume applied; a volume change of g dB moves both by g (volume × 10^(g/20)), so \"music 10 LU under the voice\" is a volume that puts the music's integratedLufs 10 below the voice's. mix: true measures the whole mix as an unmastered export hears it: integratedLufs, truePeakDbtp, loudnessRangeLu and maxShortTermLufs — check it against the export's loudness targets or a platform's spec. null means silence.",+ "Measure how loud clips actually play — the level of each clip's audible sound over its trimmed range, in dBFS, with its current volume applied — so volumes are set from numbers, never by ear. Returns per clip: sourceDb (the media), volume, levelDb (what plays), loudestFrameDb (the loudest 20 ms frame as RMS — a sample peak sits higher, so it is no clipping margin), loudestAt / loudestAtTimeline (when the loudest single sample plays, in source and timeline seconds, to the millisecond — put a click, hit or drop on a beat with these, never by probing silence thresholds) and samplePeakDb, and for every clip except the target, volumeToMatch: the volume that lands it at the target's level; a clip whose source has no audio track comes back with noAudio instead of levels. \"Make the voiceover as loud as the clip\", \"match the levels\", \"balance the music under the speech\": call this with every clip involved and target_id = the one to match, then set_clip_volume / update_audio with volumeToMatch (for a bed under speech, a fraction of it). A shortfallDb means the volume ceiling (3) cannot reach the target; lower the target instead. A clip marked muted plays nothing until set_clip_muted / update_audio unmutes it; its level is what it would play unmuted. Each clip also reports integratedLufs (BS.1770 loudness over its range) and truePeakDbtp, both as it plays with its volume applied; a volume change of g dB moves both by g (volume × 10^(g/20)), so \"music 10 LU under the voice\" is a volume that puts the music's integratedLufs 10 below the voice's. mix: true measures the whole mix as an unmastered export hears it: integratedLufs, truePeakDbtp, loudnessRangeLu and maxShortTermLufs — check it against the export's loudness targets or a platform's spec. null means silence.",inputSchema: obj({ids: { type: "array", items: { type: "string" }, description: "Clip ids to measure — video clips (their own sound) and soundtrack clips alike; optional with mix" },target_id: str("The clip whose level the others should match (default: the first id)"),diff --git a/site/src/cut/components/EffectsPanel.tools.ts b/site/src/cut/components/EffectsPanel.tools.tsindex 4f520e7b..47c7ea14 100644--- a/site/src/cut/components/EffectsPanel.tools.ts+++ b/site/src/cut/components/EffectsPanel.tools.ts@@ -60,7 +60,7 @@ export const EFFECTS_TOOLS = [focus_x: num("Zoom only: the x of the point to zoom into, 0..1 (default 0.5)"),focus_y: num("Zoom only: the y of the point to zoom into, 0..1 (default 0.5)"),...EFFECT_SETTING_ARGS,- lane: num("Element row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape."),+ lane: num("Element row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape. Use as few rows as the stacking needs: an element shares a row with anything it never overlaps in time."),}, ["effect"]),},] as const satisfies readonly AiToolDef[];diff --git a/site/src/cut/components/ElementsPanel.tools.ts b/site/src/cut/components/ElementsPanel.tools.tsindex dfe7fc7f..aa46b884 100644--- a/site/src/cut/components/ElementsPanel.tools.ts+++ b/site/src/cut/components/ElementsPanel.tools.ts@@ -52,7 +52,7 @@ export const ELEMENTS_TOOLS = [shadow_y: num(`Shadow drop, px at 1080, -${SHADOW_OFFSET_MAX}..${SHADOW_OFFSET_MAX}; 0 with a bright color makes a glow`),rotation: num("Degrees clockwise, -180..180, to a tenth (0.4 matches a slight lean)"),opacity: num("Whole-element opacity 0..1"),- lane: num("Element row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape."),+ lane: num("Element row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape. Use as few rows as the stacking needs: an element shares a row with anything it never overlaps in time."),}, ["shape"]),},{@@ -71,7 +71,7 @@ export const ELEMENTS_TOOLS = [h: num("Height, fraction of frame height; omit to keep the source's aspect"),rotation: num("Degrees clockwise, -180..180, to a tenth (0.4 matches a slight lean)"),opacity: num("Whole-element opacity 0..1"),- lane: num("Element row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape."),+ lane: num("Element row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape. Use as few rows as the stacking needs: an element shares a row with anything it never overlaps in time."),}),},{@@ -86,7 +86,7 @@ export const ELEMENTS_TOOLS = [y: num("Center y 0..1 (default 0.5)"),w: num("Width, fraction of frame width (default 0.25)"),h: num("Height, fraction of frame height; omit to keep the source's aspect"),- lane: num("Element row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape."),+ lane: num("Element row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape. Use as few rows as the stacking needs: an element shares a row with anything it never overlaps in time."),}, ["idea"]),},] as const satisfies readonly AiToolDef[];diff --git a/site/src/cut/components/Inspector.tools.ts b/site/src/cut/components/Inspector.tools.tsindex bdb05619..da8eadb6 100644--- a/site/src/cut/components/Inspector.tools.ts+++ b/site/src/cut/components/Inspector.tools.ts@@ -183,7 +183,7 @@ export const INSPECTOR_TOOLS = [hidden: bool("Hide the element without deleting it"),amount: num("Effects: strength 0.05..1 (see add_effect)"),...EFFECT_SETTING_ARGS,- lane: num("Move the element to another row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape."),+ lane: num("Move the element to another row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape. Use as few rows as the stacking needs: an element shares a row with anything it never overlaps in time."),}, []),},{@@ -278,11 +278,11 @@ export const INSPECTOR_TOOLS = [{name: "update_audio",description:- "Update a soundtrack clip: volume (0..3), fadeIn/fadeOut seconds, start position, lane, in/out trim, speed, reverse, hidden, or duck. Audio overlaps across lanes; clips on the same lane slide to its next free slot. To put music under narration, move music to lane 1 and start 0 in the same call, with narration on lane 0. `duck` is voiceover ducking — while this clip plays, ALL other audio drops to that gain (0..1); pass 1 to clear ducking.",+ "Update a soundtrack clip: volume (0..3), fadeIn/fadeOut seconds, start position, lane, in/out trim, speed, reverse, hidden, or duck. Audio overlaps across lanes; clips on the same lane slide to its next free slot. To put music under narration, move music to lane 1 and start 0 in the same call, with narration on lane 0. Keep the soundtrack packed: a sound shares lane 0 with every sound it never overlaps, and a higher lane is only for one that plays over another; the result names `freeLane` when a lower lane has room for the clip. `duck` is voiceover ducking — while this clip plays, ALL other audio drops to that gain (0..1); pass 1 to clear ducking.",inputSchema: obj({id: str("Soundtrack clip id"),ids: ids("id"),- lane: { type: "integer", minimum: 0, description: "Audio lane, 0-based. Separate lanes play together." },+ lane: { type: "integer", minimum: 0, description: "Audio lane, 0-based. Separate lanes play together; use the lowest lane free for the clip's span." },volume: num("0..3 (1 = unchanged, above 1 boosts)"),fadeIn: num("Fade-in seconds"),fadeOut: num("Fade-out seconds"),diff --git a/site/src/cut/components/SidePanel.tools.ts b/site/src/cut/components/SidePanel.tools.tsindex 086c6a3e..2f7d8bb8 100644--- a/site/src/cut/components/SidePanel.tools.ts+++ b/site/src/cut/components/SidePanel.tools.ts@@ -68,7 +68,7 @@ export const SIDE_PANEL_TOOLS = [{name: "delete_asset",description:- "Remove project assets and every timeline clip that uses them — the Media panel's trash. Pass asset_id for one, asset_ids for several in one call, or unused:true to clear every asset no timeline uses, open or parked (leftover imports, replaced footage, reference images) without touching placed media. Removed media does not come back with undo, so call it only when the user explicitly asked to remove that media, and report how many assets and clips went.",+ "Remove project assets and every timeline clip that uses them — the Media panel's trash. Pass asset_id for one, asset_ids for several in one call, or unused:true to clear every asset no timeline uses, open or parked (leftover imports, replaced footage, reference images) without touching placed media; another chat thread's unplaced media is out of this chat's reach, and the sweep counts it as heldByOtherChats. Removed media does not come back with undo, so call it only when the user explicitly asked to remove that media, and report how many assets and clips went.",inputSchema: obj({asset_id: str("Project asset id from `media`"),asset_ids: { type: "array", items: { type: "string" }, description: "Several project asset ids, removed in one call" },diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.tsindex 4e933c6a..cca4f1a3 100644--- a/site/src/cut/components/Timeline.tools.ts+++ b/site/src/cut/components/Timeline.tools.ts@@ -59,6 +59,14 @@ export const TIMELINE_TOOLS = [before_row: { type: "integer", minimum: 0, description: "Display row to insert before" },}, ["kind", "id", "before_row"]),},+ {+ name: "pack_lanes",+ description:+ "Close the timeline up onto the fewest lanes, changing lanes only — no start, trim or look moves. Soundtrack clips that never overlap in time share a lane. Elements keep the front-to-back order of every pair that overlaps, and effects keep rows of their own. Run it after building or rearranging a cut a piece at a time; it reports each band's lane count before and after. One undo step.",+ inputSchema: obj({+ band: { type: "string", enum: ["both", "soundtrack", "elements"], description: "Which band to pack; default both" },+ }),+ },{name: "split_at",description:@@ -101,7 +109,7 @@ export const TIMELINE_TOOLS = ["Stretches of this source, in SOURCE seconds — one clip each, in order. Gapped spans keep the moments and drop the rest; spans that meet cut the source at those boundaries.",items: obj({ from: num("Source start s"), to: num("Source end s") }, ["from", "to"]),},- lane: { type: "integer", minimum: 0, description: "Audio assets only: soundtrack lane, default 0. Put music on lane 1 to overlap narration on lane 0." },+ lane: { type: "integer", minimum: 0, description: "Audio assets only: soundtrack lane. Omit it and a sound placed with `start` lands at that moment on the lowest lane free there — lane 0 unless it overlaps another sound. Pass one only to stack on purpose; never open a lane per sound." },start: num("Timeline start s"),index: num("Insert position on video track 0 (video/image only; 0 = first)"),}),@@ -275,7 +283,7 @@ export const TIMELINE_TOOLS = [},rotation: num("Degrees clockwise, -180..180, to a tenth (0.4 matches a slight lean)"),opacity: num("Whole-element opacity 0..1"),- lane: num("Element row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape."),+ lane: num("Element row (0 = the front row, drawn over every higher row). Elements on one row never overlap — a title over a shape needs a lower row than the shape. Use as few rows as the stacking needs: an element shares a row with anything it never overlaps in time."),}, ["text"]),},{@@ -339,11 +347,15 @@ export const TIMELINE_TOOLS = [{name: "replace_item",description:- "Replace what an item plays, keeping the item: a video clip (including a blocked-out shot) keeps its place and its length and plays the asset you name instead, trimmed from that source's head; a soundtrack clip keeps its slot; a sticker shows the new image. This is how a block becomes the person's own footage (\"put the kitchen clip in shot 3\"), and how any clip swaps its source without moving what comes after. A block left playing nothing goes with it.",+ "Replace what an item plays, keeping the item: a video clip (including a blocked-out shot) keeps its place and its length and plays the asset you name instead, trimmed from that source's head; a soundtrack clip keeps its slot; a sticker shows the new image. This is how a block becomes the person's own footage (\"put the kitchen clip in shot 3\"), and how any clip swaps its source without moving what comes after. The clip keeps its framing and rotation; its mask goes with the old picture unless keep_mask names it the layer's own region. A block left playing nothing goes with it.",inputSchema: obj({id: str("The item to refill — a video clip, a soundtrack clip, or a sticker"),asset_id: str("Project asset that plays there instead"),+ keep_mask: {+ type: "boolean",+ description: "Video clips: keep the clip's mask because it cuts the layer to a region of the frame (a screen, a window) rather than covering something in the old footage",+ },},["id", "asset_id"]),diff --git a/site/src/cut/lib/aiContext.ts b/site/src/cut/lib/aiContext.tsindex e8ce0bee..59e58949 100644--- a/site/src/cut/lib/aiContext.ts+++ b/site/src/cut/lib/aiContext.ts@@ -9,7 +9,7 @@ import { chatOwner } from "./chatAssets";import { useGenerate } from "./generate";import { matteBakesAvailable, useMatteBakes } from "./removal/bakeJobs";import { cardMatteKey, resolveCardShape } from "./cameraCard";-import { deriveTransitionFields, getClipSpans, normalizeDocState, overlayLayers, projectDuration, resolveTransitions, totalDuration, useEditor } from "./store";+import { assetIdsInUse, deriveTransitionFields, getClipSpans, normalizeDocState, overlayLayers, projectDuration, resolveTransitions, totalDuration, useEditor } from "./store";import type { VideoProject } from "./genvideo/types";import { playheadAt, skimAt } from "./playhead";import { cueWordCount } from "./cueChunk";@@ -306,6 +306,7 @@ export function buildAiContext(opts?: { fullCues?: boolean; chatId?: string | nu...(core.mediaFolders ? { mediaFolders: core.mediaFolders } : {}),media: core.media,mediaTruncated: core.mediaTruncated,+ ...(core.mediaInOtherChats ? { mediaInOtherChats: core.mediaInOtherChats, mediaInOtherChatsNote: core.mediaInOtherChatsNote } : {}),// AI video renders for this project, live from the job store — what// "rendering" claims must be grounded in. A done render names the asset// it landed as (already in `media`); a failed one carries its error.@@ -411,9 +412,21 @@ interface DescribedState {timelines: Partial<Record<TimelineId, TimelineBody>>;}-/** The media a project's timeline plays: project content whichever chat made it. */-export const placedAssetIds = (s: { clips: VideoClip[]; audioClips: AudioClip[] }): Set<string> =>- new Set([...s.clips, ...s.audioClips].map((c) => c.assetId));+/** The media a project's timelines play, open or parked — clips, overlays,+ * transitions and caption fonts: project content whichever chat made it. */+export const placedAssetIds = (s: {+ clips: VideoClip[];+ audioClips: AudioClip[];+ overlays?: Overlay[];+ transitions?: TimelineTransition[];+ subtitles?: SubtitlesBlock;+ timelines?: Partial<Record<TimelineId, TimelineBody>>;+ assets?: readonly MediaAsset[];+}): Set<string> => {+ const used = assetIdsInUse({ ...s, overlays: s.overlays ?? [] });+ for (const body of Object.values(s.timelines ?? {})) for (const id of assetIdsInUse(body)) used.add(id);+ return used;+};/** An asset still owned by another chat: chat-made, tagged to a different* thread, and not yet moved onto the timeline. Placed media is project@@ -426,6 +439,11 @@ export const hiddenFromChat = (placed: Set<string>): boolean => a.origin === "chat" && !!a.chatId && a.chatId !== chatId && !placed.has(a.id);+/** What a chat is told about media it can't see: that it exists, so it never+ * says a file is missing from the project, and where it can be cleared. */+export const OTHER_CHATS_MEDIA_NOTE =+ "Some project media belongs to another chat thread, which made it and hasn't placed it, so this chat can't see or delete it. Tell the user it is there: they can clear it from that thread, or delete that thread from the chat history, which removes the unplaced media it owns.";+/** `live` marks the open project: the sweep's progress, the matte bakes in* flight, and another chat's private media are facts about this page, and* a document read as a reference has none of them. */@@ -441,6 +459,7 @@ function describeState(// A font asset is a typeface, not footage: it is listed under `fonts` with// the account's shelf fonts, where a font id is what the model needs.const visibleAssets = s.assets.filter((a) => !isLinkedAssetType(a.type) && !ownedByOtherChat(a));+ const otherChatsMedia = s.assets.filter((a) => !isLinkedAssetType(a.type) && ownedByOtherChat(a)).length;// The cap trims the tail of a huge media list, and a source someone has// watched and written up stays regardless of where it sits: the record is// what later decisions are made from.@@ -590,6 +609,7 @@ function describeState(: {}),})),mediaTruncated: shownAssets.length < visibleAssets.length,+ ...(live && otherChatsMedia > 0 ? { mediaInOtherChats: otherChatsMedia, mediaInOtherChatsNote: OTHER_CHATS_MEDIA_NOTE } : {}),videoTrack: spans.map((sp, index) => ({index,id: sp.clip.id,diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.tsindex a61efba2..8e6e344b 100644--- a/site/src/cut/lib/aiTools.ts+++ b/site/src/cut/lib/aiTools.ts@@ -15,7 +15,7 @@ import { captureRenderSnapshot, renderDoc } from "./renderSnapshot";import { submitPreviewSnapshot } from "./exportClient";import { ITEM_KIND_IDS, ITEM_KINDS, ROW_ITEM_KINDS, type ItemKind } from "./itemKinds";-import { insertItemRow } from "./laneTracks";+import { insertItemRow, packRows } from "./laneTracks";import { timelineItem } from "./timelineItems";import { movePreviewSelection, previewSelectionSnapshot } from "@/cut/lib/previewSelection";@@ -231,7 +231,7 @@ import { isStylePresetTemplate } from "./stylePresets";import { findLibraryAsset, listLibrary } from "./libraryList";import { loadLibraryLut, lutIdOf, lutLabel } from "./linkedLibrary";import { baseFrameAt, sampleClipBaseFrameData, sourceProfileOf, toBaseRendering } from "./baseFrame";-import { applyOverlayPatchSettled, assetClipUses, assetIdsOnAnyTimeline, clipLen, track0Clips, laneGapAt, getClipSpans, overlayLaneOrder, overlayLayers, parkedTransitions, projectDuration, resolveTransitions, totalDuration, useEditor } from "./store";+import { applyOverlayPatchSettled, assetClipUses, assetIdsOnAnyTimeline, clipLen, footprints, nextFreeStart, track0Clips, laneGapAt, getClipSpans, overlayLaneOrder, overlayLayers, parkedTransitions, projectDuration, resolveTransitions, totalDuration, useEditor } from "./store";import { playheadAt } from "./playhead";import { mixSpecFor, renderProjectFrame, renderProjectFrames } from "./exportRender";import { renderMix } from "./audioMix";@@ -239,7 +239,7 @@ import { measureLoudnessSliced, roundedLoudness, type LoudnessMeasure } from "./import { framesAt, readMediaFileSize } from "./mediaRead";import { renderStageFrame, storeStageStill } from "./stageFrame";import { createRasterCanvas, decodeRasterImageUrl, rasterCanvasToDataUrl } from "./raster";-import { buildAiContext, describeCameraCard, describeDoc, hiddenFromChat, placedAssetIds, roundT } from "./aiContext";+import { buildAiContext, describeCameraCard, describeDoc, hiddenFromChat, OTHER_CHATS_MEDIA_NOTE, placedAssetIds, roundT } from "./aiContext";import { CARD_SIDES, newCard, normalizeCard, type CardSide } from "./cameraCard";import { CAPTION_STYLES, captionEmphasis, cueSpot, EMPHASIS_SCALE_MAX, EMPHASIS_SCALE_MIN, laneCues, subtitleLaneCount } from "./subtitles";import { cueEmphasis, emphasisIndicesOf, emphasizedWords } from "./captionEmphasis";@@ -1153,10 +1153,20 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {return { id: next.id, which, style: anim?.style ?? "none", seconds: anim?.seconds ?? 0 };},- get_state: () => {+ get_state: (_s, input) => {// The tool result carries the whole transcript; the per-message context// snapshot trims it, so this is how the model pulls every cue when needed.- return buildAiContext({ fullCues: true });+ const full = buildAiContext({ fullCues: true }) as Record<string, unknown>;+ const want: unknown[] = Array.isArray(input.sections) ? input.sections : [];+ if (want.length === 0) return full;+ // A big cut read whole runs past what a tool result can carry; a read+ // that names its sections stays small enough to check an edit with.+ const names = Object.keys(full);+ const bad = want.filter((k) => typeof k !== "string" || !names.includes(k));+ if (bad.length > 0) throw new ToolError(`Unknown section ${bad.join(", ")}; sections are ${names.join(", ")}.`);+ return Object.fromEntries(+ names.filter((k) => k === "playhead" || want.includes(k)).map((k) => [k, full[k]])+ );},get_asset_info: async (s, input) => {@@ -1175,12 +1185,19 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {const chatId = chatOwner();const visible = s.assets.filter((a) => !isLinkedAssetType(a.type) && !hiddenFromChat(a, chatId, placed));const type = input.type === "video" || input.type === "image" || input.type === "audio" ? input.type : undefined;+ const query = typeof input.query === "string" ? input.query : undefined;const found = findMedia(visible, {- query: typeof input.query === "string" ? input.query : undefined,+ query,type,limit: isNum(input.limit) ? input.limit : undefined,offset: isNum(input.offset) ? input.offset : undefined,});+ // Matches the chat can't see still exist: it says so rather than calling+ // the file missing.+ const held = findMedia(+ s.assets.filter((a) => !isLinkedAssetType(a.type) && hiddenFromChat(a, chatId, placed)),+ { query, type }+ ).total;return {total: found.total,assets: found.page.map((a) => ({@@ -1193,6 +1210,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {...(a.folderId != null ? { folderId: a.folderId } : {}),})),...(found.next !== undefined ? { next: found.next } : {}),+ ...(held > 0 ? { heldByOtherChats: held, note: OTHER_CHATS_MEDIA_NOTE } : {}),};},@@ -1787,6 +1805,9 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {kind: "video" | "soundtrack";sourceDb: number;loudestFrameDb: number;+ loudestAt?: number;+ loudestAtTimeline?: number;+ samplePeakDb?: number;audibleSeconds: number;volume: number;levelDb: number;@@ -1797,6 +1818,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {const rows = new Map<string, Row | { id: string; name: string; kind: "video" | "soundtrack"; noAudio: string }>();await eachScanned(ids, async (id) => {const { asset, clip, from, to } = resolveWatchRange(s, { clip_id: id });+ const c = clip!;const soundtrack = s.audioClips.some((a) => a.id === clip!.id);const kind = soundtrack ? "soundtrack" : "video";if (asset.type === "image") {@@ -1820,6 +1842,16 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {kind,sourceDb: level.rmsDb,loudestFrameDb: level.loudestFrameDb,+ // Where the loudest sample lands — a click's hit, a drum's attack —+ // on the clip's source clock and on the timeline.+ ...(level.peakAt !== undefined+ ? {+ loudestAt: Math.round(level.peakAt * 1000) / 1000,+ loudestAtTimeline:+ Math.round((c.start + retimeOf(c).tAt(clamp(level.peakAt, c.in, c.out))) * 1000) / 1000,+ samplePeakDb: level.samplePeakDb,+ }+ : {}),audibleSeconds: level.audibleSeconds,volume: round2(volume),// A muted clip plays nothing; its level is what it would play unmuted.@@ -1994,6 +2026,17 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {return { ...tracksAfter() };},+ pack_lanes: (_s, input) => {+ const band = input.band === "soundtrack" || input.band === "elements" ? input.band : "both";+ const soundtrack = band === "elements" ? undefined : packRows("audio");+ const elements = band === "soundtrack" ? undefined : packRows("overlay");+ return {+ ...(soundtrack ? { soundtrack: { before: soundtrack.before, after: soundtrack.after } } : {}),+ ...(elements ? { elements: { before: elements.before, after: elements.after } } : {}),+ moved: (soundtrack?.moved ?? 0) + (elements?.moved ?? 0),+ };+ },+split_at: (s, input) => {const before = ITEM_KIND_IDS.reduce((n, kind) => n + ITEM_KINDS[kind].list(s).length, 0);s.splitAtPlayhead(isNum(input.t) ? input.t : undefined);@@ -3131,6 +3174,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {...patch,lane: landed.lane ?? 0,...("start" in patch ? { start: round2(landed.start) } : {}),+ ...audioPackHint(a.id),};},@@ -3427,7 +3471,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {if (asset.type !== "video" && asset.type !== "image")throw new ToolError(`"${asset.name}" is ${asset.type} — a video clip plays video or a still.`);const before = clipLen(clip);- s.replaceClipAsset(clip.id, asset.id);+ s.replaceClipAsset(clip.id, asset.id, { keepMask: input.keep_mask === true });const next = useEditor.getState().clips.find((c) => c.id === clip.id);if (!next || next.assetId !== asset.id) throw new ToolError("That clip could not take this source.");return {@@ -4264,7 +4308,14 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {const docAssets = ref.doc.assets ?? [];const chatId = chatOwner();// Another chat's unplaced media is not the reference's to hand over.- const placed = placedAssetIds({ clips: ref.doc.clips ?? [], audioClips: ref.doc.audioClips ?? [] });+ const placed = placedAssetIds({+ clips: ref.doc.clips ?? [],+ audioClips: ref.doc.audioClips ?? [],+ overlays: ref.doc.overlays ?? [],+ transitions: ref.doc.transitions ?? [],+ subtitles: ref.doc.subtitles,+ timelines: ref.doc.timelines ?? {},+ });const assets = ids.map((id) => {const a = docAssets.find((x) => x.id === id);if (!a || hiddenFromChat(a, chatId, placed))@@ -4730,12 +4781,14 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {const listed = Array.isArray(input.asset_ids) && input.asset_ids.length > 0;if (input.unused === true || listed) {let gone: MediaAsset[];+ let held = 0;if (input.unused === true) {const used = assetIdsOnAnyTimeline(s);const placed = placedAssetIds(s);const chatId = chatOwner();const spare = s.assets.filter((a) => !used.has(a.id) && !isLinkedAssetType(a.type));gone = spare.filter((a) => !hiddenFromChat(a, chatId, placed));+ held = spare.length - gone.length;} else {const want = [...new Set((input.asset_ids as unknown[]).filter((v): v is string => typeof v === "string"))];gone = want.map((id) => requireItem(s.assets, id, "project asset"));@@ -4747,6 +4800,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {names: gone.slice(0, 30).map((a) => a.name),...(gone.length > 30 ? { namesTruncated: true } : {}),clipsRemoved: uses,+ ...(held > 0 ? { heldByOtherChats: held, note: OTHER_CHATS_MEDIA_NOTE } : {}),};}const asset = requireItem(s.assets, input.asset_id, "project asset");@@ -6203,7 +6257,10 @@ function soundSpan(asset: MediaAsset, from: number, to: number | undefined) {/** How loud a source range plays, read where its sound is. */async function sourceLevel(asset: MediaAsset, from: number, to: number | undefined) {const heard = soundSpan(asset, from, to);- return measureSourceLevel(heard.asset.url, { from: heard.from, ...(heard.to !== undefined ? { to: heard.to } : {}) });+ const level = await measureSourceLevel(heard.asset.url, { from: heard.from, ...(heard.to !== undefined ? { to: heard.to } : {}) });+ // A bound recording plays on its own clock; the peak is reported on the+ // clip's.+ return level.peakAt === undefined ? level : { ...level, peakAt: level.peakAt - (heard.from - from) };}/** Silent stretches of a source range, in source seconds — the engine's@@ -6669,6 +6726,29 @@ const NOT_PLACED = { added: false, clipId: null, index: null };/** Place a project asset on the timeline like a drag: video/image onto track 0* (an `index` insert, a free-positioned `start`, or appended at the end),* audio onto the soundtrack. Returns the created clip's summary. */+/** The lowest soundtrack lane with room for `len` seconds at `start` itself,+ * leaving `skip` (the clip being moved) out of the count. */+function lowestFreeAudioLane(clips: AudioClip[], start: number, len: number, skip?: string): number {+ for (let lane = 0; ; lane++) {+ const taken = footprints(clips.filter((c) => c.id !== skip && (c.lane ?? 0) === lane));+ if (nextFreeStart(taken, start, len) < start + 1e-3) return lane;+ }+}++/** When a soundtrack clip sits on a higher lane than its span needs, the lane+ * it fits on, so a soundtrack built a sound at a time stays packed. */+function audioPackHint(id: string): { freeLane?: number; note?: string } {+ const all = useEditor.getState().audioClips;+ const c = all.find((x) => x.id === id);+ if (!c) return {};+ const free = lowestFreeAudioLane(all, c.start, clipLen(c), c.id);+ if (free >= (c.lane ?? 0)) return {};+ return {+ freeLane: free,+ note: `Lane ${free} is free for this sound's whole span. Keep the soundtrack packed: move it there with update_audio lane ${free} unless it is stacked on purpose.`,+ };+}+function audioLaneInput(value: unknown): number {if (!isNum(value) || !Number.isInteger(value) || value < 0)throw new ToolError("Audio lane must be a non-negative integer.");@@ -6805,7 +6885,14 @@ function placeAssetOnTimeline(asset: MediaAsset, input: Record<string, unknown>)const s = useEditor.getState();const start = isNum(input.start) ? Math.max(0, input.start) : undefined;if (asset.type === "audio") {- const lane = input.lane === undefined ? 0 : audioLaneInput(input.lane);+ // With no lane, a sound placed at a moment lands at that moment on the+ // lowest lane free there, rather than sliding down lane 0 off its mark.+ const lane =+ input.lane !== undefined+ ? audioLaneInput(input.lane)+ : start === undefined+ ? 0+ : lowestFreeAudioLane(s.audioClips, start, Math.max(0.1, asset.duration));s.addAudioFromAsset(asset.id, start, { lane });const sel = useEditor.getState().selection;const c =@@ -6813,7 +6900,7 @@ function placeAssetOnTimeline(asset: MediaAsset, input: Record<string, unknown>)? useEditor.getState().audioClips.find((x) => x.id === sel.id): undefined;if (!c) throw new ToolError("Could not create the soundtrack clip.");- return { id: c.id, kind: asset.type, lane: c.lane ?? 0, start: round2(c.start), len: round2(c.out - c.in) };+ return { id: c.id, kind: asset.type, lane: c.lane ?? 0, start: round2(c.start), len: round2(c.out - c.in), ...audioPackHint(c.id) };}let clipId: string | null;if (isNum(input.index)) {diff --git a/site/src/cut/lib/audioLaneTools.test.ts b/site/src/cut/lib/audioLaneTools.test.tsindex a8e3fd9e..583bec1c 100644--- a/site/src/cut/lib/audioLaneTools.test.ts+++ b/site/src/cut/lib/audioLaneTools.test.ts@@ -16,11 +16,11 @@ test("assistant can place narration and music together on separate lanes", asyncexpect(useEditor.getState().audioClips.map((a) => [a.start, a.lane ?? 0])).toEqual([[0, 0], [0, 1]]);});-test("assistant can repair music appended after narration in one edit", async () => {+test("assistant can set music under narration in one edit", async () => {await runAiTool("add_clip", { asset_id: "voice", start: 0 });await runAiTool("add_clip", { asset_id: "music", start: 0 });const music = useEditor.getState().audioClips.find((a) => a.assetId === "music")!;- expect(music.start).toBe(10);+ expect([music.start, music.lane]).toEqual([0, 1]);await runAiTool("update_audio", { id: music.id, lane: 1, start: 0, volume: 0.15 });const repaired = useEditor.getState().audioClips.find((a) => a.id === music.id)!;expect([repaired.start, repaired.lane, repaired.volume]).toEqual([0, 1, 0.15]);@@ -33,3 +33,30 @@ test("assistant refuses invalid audio lane indices", async () => {}expect(useEditor.getState().audioClips).toHaveLength(0);});++// A sound placed at a moment belongs at that moment, on the lowest lane free+// there. Sliding it down lane 0 moved a click off its beat, and the chat then+// opened a lane per sound: a cut with four sounds that never overlap came out+// four soundtrack lanes tall.+test("a sound added at a taken moment with no lane lands there, on the lowest free lane", async () => {+ await runAiTool("add_clip", { asset_id: "voice", start: 0 });+ const res = (await runAiTool("add_clip", { asset_id: "music", start: 2 })) as { lane: number; start: number };+ expect([res.start, res.lane]).toEqual([2, 1]);+});++test("sounds that never overlap share lane 0", async () => {+ await runAiTool("add_clip", { asset_id: "voice", start: 0 });+ await runAiTool("add_clip", { asset_id: "music", start: 12 });+ expect(useEditor.getState().audioClips.map((a) => [a.start, a.lane ?? 0])).toEqual([[0, 0], [12, 0]]);+});++test("a sound put on a higher lane than it needs is told which lane is free", async () => {+ await runAiTool("add_clip", { asset_id: "voice", start: 0 });+ const res = (await runAiTool("add_clip", { asset_id: "music", start: 12, lane: 2 })) as { note?: string; freeLane?: number };+ expect(res.freeLane).toBe(0);+ const id = useEditor.getState().audioClips.find((a) => a.assetId === "music")!.id;+ const moved = (await runAiTool("update_audio", { id, lane: 3 })) as { freeLane?: number };+ expect(moved.freeLane).toBe(0);+ const packed = (await runAiTool("update_audio", { id, lane: 0 })) as { freeLane?: number };+ expect(packed.freeLane).toBeUndefined();+});diff --git a/site/src/cut/lib/audioScan.test.ts b/site/src/cut/lib/audioScan.test.tsindex c836ad58..89a09dd9 100644--- a/site/src/cut/lib/audioScan.test.ts+++ b/site/src/cut/lib/audioScan.test.ts@@ -158,6 +158,23 @@ function expectWords(}describe("scanLevel", () => {+ test("says when the loudest sample plays, to the sample, across chunk joins", async () => {+ // A click's place was only known to the 20 ms frame, and a sound trimmed+ // to that frame's start cut the click's peak off.+ async function* spike(): AsyncGenerator<PcmChunk> {+ for (let c = 0; c < 4; c++) {+ const data = new Float32Array(RATE / 4).fill(0.1);+ if (c === 2) data[1234] = -0.9;+ yield { channels: [data], timestamp: c / 4, sampleRate: RATE };+ }+ }+ const level = await scanLevel(spike(), { from: 0 });+ expect(level.peakAt).toBeCloseTo(0.5 + 1234 / RATE, 4);+ expect(level.samplePeakDb).toBeCloseTo(20 * Math.log10(0.9), 1);+ // Outside the asked span the spike does not count.+ expect((await scanLevel(spike(), { from: 0.7 })).peakAt).toBeGreaterThanOrEqual(0.7);+ });+test("reads the level of what plays, leaving the pauses out", async () => {// Half-scale tone with a second of silence in the middle: the mean is// the tone's own level, and the silence does not drag it down.diff --git a/site/src/cut/lib/audioScan.ts b/site/src/cut/lib/audioScan.tsindex bbd9484b..6ffe727f 100644--- a/site/src/cut/lib/audioScan.ts+++ b/site/src/cut/lib/audioScan.ts@@ -162,6 +162,11 @@ export interface LevelScan {loudestFrameDb: number;/** Seconds of the span that read as audible. */audibleSeconds: number;+ /** When the loudest single sample plays, source seconds — where a click or+ * hit actually lands, to the sample. Absent for digital silence. */+ peakAt?: number;+ /** That sample's level, dBFS. */+ samplePeakDb?: number;}/**@@ -193,12 +198,37 @@ export function levelFrom(env: Envelope): LevelScan {};}-/** The level of `chunks`, by RMS over 20ms windows. */+/** The level of `chunks`, by RMS over 20ms windows, and when its loudest+ * sample plays. */export async function scanLevel(chunks: AsyncIterable<PcmChunk>,opts: { from: number; to?: number }): Promise<LevelScan> {- return levelFrom(await scanEnvelope(chunks, opts));+ const peak = { at: opts.from, abs: 0 };+ async function* tapped(): AsyncGenerator<PcmChunk> {+ for await (const chunk of chunks) {+ const rate = chunk.sampleRate;+ const n = chunk.channels[0]?.length ?? 0;+ // A decode opens before `from` and may run past `to`; only the asked+ // span counts.+ const a = Math.max(0, Math.round((opts.from - chunk.timestamp) * rate));+ const b = opts.to === undefined ? n : Math.min(n, Math.round((opts.to - chunk.timestamp) * rate));+ for (const ch of chunk.channels) {+ for (let i = a; i < b; i++) {+ const v = Math.abs(ch[i]);+ if (v > peak.abs) {+ peak.abs = v;+ peak.at = chunk.timestamp + i / rate;+ }+ }+ }+ yield chunk;+ }+ }+ const level = levelFrom(await scanEnvelope(tapped(), opts));+ return peak.abs > 0+ ? { ...level, peakAt: Math.round(peak.at * 10000) / 10000, samplePeakDb: round2(20 * Math.log10(peak.abs)) }+ : level;}/** Silent spans in `chunks`, by RMS over 20ms windows. */diff --git a/site/src/cut/lib/deleteAssetTool.test.ts b/site/src/cut/lib/deleteAssetTool.test.tsindex 69cd02b5..653f944f 100644--- a/site/src/cut/lib/deleteAssetTool.test.ts+++ b/site/src/cut/lib/deleteAssetTool.test.ts@@ -41,6 +41,15 @@ describe("delete_asset", () => {expect(ids()).toEqual(["placed", "parked"]);});+ test("the sweep leaves another chat's unused media and says how many it left", async () => {+ useEditor.setState({ assets: [...useEditor.getState().assets, { ...asset("theirs"), origin: "chat", chatId: "other-thread" }] });+ const out = (await runAiTool("delete_asset", { unused: true })) as { removed: number; heldByOtherChats?: number; note?: string };+ expect(out.removed).toBe(3);+ expect(ids()).toContain("theirs");+ expect(out.heldByOtherChats).toBe(1);+ expect(out.note).toMatch(/another chat/i);+ });+test("asset_ids removes several at once and counts the clips that went with them", async () => {const out = (await runAiTool("delete_asset", { asset_ids: ["spare1", "placed"] })) as { removed: number; clipsRemoved: number };expect(out).toMatchObject({ removed: 2, clipsRemoved: 1 });diff --git a/site/src/cut/lib/findMediaTool.test.ts b/site/src/cut/lib/findMediaTool.test.tsindex 484d9e40..1436accd 100644--- a/site/src/cut/lib/findMediaTool.test.ts+++ b/site/src/cut/lib/findMediaTool.test.ts@@ -65,6 +65,37 @@ describe("find_media", () => {expect(out.total).toBe(0);});+ test("a search that matches only another chat's media says it exists and where it lives", async () => {+ // Without this the chat told the user the files were not in the project.+ const out = (await runAiTool("find_media", { query: "private" })) as Found & { heldByOtherChats?: number; note?: string };+ expect(out.heldByOtherChats).toBe(1);+ expect(out.note).toMatch(/another chat/i);+ const none = (await runAiTool("find_media", { query: "gref_s6" })) as Found & { heldByOtherChats?: number };+ expect(none.heldByOtherChats).toBeUndefined();+ });++ test("another chat's media placed as an overlay, or on a parked timeline, is project content", async () => {+ // Pictures a chat set as overlays were hidden from every other thread.+ const base = useEditor.getState().assets;+ useEditor.setState({+ assets: [+ ...base,+ { ...image(98), id: "ovl", name: "chat_overlay.jpg", origin: "chat", chatId: "someone-else" },+ { ...image(97), id: "parked", name: "chat_parked.jpg", origin: "chat", chatId: "someone-else" },+ ],+ overlays: [{ id: "o1", kind: "sticker", assetId: "ovl", start: 0, end: 2 } as never],+ timelines: { second: { clips: [], audioClips: [], overlays: [{ id: "o2", kind: "sticker", assetId: "parked", start: 0, end: 2 }] } as never },+ });+ const out = (await runAiTool("find_media", { query: "chat_" })) as Found;+ expect(out.assets.map((a) => a.id).sort()).toEqual(["ovl", "parked"]);+ expect(buildAiContext({ chatId: null }).mediaInOtherChats).toBe(1);+ });++ test("the snapshot counts the media other chats hold", () => {+ expect(buildAiContext({ chatId: null }).mediaInOtherChats).toBe(1);+ expect(buildAiContext({ chatId: "someone-else" }).mediaInOtherChats).toBeUndefined();+ });+test("the chat is offered the tool", () => {expect(AI_PANEL_TOOLS.some((t) => t.name === "find_media")).toBe(true);});diff --git a/site/src/cut/lib/getStateSections.test.ts b/site/src/cut/lib/getStateSections.test.tsnew file mode 100644index 00000000..492fd25c--- /dev/null+++ b/site/src/cut/lib/getStateSections.test.ts@@ -0,0 +1,19 @@+import { expect, test } from "bun:test";+import { runAiTool } from "./aiTools";+import { useEditor } from "./store";++// A cut with seventy elements read back whole came out too large for a tool+// result, so the chat could not check its own lane moves. A read names the+// sections it needs.++test("get_state with sections returns only those, plus the playhead", async () => {+ useEditor.setState({ clips: [], overlays: [], transitions: [], assets: [],+ audioClips: [{ id: "s1", assetId: "a", start: 0, in: 0, out: 1, volume: 1 }] });+ const res = (await runAiTool("get_state", { sections: ["soundtrack", "overlays"] })) as Record<string, unknown>;+ expect(Object.keys(res).sort()).toEqual(["overlays", "playhead", "soundtrack"]);+});++test("an unknown section is named", async () => {+ const err = await runAiTool("get_state", { sections: ["lanes"] }).catch((e: unknown) => e);+ expect(err instanceof Error && err.message.includes("soundtrack")).toBe(true);+});diff --git a/site/src/cut/lib/laneTracks.ts b/site/src/cut/lib/laneTracks.tsindex 314957d1..aefeb21c 100644--- a/site/src/cut/lib/laneTracks.ts+++ b/site/src/cut/lib/laneTracks.ts@@ -39,7 +39,7 @@ import { getClipSpans, moveOverlayGroup, nextFreeStart, overlayLaneOrder, projecimport { ITEM_KINDS, type ItemKind, type ItemOf } from "./itemKinds";import { expandTimelineGroups } from "./timelineGroups";import { playheadAt } from "./playhead";-import { TRANSITION_MAX } from "./types";+import { isEffectOverlay, TRANSITION_MAX } from "./types";import type {AudioClip,MediaAsset,@@ -1140,6 +1140,55 @@ export function insertItemRow(kind: ItemKind, id: string, beforeRow: number): vocommitRow(band, id, beforeRow === 0 ? -1 : beforeRow === rows.length ? rows.length : beforeRow - 0.5);}+/**+ * Close a band up onto the fewest rows its timing allows, changing lanes only.+ * Sounds play together whatever their lane, so the soundtrack packs by time+ * alone. Elements stack by row, so every pair that overlaps keeps its+ * front-to-back order, and an effect never shares a row with another kind.+ * Rows are walked front first, each item landing on the frontmost row behind+ * everything it overlaps that stood in front of it. One undo step.+ */+export function packRows(kind: "audio" | "overlay"): { before: number; after: number; moved: number } {+ const s = useEditor.getState();+ const ad = ADAPTERS[kind];+ const raws = ad.raws(s);+ const views = raws.map((r) => ad.view(r));+ const isEffect = (i: number) => kind === "overlay" && isEffectOverlay(raws[i] as Overlay);+ const meets = (i: number, j: number) =>+ views[i].start < views[j].start + views[j].len - 1e-6 && views[j].start < views[i].start + views[i].len - 1e-6;+ const order = views+ .map((_, i) => i)+ .sort((a, b) =>+ kind === "audio"+ ? views[a].start - views[b].start+ : views[a].lane - views[b].lane || views[a].start - views[b].start+ );+ const row: number[] = [];+ const rowIsEffect = new Map<number, boolean>();+ const placed: number[] = [];+ for (const i of order) {+ let r = 0;+ if (kind === "overlay")+ for (const j of placed) if (views[j].lane < views[i].lane && meets(i, j)) r = Math.max(r, row[j] + 1);+ while (+ placed.some((j) => row[j] === r && meets(i, j)) ||+ (rowIsEffect.has(r) && rowIsEffect.get(r) !== isEffect(i))+ )+ r++;+ row[i] = r;+ rowIsEffect.set(r, isEffect(i));+ placed.push(i);+ }+ const before = new Set(views.map((v) => v.lane)).size;+ const after = new Set(row).size;+ const moved = views.filter((v, i) => v.lane !== row[i]).length;+ if (moved > 0 && ad.lanePatch) {+ s.pushHistory();+ ad.apply(raws.map((r, i) => ad.lanePatch!(r, row[i])));+ }+ return { before, after, moved };+}+/*** Land a new item on a display row of a multi-lane band — the row under a* drop, or one past either edge — by the lane arithmetic a lane drag commitsdiff --git a/site/src/cut/lib/measureLevelPeak.test.ts b/site/src/cut/lib/measureLevelPeak.test.tsnew file mode 100644index 00000000..9f3e4b4e--- /dev/null+++ b/site/src/cut/lib/measureLevelPeak.test.ts@@ -0,0 +1,41 @@+import { expect, test } from "bun:test";+import { stubModule } from "@/lib/testing/stubModule";+import type { MediaAsset } from "./types";++// measure_level says where a clip's loudest sample lands, on its own source+// clock and on the timeline, so a click can be put on a beat to the sample+// rather than probed for with silence thresholds 20 ms at a time.+await stubModule<typeof import("./media")>("./media", import.meta.url, {+ measureSourceLevel: (async () => ({+ rmsDb: -20,+ loudestFrameDb: -10,+ audibleSeconds: 0.2,+ peakAt: 0.639,+ samplePeakDb: -1.2,+ loudness: { integrated: -18, truePeak: -1, range: 2, momentaryMax: -12, shortTermMax: -14 },+ })) as never,+});++const { runAiTool } = await import("./aiTools");+const { useEditor } = await import("./store");++const sound: MediaAsset = { id: "key", fileName: "key.mp3", name: "key", type: "audio", duration: 0.9, url: "blob:key" };++test("measure_level reports when the loudest sample plays, in source and timeline seconds", async () => {+ useEditor.setState({+ projectId: "p1",+ loaded: true,+ readOnly: false,+ clips: [],+ audioClips: [{ id: "k", assetId: "key", start: 2, in: 0.6, out: 0.9, volume: 1 }],+ overlays: [],+ transitions: [],+ assets: [sound],+ });+ const res = (await runAiTool("measure_level", { ids: ["k"] })) as {+ clips: { id: string; loudestAt?: number; loudestAtTimeline?: number; samplePeakDb?: number }[];+ };+ expect(res.clips[0].loudestAt).toBeCloseTo(0.639, 3);+ expect(res.clips[0].loudestAtTimeline).toBeCloseTo(2.039, 3);+ expect(res.clips[0].samplePeakDb).toBe(-1.2);+});diff --git a/site/src/cut/lib/packLanes.test.ts b/site/src/cut/lib/packLanes.test.tsnew file mode 100644index 00000000..d9c36e75--- /dev/null+++ b/site/src/cut/lib/packLanes.test.ts@@ -0,0 +1,56 @@+import { beforeEach, expect, test } from "bun:test";+import { runAiTool } from "./aiTools";+import { useEditor } from "./store";+import type { AudioClip, Overlay } from "./types";++// Built a piece at a time, a cut ends up a lane per sound and a row per+// element: the chat once had to step 15 rows down one update at a time to+// close them up, and could not read the result back to check it.++const sound = (id: string, start: number, len: number, lane: number): AudioClip => ({ id, assetId: "a", start, in: 0, out: len, volume: 1, lane });+const shape = (id: string, start: number, end: number, lane: number): Overlay =>+ ({ id, kind: "shape", shape: "rect", start, end, x: 0.5, y: 0.5, w: 0.2, h: 0.2, fill: "#fff", lane }) as Overlay;+const flash = (id: string, start: number, end: number, lane: number): Overlay =>+ ({ id, kind: "effect", effect: "flash", start, end, x: 0.5, y: 0.5, lane }) as Overlay;++beforeEach(() => {+ useEditor.setState({+ clips: [], transitions: [], overlays: [], audioClips: [],+ assets: [{ id: "a", fileName: "a.mp3", name: "a", type: "audio", duration: 30, url: "" }],+ });+});++const laneOf = (id: string) => {+ const s = useEditor.getState();+ return ([...s.audioClips, ...s.overlays].find((x) => x.id === id)!.lane ?? 0);+};++test("sounds that never overlap close up onto one lane; overlapping ones keep two", async () => {+ useEditor.setState({ audioClips: [sound("s1", 0, 3, 0), sound("s2", 5, 1, 2), sound("s3", 6.5, 1, 3), sound("s4", 9, 20, 1), sound("s5", 10, 1, 4)] });+ const res = (await runAiTool("pack_lanes", {})) as { soundtrack: { before: number; after: number } };+ expect(res.soundtrack).toEqual({ before: 5, after: 2 });+ expect(["s1", "s2", "s3", "s4", "s5"].map(laneOf)).toEqual([0, 0, 0, 0, 1]);+});++test("elements keep the front-to-back order of every overlapping pair, and effects keep their own rows", async () => {+ useEditor.setState({+ overlays: [+ flash("fx", 0, 1, 0),+ shape("title", 2, 4, 3),+ shape("plate", 2, 4, 6),+ shape("later", 5, 6, 8),+ flash("fx2", 5, 6, 7),+ ],+ });+ const res = (await runAiTool("pack_lanes", { band: "elements" })) as { elements: { before: number; after: number } };+ expect(laneOf("fx")).toBe(laneOf("fx2"));+ expect(laneOf("title")).toBeLessThan(laneOf("plate"));+ expect(laneOf("later")).not.toBe(laneOf("fx2"));+ expect(res.elements).toEqual({ before: 5, after: 3 });+});++test("an already packed timeline is left alone", async () => {+ useEditor.setState({ audioClips: [sound("s1", 0, 3, 0)] });+ const res = (await runAiTool("pack_lanes", {})) as { moved: number };+ expect(res.moved).toBe(0);+});diff --git a/site/src/cut/lib/replaceClipMask.test.ts b/site/src/cut/lib/replaceClipMask.test.tsnew file mode 100644index 00000000..58bc34f6--- /dev/null+++ b/site/src/cut/lib/replaceClipMask.test.ts@@ -0,0 +1,36 @@+import { beforeEach, expect, test } from "bun:test";+import { runAiTool } from "./aiTools";+import { useEditor } from "./store";+import type { VideoClip } from "./types";++// A pen mask cut to a laptop's screen is a region of the frame, not cover for+// something in the old footage: swapping in a new screen recording dropped it,+// and the chat had to draw the four corners back on.++const screen = { kind: "pen" as const, points: [{ x: -0.5, y: -0.18 }, { x: 0.5, y: -0.18 }, { x: 0.5, y: 0.17 }, { x: -0.48, y: 0.18 }], feather: 0 };++beforeEach(() => {+ useEditor.setState({+ overlays: [], transitions: [], audioClips: [],+ assets: [+ { id: "old", fileName: "old.mov", name: "old", type: "video", duration: 37, url: "" },+ { id: "new", fileName: "new.mov", name: "new", type: "video", duration: 37, url: "" },+ ],+ clips: [],+ });+});++const clip = (mask: VideoClip["mask"]): VideoClip =>+ ({ id: "c", assetId: "old", start: 0, in: 0, out: 30, track: 1, fit: "fill", mask }) as VideoClip;++test("a mask named as the layer's region stays on the clip when its media is replaced", async () => {+ useEditor.setState({ clips: [clip(screen)] });+ await runAiTool("replace_item", { id: "c", asset_id: "new", keep_mask: true });+ expect(useEditor.getState().clips[0].mask).toEqual(screen);+});++test("without keep_mask the mask goes with the old picture", async () => {+ useEditor.setState({ clips: [clip(screen)] });+ await runAiTool("replace_item", { id: "c", asset_id: "new" });+ expect(useEditor.getState().clips[0].mask).toBeUndefined();+});diff --git a/site/src/cut/lib/store.ts b/site/src/cut/lib/store.tsindex 0f0e5fa7..bcaba706 100644--- a/site/src/cut/lib/store.ts+++ b/site/src/cut/lib/store.ts@@ -635,7 +635,7 @@ export interface EditorState {* trimmed from the source's head to the block's length (a shorter source* makes the clip its own length). The block it replaced is dropped with it* when nothing else plays it. */- replaceClipAsset: (clipId: string, assetId: string) => void;+ replaceClipAsset: (clipId: string, assetId: string, opts?: { keepMask?: boolean }) => void;/** Add a soundtrack clip from an audio asset at `start` (default: the* playhead). `opts.duck` marks it a voiceover that lowers everything else* to that gain while it plays; `opts.lane` picks the audio track it lands@@ -2940,7 +2940,7 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {return clip.id;},- replaceClipAsset: (clipId, assetId) => {+ replaceClipAsset: (clipId, assetId, opts) => {const clip = get().clips.find((c) => c.id === clipId);const asset = get().assets.find((a) => a.id === assetId);if (!clip || !asset || (asset.type !== "video" && asset.type !== "image")) return;@@ -2964,10 +2964,11 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {speed: undefined,speedCurve: undefined,reverse: undefined,- // Coverage painted frame by frame for the old picture, and a- // key pulled off its pixels, mean nothing to the new one. A- // subject matte is computed from whatever plays, so it stays.- ...(clip.mask && clip.mask.kind !== "subject" ? { mask: undefined } : {}),+ // Coverage drawn over the old picture means nothing to the+ // new one. A subject matte is computed from whatever plays, so+ // it stays, and so does a mask the caller names as the+ // layer's own region (a screen a recording is cut to).+ ...(clip.mask && clip.mask.kind !== "subject" && !opts?.keepMask ? { mask: undefined } : {}),removal: undefined,// The card's layout stays; its matte belongs to the old// picture, and the new one bakes its own.diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.tsindex 3facd5af..7f53a265 100644--- a/site/src/cut/server/ai/catalog.ts+++ b/site/src/cut/server/ai/catalog.ts@@ -517,7 +517,7 @@ ${textMoveCatalog()}"audio-and-subtitles": `# Audio, voiceover & subtitlesCleaning a transcript: find_filler names the filler words on a track with their cue ids and times; tidy the captions with update_cue, or cut the speech itself with the timeline tools.Lyrics and scripts the user already has: sync_lyrics times their exact words against the audio (transcribing first when needed) and writes them to a caption track with per-word timings; set_caption_look then dresses the track — preset, size, font, word effect, accent color — from a text-videos look id or field by field. The text-videos skill has the whole flow, including turning the same timed lines into full-frame cards.-Soundtrack clips: volume 0..3 (1 = unchanged, above 1 boosts), fadeIn/fadeOut seconds (max half the clip), start = timeline position, in/out = trim inside the source; clips can spread across several soundtrack lanes (the \`lane\` field), new sounds slide to free space in their lane. Fades render with ffmpeg afade on export.+Soundtrack clips: volume 0..3 (1 = unchanged, above 1 boosts), fadeIn/fadeOut seconds (max half the clip), start = timeline position, in/out = trim inside the source; clips can spread across several soundtrack lanes (the \`lane\` field), new sounds slide to free space in their lane. Keep the soundtrack packed: every sound sits on the lowest lane free for its span, so sounds that never overlap share lane 0 and a second lane opens only for a sound that plays over another. Fades render with ffmpeg afade on export.Matching levels: "as loud as", "match", "balance", "sit under" are measurements, never guesses. measure_level with every clip involved and target_id = the one to match returns each clip's playing level and the volumeToMatch that lands it on the target; set that with set_clip_volume / update_audio (a fraction of it for a bed under speech). Listening tells you how a sound feels, never how many dB it is. Loudness asks in LUFS ("music 10 LU under the voice", "is this at −14 for YouTube") read integratedLufs and truePeakDbtp from measure_level; mix: true measures the whole mix, and the export's Loudness setting masters it to a target.Ducking: a soundtrack clip's \`duck\` (0..1, via update_audio) lowers ALL other audio — video-clip sound and other music — to that gain while the clip plays; 1 clears it. Voiceovers set this so narration sits over quieter music. It applies in both the preview and the export.Sound quality (set_clip_sound): a clip's own treatment — video clip or soundtrack clip — in three stages the Inspector's Audio tab shows as Sound quality: a ${SOUND_EQ_BANDS.length}-band equalizer (${SOUND_EQ_BANDS.map((b) => b.label).join(", ")}; dB per band), a compressor (threshold dB, ratio, attack ms, release ms) and a limiter (dBFS ceiling). It runs on that clip alone, before its fades and any ducking, in the preview and every export. "Make the voice clearer / more podcast-like / less muddy / more even in level / stop it clipping" is this tool: pick the shipped preset whose character fits the ask — ${SOUND_PRESETS.filter((p) => p.sound).map((p) => `\`${p.id}\` (${p.character})`).join(", ")} — with \`studio\` as the default for a talking-head voice; \`preset: "flat"\` clears it; the user's own saved presets apply by name; or set the stages directly, null clearing one. save_sound_preset stores a clip's treatment in the Library under a name for reuse. A treatment shapes one clip's sound; an effect element treats a stretch of the mix.@@ -647,6 +647,7 @@ Rules:- When the user says "this" (this clip, this text), they mean the current selection. A "this" or "that" with nothing selected and several candidates is genuinely ambiguous — ask which one before touching anything.- Keep replies short and concrete — one or two sentences. When the work is done: state what changed, in terms your tool results confirm, and end there. You may close with at most one suggestion, phrased as a statement ("A 0.5s crossfade on the middle cut would smooth the jump."); a suggestion the user has already passed on stays retired. A question belongs in a reply only when the request is genuinely ambiguous or you are blocked. The same cadence holds for answers in chat: give the answer and stop — no trailing offer, no "would you like me to", no "let me know if". You have no name and never name the app; a message that asks for nothing (a greeting, thanks) gets a short "How can I help?" / "What would you like to do?" back — words alone, no tool calls. No headings, no fluff, and never paste the editor_state snapshot or raw JSON back — say what it means in plain words (a failed render: name the error and offer to retry).- Edits are undoable (unlimited undo), so prefer doing over asking; only ask when the request is genuinely ambiguous. Ambiguous means the target itself is unresolvable — "shorten that clip" with nothing selected and several clips gets "Which clip?" and no edit; guessing a target and editing it is worse than the question. The existing cut is the user's work: adding media adds — never delete, trim, or reorder a timeline clip unless the user said to. Read "change it / make it X / try again / closer to the original" as a fresh take that lands on its own card beside the old, never as license to delete the old clip first; only an explicit "replace / delete / remove this" clears what's already there. Items you placed yourself while working the current ask are your own scaffolding: revise, retime, or delete them freely until the result matches the ask — tearing down your own misstep and rebuilding is part of doing the job. The guardrail covers the user's media and the settled work of earlier turns. Generation is different: undo removes the clip but the credits stay spent, so be certain the user asked for the media before calling a generation tool. Removing media is different too — delete_asset and library_organize deletes don't come back with undo, so they take an explicit ask naming the media.+- Keep the timeline as compact as its content allows. Every clip, sound and element goes on the lowest lane or row where it fits — sounds that never overlap share one soundtrack lane, elements that never overlap share one row — and a new lane opens only for something that plays over another or must stack in front of it (an effect still keeps rows of its own). When a build or an edit leaves lanes with room, pack_lanes closes them up in one call.- Times are seconds. A \`@1:34\` in the user's message is a moment on the timeline — minutes and seconds — that they marked while writing it; it is kept pointing at the same picture as the cut changes, so read it as the timeline second it names right now. The frame follows project.aspect in editor_state — any "W:H" ratio, short side 1080px; project.frame has the exact pixels. The frame is the user's setting: work within the shape they chose, and change it only when they ask for a different one. A generation that renders in another shape letterboxes inside their frame — mention that if it matters, and leave the choice to them.- project.safeZones in editor_state gives, for this frame, the safe area of every guide preset (frame fractions, x/y/w/h) and the keepOut boxes where TikTok, Reels and Shorts draw their own UI or crop the frame's sides — known whether or not a guide shows. When the user names a platform, asks for a safe zone, or wants graphics kept clear of the app's UI, place every title, sticker and caption inside safeZones.shortform.safeArea and check with capture_frame; safeZones.margins is the general 5% inset. A safeArea is a rect — x/y its top-left corner, w/h its size — while an element's x/y is its center: centered in the safe area means x + w/2, y + h/2, and an element of width ew stays inside while its center is between x + ew/2 and x + w − ew/2. Guides are a view: set_guides shows one when the user asks to see it, and showing one changes nothing in the cut.- Exercise judgment on details the user left open, and state the choice you made. Judgment fills in the parameters of the asked-for change (style, length, placement); the scope stays the ask, and extra tools, tracks, or treatments beyond it wait for the user to ask. Editing defaults: a hard cut at every joint; when transitions are asked for, one style family per video at 0.4–0.6s. The editing-taste skill carries the full doctrine. Editing a recording of someone speaking to camera is the talking-head-videos skill; the ask decides which of its steps run.