Skip to content

Open source CapCut alternative.

TypeScript

feat(site): teach the chat the smooth flag on set_speed and set_speed_curve

David Le2ae78dae223f
diff --git a/site/src/cut/components/Inspector.tools.ts b/site/src/cut/components/Inspector.tools.ts
index 41b49c98..8b4914a9 100644
--- a/site/src/cut/components/Inspector.tools.ts
+++ b/site/src/cut/components/Inspector.tools.ts
@@ -283,13 +283,13 @@ export const INSPECTOR_TOOLS = [
{
name: "set_speed",
description:
- "Set a video clip's playback speed, one rate across the whole clip, and/or play it backward. Faster shortens the clip on the timeline; slower stretches it. Later clips, titles, captions, and soundtrack shift to stay in sync. A clip carrying a speed curve loses it when speed is passed: the rate becomes uniform again (set_speed_curve is the tool for a rate that changes through the footage). `reverse: true` plays the clip's trim backward, picture and sound, at its rate — the clip keeps its length and its curve, and its head now shows source `out`; `reverse: false` turns it forward again.",
- inputSchema: obj({ clipId: str("Video clip id"), speed: num("Playback rate (1 = normal, no upper limit)"), reverse: bool("Play the footage backward (true) or forward (false)") }, ["clipId"]),
+ "Set a video clip's playback speed, one rate across the whole clip, and/or play it backward. Faster shortens the clip on the timeline; slower stretches it. Later clips, titles, captions, and soundtrack shift to stay in sync. A clip carrying a speed curve loses it when speed is passed: the rate becomes uniform again (set_speed_curve is the tool for a rate that changes through the footage). `reverse: true` plays the clip's trim backward, picture and sound, at its rate — the clip keeps its length and its curve, and its head now shows source `out`; `reverse: false` turns it forward again. `smooth: true` smooths the clip's slow motion: wherever its rate is under 1×, the frames between source frames are synthesized (the export estimates the motion between them) so the slow stretch moves without stepping; at 1× and above nothing changes. Turn it on for real slow motion from ordinary footage; `smooth: false` shows the source frames as they are.",
+ inputSchema: obj({ clipId: str("Video clip id"), speed: num("Playback rate (1 = normal, no upper limit)"), reverse: bool("Play the footage backward (true) or forward (false)"), smooth: bool("Smooth slow motion: synthesize the frames between source frames wherever the rate is under 1×") }, ["clipId"]),
},
{
name: "set_speed_curve",
description:
- `Give a video clip (any track) a speed curve: its rate changes through the footage instead of being one number, so one clip can race through a walk, land on a beat, and hold on a face. Nodes are {at, speed}: \`at\` in SOURCE seconds inside the clip's trim — the unit detect_beats, trim_clip and watch_video speak, so beat times drop straight in — and speed ${SPEED_CURVE_MIN}–${SPEED_CURVE_MAX}×. The rate curves smoothly between nodes and holds flat past the outermost ones; the whole list replaces the clip's curve, and an empty list clears it (the clip keeps its length at one uniform rate). Or pass a preset to lay a named ramp over the trim: ${speedCurvePresetCatalogText()}. Technique: give a peak a beat of normal speed on either side or it reads as a glitch; below about 0.5× ordinary footage shows its own frame rate, so real slow motion wants high-frame-rate source (say so instead of shipping judder); sound stretches with its pitch kept, which holds up on speech to roughly 0.5–2×. The clip's timeline length becomes the integral of the curve and later items shift like a trim. Returns the resulting nodes, the clip's length before and after, and the rows that moved. In the UI the Speed Curve row opens the same graph over the timeline.`,
+ `Give a video clip (any track) a speed curve: its rate changes through the footage instead of being one number, so one clip can race through a walk, land on a beat, and hold on a face. Nodes are {at, speed}: \`at\` in SOURCE seconds inside the clip's trim — the unit detect_beats, trim_clip and watch_video speak, so beat times drop straight in — and speed ${SPEED_CURVE_MIN}–${SPEED_CURVE_MAX}×. The rate curves smoothly between nodes and holds flat past the outermost ones; the whole list replaces the clip's curve, and an empty list clears it (the clip keeps its length at one uniform rate). Or pass a preset to lay a named ramp over the trim: ${speedCurvePresetCatalogText()}. Technique: give a peak a beat of normal speed on either side or it reads as a glitch; below about 0.5× ordinary footage shows its own frame rate, so pass \`smooth: true\` with the curve — it synthesizes the frames between source frames wherever the rate is under 1× and leaves the faster stretches alone — unless the source is high-frame-rate already; sound stretches with its pitch kept, which holds up on speech to roughly 0.5–2×. The clip's timeline length becomes the integral of the curve and later items shift like a trim. Returns the resulting nodes, the clip's length before and after, and the rows that moved. In the UI the Speed Curve row opens the same graph over the timeline.`,
inputSchema: obj(
{
clipId: str("Video clip id"),
@@ -309,6 +309,7 @@ export const INSPECTOR_TOOLS = [
enum: SPEED_CURVE_PRESET_IDS,
description: "A named ramp laid over the clip's trim, instead of nodes",
},
+ smooth: bool("Smooth slow motion: synthesize the frames between source frames wherever the curve is under 1×"),
},
["clipId"]
),
diff --git a/site/src/cut/lib/aiContext.ts b/site/src/cut/lib/aiContext.ts
index 3f911e68..3b62b51f 100644
--- a/site/src/cut/lib/aiContext.ts
+++ b/site/src/cut/lib/aiContext.ts
@@ -482,9 +482,9 @@ export function buildAiContext(opts?: { fullCues?: boolean; chatId?: string | nu
/** A clip's rate as the model reads it: one number, or the average plus the
* curve's nodes as [sourceSeconds, rate] when the rate changes through the
* footage. */
-function describeRate(c: { speed?: number; speedCurve?: SpeedNode[]; reverse?: boolean; in: number; out: number }) {
+function describeRate(c: { speed?: number; speedCurve?: SpeedNode[]; reverse?: boolean; smoothSlow?: boolean; in: number; out: number }) {
const nodes = speedCurveOf(c);
- const reverse = c.reverse ? { reverse: true } : {};
+ const reverse = { ...(c.reverse ? { reverse: true } : {}), ...(c.smoothSlow ? { smooth: true } : {}) };
if (!nodes) return { speed: r(c.speed ?? 1), ...reverse };
return {
speed: r(retimeOf(c).rate),
@@ -541,7 +541,7 @@ function describeOverlayClip(c: VideoClip, assets: Map<string, { name: string }>
: {}),
...(c.rotation ? { rotation: c.rotation } : {}),
...((c.opacity ?? 1) < 1 ? { opacity: r(c.opacity ?? 1) } : {}),
- ...(retimeOf(c).rate !== 1 || c.speedCurve || c.reverse ? describeRate(c) : {}),
+ ...(retimeOf(c).rate !== 1 || c.speedCurve || c.reverse || c.smoothSlow ? describeRate(c) : {}),
...(c.grade ? { colorGrade: c.grade } : {}),
...(c.mask ? { mask: c.mask } : {}),
...(c.boxStyle ? { boxStyle: c.boxStyle } : {}),
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index 054bff3b..31f865d2 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -955,6 +955,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
out: round2(clip.out),
speed: round2(speed),
...(clip.reverse ? { reverse: true } : {}),
+ ...(clip.smoothSlow ? { smooth: true } : {}),
...(speedCurveOf(clip)
? {
speedCurve: speedCurveOf(clip)!.map(([at, rate]) => [round2(at), round2(rate)]),
@@ -3585,16 +3586,19 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
set_speed: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
const reverse = typeof input.reverse === "boolean" ? input.reverse : undefined;
- if (!isNum(input.speed) && reverse === undefined)
- throw new ToolError("speed (e.g. 1.5) or reverse is required.");
+ const smooth = typeof input.smooth === "boolean" ? input.smooth : undefined;
+ if (!isNum(input.speed) && reverse === undefined && smooth === undefined)
+ throw new ToolError("speed (e.g. 1.5), reverse, or smooth is required.");
const before = clipLen(clip);
if (isNum(input.speed)) s.setClipSpeed(clip.id, input.speed);
if (reverse !== undefined) s.setClipReverse(clip.id, reverse);
+ if (smooth !== undefined) s.setClipSmoothSlow(clip.id, smooth);
const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
return {
id: next.id,
speed: next.speed ?? 1,
...(next.reverse ? { reverse: true } : {}),
+ ...(next.smoothSlow ? { smooth: true } : {}),
lenBefore: round2(before),
lenAfter: round2(clipLen(next)),
...tracksAfter(),
@@ -3633,11 +3637,14 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
s.setClipSpeedCurve(clip.id, nodes);
}
}
+ // After the curve, so a rejected curve leaves the flag alone too.
+ if (typeof input.smooth === "boolean") s.setClipSmoothSlow(clip.id, input.smooth);
const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
const curve = speedCurveOf(next);
return {
id: next.id,
speed: round2(retimeOf(next).rate),
+ ...(next.smoothSlow ? { smooth: true } : {}),
nodes: curve ? curve.map(([at, speed]) => ({ at: round2(at), speed: round2(speed) })) : [],
lenBefore: round2(before),
lenAfter: round2(clipLen(next)),
@@ -4140,7 +4147,7 @@ function resolveWatchTarget(
input: Record<string, unknown>
): {
asset: MediaAsset;
- clip: { id: string; start: number; in: number; out: number; speed?: number; speedCurve?: SpeedNode[]; reverse?: boolean } | null;
+ clip: { id: string; start: number; in: number; out: number; speed?: number; speedCurve?: SpeedNode[]; reverse?: boolean; smoothSlow?: boolean } | null;
} {
if (input.clip_id !== undefined && input.clip_id !== null) {
const id = String(input.clip_id);
@@ -4167,7 +4174,7 @@ function resolveWatchRange(
): {
projectId: string;
asset: MediaAsset;
- clip: { id: string; start: number; in: number; out: number; speed?: number; speedCurve?: SpeedNode[]; reverse?: boolean } | null;
+ clip: { id: string; start: number; in: number; out: number; speed?: number; speedCurve?: SpeedNode[]; reverse?: boolean; smoothSlow?: boolean } | null;
speed: number;
from: number;
to: number | undefined;
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index 979a3cce..fa7a6d25 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -100,7 +100,7 @@ Times are in seconds on the shared timeline. The playhead is currentTime; a skim
- A clip's timeline length is (out-in)/speed — or, for a clip with a speed curve, the integral of its curve over the trim (editor_state reports \`len\` either way, and lists a curved clip's \`speedCurve\` nodes as [sourceSeconds, rate]). A clip with \`reverse: true\` plays its trim backward at that same rate — same length, head showing source \`out\`, sound running backward too. Total duration runs to the last clip's end, gaps included. Transitions blend at cuts and never change layout or length.
- trim_clip changes in/out inside the source media. in >= 0, out <= source duration, out-in >= 0.1.
- Resizes ripple the same way deletes do: while track 0 is the only video track, a track-0 trim or speed change slides everything past the clip's tail — clips, titles, captions, soundtrack — with the moved edge, in both directions, every existing gap keeping its width. With overlay video tracks present, growth pushes the track-0 run and a shrink leaves a gap.
-- set_speed sets a clip's playback rate; it changes the clip's timeline length and carries the timeline behind it like a trim. set_speed with \`reverse: true\` plays the clip backward (a rewind, a drop that rises, a pour that un-pours); it changes nothing about length or the curve. set_speed_curve makes the rate change through the footage — nodes of {at: source seconds, speed} or a named preset ramp — for speed ramps, whips, holds, and hits on the beat: detect_beats first, then place nodes at the beats it returns (same unit, no arithmetic). set_speed on a curved clip makes it uniform again.
+- set_speed sets a clip's playback rate; it changes the clip's timeline length and carries the timeline behind it like a trim. set_speed with \`reverse: true\` plays the clip backward (a rewind, a drop that rises, a pour that un-pours); it changes nothing about length or the curve. set_speed_curve makes the rate change through the footage — nodes of {at: source seconds, speed} or a named preset ramp — for speed ramps, whips, holds, and hits on the beat: detect_beats first, then place nodes at the beats it returns (same unit, no arithmetic). set_speed on a curved clip makes it uniform again. Either tool takes \`smooth: true\` — smooth slow motion — which synthesizes the frames between source frames wherever the clip's rate is under 1× (the export estimates the motion between them) and leaves the rest of the clip alone; editor_state shows it as \`smooth\`. Ordinary footage slowed below about 0.5× wants it on.
- set_transition joins a clip into the next one (0–2s, ${TRANSITION_STYLE_IDS.length} styles); set_animation animates one clip's own entrance/exit — read the transitions-and-fades skill before styling cuts. Splitting or deleting clears the affected transition. A transition is a bar on its own row at an absolute time, so an edit can leave one lining up with nothing: every timeline mutation reports those as \`parkedTransitions\`, and clearing them (remove_transition) is part of finishing the edit. Grading a clip's picture is the color tools (set_color_preset, set_color_grade, set_color_curves, set_color_wheels, set_color_hsl — read the color-grading skill); a stylized post treatment (grain, VHS, halation…) is add_effect, dropped over the stretch of timeline it covers — as is a treatment on the sound (echo, reverb, telephone…).
- Cutting a clip's background away — the AI person matte, ink around the silhouette, a fill behind the subject — is the cutout suite (set_removal, set_removal_stroke, set_removal_background); read the background-removal skill.
- split_at cuts the track-0 clip under that time into two clips at the exact frame. With a soundtrack or overlay clip selected it splits that instead.