Skip to content

Open source CapCut alternative.

TypeScript

feat(site): find_filler tool [rebuild]

David Le487ee7ce3c2e
diff --git a/site/scripts/lib/cut-eval/cases.ts b/site/scripts/lib/cut-eval/cases.ts
index e01b0931..91cc80e9 100644
--- a/site/scripts/lib/cut-eval/cases.ts
+++ b/site/scripts/lib/cut-eval/cases.ts
@@ -549,6 +549,7 @@ export function cases(audio: { dataBase64: string; mimeType: string }): EvalCase
bucket: "multi-tool",
input: () => [userTurn("cut the filler words out of my video", { state: FILLER_STATE })],
reply: /filler|um|uh|cut|remove|trim/i,
+ requiredTools: ["find_filler"],
anyTools: ["split_at", "delete_item", "trim_clip"],
state: FILLER_STATE,
simulate: () => makeTimelineSim(FILLER_STATE),
diff --git a/site/scripts/lib/cut-eval/fixtures.ts b/site/scripts/lib/cut-eval/fixtures.ts
index d145ff7a..1fdb8eb4 100644
--- a/site/scripts/lib/cut-eval/fixtures.ts
+++ b/site/scripts/lib/cut-eval/fixtures.ts
@@ -22,6 +22,7 @@ export const SAFE_TOOLS = new Set([
"capture_frame",
"watch_video",
"detect_silence",
+ "find_filler",
"detect_beats",
"listen_audio",
"measure_level",
@@ -204,14 +205,37 @@ export const VOICE_REF = {
/** The base snapshot with a spoken transcript on track 0 — filler words at
* known cue timings, so "cut the filler" has real ranges to act on. */
+const timed = (id: string, start: number, end: number, text: string) => {
+ const parts = text.split(/\s+/).filter(Boolean);
+ const step = (end - start) / parts.length;
+ return {
+ id,
+ start,
+ end,
+ text,
+ words: parts.map((w, i) => ({ w, t0: Math.round((start + step * i) * 100) / 100, t1: Math.round((start + step * (i + 1)) * 100) / 100 })),
+ };
+};
+
export const FILLER_CUES = [
- { id: "cue1", start: 0, end: 1.8, text: "Um, so today we're" },
- { id: "cue2", start: 1.8, end: 4.2, text: "at the beach with the dogs" },
- { id: "cue3", start: 4.2, end: 5.6, text: "and, uh, you know," },
- { id: "cue4", start: 5.6, end: 9, text: "they absolutely love the water" },
- { id: "cue5", start: 9, end: 12.5, text: "so let's watch them play" },
+ timed("cue1", 0, 1.8, "Um, so today we're"),
+ timed("cue2", 1.8, 4.2, "at the beach with the dogs"),
+ timed("cue3", 4.2, 5.6, "and, uh, you know,"),
+ timed("cue4", 5.6, 9, "they absolutely love the water"),
+ timed("cue5", 9, 12.5, "so let's watch them play"),
];
+/** The filler words a judge finds in FILLER_CUES, served to the eval as
+ * find_filler's result: the ums and the stranded "you know". */
+export const FILLER_WORDS = (() => {
+ const picks = new Set(["um,", "uh,", "you", "know,"]);
+ return FILLER_CUES.flatMap((c) =>
+ c.words
+ .filter((w) => picks.has(w.w.toLowerCase()))
+ .map((w) => ({ cue_id: c.id, word: w.w, start: w.t0, end: w.t1 }))
+ );
+})();
+
export const FILLER_STATE = {
...EDITOR_STATE,
subtitles: {
@@ -726,6 +750,7 @@ export function serveSafeTool(name: string, state: unknown): unknown {
if (name === "library_list") return { folders: [], assets: [], templates: [] };
if (name === "notes_list") return { notes: [], folders: [], labels: [] };
if (name === "detect_silence") return { silences: [] };
+ if (name === "find_filler") return { fillers: FILLER_WORDS, count: FILLER_WORDS.length, track: 0 };
if (name === "measure_level") return { targetId: "", clips: [] };
if (name === "detect_beats") return { bpm: 0, beats: [] };
return { ok: true };
diff --git a/site/src/cut/components/AiPanel.tools.ts b/site/src/cut/components/AiPanel.tools.ts
index f81043f0..5da5ddba 100644
--- a/site/src/cut/components/AiPanel.tools.ts
+++ b/site/src/cut/components/AiPanel.tools.ts
@@ -58,7 +58,7 @@ export const AI_PANEL_TOOLS = [
{
name: "detect_silence",
description:
- "Find silent stretches in a source's audio — dead air, long pauses, gaps between takes. Returns [{start,end,duration}] in SOURCE seconds, plus each one's timeline times when clip_id is passed. Cheap and image-free; pair it with the transcript's cue timings to find filler, then cut with split_at / trim_clip / delete_item — place speech cuts inside these spans (cue timings drift from the audio), and read the watching-and-cutting skill for the pacing rules.",
+ "Find silent stretches in a source's audio — dead air, long pauses, gaps between takes. Returns [{start,end,duration}] in SOURCE seconds, plus each one's timeline times when clip_id is passed. Cheap and image-free; find_filler finds the filler words, then cut with split_at / trim_clip / delete_item — place speech cuts inside these spans (cue timings drift from the audio), and read the watching-and-cutting skill for the pacing rules.",
inputSchema: obj({
clip_id: str("Clip id — video, overlay, or soundtrack; scopes to its trimmed range and maps results to timeline seconds"),
asset_id: str("Project asset id (video or audio)"),
@@ -68,6 +68,15 @@ export const AI_PANEL_TOOLS = [
min_silence: num("Shortest silent stretch to report, seconds (default 0.35)"),
}),
},
+ {
+ name: "find_filler",
+ description:
+ "Find the filler words in a subtitle track's transcript — um, uh, a stranded 'like' or 'you know', stutters, repeated words — each judged in its sentence. Returns [{cue_id, word, start, end}] in TIMELINE seconds (the cues' own word timings), plus each word's source times when clip_id is passed, so you can cut the speech with split_at / delete_item / trim_clip or tidy the captions with update_cue. Needs a transcribed track (subtitles_generate) first. Words marked estimated sit in cues that lost their word timings.",
+ inputSchema: obj({
+ track: num("Subtitle track, 0-based (default: the active one)"),
+ clip_id: str("Video clip id — scopes to its timeline span and maps each word to source seconds"),
+ }),
+ },
{
name: "measure_level",
description:
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index 1619f94a..242a91ae 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -1101,6 +1101,88 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
};
},
+ find_filler: async (s, input) => {
+ const lane = targetSubtitleTrack(input);
+ const cues = laneCues(s.subtitles, lane).filter((c) => c.text.trim());
+ if (cues.length === 0) throw new ToolError(`Subtitle track ${lane} has no cues — transcribe first.`);
+ const clipId = typeof input.clip_id === "string" ? input.clip_id : undefined;
+ const clip = clipId ? s.clips.find((c) => c.id === clipId) : undefined;
+ if (clipId && !clip) throw new ToolError(`No clip ${clipId}.`);
+ // Every word with its timeline span: the transcriber's own timings, or an
+ // even split of the cue when a hand edit dropped them.
+ const words = cues.flatMap((c) => {
+ const timed = c.words && c.words.length > 0;
+ return cueWords([c]).map((w, i) => ({
+ key: `word::${c.id}::${i}`,
+ cueId: c.id,
+ i,
+ w: w.w,
+ t0: w.t0,
+ t1: w.t1,
+ estimated: !timed,
+ }));
+ });
+ if (clip) {
+ const inRange = words.filter((w) => w.t1 > clip.start && w.t0 < clip.start + retimeOf(clip).tAt(clip.out));
+ words.splice(0, words.length, ...inRange);
+ }
+ if (words.length === 0) throw new ToolError("No transcript words in that range.");
+ const questions = Object.fromEntries(
+ words.map((w) => [
+ w.key,
+ noul(
+ `In \`cues\`, is word ${w.i} of cue "${w.cueId}" a disfluency or false start a careful editor would cut out of the speech — an um, uh, er, hmm; a stranded "like", "you know", "I mean", "sort of"; a stutter or an immediately repeated word — judged in the sentence it sits in?`,
+ {
+ true: "The word carries no meaning in its sentence and the speech reads cleaner without it.",
+ false: "The word is part of what is being said, including a \"like\" or \"so\" doing real work in the sentence.",
+ },
+ ),
+ ]),
+ );
+ const state = (keys: string[]) => {
+ const wanted = new Set(keys);
+ return {
+ cues: cues
+ .map((c) => ({
+ id: c.id,
+ text: c.text,
+ words: words.filter((w) => w.cueId === c.id).map((w) => ({ i: w.i, w: w.w, judged: wanted.has(w.key) })),
+ }))
+ .filter((c) => c.words.some((w) => w.judged)),
+ };
+ };
+ const post = (payload: Record<string, unknown>, signal?: AbortSignal) =>
+ hostedPost("/api/inference/judge", payload, signal);
+ let answers: Record<string, { type: string; noul?: number }>;
+ try {
+ answers = (await askJudgeChunked(post, state, questions, FILLER_JUDGE_CHUNK)) as Record<string, { type: string; noul?: number }>;
+ } catch (e) {
+ throw new ToolError(`Filler detection is unavailable: ${e instanceof Error ? e.message : String(e)}`);
+ }
+ const floor = cutJudge().fillerCut;
+ const toSource = clip ? (t: number) => round2(retimeOf(clip).srcAt(clamp(t - clip.start, 0, Infinity))) : null;
+ const fillers = words
+ .filter((w) => (answers[w.key]?.noul ?? 0) >= floor)
+ .map((w) => ({
+ cue_id: w.cueId,
+ word: w.w,
+ start: round2(w.t0),
+ end: round2(w.t1),
+ ...(w.estimated ? { estimated: true } : {}),
+ ...(toSource ? { source: { start: toSource(w.t0), end: toSource(w.t1) } } : {}),
+ }));
+ return {
+ fillers,
+ count: fillers.length,
+ track: lane,
+ ...(clip ? { clip: { id: clip.id, timelineStart: round2(clip.start), in: round2(clip.in), out: round2(clip.out) } } : {}),
+ ...(fillers.some((f) => f.estimated)
+ ? { note: "Words marked estimated sit in cues with no word timings; their spans split the cue evenly, so confirm them with listen_audio before a tight cut." }
+ : {}),
+ ...(fillers.length === 0 ? { note: "No filler words on this track." } : {}),
+ };
+ },
+
detect_beats: async (s, input) => {
const { asset, clip, speed, from, to } = resolveWatchRange(s, input);
if (assetIsSilent(asset)) throw new ToolError(`"${asset.name}" carries no audio.`);
@@ -4282,6 +4364,7 @@ async function rankStock<T extends { id: string; kind: string; category: string;
}
const STOCK_JUDGE_CHUNK = 50;
+const FILLER_JUDGE_CHUNK = 60;
export const MEDIA_RUNTIME_TOOLS: ReadonlySet<string> = new Set([
"convert_media",
diff --git a/site/src/cut/lib/pi/mutationLedger.ts b/site/src/cut/lib/pi/mutationLedger.ts
index 28b3807b..775d5a58 100644
--- a/site/src/cut/lib/pi/mutationLedger.ts
+++ b/site/src/cut/lib/pi/mutationLedger.ts
@@ -15,6 +15,7 @@ const READ_ONLY = new Set([
"watch_video",
"note_source",
"detect_silence",
+ "find_filler",
"detect_beats",
"listen_audio",
"measure_level",
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index 744d82a0..5e7f464a 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -282,6 +282,7 @@ Panel geography, for hand-instructions: the Cutout tab is one flat pane. One seg
Recipes: "remove the background" on a person = auto — green-screen and plain-backdrop shots included, the matte keys them. "Cut out the dog" = custom with subject "the dog". "Put me on a beach" = auto + the beach behind (background image or lower track). Sticker/die-cut = auto + white cut stroke. A bold silhouette pop = offset stroke in a loud color. Turning it off = set_removal mode "off" — the picture shows plain, and the whole cutout (matte, stroke, fill) stays stored, so any on-mode brings it straight back.`,
"watching-and-cutting": `# Watching footage & cutting by content
+Filler words: find_filler judges every transcript word in its sentence and returns the ums, uhs and stranded "you know"s with their timeline times, so a "cut the filler" sweep is one call and then the cuts.
When a request depends on what the footage actually contains — "cut the dead air", "clip the best moment", "remove the boring part", "split where the scene changes" — watch it first. Never guess at content you haven't seen.
Your eyes and ears:
@@ -466,6 +467,7 @@ ${textMoveCatalog()}
**How to use this list.** Pick a look for the base design, one composition device, one rhythm device and one motion device, and hold them for the whole video. Then break the pattern exactly once, on the hook. That single break is what a viewer remembers.`,
"audio-and-subtitles": `# Audio, voiceover & subtitles
+Cleaning a transcript: find_filler names the filler words on a track with their cue ids and times; tidy the captions with update_cue, or cut the speech itself with the timeline tools.
Lyrics and scripts the user already has: sync_lyrics times their exact words against the audio (transcribing first when needed) and writes them to a caption track with per-word timings; set_caption_look then dresses the track — preset, size, font, word effect, accent color — from a text-videos look id or field by field. The text-videos skill has the whole flow, including turning the same timed lines into full-frame cards.
Soundtrack clips: volume 0..3 (1 = unchanged, above 1 boosts), fadeIn/fadeOut seconds (max half the clip), start = timeline position, in/out = trim inside the source; clips can spread across several soundtrack lanes (the \`lane\` field), new sounds slide to free space in their lane. Fades render with ffmpeg afade on export.
Matching levels: "as loud as", "match", "balance", "sit under" are measurements, never guesses. measure_level with every clip involved and target_id = the one to match returns each clip's playing level and the volumeToMatch that lands it on the target; set that with set_clip_volume / update_audio (a fraction of it for a bed under speech). Listening tells you how a sound feels, never how many dB it is.