Open source CapCut alternative.
TypeScript
fix(cut): let a text-sequence line run one frame where it is given [rebuild]
David Le0a7f057cb2ef
diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.tsindex 8b41010d..008d2f37 100644--- a/site/src/cut/components/Timeline.tools.ts+++ b/site/src/cut/components/Timeline.tools.ts@@ -290,7 +290,7 @@ export const TIMELINE_TOOLS = [{name: "add_text_sequence",description:- "Place a run of text on the timeline in one call — plain lines timed to speech, a lyric video, a kinetic-typography passage, a quote sequence. Design is opt-in: with no `look` the run comes out plain — one readable face, one place in the frame, a short fade, no cards, no moves, and the project's frame color untouched. That is what a bare \"add the text\" or \"put the words on screen\" asks for. Name a `look` when the user asked for a designed run (a lyric video, kinetic type, a style, a vibe, \"make it interesting\"): `look` carries the design of one line (frame color, cards, type, entrance) AND the ensemble the run walks across its whole length: where successive lines land, which faces they rotate through, which entrances, and what each line does while it holds. So one such call already produces a varied, composed run without a hundred repair calls after it. Lines come from `lines` ({text, start, end} in timeline seconds), or from the caption track with from_captions when a sync already timed them. In a designed run, mark the loud lines with `emphasis: \"hero\"` and the quiet ones `\"whisper\"`, and label passages with `section` (\"verse\", \"chorus\") — a new section restarts the type and color rotas, so a chorus does not look like the verse before it. Anything you set per line (font, x/y, rotation, in_style, move, color, size) wins over the ensemble. The words land on `lane` and each card on the row under them. A look that paints a frame color sets the project background to it unless background is false; the plain and over-footage looks leave the background alone unless background is true. Look ids, layouts and moves are in the text-videos and text-creativity skills.",+ "Place a run of text on the timeline in one call — plain lines timed to speech, a lyric video, a kinetic-typography passage, a quote sequence. Design is opt-in: with no `look` the run comes out plain — one readable face, one place in the frame, a short fade, no cards, no moves, and the project's frame color untouched. That is what a bare \"add the text\" or \"put the words on screen\" asks for. Name a `look` when the user asked for a designed run (a lyric video, kinetic type, a style, a vibe, \"make it interesting\"): `look` carries the design of one line (frame color, cards, type, entrance) AND the ensemble the run walks across its whole length: where successive lines land, which faces they rotate through, which entrances, and what each line does while it holds. So one such call already produces a varied, composed run without a hundred repair calls after it. Lines come from `lines` ({text, start, end} in timeline seconds), or from the caption track with from_captions when a sync already timed them. Each line keeps the start and end you give it, on its frame, and runs one frame (1/60 s) or longer, so a label per shot of a fast cut lands on that shot's frames however short; a line under 0.2 s cuts in and out (no entrance or exit, which would leave it blank on so few frames). In a designed run, mark the loud lines with `emphasis: \"hero\"` and the quiet ones `\"whisper\"`, and label passages with `section` (\"verse\", \"chorus\") — a new section restarts the type and color rotas, so a chorus does not look like the verse before it. Anything you set per line (font, x/y, rotation, in_style, move, color, size) wins over the ensemble. The words land on `lane` and each card on the row under them. A look that paints a frame color sets the project background to it unless background is false; the plain and over-footage looks leave the background alone unless background is true. Look ids, layouts and moves are in the text-videos and text-creativity skills.",inputSchema: obj({look: str("Look id from the text-videos skill. Omit it for the plain run — that is the default and the right answer whenever the user did not ask for a design."),variation: {diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.tsindex 53a0bb57..023201e9 100644--- a/site/src/cut/lib/aiTools.ts+++ b/site/src/cut/lib/aiTools.ts@@ -6585,8 +6585,8 @@ function textSequenceLines(const lane = isNum(input.from_captions) ? Math.max(0, Math.round(input.from_captions)) : 0;return laneCues(s.subtitles, lane).map((c) => ({text: c.text,- start: roundT(c.start),- end: roundT(c.end),+ start: c.start,+ end: Math.max(c.end, c.start + CLIP_MIN_SECONDS),}));}const raw = Array.isArray(input.lines) ? input.lines : [];@@ -6614,17 +6614,18 @@ function textSequenceLines(.filter((l) => l.text.length > 0);// A line with no times follows the one before it; the first starts at the// playhead. Every line gets a real span, so nothing lands on top of- // anything.+ // anything. Times stay on the frames they were given (the input snap put+ // them there): a label per shot holds that shot's frames, one or more.let cursor = playheadAt();return lines.map((l, i) => {const start = Number.isNaN(l.start) ? cursor : l.start;const nextStart = lines[i + 1]?.start ?? NaN;const fallbackEnd = Number.isNaN(nextStart) ? start + 2 : nextStart;- // Every line lands at least 0.2s long, whichever way its end was+ // Every line lands at least one frame long, whichever way its end was// decided — a following line that starts earlier cannot invert it.- const end = Math.max(Number.isNaN(l.end) ? fallbackEnd : l.end, start + 0.2);+ const end = Math.max(Number.isNaN(l.end) ? fallbackEnd : l.end, start + CLIP_MIN_SECONDS);cursor = end;- return { ...l, start: roundT(start), end: roundT(end) } as ComposeLine;+ return { ...l, start, end } as ComposeLine;});}diff --git a/site/src/cut/lib/oneFrameLabel.test.ts b/site/src/cut/lib/oneFrameLabel.test.tsnew file mode 100644index 00000000..28e1ee1b--- /dev/null+++ b/site/src/cut/lib/oneFrameLabel.test.ts@@ -0,0 +1,63 @@+import { beforeAll, beforeEach, expect, test } from "bun:test";+import { installSkiaRaster } from "./headless/skiaRaster";+import { renderProjectFrame } from "./exportRender";+import { renderElementFrames } from "./textRender";+import { createRasterCanvas, decodeRasterImage } from "./raster";+import { runAiTool } from "./aiTools";+import { useEditor } from "./store";+import { emptySubtitles, isTextOverlay } from "./types";+import type { ExportDoc } from "./renderSnapshot";+import { evalOverlayAnim } from "@donkeycut/effects-kit";++// A label per shot in a reel cut every frame or two runs one frame: it has to+// show on that frame, in the export and in the frames the ffmpeg path burns.++const size = { width: 320, height: 180 };++beforeAll(async () => {+ expect(await installSkiaRaster()).toBe(true);+});++beforeEach(() =>+ useEditor.setState({+ projectId: "p1",+ aspect: "16:9",+ assets: [],+ clips: [],+ audioClips: [],+ overlays: [],+ subtitles: emptySubtitles(),+ background: "#000000",+ })+);++const ink = (data: Uint8ClampedArray, channel: number) => {+ let n = 0;+ for (let i = channel; i < data.length; i += 4) if (data[i] > 128) n++;+ return n;+};++for (const look of ["serif-mood", "lyric-card", "over-footage"]) {+ test(`a one-frame label in ${look} shows on its frame`, async () => {+ await runAiTool("add_text_sequence", { look, cards: false, lines: [{ text: "SHOT", start: 1, end: 61 / 60, size: 160 }] });+ const word = useEditor.getState().overlays.filter(isTextOverlay)[0];+ expect(evalOverlayAnim(word.anim, 0, word.end - word.start, true).alpha).toBe(1);++ const doc: ExportDoc = {+ aspect: "16:9", assets: [], clips: [], audioClips: [], overlays: [word],+ subtitles: { cues: [], showOnVideo: false, showOnTimeline: false },+ };+ const frame = await renderProjectFrame(doc, 1, size, () => "");+ const exported = (frame.getContext("2d") as CanvasRenderingContext2D).getImageData(0, 0, size.width, size.height).data;+ expect(ink(exported, 0) + ink(exported, 1) + ink(exported, 2)).toBeGreaterThan(50);++ const frames = await renderElementFrames(word, size.width, size.height, 30);+ const first = await decodeRasterImage(frames.images[frames.entries[0].image]);+ expect(first).not.toBeNull();+ const sheet = createRasterCanvas(size.width, size.height);+ const sctx = sheet.getContext("2d") as CanvasRenderingContext2D;+ sctx.drawImage(first!.source, frames.x, frames.y);+ const sampled = sctx.getImageData(0, 0, size.width, size.height).data;+ expect(ink(sampled, 3)).toBeGreaterThan(50);+ });+}diff --git a/site/src/cut/lib/textCompose.ts b/site/src/cut/lib/textCompose.tsindex cf773450..2af01953 100644--- a/site/src/cut/lib/textCompose.ts+++ b/site/src/cut/lib/textCompose.ts@@ -195,6 +195,10 @@ export interface ComposedLine {const clampPos = (v: number) => Math.min(0.9, Math.max(0.1, v));const round2 = (n: number) => Math.round(n * 100) / 100;+/** A line held for less than this cuts in and out: an entrance or exit ramp+ * starts at nothing, so on a label that runs a frame or a few (one per shot+ * of a fast cut) the ramp would be all there is and the words never show. */+export const TEXT_RAMP_MIN_SECONDS = 0.2;const step = <T,>(arr: readonly T[], n: number): T => arr[((n % arr.length) + arr.length) % arr.length];/** Comparison form of a line: letters and digits only. Two lines that read@@ -349,9 +353,10 @@ export function composeTextRun(const moveId =line.move ??(emphasis === "hero" ? ens?.heroMove ?? step(moves, i + sec) : step(moves, i + sec));+ const ramps = line.end - line.start >= TEXT_RAMP_MIN_SECONDS;const anim: OverlayAnim = {- in: { style: inStyle, seconds: round2(inSeconds) },- ...(look.motion.out ? { out: look.motion.out } : {}),+ ...(ramps ? { in: { style: inStyle, seconds: round2(inSeconds) } } : {}),+ ...(ramps && look.motion.out ? { out: look.motion.out } : {}),...(look.motion.loop? { loop: { style: look.motion.loop.style, speed: look.motion.loop.speed ?? 1 } }: {}),diff --git a/site/src/cut/lib/textVideo.test.ts b/site/src/cut/lib/textVideo.test.tsindex 72b8ab72..a0f70d7a 100644--- a/site/src/cut/lib/textVideo.test.ts+++ b/site/src/cut/lib/textVideo.test.ts@@ -166,6 +166,36 @@ describe("add_text_sequence", () => {for (const w of overlays().filter(isTextOverlay)) expect(w.end).toBeGreaterThan(w.start);});+ test("one label per shot keeps each shot's own frames, however short", async () => {+ // Shots of one to five frames at 30 fps, one label each.+ const shots = [+ [0, 1 / 30],+ [1 / 30, 3 / 30],+ [3 / 30, 4 / 30],+ [4 / 30, 9 / 30],+ [9 / 30, 10 / 30],+ ];+ await runAiTool("add_text_sequence", {+ look: "serif-mood",+ lines: shots.map(([start, end], i) => ({ text: `SHOT ${i + 1}`, start: Number(start.toFixed(4)), end: Number(end.toFixed(4)) })),+ });+ const words = overlays()+ .filter(isTextOverlay)+ .sort((a, b) => a.start - b.start);+ expect(words.length).toBe(shots.length);+ words.forEach((w, i) => {+ expect(w.start).toBeCloseTo(shots[i][0], 9);+ expect(w.end).toBeCloseTo(shots[i][1], 9);+ });+ });++ test("a line runs one frame at the least", async () => {+ await runAiTool("add_text_sequence", { look: "serif-mood", lines: [{ text: "BLINK", start: 1, end: 1 }] });+ const [w] = overlays().filter(isTextOverlay);+ expect(w.start).toBe(1);+ expect(w.end).toBeCloseTo(61 / 60, 9);+ });+test("a run with nothing to place is refused", async () => {let message = "";try {@@ -330,9 +360,12 @@ describe("composing a run", () => {TEXT_LOOKS["over-footage"],{ variation: "bold", frame: { w: 1920, h: 1080 }, cards: [] });- const widest = composed[0].text.split("\n").reduce((n, l) => Math.max(n, l.length), 0);+ const c = composed[0];+ // Measured the way the run measures it, so the check holds whether the+ // process has a canvas to ask (a headless raster installed) or not.+ const widest = Math.max(...c.text.split("\n").map((l) => measureLine(l, textFontOf(c, c.size), c.size)));// Centered at x 0.18 there is only 36% of the frame to spread into.- expect(widest * composed[0].size * 0.5).toBeLessThanOrEqual(1920 * 0.4);+ expect(widest).toBeLessThanOrEqual(1920 * 0.4);});test("a repeated hook steps up rather than repeating flat", async () => {diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.tsindex f4cb8706..231f4b11 100644--- a/site/src/cut/server/ai/catalog.ts+++ b/site/src/cut/server/ai/catalog.ts@@ -444,7 +444,7 @@ With no lyrics text, transcribe with subtitles_generate and work from the cues i**When the words are off the audio.** \"The subtitles are late\", \"sync the text to the audio\", \"it drifts by the end\" — that is align_to_audio, and it is the only honest answer to it. It renders the cut's audible mix, reads a 10ms energy envelope to find where speech really starts and stops, and moves each caption or element edge onto the speech edge it belongs to; an edge with nothing audible near it keeps its time. Re-transcribing instead throws away the user's own wording, and retiming cue by cue with update_cue is guessing at what the waveform already knows — reach for update_cue for the one line the pass could not place. The pass reports \`evidence: false\` when the mix gave it nothing to read (a bed of music with no quiet in it, digital silence, or speech that does not line up with these times at all); say that plainly rather than reporting a fix, and check by ear with listen_audio.-**Timing.** One line per screen beats two. A line needs about 0.4s minimum to read; 1.5–2.5s is the normal beat for a spoken line, and a sung line takes whatever the vocal takes. Lines never overlap on a row — the sequence tool lays them end to end from the times you give it.+**Timing.** One line per screen beats two. A line needs about 0.4s minimum to read; 1.5–2.5s is the normal beat for a spoken line, and a sung line takes whatever the vocal takes. Lines never overlap on a row — the sequence tool lays them end to end from the times you give it, each exactly where you put it. A label per shot of a fast cut can be as short as its shot (one frame and up); under 0.2s it cuts in and out with no entrance.**Fixing a run.** update_overlay retimes or recolors any line, set_overlay_animation swaps its entrance, set_overlay_keyframes gives it a path. Reach for those to fix a handful of lines. When the whole run is wrong, delete it and re-compose with different look, layout, or variation — that is one call against a hundred.`,