Skip to content

Open source CapCut alternative.

TypeScript

feat(site): a handover on the sound instead of the picture [rebuild]

David Leb871dc8a85c3
diff --git a/docs/guides/cut/README.md b/docs/guides/cut/README.md
index e08c5ec5..0f40fa70 100644
--- a/docs/guides/cut/README.md
+++ b/docs/guides/cut/README.md
@@ -38,6 +38,8 @@ The client probes the engine's health endpoint (dedicated port first, dev server
**An effect treats what runs under it.** An effect is a stretch of the timeline rather than a property of a clip, so it can cover part of a shot, a whole one, or a run of them, and it is trimmed and moved like any other element. Some of them treat the picture — a grade, a grain, a shake, a push in — and some treat the sound: everything audible under the window, clip audio, upper tracks and soundtrack, mixed and then filtered as one. The sound treatments are described as data — a dry level, delay taps, filter bands, a tremolo, a bit crush — and three renderers read that one description: the Web Audio graph the preview plays, the same graph folded offline for an in-tab export, and an ffmpeg chain for the engine and the cloud worker. That is what keeps the export sounding like the preview; the ffmpeg side cuts the window out of the mix, runs the chain over it, and pads the result back to the length it went in at, so a chain that rings past its window cannot walk the rest of the sound late. The window's edges cross over rather than switch, in both mixes, so an effect coming in cannot click.
+**A handover is on the picture or on the sound.** A transition is a blend at the cut rather than an overlap: the clips still abut, the layout never moves, and the incoming clip starts exactly at the cut. Most styles blend the picture across the outgoing clip's last seconds. One hands over on the sound instead — the picture cuts, the outgoing clip's sound fades out into the cut and the incoming clip's fades in out of it. The two windows sit on separate fields, so everything that draws the picture reads a plain cut and only the mix ramps: the preview, the in-tab export, and the ffmpeg graph the engine and the worker build all follow from that one split. A sound handover needs a second clip's sound to cross into, so a bar of it parked on a clip's open head or tail plays nothing. It is picked where the other work on the sound is picked — the Effects tab's Sound family — while the styles that blend the picture are picked from the Transitions tab.
+
## The preview
The preview decodes the cut itself. Every clip on screen owns a decoder reading its file directly — the container is parsed in the page and frames come off WebCodecs — and a small ring of already-decoded frames sits beside it. Drawing a frame is a question answered from that ring, never a wait: playing walks each file forward and keeps the ring ahead of the playhead, scrubbing asks for one moment and the newest ask wins, and if the exact frame has not landed the nearest one already decoded goes up and is replaced when the real one arrives. A held frame is worth more than a stall.
diff --git a/site/scripts/lib/cut-eval/cases.ts b/site/scripts/lib/cut-eval/cases.ts
index aa346f2d..2103bd9c 100644
--- a/site/scripts/lib/cut-eval/cases.ts
+++ b/site/scripts/lib/cut-eval/cases.ts
@@ -454,6 +454,30 @@ export function cases(audio: { dataBase64: string; mimeType: string }): EvalCase
return { id: "fx1", effect: "telephone", start, end, amount: Number(args.amount) || 0.5 };
},
},
+ {
+ // "Only the audio" at a cut is a transition style, not an effect and not
+ // per-clip fades: audiocross leaves the picture a hard cut and dissolves
+ // the sound across it.
+ name: "sound-dissolve-single-tool",
+ bucket: "single-tool",
+ input: () => [
+ userTurn("keep the cut between the two clips but cross dissolve just the audio", {
+ state: TWO_CLIP_STATE,
+ }),
+ ],
+ reply: /audio|sound|dissolve/i,
+ requiredTools: ["set_transition"],
+ maxToolCalls: 3,
+ state: TWO_CLIP_STATE,
+ simulate: () => (name, args) => {
+ if (name !== "set_transition") return undefined;
+ if (args.style !== "audiocross")
+ throw new Error(`style ${args.style} — expected audiocross`);
+ const seconds = Number(args.seconds ?? 0);
+ if (!(seconds > 0)) throw new Error(`seconds ${seconds} — a dissolve needs a window`);
+ return { clipId: String(args.clipId), seconds, style: "audiocross" };
+ },
+ },
{
// Last pick wins per edge: swapping a crossfading joint for an entrance
// animation is a set_animation — the override clears the transition
diff --git a/site/src/cut/components/Timeline.tsx b/site/src/cut/components/Timeline.tsx
index 48b83a32..29f3df37 100644
--- a/site/src/cut/components/Timeline.tsx
+++ b/site/src/cut/components/Timeline.tsx
@@ -59,6 +59,7 @@ import { cn } from "@/lib/utils";
const TRANSITION_ICONS: Record<TransitionStyle, LucideIcon> = {
crossfade: Blend,
+ audiocross: AudioLines,
crosszoom: Expand,
dipblack: Moon,
dipwhite: Sun,
diff --git a/site/src/cut/hooks/usePlayback.ts b/site/src/cut/hooks/usePlayback.ts
index 77c45cb4..bd707b4e 100644
--- a/site/src/cut/hooks/usePlayback.ts
+++ b/site/src/cut/hooks/usePlayback.ts
@@ -13,7 +13,7 @@ import { clipCovers, projectFadeSeconds, rectOf } from "@/cut/lib/types";
import type { ClipSpan, MediaAsset, VideoClip } from "@/cut/lib/types";
import { SubjectMaskCompositor } from "@/cut/lib/behindPass";
import { FrameCompositor, MISSING_FRAME, PENDING_FRAME, type Frame } from "@/cut/lib/composite";
-import { duckGainAt, overlayPlan, trackZeroPlan } from "@/cut/lib/framePlan";
+import { duckGainAt, overlayPlan, soundCrossGain, trackZeroPlan } from "@/cut/lib/framePlan";
import { type ClipFrameSource, FrameSourcePool, mappingKey, walkCostMs } from "@/cut/lib/frameSource";
import { liveAudioFxAt } from "@/cut/lib/audioEffects";
import { PreviewMixer, type Voice } from "@/cut/lib/previewMixer";
@@ -301,7 +301,8 @@ class Engine {
*
* The same ramps that dim the picture dim the sound: a clip fading out of a
* dissolve takes its audio with it, an upper-track clip's transition carries
- * its own, and a live voiceover ducks the rest.
+ * its own, and a live voiceover ducks the rest. A sound dissolve is the
+ * ramp on its own — the picture cuts and only these gains move.
*/
private voicesAt(t: number, spans: ClipSpan[], master: ClipSpan | undefined): Voice[] {
const s = useEditor.getState();
@@ -316,7 +317,7 @@ class Engine {
in: master.clip.in,
out: master.clip.out,
speed: clipSpeed(master.clip),
- gain: plan.gain * duck * (master.clip.volume ?? 1),
+ gain: plan.gain * soundCrossGain(spans, master, t) * duck * (master.clip.volume ?? 1),
});
}
for (const { clip, asset, gain } of this.liveOverlays(t)) {
@@ -425,6 +426,7 @@ class Engine {
start: clip.start,
len: Math.max(0.1, (clip.out - clip.in) / clipSpeed(clip)),
transitionOut: 0,
+ soundOut: 0,
};
const frame = this.frameFor(span, t, playing);
if (frame.kind !== "ready") continue;
diff --git a/site/src/cut/lib/audioMix.test.ts b/site/src/cut/lib/audioMix.test.ts
index afc334d4..fdaab38f 100644
--- a/site/src/cut/lib/audioMix.test.ts
+++ b/site/src/cut/lib/audioMix.test.ts
@@ -34,6 +34,20 @@ describe("foldClips", () => {
expect(geo[1].fadeIn).toBe(0);
});
+ test("a sound dissolve crosses the cut: one clip out, the next one in", () => {
+ const geo = foldClips([clip({ soundCross: 1 }), clip()]);
+ expect(geo[0].fadeOut).toBe(1);
+ expect(geo[1].fadeIn).toBe(1);
+ // The picture cut claims no layout here either.
+ expect(geo[1].at).toBe(4);
+ });
+
+ test("a sound dissolve is clamped by both clips it crosses", () => {
+ const geo = foldClips([clip({ out: 1, soundCross: 3 }), clip({ out: 2 })]);
+ expect(geo[0].fadeOut).toBeCloseTo(0.9, 5);
+ expect(geo[1].fadeIn).toBeCloseTo(0.9, 5);
+ });
+
test("scales a blend down so it cannot swallow its clip", () => {
const geo = foldClips([clip({ out: 1, transition: 2 }), clip({ out: 1 })]);
expect(geo[0].fadeOut).toBeLessThanOrEqual(0.9);
diff --git a/site/src/cut/lib/audioMix.ts b/site/src/cut/lib/audioMix.ts
index a9a04832..fb002b0a 100644
--- a/site/src/cut/lib/audioMix.ts
+++ b/site/src/cut/lib/audioMix.ts
@@ -33,6 +33,10 @@ export interface MixClip {
volume?: number;
/** Cross-dissolve overlap into the next clip, timeline seconds. */
transition?: number;
+ /** Sound dissolve into the next clip, timeline seconds: this clip fades out
+ * over the window before the cut and the next one fades in over the window
+ * after it. Set instead of `transition` when the handover is on the sound. */
+ soundCross?: number;
}
/** A clip placed at an absolute time: an upper-track video's audio, a
@@ -94,6 +98,9 @@ const itemDur = (a: MixItem) => Math.max(0, (a.out - a.in) / speedOf(a.speed));
* transition edges are. Clips abut — a transition claims no layout — and its
* sound is the outgoing clip fading across the blend window while the next
* enters clean at the cut, exactly like the engine's tail-fade + hard join.
+ *
+ * A sound dissolve is the same fade out with the next clip's fade in against
+ * it: the picture cuts and the sound crosses over the cut.
*/
export function foldClips(clips: MixClip[]) {
const geo = clips.map((clip) => ({ clip, at: 0, dur: clipDur(clip), fadeIn: 0, fadeOut: 0 }));
@@ -101,8 +108,15 @@ export function foldClips(clips: MixClip[]) {
geo.forEach((g, j) => {
g.at = acc;
acc += g.dur;
- const fade = geo[j + 1] ? Math.min(g.clip.transition ?? 0, g.dur * 0.9) : 0;
+ const next = geo[j + 1];
+ if (!next) return;
+ const fade = Math.min(g.clip.transition ?? 0, g.dur * 0.9);
if (fade > 0.01) g.fadeOut = fade;
+ const cross = Math.min(g.clip.soundCross ?? 0, g.dur * 0.9, next.dur * 0.9);
+ if (cross > 0.01) {
+ g.fadeOut = Math.max(g.fadeOut, cross);
+ next.fadeIn = Math.max(next.fadeIn, cross);
+ }
});
return geo;
}
diff --git a/site/src/cut/lib/exportClient.ts b/site/src/cut/lib/exportClient.ts
index d343e61b..ad3f432b 100644
--- a/site/src/cut/lib/exportClient.ts
+++ b/site/src/cut/lib/exportClient.ts
@@ -491,6 +491,9 @@ async function buildExportPayload(
frame: sp.clip.frame,
speed: clipSpeed(sp.clip),
transition: sp.transitionOut,
+ // A sound dissolve carries its own window: the picture cuts, so the
+ // server's blend must not see one.
+ soundCross: sp.soundOut,
// The style rides along with the overlap; the server resolves it to an
// xfade name (and the cross-zoom ramps) itself, so the spec carries only
// the id.
@@ -598,7 +601,16 @@ async function buildExportPayload(
const overlayClipOf = new Map<object, VideoClip>();
const overlayVideos = overlayTracks.flatMap((track) => {
const trackSpans = getClipSpans(doc.clips, doc.assets, track);
- const ramps = trackSpans.map(() => ({ headFade: 0, tailFade: 0, headZoom: 0, tailZoom: 0 }));
+ const ramps = trackSpans.map(() => ({
+ headFade: 0,
+ tailFade: 0,
+ headZoom: 0,
+ tailZoom: 0,
+ // Sound-only ramps: a dissolve on the sound leaves the picture alone,
+ // so its windows ride apart from the alpha fades.
+ headSound: 0,
+ tailSound: 0,
+ }));
trackSpans.forEach((sp, i) => {
const r = ramps[i];
const applyAnim = (a: ClipAnim | undefined, side: "head" | "tail") => {
@@ -622,6 +634,10 @@ async function buildExportPayload(
nr.headZoom = Math.max(nr.headZoom, sp.transitionOut);
}
}
+ if (sp.soundOut > 0 && trackSpans[i + 1]) {
+ r.tailSound = Math.max(r.tailSound, sp.soundOut);
+ ramps[i + 1].headSound = Math.max(ramps[i + 1].headSound, sp.soundOut);
+ }
});
return trackSpans
.map((sp, i) => ({ c: sp.clip, ramp: ramps[i] }))
diff --git a/site/src/cut/lib/exportRender.ts b/site/src/cut/lib/exportRender.ts
index 1ae7380b..22c7d0ca 100644
--- a/site/src/cut/lib/exportRender.ts
+++ b/site/src/cut/lib/exportRender.ts
@@ -452,6 +452,11 @@ function overlayRamps(spans: ClipSpan[]): { head: number; tail: number }[] {
ramps[i].tail = Math.max(ramps[i].tail, sp.transitionOut);
ramps[i + 1].head = Math.max(ramps[i + 1].head, sp.transitionOut);
}
+ // A sound dissolve is these ramps and nothing else: the picture cuts.
+ if (sp.soundOut > 0 && spans[i + 1]) {
+ ramps[i].tail = Math.max(ramps[i].tail, sp.soundOut);
+ ramps[i + 1].head = Math.max(ramps[i + 1].head, sp.soundOut);
+ }
});
return ramps;
}
@@ -522,6 +527,7 @@ export function mixSpecFor(doc: ExportDoc, resolve: (asset: MediaAsset) => strin
speed: sp.clip.speed,
volume: sp.clip.volume,
transition: sp.transitionOut,
+ soundCross: sp.soundOut,
},
]);
diff --git a/site/src/cut/lib/framePlan.test.ts b/site/src/cut/lib/framePlan.test.ts
index 2a013010..90c5974d 100644
--- a/site/src/cut/lib/framePlan.test.ts
+++ b/site/src/cut/lib/framePlan.test.ts
@@ -1,5 +1,13 @@
import { describe, expect, test } from "bun:test";
-import { clipAnimFx, duckGainAt, prerollLead, PREROLL_LEAD_S, trackZeroPlan } from "./framePlan";
+import {
+ clipAnimFx,
+ duckGainAt,
+ overlayTransitionFx,
+ prerollLead,
+ PREROLL_LEAD_S,
+ soundCrossGain,
+ trackZeroPlan,
+} from "./framePlan";
import { TRANSITION_ZOOM } from "./types";
import type { AudioClip, ClipSpan, MediaAsset, VideoClip } from "./types";
@@ -36,12 +44,59 @@ function spansOf(count: number, len = 4, transitions: number[] = []): ClipSpan[]
start: at,
len,
transitionOut,
+ soundOut: 0,
});
at += len; // clips abut — a transition is a blend at the cut, never overlap
}
return spans;
}
+/** The same run, handing over on the sound: the picture cuts, the window
+ * lands on `soundOut`. */
+function soundSpansOf(count: number, len = 4, crosses: number[] = []): ClipSpan[] {
+ const spans = spansOf(count, len);
+ return spans.map((sp, i) => ({ ...sp, soundOut: crosses[i] ?? 0 }));
+}
+
+describe("soundCrossGain", () => {
+ test("crosses the level over the cut while the picture stays a cut", () => {
+ const spans = soundSpansOf(2, 4, [2]);
+ const cut = spans[1].start;
+ // Nothing moves until the window opens.
+ expect(soundCrossGain(spans, spans[0], cut - 2.5)).toBeCloseTo(1, 5);
+ // The outgoing clip ramps down into the cut…
+ expect(soundCrossGain(spans, spans[0], cut - 1)).toBeCloseTo(0.5, 5);
+ expect(soundCrossGain(spans, spans[0], cut - 0.001)).toBeCloseTo(0, 2);
+ // …and the incoming one ramps up out of it.
+ expect(soundCrossGain(spans, spans[1], cut)).toBeCloseTo(0, 5);
+ expect(soundCrossGain(spans, spans[1], cut + 1)).toBeCloseTo(0.5, 5);
+ expect(soundCrossGain(spans, spans[1], cut + 2)).toBeCloseTo(1, 5);
+ // The picture never blends: no overlap, no incoming picture.
+ expect(trackZeroPlan(spans[0], spans, cut - 1).p).toBe(0);
+ expect(trackZeroPlan(spans[0], spans, cut - 1).incoming).toBe(null);
+ expect(trackZeroPlan(spans[0], spans, cut - 1).masterAlpha).toBe(1);
+ });
+
+ test("leaves a run with no sound dissolve at full level", () => {
+ const spans = soundSpansOf(2, 4);
+ expect(soundCrossGain(spans, spans[0], 3.9)).toBe(1);
+ expect(soundCrossGain(spans, spans[1], 4.1)).toBe(1);
+ });
+});
+
+describe("overlayTransitionFx", () => {
+ test("an upper-track sound dissolve moves the level and not the picture", () => {
+ const spans = soundSpansOf(2, 4, [2]);
+ const cut = spans[1].start;
+ const out = overlayTransitionFx(spans[0], undefined, spans[1], cut - 1);
+ expect(out.gain).toBeCloseTo(0.5, 5);
+ expect(out.alpha).toBe(1);
+ const inc = overlayTransitionFx(spans[1], spans[0], undefined, cut + 1);
+ expect(inc.gain).toBeCloseTo(0.5, 5);
+ expect(inc.alpha).toBe(1);
+ });
+});
+
describe("trackZeroPlan", () => {
test("reports no transition in the middle of a clip", () => {
const spans = spansOf(2);
diff --git a/site/src/cut/lib/framePlan.ts b/site/src/cut/lib/framePlan.ts
index a61a153b..b16ff6d2 100644
--- a/site/src/cut/lib/framePlan.ts
+++ b/site/src/cut/lib/framePlan.ts
@@ -67,6 +67,16 @@ export function overlayTransitionFx(
zoom = 1 + (TRANSITION_ZOOM - 1) * p;
}
}
+ // A sound dissolve at either edge: the picture cuts and only the level
+ // moves, so the clip arrives at full opacity with its sound coming up.
+ if (prev && prev.soundOut > 0) {
+ const rel = t - span.start;
+ if (rel < prev.soundOut) gain = Math.min(gain, Math.max(0, rel / prev.soundOut));
+ }
+ if (next && span.soundOut > 0) {
+ const left = span.start + span.len - t;
+ if (left < span.soundOut) gain = Math.min(gain, Math.max(0, left / span.soundOut));
+ }
// The clip's own entrance/exit animations. A transitioned joint owns its
// edges: the transition plays there and the adjacent animation is held
// (running both would fight over the same window).
@@ -195,6 +205,31 @@ export function duckGainAt(audioClips: AudioClip[], t: number): number {
return g;
}
+/**
+ * The gain a sound dissolve puts on the clip playing at `t`.
+ *
+ * The picture cuts and the sound crosses it: the outgoing clip ramps down over
+ * the window before the cut, and the clip after it ramps up over the window
+ * after it. The preview reads this per frame; the offline fold schedules the
+ * same two ramps.
+ */
+export function soundCrossGain(spans: ClipSpan[], master: ClipSpan, t: number): number {
+ const i = spans.indexOf(master);
+ if (i < 0) return 1;
+ let g = 1;
+ const out = master.soundOut;
+ if (out > 0.01) {
+ const from = master.start + master.len - out;
+ if (t > from) g = Math.min(g, Math.max(0, (master.start + master.len - t) / out));
+ }
+ const prev = spans[i - 1];
+ const inc = prev?.soundOut ?? 0;
+ if (inc > 0.01 && t < master.start + inc) {
+ g = Math.min(g, Math.max(0, (t - master.start) / inc));
+ }
+ return g;
+}
+
/** How much of a clip's animation window is still running at `rel`, for the
* sides that show a neighbour's frame behind them. */
function animLive(a: ClipAnim | undefined, side: "in" | "out", rel: number, len: number) {
diff --git a/site/src/cut/lib/store.test.ts b/site/src/cut/lib/store.test.ts
index d1bb634d..bb99a7a7 100644
--- a/site/src/cut/lib/store.test.ts
+++ b/site/src/cut/lib/store.test.ts
@@ -974,6 +974,36 @@ describe("bars and clip edges", () => {
expect(clipById(b.id).start).toBeCloseTo(4);
});
+ test("a sound dissolve hands over on the audio and leaves the picture cut", () => {
+ const av = asset(4);
+ const a = vclip({ track: 0, start: 0, out: 4, assetId: av.id });
+ const b = vclip({ track: 0, start: 4, out: 4, assetId: av.id });
+ useEditor.setState({ assets: [av], clips: [a, b] });
+ s().setClipTransition(a.id, 0.8, "audiocross");
+ const spans = getClipSpans(s().clips, s().assets, 0);
+ // The window rides the sound field; every reader of the picture blend
+ // sees a plain cut.
+ expect(spans[0].transitionOut).toBe(0);
+ expect(spans[0].soundOut).toBeCloseTo(0.8);
+ // Switching the same bar to a picture style moves the window back.
+ s().setClipTransition(a.id, 0.8, "crossfade");
+ const after = getClipSpans(s().clips, s().assets, 0);
+ expect(after[0].transitionOut).toBeCloseTo(0.8);
+ expect(after[0].soundOut).toBe(0);
+ });
+
+ test("a sound dissolve on an open edge plays nothing", () => {
+ const av = asset(4);
+ const a = vclip({ track: 0, start: 0, out: 4, assetId: av.id });
+ useEditor.setState({ assets: [av], clips: [a] });
+ // Dropped on a clip's head there is no next clip's sound to cross into,
+ // so it becomes no entrance animation at all.
+ const bar = s().addTransition({ start: 0, seconds: 0.6, style: "audiocross" });
+ expect(bar).toBeTruthy();
+ expect(clipById(a.id).animIn).toBeUndefined();
+ expect(clipById(a.id).animOut).toBeUndefined();
+ });
+
test("a pre-bar doc's clip fields become bars once, and derive back unchanged", () => {
const av = asset(4);
const a = vclip({
@@ -1089,6 +1119,35 @@ describe("bars and clip edges", () => {
expect(clipById(a.id).animOut).toEqual({ style: "fade", seconds: 0.8 });
});
+ test("a sound dissolve on an open edge reads as parked, not as a live bar", () => {
+ const av = asset(8);
+ const a = vclip({ track: 0, start: 0, out: 4, assetId: av.id });
+ const b = vclip({ track: 0, start: 6, out: 4, assetId: av.id });
+ useEditor.setState({ assets: [av], clips: [a, b] });
+ // A sound dissolve dropped on a's open tail. It has no picture ramp and
+ // nothing to blend into, so it plays nothing and has to say so.
+ const id = s().addTransition({ start: 3.2, seconds: 0.8, style: "audiocross" });
+ expect(parkedTransitions(s().clips, s().transitions).map((t) => t.id)).toEqual([id]);
+ expect(clipById(a.id).animOut).toBeUndefined();
+ // The same bar on a real cut plays it, and stops reading as parked:
+ // pulling b up to a's tail turns that open edge into a handover.
+ s().updateClip(b.id, { start: 4 });
+ expect(parkedTransitions(s().clips, s().transitions)).toEqual([]);
+ expect(clipById(a.id).transition).toBeCloseTo(0.8);
+ expect(clipById(a.id).transitionStyle).toBe("audiocross");
+ });
+
+ test("a sound dissolve on an open edge leaves the boundary for a bar that plays it", () => {
+ const av = asset(8);
+ const a = vclip({ track: 0, start: 0, out: 4, assetId: av.id });
+ useEditor.setState({ assets: [av], clips: [a] });
+ const sound = s().addTransition({ start: 3.2, seconds: 0.8, style: "audiocross" });
+ // A picture bar on the same tail takes the edge the sound bar cannot.
+ s().addTransition({ start: 3.2, seconds: 0.8, style: "crossfade" });
+ expect(clipById(a.id).animOut).toEqual({ style: "fade", seconds: 0.8 });
+ expect(parkedTransitions(s().clips, s().transitions).map((t) => t.id)).toEqual([sound]);
+ });
+
test("a copied bar pastes onto the cut nearest the playhead", () => {
const av = asset(12);
const a = vclip({ track: 0, start: 0, out: 4, assetId: av.id });
diff --git a/site/src/cut/lib/store.ts b/site/src/cut/lib/store.ts
index f613058d..9e1553f5 100644
--- a/site/src/cut/lib/store.ts
+++ b/site/src/cut/lib/store.ts
@@ -60,7 +60,7 @@ import { useGenNotify } from "./genNotify";
import { clampPlayhead, playheadAt, previewAt, setPlayhead, setSkim } from "./playhead";
import { engineTranscribeSamples, withEngineStt } from "./localStt";
import { trackLocale } from "./subtitles";
-import { ANIM_STYLE_IDS, animStyleOfTransition, clipPoseAt, DEFAULT_BACKGROUND, emptySubtitles, frameOf, IMAGE_CLIP_SECONDS, isEffectOverlay, isStickerOverlay, MAX_SUBTITLE_LANES, mediaUrl, migrateBehindSubject, migrateLegacyTransitions, normalizeAspect, overlayAnimStyle, projectBackground, SPEED_FLOOR, SPEED_MIN, stampOverlayKinds, stripDefaultOverlayKinds, TRANSITION_MAX, TRANSITION_STYLE_IDS, transitionStyleOfAnim } from "./types";
+import { ANIM_STYLE_IDS, animStyleOfTransition, clipPoseAt, DEFAULT_BACKGROUND, emptySubtitles, frameOf, IMAGE_CLIP_SECONDS, isAudioTransition, isEffectOverlay, isStickerOverlay, MAX_SUBTITLE_LANES, mediaUrl, migrateBehindSubject, migrateLegacyTransitions, normalizeAspect, overlayAnimStyle, projectBackground, SPEED_FLOOR, SPEED_MIN, stampOverlayKinds, stripDefaultOverlayKinds, TRANSITION_MAX, TRANSITION_STYLE_IDS, transitionStyleOfAnim } from "./types";
import { liftMoveTracks } from "./textMotion";
import { readTextStyle } from "./textStyle";
import { loadUiState, saveUiState, type ProjectUiState } from "./uiState";
@@ -4698,6 +4698,15 @@ function transitionBoundaries(clips: VideoClip[]): TransitionBoundary[] {
return out;
}
+/** Whether a bar's style has anything to play at this kind of boundary. A
+ * cut hands one clip's sound and picture to the next, so every style plays
+ * one. An open head or tail has only the single clip's edge to ramp, which is
+ * a picture ramp — a sound dissolve has nothing to blend there, so it never
+ * claims the boundary: it reads as parked, which is what it is, and the AI's
+ * debris report names it. */
+const playsBoundary = (style: TransitionStyle, kind: TransitionRole["kind"]): boolean =>
+ kind === "cut" || animStyleOfTransition(style) !== null;
+
/**
* Match each transition bar to the boundaries it lines up with, by time
* alone: a bar plays every cut or open tail its end sits on and every open
@@ -4723,6 +4732,7 @@ export function resolveTransitions(
.filter(
(b) =>
!taken.has(b) &&
+ playsBoundary(t.style, b.kind) &&
Math.abs((b.kind === "in" ? t.start : t.start + t.seconds) - b.at) <= TOUCH_EPS
)
.sort((a, b) => rank[a.kind] - rank[b.kind]);
@@ -4813,7 +4823,9 @@ export function deriveTransitionFields(
const transitionStyle = cut && cut.style !== "crossfade" ? cut.style : undefined;
const animOf = (t: TimelineTransition | undefined): ClipAnim | undefined => {
if (!t) return undefined;
- const style = animStyleOfTransition(t.style);
+ // Only a style with an edge ramp reaches an open edge — a sound
+ // dissolve never takes one (`playsBoundary`).
+ const style = animStyleOfTransition(t.style)!;
return {
// Upper tracks composite through alpha, so their edges render the
// ramps alpha can express.
@@ -5049,13 +5061,18 @@ function buildClipSpans(clips: VideoClip[], assets: MediaAsset[], track: number)
const len = clipLen(clip);
const next = present[i + 1]?.clip;
// The blend into the next clip, live only at a cut the pair actually
- // makes; clips dragged apart dissolve into nothing.
+ // makes; clips dragged apart dissolve into nothing. A sound style hands
+ // over on the audio alone, so its window lands on the other field and
+ // every reader of the picture blend sees a plain cut.
+ const overlap = transitionOverlap(clip, next);
+ const onSound = isAudioTransition(clip.transitionStyle);
spans.push({
clip,
asset,
start: clip.start,
len,
- transitionOut: transitionOverlap(clip, next),
+ transitionOut: onSound ? 0 : overlap,
+ soundOut: onSound ? overlap : 0,
});
}
return spans;
diff --git a/site/src/cut/lib/types.test.ts b/site/src/cut/lib/types.test.ts
index 61eebdfa..fae7b449 100644
--- a/site/src/cut/lib/types.test.ts
+++ b/site/src/cut/lib/types.test.ts
@@ -1,5 +1,5 @@
import { describe, expect, test } from "bun:test";
-import { clipCovers, contentRect, migrateLegacyTransitions, normalizeAspect, TRANSITION_STYLE_IDS, type VideoClip } from "./types";
+import { animStyleOfTransition, AUDIO_TRANSITION_STYLES, clipCovers, contentRect, migrateLegacyTransitions, normalizeAspect, TRANSITION_STYLE_GROUPS, TRANSITION_STYLE_IDS, TRANSITION_STYLE_LABELS, type VideoClip } from "./types";
/** A minimal track-0 clip; legacy docs carry retired transitionStyle strings. */
function clip(
@@ -89,6 +89,20 @@ describe("migrateLegacyTransitions", () => {
expect(out[0].animOut).toBeUndefined();
});
+ test("every style a bar can carry is a style the tools can name", () => {
+ // The picker groups are the Transitions tab's layout, and the sound
+ // styles are picked elsewhere — so the id list is the groups plus them,
+ // never the groups alone (the tool enum and the doc validator read it).
+ for (const id of AUDIO_TRANSITION_STYLES) {
+ expect(TRANSITION_STYLE_IDS).toContain(id);
+ expect(TRANSITION_STYLE_GROUPS.flatMap((g) => g.ids)).not.toContain(id);
+ expect(TRANSITION_STYLE_LABELS[id]).toBeTruthy();
+ // A sound handover has no picture blend to fall back to on an open edge.
+ expect(animStyleOfTransition(id)).toBeNull();
+ }
+ expect(new Set(TRANSITION_STYLE_IDS).size).toBe(TRANSITION_STYLE_IDS.length);
+ });
+
test("current style ids are never treated as legacy", () => {
for (const id of TRANSITION_STYLE_IDS) {
const clips = [clip("a", 0, { transition: 0.5, transitionStyle: id }), clip("b", 4)];
diff --git a/site/src/cut/lib/types.ts b/site/src/cut/lib/types.ts
index 8481e641..0c7d592f 100644
--- a/site/src/cut/lib/types.ts
+++ b/site/src/cut/lib/types.ts
@@ -604,7 +604,8 @@ export type TransitionStyle =
| "circleopen"
| "circleclose"
| "splitopen"
- | "splitclose";
+ | "splitclose"
+ | "audiocross";
/** The ffmpeg xfade transition each style renders with on export. Doubles as
* the sanitizing allowlist: the server looks the style up here and falls back
@@ -628,9 +629,23 @@ export const TRANSITION_XFADE: Record<TransitionStyle, string> = {
circleclose: "circleclose",
splitopen: "vertopen",
splitclose: "vertclose",
+ // The sound dissolve never reaches xfade — the picture cuts and the renders
+ // branch before this map. The entry keeps the record total.
+ audiocross: "fade",
};
-/** Picker layout: styles grouped by family, in display order. */
+/** The styles that hand over on the sound alone: the picture cuts, and the
+ * outgoing fades out into the cut while the incoming fades in out of it. They
+ * take no picture blend, so a clip's own entrance and exit animations still
+ * play across a cut one of them sits on. */
+export const AUDIO_TRANSITION_STYLES: TransitionStyle[] = ["audiocross"];
+
+export const isAudioTransition = (style: string | undefined): boolean =>
+ !!style && (AUDIO_TRANSITION_STYLES as string[]).includes(style);
+
+/** Picker layout for the Transitions tab: the styles that blend the picture,
+ * grouped by family, in display order. The sound styles are picked from the
+ * Effects tab's Sound family, beside the other treatments on the sound. */
export const TRANSITION_STYLE_GROUPS: { label: string; ids: TransitionStyle[] }[] = [
{ label: "Fade", ids: ["crossfade", "dipblack", "dipwhite", "blur"] },
{ label: "Zoom", ids: ["crosszoom"] },
@@ -639,9 +654,11 @@ export const TRANSITION_STYLE_GROUPS: { label: string; ids: TransitionStyle[] }[
{ label: "Shape", ids: ["circleopen", "circleclose", "splitopen", "splitclose"] },
];
-export const TRANSITION_STYLE_IDS: TransitionStyle[] = TRANSITION_STYLE_GROUPS.flatMap(
- (g) => g.ids
-);
+/** Every style a bar can carry, whichever tab picks it. */
+export const TRANSITION_STYLE_IDS: TransitionStyle[] = [
+ ...TRANSITION_STYLE_GROUPS.flatMap((g) => g.ids),
+ ...AUDIO_TRANSITION_STYLES,
+];
export const TRANSITION_STYLE_LABELS: Record<TransitionStyle, string> = {
crossfade: "Cross fade",
@@ -661,6 +678,7 @@ export const TRANSITION_STYLE_LABELS: Record<TransitionStyle, string> = {
circleclose: "Circle close",
splitopen: "Split open",
splitclose: "Split close",
+ audiocross: "Sound dissolve",
};
/** Peak scale the zoom transitions push into (preview and export). */
@@ -723,9 +741,10 @@ export function overlayAnimStyle(style: AnimStyle): "fade" | "zoom" {
* against it, so the same drag reads as the clip arriving or leaving. The
* directional styles keep their direction, the zooms keep their push, and the
* shaped wipes — which need a second picture to reveal — come through as the
- * ramp closest to them.
+ * ramp closest to them. A sound style becomes nothing: it hands one clip's
+ * sound to the next one's, and an open edge has no next one.
*/
-const EDGE_ANIM: Record<TransitionStyle, AnimStyle> = {
+const EDGE_ANIM: Record<TransitionStyle, AnimStyle | null> = {
crossfade: "fade",
dipblack: "fade",
dipwhite: "fade",
@@ -743,6 +762,7 @@ const EDGE_ANIM: Record<TransitionStyle, AnimStyle> = {
wiperight: "slideright",
wipeup: "slideup",
wipedown: "slidedown",
+ audiocross: null,
};
const ANIM_TRANSITION: Record<AnimStyle, TransitionStyle> = {
@@ -755,7 +775,8 @@ const ANIM_TRANSITION: Record<AnimStyle, TransitionStyle> = {
slidedown: "pushdown",
};
-export const animStyleOfTransition = (style: TransitionStyle): AnimStyle => EDGE_ANIM[style];
+export const animStyleOfTransition = (style: TransitionStyle): AnimStyle | null =>
+ EDGE_ANIM[style];
export const transitionStyleOfAnim = (style: AnimStyle): TransitionStyle =>
ANIM_TRANSITION[style] ?? "crossfade";
@@ -1165,6 +1186,13 @@ export interface ClipSpan {
* over this clip's live tail. Spans never intersect — the next one starts
* exactly at this one's end, and plays from its head there. */
transitionOut: number;
+ /** Sound dissolve into the next span, in timeline seconds: this clip's
+ * audio fades out over the window before the cut and the next clip's fades
+ * in over the window after it, while the picture cuts. Zero unless the
+ * transition at the cut is one of the sound styles, and never set at the
+ * same time as `transitionOut` — a handover blends the picture or the
+ * sound. */
+ soundOut: number;
}
/** The document persisted as project.json inside each project folder. */
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index 503a9a66..5c761e3f 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -148,18 +148,18 @@ Cutting speech well: pace is part of the message. Keep a beat of the speaker's o
"transitions-and-fades": `# Transitions, animations, looks & fades
Route the ask to the right feature:
-- set_transition: a styled join between one video clip and the next, blended across the cut. It never moves clips or shortens anything, and it plays only while the pair touches.
+- set_transition: a styled join between one video clip and the next, blended across the cut. It never moves clips or shortens anything, and it plays only while the pair touches. One style, audiocross, blends the sound alone — the picture cuts, the outgoing clip's sound fades out into the cut and the incoming clip's fades in out of it (it plays at cuts only, so it does nothing on a clip's open head or tail). In the UI it is picked from the Effects tab's Sound family, beside the treatments on the sound; every style that blends the picture is picked from the Transitions tab.
- set_animation: one clip's own entrance ("in") or exit ("out") — the clip fades/zooms/pops/slides on its own edge. Never moves neighbors. For "fade this clip in", "make it slide in", use this. A wipe is a transition, not an animation.
- set_project_fade: the whole video fades in from black at the start and/or out to black at the end — picture and full mix (titles, captions, soundtrack). Survives clip reordering. For "fade in the video", "fade to black at the end", use this.
- set_background: the color of the frame itself, behind everything, for the whole cut. For "make the background white", or any cut whose picture is titles and shapes rather than footage. Covered in the graphics skill.
- update_audio fadeIn/fadeOut: audio-only ramps on one soundtrack clip ("fade the music out").
- An ask beyond every preset here — a reference template, a grid or mosaic reveal, a multi-panel build — is a construction job: the graphics skill's composite-builds section shows how overlay video tracks, masks, and keyframes compose into it.
-set_transition styles (clipId = the leading clip; 0 clears, max 2s): crossfade blends; crosszoom adds a zoom punch; dipblack/dipwhite dip through a solid color; blur defocuses through the cut; pushleft/right/up/down shove the old frame out; wipeleft/right/up/down reveal with a hard traveling edge; circleopen/close and splitopen/close are shape reveals. Directional names describe the motion.
+set_transition styles (clipId = the leading clip; 0 clears, max 2s): crossfade blends; crosszoom adds a zoom punch; dipblack/dipwhite dip through a solid color; blur defocuses through the cut; pushleft/right/up/down shove the old frame out; wipeleft/right/up/down reveal with a hard traveling edge; circleopen/close and splitopen/close are shape reveals; audiocross cuts the picture and dissolves only the sound across the cut, for a handover the eye takes as a cut and the ear takes as a blend. Directional names describe the motion.
set_animation styles: fade (audio follows), zoom, pop, slideleft/right/up/down — the slide names are the motion direction (slideleft enters from the right edge moving left, or exits off the left). Each edge holds one effect and the last pick wins: set_animation on a transitioned edge replaces that transition, set_transition clears the animations adjacent to its joint.
Bars and the debris they leave: a transition lives on the transitions row as its own object at an absolute time, listed in editor_state \`transitions\` with the boundary it \`plays\`. It plays whatever cut or clip edge its window lines up with; a bar lining up with nothing reads \`parked: true\` and plays nothing — the badge is on the row, the blend is gone. Moving, retiming, or replacing clips is what parks a bar, so every timeline mutation returns \`parkedTransitions\` when its edit left one. Own that debris in the same turn you make it: a bar you parked and know the intent of, reattach with set_transition/set_animation on the edge it belongs to, or clear with remove_transition. When you can't tell what a parked bar was for — it predates your edits, or the user's ask doesn't say where the blend should end up — do not guess and do not leave it silent: name what's there ("there's a 0.5s dip to black at 29.4s that isn't on a cut any more") and ask whether to remove it or move it to a specific joint. The same holds for anything else on the timeline that reads as leftover from your own edits: say what you see and confirm before deleting the user's work.
-Picking for a vibe: "smooth" → crossfade; "punchy/energetic" → crosszoom or a push; "dramatic scene change" → dipblack 0.5–0.8s; "dreamy" → blur (or the dreamy look); "retro" → vhs or vintage look. Between clips 0.4–0.8s reads well; 1s+ is slow and cuts total duration. The editing-taste skill covers when a joint should stay a hard cut and how to keep transition styling consistent across the video.
+Picking for a vibe: "smooth" → crossfade; "let the sound carry over a hard cut", "audio crossfade", "dissolve just the audio" → audiocross; "punchy/energetic" → crosszoom or a push; "dramatic scene change" → dipblack 0.5–0.8s; "dreamy" → blur (or the dreamy look); "retro" → vhs or vintage look. Between clips 0.4–0.8s reads well; 1s+ is slow and cuts total duration. The editing-taste skill covers when a joint should stay a hard cut and how to keep transition styling consistent across the video.
UI: select a clip → Inspector "Effects" opens the panel (Transition / In / Out / Looks tabs). Preview and export render the same treatment.
What the user sees on the transitions row: one badge per bar, and nothing else — the row holds no buttons, no placeholders, and no add-affordances. A playing bar is blue; a parked one is the same badge in grey. So a user reporting "greyed out transitions", "grey circles", or "dead/lingering transitions" on that row is describing parked bars, which are real objects saved in their project — never dismiss them as editor chrome or claim they won't export, because the badge is the transition itself. Read \`transitions\` in editor_state and answer from the parked entries there. Grey elsewhere means something different: a clip, element, or track shown grayed is hidden (the \`hidden\` flag), which does drop out of playback and export.`,
@@ -172,7 +172,7 @@ Titles (add_title / update_overlay): text, size (frame px; the design short side
Shapes (add_shape): rect, ellipse, line, arrow. w/h are frame fractions; rect/ellipse take fill + fill_opacity, corner radius (rect), and an outline (stroke_color/stroke_width); line/arrow draw in \`fill\` with h as their thickness, and rotation gives them their direction (0° points right, 90° points down).
Stickers (add_sticker): a project image asset (generated cutouts in the Elements panel carry origin "sticker"); w is a frame-width fraction and height follows the source's aspect. Lottie JSON uploads become animated stickers that loop for the element's duration.
Custom stickers (create_sticker): one call generates the image, removes the background, adds the die-cut outline, and places it — use it when the user asks for a sticker of something they don't have (it spends credits like image generation).
-Effects (add_effect): time-ranged treatments over the finished cut. Three families, one list: the picture treatments (${EFFECT_TREATMENTS.join(", ")}), the graded looks (${LOOK_EFFECTS.join(", ")}), and the audio treatments (${AUDIO_EFFECT_IDS.join(", ")}). A picture effect treats the footage and everything laid over it; an audio effect treats everything audible under its window at once — clip sound, upper-track sound and the soundtrack — so "make this bit sound like a phone call", "put an echo on that line", or "muffle the music here" is an effect element over that stretch, and a per-clip volume, fade or duck is the audio tools instead. A cut's own audio crossfade at a join belongs to the transition on that cut, not here. Placed like any element and tuned with \`amount\` (update_overlay); an effect added over a clip opens covering that clip, and trims like anything else. One effect at a time reads best, amounts under ~0.5. Effects are stylized post treatments on a stretch of timeline; pure color — exposure, warmth, a cinematic or film palette on one clip — is the per-clip grading tools instead (color-grading skill), whose preset catalog is separate from the look effects.
+Effects (add_effect): time-ranged treatments over the finished cut. Three families, one list: the picture treatments (${EFFECT_TREATMENTS.join(", ")}), the graded looks (${LOOK_EFFECTS.join(", ")}), and the audio treatments (${AUDIO_EFFECT_IDS.join(", ")}). A picture effect treats the footage and everything laid over it; an audio effect treats everything audible under its window at once — clip sound, upper-track sound and the soundtrack — so "make this bit sound like a phone call", "put an echo on that line", or "muffle the music here" is an effect element over that stretch, and a per-clip volume, fade or duck is the audio tools instead. A handover on the sound at a cut belongs to the transition on that cut — the audiocross style — not here. Placed like any element and tuned with \`amount\` (update_overlay), and trims like anything else: a picture effect dropped over a clip opens covering that clip, while an audio effect opens at the standard length wherever it was placed — a treatment on the sound is aimed at a moment, so give it the window you mean. One effect at a time reads best, amounts under ~0.5. Effects are stylized post treatments on a stretch of timeline; pure color — exposure, warmth, a cinematic or film palette on one clip — is the per-clip grading tools instead (color-grading skill), whose preset catalog is separate from the look effects.
Advanced text (update_overlay): italic, align (multi-line), letter_spacing (em), line_height, stretch_x/stretch_y (glyph stretch multipliers, the side grips' axis-only resize), stroke_color/stroke_width (em outline behind the fill), and richer shadows. Bundled Google families join the system set — ${GOOGLE_FONT_IDS.join(", ")} — and so do the user's own: a .ttf/.otf/.woff/.woff2 file dropped on the Library is offered in every project under a font id like "font:<id>". Editor_state lists the ones this account has under \`fonts\` — use an id from there instead of guessing at a family name, and when someone asks for a typeface that is not in the list, tell them the Library takes font files.
Animation (set_overlay_animation): every element takes preset In/Out ramps (${OVERLAY_ANIM_STYLE_IDS.join(", ")} — typewriter for titles only; the per-letter styles sweep a title character by character), a Loop (${OVERLAY_LOOP_STYLE_IDS.join(", ")}) that runs its whole duration, and a Move that says what it does WHILE it holds. Entrances 0.3–0.6s read well; one loop per scene, sparingly; a move on the lines that matter. All four are slots on the element, applied internally; the keyframe track (set_overlay_keyframes) is the separate power tier for motion the names do not cover, and slots compose over it.
Keyframes (set_overlay_keyframes): for motion no preset covers — a path across the frame, a slow push in, a drift that holds first. A key is a whole pose (position, scale, rotation, opacity) at a time in seconds from the element's start; the pose moves linearly between keys and holds outside them, and presets still compose on top. Two or three keys carry almost everything; reach for a preset first and keyframes when the user describes a specific path or timing. In the UI the diamond in the inspector adds a key at the playhead, and dragging the element then records into it. set_clip_keyframes is the same track on a video clip (any track): pans, push-ins, picture-in-picture flights, spins, and fades on the footage itself.
@@ -300,7 +300,7 @@ ${textMoveCatalog()}
Lyrics and scripts the user already has: sync_lyrics times their exact words against the audio (transcribing first when needed) and writes them to a caption track with per-word timings; set_caption_look then dresses the track — preset, size, font, karaoke word highlight, accent color — from a text-videos look id or field by field. The text-videos skill has the whole flow, including turning the same timed lines into full-frame cards.
Soundtrack clips: volume 0..1.5, fadeIn/fadeOut seconds (max half the clip), start = timeline position, in/out = trim inside the source; clips can spread across several soundtrack lanes (the \`lane\` field), new sounds slide to free space in their lane. Fades render with ffmpeg afade on export.
Ducking: a soundtrack clip's \`duck\` (0..1, via update_audio) lowers ALL other audio — video-clip sound and other music — to that gain while the clip plays; 1 clears it. Voiceovers set this so narration sits over quieter music. It applies in both the preview and the export.
-Audio effects (add_effect): the treatments over the sound — ${AUDIO_EFFECT_IDS.join(", ")} — placed on the timeline as elements, each treating everything audible under its window at once. Reach for one when the ask is how a stretch SOUNDS ("make this line sound like a phone call", "echo on that word", "muffle the music behind the voiceover"); \`amount\` says how far the treatment goes, and the effect trims and moves like any element. Loudness, fades and ducking on one clip stay with the audio tools; the crossfade across a cut stays with the transition on that cut.
+Audio effects (add_effect): the treatments over the sound — ${AUDIO_EFFECT_IDS.join(", ")} — placed on the timeline as elements, each treating everything audible under its window at once. Reach for one when the ask is how a stretch SOUNDS ("make this line sound like a phone call", "echo on that word", "muffle the music behind the voiceover"); \`amount\` says how far the treatment goes, and the effect trims and moves like any element. Loudness, fades and ducking on one clip stay with the audio tools; a dissolve across a cut stays with the transition on that cut (the audiocross style).
Voiceover (Donkey's hosted speech model — signed in, spends credits, like image/video generation):
- voiceover_generate(script, voice?, direction?, duck?, add_to_timeline?, start?, language?): synthesizes the script into a playable chat card; add_to_timeline:true (or a start) also drops it on the soundtrack at the playhead (or start) when the user asked for it in the cut. Defaults to a 0.4 duck when placed so it sits over other audio. Voices are Gemini's prebuilt set — list_voices returns them (id + one-word character like Warm, Upbeat, Gravelly); omit voice for a good default. \`direction\` steers delivery in natural language ("Say warmly, like an old friend") and may ask for another language — "say it in Spanish" translates the script before synthesis; the script itself can carry inline tags like [whispers], [excited], [laughs]. \`language\` only pins pronunciation of the text as written.
- read_subtitles_aloud(voice?, direction?, duck?, track?): speaks one subtitle track's cues, each line at its own cue time — captions become narration. Needs cues first. The Inspector offers the same per-clip: "Generate audio for clip" transcribes just that clip when it has no cues yet, then voices it.
diff --git a/site/src/cut/server/exportPipeline.test.ts b/site/src/cut/server/exportPipeline.test.ts
index 9f0abed2..2b03104b 100644
--- a/site/src/cut/server/exportPipeline.test.ts
+++ b/site/src/cut/server/exportPipeline.test.ts
@@ -186,6 +186,52 @@ describe("export filtergraph timebases", () => {
expect(xfadeMismatches(g)).toEqual([]);
});
+ test("a sound dissolve cuts the picture and crosses the sound", async () => {
+ const g = await graphFor({
+ clips: [
+ clip("a.mp4", { soundCross: 0.5, transitionStyle: "audiocross" }),
+ clip("b.mp4"),
+ ],
+ });
+ const graph = g.join(";");
+ // No blend on the picture at all — the join is a plain concat.
+ expect(graph).not.toContain("xfade=");
+ expect(graph).not.toContain("tpad=start_duration");
+ // The outgoing clip's sound fades into the cut, the incoming clip's out
+ // of it, and the pieces concat back to the same length.
+ expect(g).toContain("[a0]afade=t=out:st=3.500:d=0.500[ah1]");
+ expect(g).toContain("[a1]afade=t=in:st=0:d=0.500[ai1]");
+ expect(g).toContain("[ah1][ai1]concat=n=2:v=0:a=1[aj1]");
+ });
+
+ test("a sound dissolve joined into a picture transition keeps the streams matched", async () => {
+ const g = await graphFor({
+ clips: [
+ clip("a.mp4", { soundCross: 0.5, transitionStyle: "audiocross" }),
+ clip("b.mp4", { transition: 0.5, transitionStyle: "crossfade" }),
+ clip("c.mp4"),
+ ],
+ });
+ expect(xfadeMismatches(g)).toEqual([]);
+ });
+
+ test("an upper-track sound dissolve ramps the sound and never the picture", async () => {
+ const g = await graphFor({
+ clips: [clip("a.mp4", { out: 8 })],
+ overlayVideos: [
+ { file: "u1.mp4", in: 0, out: 4, start: 0, track: 1, muted: false, tailSound: 0.5 },
+ { file: "u2.mp4", in: 0, out: 4, start: 4, track: 1, muted: false, headSound: 0.5 },
+ ],
+ });
+ const graph = g.join(";");
+ // The sound crosses at the cut between the two upper-track clips…
+ expect(graph).toContain("afade=t=out:st=3.500:d=0.500");
+ expect(graph).toContain("afade=t=in:st=0:d=0.500");
+ // …and the picture keeps its opacity throughout: no alpha ramp anywhere.
+ expect(graph).not.toContain("fade=t=in:st=0:d=0.500:alpha=1");
+ expect(graph).not.toContain("alpha=1");
+ });
+
test("every join transitioned, styles mixed", async () => {
const g = await graphFor({
clips: [
diff --git a/site/src/cut/server/exportPipeline.ts b/site/src/cut/server/exportPipeline.ts
index b0697e2a..41e97219 100644
--- a/site/src/cut/server/exportPipeline.ts
+++ b/site/src/cut/server/exportPipeline.ts
@@ -50,6 +50,12 @@ export interface ExportSpec {
speed?: number; // playback rate, default 1
/** Transition into the next clip, in timeline seconds (overlap). */
transition?: number;
+ /** Sound dissolve into the next clip, in timeline seconds. The picture
+ * cuts; this clip's sound fades out over the window before the cut and
+ * the next clip's fades in over the window after it. Carried apart from
+ * `transition` so a join that blends only the sound is not read as a
+ * picture blend anywhere. */
+ soundCross?: number;
/** Transition look id, resolved to an xfade name through the
* TRANSITION_XFADE allowlist (unknown ids render as a plain fade). Cross
* zoom renders as the fade plus zoom ramps on both segments' overlap
@@ -127,6 +133,10 @@ export interface ExportSpec {
tailFade?: number;
headZoom?: number;
tailZoom?: number;
+ /** Sound dissolve ramps, timeline seconds from this overlay's head/tail:
+ * the level crosses at the cut and the picture is left alone. */
+ headSound?: number;
+ tailSound?: number;
/** A still image: looped for the clip's length instead of trimmed. */
image?: boolean;
/** Manual color adjustments, baked into this overlay's segment. */
@@ -1274,8 +1284,10 @@ export async function runExport(
// incoming segment's head with its cloned first frame and runs the xfade
// across the outgoing clip's last blend-window seconds: the held frame
// arrives over the live tail, and the real segment starts exactly at the
- // cut. Its sound is a tail fade on the outgoing side and a hard join. The
- // rest hard-cut (concat). Fold left so mixed sequences chain correctly.
+ // cut. Its sound is a tail fade on the outgoing side and a hard join. A
+ // sound dissolve cuts the picture and crosses the sound instead — a tail
+ // fade against the incoming clip's head fade. The rest hard-cut (concat).
+ // Fold left so mixed sequences chain correctly.
let vAcc = segLabel[0];
let aAcc = "a0";
let acc = clipDur(spec.clips[0]); // running timeline length of the accumulator
@@ -1284,9 +1296,19 @@ export async function runExport(
const durJ = clipDur(spec.clips[j]);
// The blend can't exceed most of either clip, matching the editor clamp.
const d = Math.min(prev.transition ?? 0, acc * 0.9, durJ * 0.9);
+ const cross = Math.min(prev.soundCross ?? 0, acc * 0.9, durJ * 0.9);
const vOut = `vj${j}`;
const aOut = `aj${j}`;
- if (d > 0.01) {
+ if (cross > 0.01) {
+ // The picture cuts: concat, re-stamped for a later xfade the same way
+ // a plain join is. The sound crosses the cut.
+ filters.push(`[${vAcc}][${segLabel[j]}]concat=n=2:v=1:a=0,fps=${fps}[${vOut}]`);
+ filters.push(
+ `[${aAcc}]afade=t=out:st=${num(Math.max(0, acc - cross))}:d=${num(cross)}[ah${j}]`
+ );
+ filters.push(`[a${j}]afade=t=in:st=0:d=${num(cross)}[ai${j}]`);
+ filters.push(`[ah${j}][ai${j}]concat=n=2:v=0:a=1[${aOut}]`);
+ } else if (d > 0.01) {
const offset = Math.max(0, acc - d);
// The style id resolves through the allowlist map; anything unknown
// (or an old spec without a style) renders as a plain fade.
@@ -1331,6 +1353,10 @@ export async function runExport(
const tz = Math.max(0, Math.min(oc.tailZoom ?? 0, olen - hz));
const hf = Math.max(0, Math.min(oc.headFade ?? 0, olen));
const tf = Math.max(0, Math.min(oc.tailFade ?? 0, olen - hf));
+ // A sound dissolve's own ramps: the sound crosses at the cut, the picture
+ // cuts with it.
+ const hs = Math.max(0, Math.min(oc.headSound ?? 0, olen));
+ const ts = Math.max(0, Math.min(oc.tailSound ?? 0, olen - hs));
const ramped = hz > 0.01 || tz > 0.01;
const maskIdx = overlayMaskInput.get(oc);
const subjMask = subjectActive && matteConsumers > 0 ? oc.mask?.subject : undefined;
@@ -1493,9 +1519,12 @@ export async function runExport(
const tempo = ospeed !== 1 ? `${atempoChain(ospeed)},` : "";
const vol = (oc.volume ?? 1) !== 1 ? `volume=${num(oc.volume ?? 1)},` : "";
// The picture's fade edges carry the sound with them; zoom edges don't.
+ // A sound dissolve's ramps are here and nowhere else.
const afades =
(hf > 0.01 ? `afade=t=in:st=0:d=${num(hf)},` : "") +
- (tf > 0.01 ? `afade=t=out:st=${num(Math.max(0, olen - tf))}:d=${num(tf)},` : "");
+ (tf > 0.01 ? `afade=t=out:st=${num(Math.max(0, olen - tf))}:d=${num(tf)},` : "") +
+ (hs > 0.01 ? `afade=t=in:st=0:d=${num(hs)},` : "") +
+ (ts > 0.01 ? `afade=t=out:st=${num(Math.max(0, olen - ts))}:d=${num(ts)},` : "");
const delayMs = Math.max(0, Math.round(oc.start * 1000));
const lab = `ovs${k}`;
filters.push(
diff --git a/site/src/cut/server/transcribe.ts b/site/src/cut/server/transcribe.ts
index 8bc031e2..f3f0f4e6 100644
--- a/site/src/cut/server/transcribe.ts
+++ b/site/src/cut/server/transcribe.ts
@@ -362,7 +362,9 @@ async function runTranscribe(job: TranscribeJob, spec: TranscribeSpec) {
// Join clip audio exactly as the timeline (and export) do: clips abut, a
// transition fades the outgoing clip's tail while the next enters clean at
// the cut, so the chain is a flat concat with tail fades — the mix runs
- // exactly the timeline's length and every cue lands where it plays.
+ // exactly the timeline's length and every cue lands where it plays. A
+ // sound dissolve's own fades stay out of it: they change no cue's time,
+ // and words are transcribed best at the level they were spoken.
// Skipped entirely for an audio-only cut (no clips) — there is no clip-audio
// chain to build, only the soundtrack, mixed below. (`spec.clips[0]` would be
// undefined otherwise.)