Skip to content

Open source CapCut alternative.

TypeScript

feat(site): a clip frames itself inside its box — zoom, pan, turn, fade, shadow [rebuild]

David Le3cbe9752e9c3
diff --git a/docs/guides/cut/README.md b/docs/guides/cut/README.md
index da5f2246..fe5afe5d 100644
--- a/docs/guides/cut/README.md
+++ b/docs/guides/cut/README.md
@@ -30,6 +30,8 @@ The client probes the engine's health endpoint (dedicated port first, dev server
**A cut needs no footage.** The frame starts as a color the project owns, and everything else — clips, titles, shapes, stickers — draws over it. So a video made entirely of text and graphics is an ordinary project: it plays, exports, shares, and renders on the worker like any other, and the same color letterboxes a fitted clip and fills a gap on track 0. Elements stack by row: two on one row slide clear of each other in time, so a title that sits over a card lives a row above it.
+**Framing is per clip.** Every video clip meets its box — the whole frame, or the region it was given — through one geometry: fitted with margins or covering with a crop, zoomed in further, panned to choose what the crop keeps, turned, faded, cornered, masked, and shadowed. One function computes where the picture lands, the preview draws that rectangle, and the export graph scales and crops to the same numbers, so a reframed clip looks the same on every surface, freeze frames and headless renders included. On the stage the selected clip wears two outlines: a blue box for its layout, with grips to move, resize, and rotate it, and a grey dashed one standing where the source really lands, so the part a crop discards stays visible — its corners zoom the picture and its interior pans it. The shadow is cast by the shape the clip actually shows, mask included, which is what makes a circle-masked camera bubble throw a circular shadow.
+
**Words as the picture.** A lyric video, a kinetic-typography passage, a quote sequence: the assistant places a whole run of lines in one call, carrying a named look. A look is the design of the whole run rather than of one line — the frame color, the cards it cycles, and the ensemble the run walks across its length: the faces it rotates through, the entrances, where successive lines land in the frame, and what each line does while it holds. That is what stops a hundred lines coming out identical and dead center, which is the failure a text video actually has. The looks live in one registry the tools and the skills both read, so adding a look teaches the assistant a new style with no other change. Words ride one of two carriers — a caption track, which holds per-word timings and can light each word as it is sung, or elements, which give each line its own card, color and motion. Text the user already has (lyrics, a script) is timed by aligning it to a transcript of the audio: the recognizer supplies the clock and the user's wording is kept exactly, which is what makes it survive singing.
**Motion is data.** Every animation in the editor — an entrance, an exit, a loop, and the moves an element makes while it holds — is one format: a pose keyframed over a normalized window with cubic-bezier easing, plus a range selector that says how the motion is handed from one character to the next. That is the After Effects text-animator model, the one Lottie serializes, and picking it means a preset and a hand-keyed track are the same object at two levels of convenience: a named move writes the ordinary pose track the user can then drag, and anything no name covers is keyed directly. The library is a JSON asset in the effects kit rather than code, and the menus, tool enums, prompts, and the frame sampler's crop budget are all read from it, so adding an animation is adding an entry. One evaluator samples it, which is what keeps the preview, the export, and the headless worker showing the same motion. A slot may also carry a preset of its own rather than naming one from the library, in the same shape a library entry has — the evaluator cannot tell the two apart, so an animation composed inside a project plays wherever a shipped one does.
diff --git a/site/src/cut/components/Inspector.tools.ts b/site/src/cut/components/Inspector.tools.ts
index 797d4f9c..6fd957b8 100644
--- a/site/src/cut/components/Inspector.tools.ts
+++ b/site/src/cut/components/Inspector.tools.ts
@@ -138,14 +138,32 @@ export const INSPECTOR_TOOLS = [
{
name: "set_framing",
description:
- "Set how a video clip meets the 9:16 frame: 'fit' letterboxes the whole picture (default), 'fill' scales it to cover the frame and crops the overflow. In fill mode panX/panY (-1..1, 0=centered) choose which part stays visible — e.g. panY=-1 keeps the top.",
+ "Set how a video clip meets its box (the project frame, or its region): 'fit' letterboxes the whole picture (default), 'fill' scales it to cover the box and crops the overflow. zoom pushes further in from there, 1..4. Whenever the picture overflows, panX/panY (-1..1, 0=centered) choose what stays visible — e.g. panY=-1 keeps the top. Landscape footage in a vertical project usually wants fill plus a pan that holds the subject.",
inputSchema: obj({
clipId: str("Video clip id"),
mode: { type: "string", enum: ["fit", "fill"], description: "Framing mode" },
- panX: num("Crop pan -1 (left) .. 1 (right), fill mode only"),
- panY: num("Crop pan -1 (top) .. 1 (bottom), fill mode only"),
+ zoom: num("Zoom past the fitted size, 1 (none) .. 4"),
+ panX: num("Crop pan -1 (left) .. 1 (right), when the picture overflows"),
+ panY: num("Crop pan -1 (top) .. 1 (bottom), when the picture overflows"),
}, ["clipId", "mode"]),
},
+ {
+ name: "set_clip_style",
+ description:
+ "Style a video clip's box: rounded corners, a border ring along its edge, and a drop shadow cast by the clip's shape (its box, corners, and mask together — a circle-masked clip casts a circular shadow). Lengths are design px at the 1080 short side. Only the fields you pass change; pass clear:true to remove all styling.",
+ inputSchema: obj({
+ clipId: str("Video clip id"),
+ radius: num("Corner radius, design px (0 squares the corners)"),
+ border_width: num("Border stroke width, design px (0 removes the ring)"),
+ border_color: str("Border color (hex)"),
+ shadow_blur: num("Shadow blur radius, design px; 0 with no offset removes the shadow"),
+ shadow_x: num("Shadow x offset, design px (default 0)"),
+ shadow_y: num("Shadow y offset, design px (default 0)"),
+ shadow_color: str("Shadow ink (hex, default #000000)"),
+ shadow_opacity: num("Shadow opacity 0..1 (default 0.35)"),
+ clear: bool("Remove the box styling entirely"),
+ }, ["clipId"]),
+ },
{
name: "set_speed",
description:
diff --git a/site/src/cut/components/Inspector.tsx b/site/src/cut/components/Inspector.tsx
index 334b7edc..9a115a64 100644
--- a/site/src/cut/components/Inspector.tsx
+++ b/site/src/cut/components/Inspector.tsx
@@ -45,7 +45,7 @@ import {
} from "@donkeycut/effects-kit";
import { clipWindow, useEditor, type EditorState } from "@/cut/lib/store";
import { usePreviewTime } from "@/cut/lib/playhead";
-import { clipKeyed, clipPoseAt } from "@/cut/lib/types";
+import { CLIP_MAX_ZOOM, clipKeyed, clipPoseAt, clipZoom, contentRect } from "@/cut/lib/types";
import { AnimationCard, AnimationTiles } from "@/cut/components/AnimationTiles";
import { ColorField } from "@/cut/components/ColorField";
import { NumberField } from "@/cut/components/NumberField";
@@ -88,6 +88,7 @@ import {
clampOverlayPos,
type AudioClip,
type BoxStyle,
+ type ClipShadow,
type ColorGrade,
type FrameRect,
type LayoutId,
@@ -402,17 +403,39 @@ const formatPercent = (v: number) => `${Math.round(v * 100)}%`;
/** One-click frame layouts (Full / Top / Bottom / Left / Right / PiP) shared by
* the video-clip and overlay-clip panels. Picking one regions the clip so two
* videos can share the frame; "Full" clears the region. */
+/** The clip's box narrowed to the source's own shape: the picture area a fit
+ * clip already draws inside `rect`, which is the box that shows the whole shot
+ * with no margins around it. */
+function sourceAspectRect(
+ rect: FrameRect,
+ srcW: number,
+ srcH: number,
+ frame: { w: number; h: number }
+): FrameRect {
+ const px = contentRect(
+ { x: rect.x * frame.w, y: rect.y * frame.h, w: rect.w * frame.w, h: rect.h * frame.h },
+ srcW,
+ srcH,
+ false
+ );
+ return { x: px.x / frame.w, y: px.y / frame.h, w: px.w / frame.w, h: px.h / frame.h };
+}
+
function LayoutButtons({
rect,
onPick,
+ onSourceAspect,
}: {
rect: FrameRect;
onPick: (frame: FrameRect | undefined, fit: "fit" | "fill") => void;
+ /** Narrow the box to the source's own shape, dropping the margins. */
+ onSourceAspect?: () => void;
}) {
const currentId =
(Object.keys(LAYOUTS) as LayoutId[]).find((id) => LAYOUTS[id].label === regionLabel(rect)) ??
"full";
return (
+ <>
<div className="flex items-center justify-between gap-2 py-1">
<span className="text-[11.5px] font-medium text-muted-foreground">Layout</span>
<Select
@@ -435,12 +458,27 @@ function LayoutButtons({
</SelectContent>
</Select>
</div>
+ {onSourceAspect && (
+ <Row label="Shape">
+ <button
+ className="clip-source-aspect rounded-md border border-input px-2 py-0.5 text-[11.5px] text-muted-foreground transition-colors hover:text-foreground"
+ onClick={onSourceAspect}
+ >
+ Match source
+ </button>
+ </Row>
+ )}
+ </>
);
}
function ClipPanel({ clip }: { clip: VideoClip }) {
const asset = useEditor((s) => s.assets.find((a) => a.id === clip.assetId));
+ const aspect = useEditor((s) => s.aspect);
const updateClip = useEditor((s) => s.updateClip);
+ const zoomCk = useSliderCheckpoint();
+ const turnCk = useSliderCheckpoint();
+ const fadeCk = useSliderCheckpoint();
const [speedDraft, setSpeedDraft] = useState<number | null>(null);
const [volumeDraft, setVolumeDraft] = useState<number | null>(null);
// The panel can push into the color-grade subview, which remembers which
@@ -607,7 +645,38 @@ function ClipPanel({ clip }: { clip: VideoClip }) {
))}
</div>
</Row>
- {clip.fit === "fill" && ((clip.panX ?? 0) !== 0 || (clip.panY ?? 0) !== 0) && (
+ <Row label="Zoom">
+ <ValueSlider
+ label="Zoom"
+ sliderClassName="clip-zoom data-horizontal:w-24"
+ valueClassName="w-9 text-muted-foreground"
+ value={clipZoom(clip)}
+ min={1}
+ max={CLIP_MAX_ZOOM}
+ step={0.01}
+ snap={[1]}
+ keyStep={0.05}
+ format={formatPercent}
+ parse={parsePercentInput}
+ onDraft={(v) => {
+ zoomCk.begin();
+ useEditor.getState().updateClipTransient(clip.id, {
+ zoom: v > 1.001 ? v : undefined,
+ });
+ }}
+ onCommit={(v) => {
+ zoomCk.begin();
+ updateClip(clip.id, { zoom: v > 1.001 ? v : undefined });
+ zoomCk.end();
+ }}
+ />
+ <ResetButton
+ title="Reset zoom"
+ show={clipZoom(clip) > 1}
+ onClick={() => updateClip(clip.id, { zoom: undefined, panX: 0, panY: 0 })}
+ />
+ </Row>
+ {((clip.panX ?? 0) !== 0 || (clip.panY ?? 0) !== 0) && (
<Row label="Position">
<button
className="clip-recenter rounded-md border border-input px-2 py-0.5 text-[11.5px] text-muted-foreground transition-colors hover:text-foreground"
@@ -617,12 +686,85 @@ function ClipPanel({ clip }: { clip: VideoClip }) {
</button>
</Row>
)}
+ <Row label="Rotation">
+ <ValueSlider
+ label="Rotation"
+ sliderClassName="clip-rotation data-horizontal:w-24"
+ valueClassName="w-9 text-muted-foreground"
+ value={clip.rotation ?? 0}
+ min={-180}
+ max={180}
+ step={1}
+ snap={[-90, 0, 90]}
+ format={(v) => `${Math.round(v)}°`}
+ parse={parseNumberInput}
+ onDraft={(v) => {
+ turnCk.begin();
+ useEditor.getState().updateClipTransient(clip.id, {
+ rotation: Math.round(v) || undefined,
+ });
+ }}
+ onCommit={(v) => {
+ turnCk.begin();
+ updateClip(clip.id, { rotation: Math.round(v) || undefined });
+ turnCk.end();
+ }}
+ />
+ <ResetButton
+ title="Reset rotation"
+ show={!!clip.rotation}
+ onClick={() => updateClip(clip.id, { rotation: undefined })}
+ />
+ </Row>
+ <Row label="Opacity">
+ <ValueSlider
+ label="Opacity"
+ sliderClassName="clip-opacity data-horizontal:w-24"
+ valueClassName="w-9 text-muted-foreground"
+ value={clip.opacity ?? 1}
+ min={0}
+ max={1}
+ step={0.01}
+ snap={[1]}
+ format={formatPercent}
+ parse={parsePercentInput}
+ onDraft={(v) => {
+ fadeCk.begin();
+ useEditor.getState().updateClipTransient(clip.id, {
+ opacity: v >= 0.999 ? undefined : v,
+ });
+ }}
+ onCommit={(v) => {
+ fadeCk.begin();
+ updateClip(clip.id, { opacity: v >= 0.999 ? undefined : v });
+ fadeCk.end();
+ }}
+ />
+ <ResetButton
+ title="Reset opacity"
+ show={(clip.opacity ?? 1) < 1}
+ onClick={() => updateClip(clip.id, { opacity: undefined })}
+ />
+ </Row>
<LayoutButtons
rect={rectOf(clip)}
onPick={(frame, fit) => updateClip(clip.id, { frame, fit })}
+ onSourceAspect={
+ asset?.width && asset?.height
+ ? () =>
+ updateClip(clip.id, {
+ frame: sourceAspectRect(rectOf(clip), asset.width!, asset.height!, frameOf(aspect)),
+ fit: "fit",
+ zoom: undefined,
+ panX: 0,
+ panY: 0,
+ })
+ : undefined
+ }
/>
<ClipTransformSection clip={clip} />
<ClipBorderSection clip={clip} />
+ <ClipShadowSection clip={clip} />
<ClipMaskSection clip={clip} />
<ClipGeneratedAudio clip={clip} />
</div>
@@ -2559,6 +2701,121 @@ function ClipBorderSection({ clip }: { clip: VideoClip }) {
);
}
+/** The clip's drop shadow: cast by whatever shape the clip really shows — the
+ * box, its rounded corners, and its mask together. */
+function ClipShadowSection({ clip }: { clip: VideoClip }) {
+ const blurCk = useSliderCheckpoint();
+ const offsetCk = useSliderCheckpoint();
+ const opacityCk = useSliderCheckpoint();
+ const st = () => useEditor.getState();
+ const sh = clip.boxStyle?.shadow;
+ const write = (patch: Partial<ClipShadow>) => {
+ const cur = st().clips.find((c) => c.id === clip.id)?.boxStyle;
+ st().updateClipTransient(clip.id, {
+ boxStyle: { ...cur, shadow: { blur: cur?.shadow?.blur ?? 24, ...cur?.shadow, ...patch } },
+ });
+ };
+ return (
+ <Section
+ title="Shadow"
+ info="Cast a drop shadow from the clip's shape — its box, corners, and mask together, so a circle-masked clip throws a circular shadow. Sizes are design pixels, matched between preview and export."
+ enabled={!!sh}
+ onEnabledChange={(v) =>
+ st().updateClip(clip.id, {
+ boxStyle: v
+ ? { ...clip.boxStyle, shadow: { blur: 24, y: 12 } }
+ : clip.boxStyle?.radius || clip.boxStyle?.borderWidth
+ ? { ...clip.boxStyle, shadow: undefined }
+ : undefined,
+ })
+ }
+ >
+ {sh && (
+ <>
+ <Row label="Blur">
+ <ScrubValue
+ label="Shadow blur"
+ className="w-9 text-muted-foreground"
+ value={sh.blur}
+ min={0}
+ max={120}
+ step={1}
+ format={(v) => String(Math.round(v))}
+ parse={parseNumberInput}
+ onScrub={(v) => {
+ blurCk.begin();
+ write({ blur: v });
+ }}
+ onCommit={(v) => {
+ blurCk.begin();
+ write({ blur: v });
+ blurCk.end();
+ }}
+ />
+ </Row>
+ <Row label="Offset">
+ {(["x", "y"] as const).map((axis) => (
+ <span key={axis} className="flex items-center gap-1">
+ <span className="text-[11px] text-muted-foreground/70 uppercase">{axis}</span>
+ <ScrubValue
+ label={`Shadow ${axis.toUpperCase()} offset`}
+ className="w-9 text-muted-foreground"
+ value={sh[axis] ?? 0}
+ min={-120}
+ max={120}
+ step={1}
+ format={(v) => String(Math.round(v))}
+ parse={parseNumberInput}
+ onScrub={(v) => {
+ offsetCk.begin();
+ write({ [axis]: Math.round(v) || undefined });
+ }}
+ onCommit={(v) => {
+ offsetCk.begin();
+ write({ [axis]: Math.round(v) || undefined });
+ offsetCk.end();
+ }}
+ />
+ </span>
+ ))}
+ </Row>
+ <Row label="Opacity">
+ <ValueSlider
+ label="Shadow opacity"
+ sliderClassName="data-horizontal:w-24"
+ valueClassName="w-9 text-muted-foreground"
+ value={sh.opacity ?? 0.35}
+ min={0}
+ max={1}
+ step={0.01}
+ format={formatPercent}
+ parse={parsePercentInput}
+ onDraft={(v) => {
+ opacityCk.begin();
+ write({ opacity: v });
+ }}
+ onCommit={(v) => {
+ opacityCk.begin();
+ write({ opacity: v });
+ opacityCk.end();
+ }}
+ />
+ </Row>
+ <Row label="Color">
+ <ColorField
+ value={sh.color ?? "#000000"}
+ label="Shadow color"
+ onBegin={() => st().pushHistory()}
+ onLive={(c) => write({ color: c })}
+ onCommit={(c) => write({ color: c })}
+ />
+ </Row>
+ </>
+ )}
+ </Section>
+ );
+}
+
function ClipMaskSection({ clip }: { clip: VideoClip }) {
const clips = useEditor((s) => s.clips);
const assets = useEditor((s) => s.assets);
diff --git a/site/src/cut/components/Preview.tsx b/site/src/cut/components/Preview.tsx
index db126a66..d94831a8 100644
--- a/site/src/cut/components/Preview.tsx
+++ b/site/src/cut/components/Preview.tsx
@@ -19,11 +19,11 @@ import {
readPoster,
} from "@/cut/lib/posterCache";
import { setPreviewCanvas } from "@/cut/lib/previewCanvas";
-import { clipKeyed, clipPoseAt, frameOf, isFullRect, rectOf, REGION_MAX_SCALE, type Aspect, type ClipSpan, type FrameRect, type MediaAsset, type VideoClip } from "@/cut/lib/types";
+import { CLIP_MAX_ZOOM, clipCovers, clipKeyed, clipPoseAt, clipZoom, contentRect, frameOf, isFullRect, rectOf, REGION_MAX_SCALE, type Aspect, type ClipSpan, type FrameRect, type MediaAsset, type VideoClip } from "@/cut/lib/types";
import { hasMaskKeys, type MaskKey } from "@donkeycut/effects-kit";
import { cn } from "@/lib/utils";
import { MaskGizmoCore, OverlayLayer } from "./OverlayLayer";
-import { HANDLE_AXIS, TransformHandles, type ResizeHandle } from "./TransformHandles";
+import { CORNER_HANDLES, HANDLE_AXIS, TransformHandles, type ResizeHandle } from "./TransformHandles";
import {
StageEffectPaint,
StagePictureFx,
@@ -31,22 +31,47 @@ import {
useEffectLanes,
} from "./StageEffects";
-/** The clip under the playhead, when it overflows the frame in fill mode. */
+/** How far a clip's picture hangs past its box, in the box's units. Zero on an
+ * axis with nothing to spare. */
+function overflowOf(
+ clip: VideoClip,
+ asset: { width?: number; height?: number },
+ box: FrameRect
+): { ox: number; oy: number } {
+ if (!asset.width || !asset.height) return { ox: 0, oy: 0 };
+ const pic = contentRect(
+ box,
+ asset.width,
+ asset.height,
+ clipCovers(clip),
+ clipZoom(clip),
+ clip.panX ?? 0,
+ clip.panY ?? 0
+ );
+ return { ox: Math.max(0, pic.w - box.w), oy: Math.max(0, pic.h - box.h) };
+}
+
+/** Play or pause, rewinding first when the playhead is parked at the end. */
+function togglePlayback() {
+ const s = useEditor.getState();
+ const total = projectDuration(s);
+ if (!total) return;
+ if (!s.playing && playheadAt() >= total - 0.01) s.seek(0);
+ s.setPlaying(!s.playing);
+}
+
+/** The clip under the playhead, when its picture overflows the whole frame —
+ * covering it or zoomed into it. A regioned clip is panned from its own
+ * preview handle instead. */
function pannableSpan(
s: { clips: VideoClip[]; assets: MediaAsset[]; aspect: Aspect },
t: number
): ClipSpan | null {
const spans = getClipSpans(s.clips, s.assets);
const span = spans.find((sp) => t >= sp.start && sp.start + sp.len > t) ?? spans[spans.length - 1];
- // Pan only makes sense for a full-frame fill clip; a regioned clip is moved
- // with its own preview handle instead.
- if (!span || span.clip.fit !== "fill" || !isFullRect(rectOf(span.clip))) return null;
- const { width, height } = span.asset;
- if (!width || !height) return null;
+ if (!span || !isFullRect(rectOf(span.clip))) return null;
const frame = frameOf(s.aspect);
- const scale = Math.max(frame.w / width, frame.h / height);
- const ox = width * scale - frame.w;
- const oy = height * scale - frame.h;
+ const { ox, oy } = overflowOf(span.clip, span.asset, { x: 0, y: 0, w: frame.w, h: frame.h });
return ox > 1 || oy > 1 ? span : null;
}
@@ -111,7 +136,9 @@ export function Preview() {
const canvasRef = useRef<HTMLCanvasElement>(null);
const wrapRef = useRef<HTMLDivElement>(null);
const stageRef = useRef<HTMLDivElement>(null);
- const [stage, setStage] = useState({ w: 270, h: 480 });
+ // The picture's box plus the empty room around it in the pane, which is where
+ // the outline of an oversized picture gets to show.
+ const [stage, setStage] = useState({ w: 270, h: 480, padX: 0, padY: 0 });
const aspect = useEditor((s) => s.aspect);
const frame = frameOf(aspect);
@@ -153,7 +180,14 @@ export function Preview() {
const availW = Math.max(120, r.width - pad);
const availH = Math.max(120, r.height - pad);
const scale = Math.min(availW / rw, availH / rh);
- setStage({ w: Math.floor(scale * rw), h: Math.floor(scale * rh) });
+ const w = Math.floor(scale * rw);
+ const h = Math.floor(scale * rh);
+ setStage({
+ w,
+ h,
+ padX: Math.max(0, Math.floor((r.width - w) / 2)),
+ padY: Math.max(0, Math.floor((r.height - h) / 2)),
+ });
};
fit();
const ro = new ResizeObserver(fit);
@@ -161,24 +195,13 @@ export function Preview() {
return () => ro.disconnect();
}, [aspect]);
- const togglePlayback = () => {
- const s = useEditor.getState();
- const total = projectDuration(s);
- if (!total) return;
- if (!s.playing && playheadAt() >= total - 0.01) s.seek(0);
- s.setPlaying(!s.playing);
- };
-
// Drag a fill-mode clip inside the frame to choose the visible crop.
const panDrag = (e: React.PointerEvent) => {
const s = useEditor.getState();
const span = pannableSpan(s, previewAt());
if (!span) return false;
const fr = frameOf(s.aspect);
- const { width = 1, height = 1 } = span.asset;
- const scale = Math.max(fr.w / width, fr.h / height);
- const ox = width * scale - fr.w;
- const oy = height * scale - fr.h;
+ const { ox, oy } = overflowOf(span.clip, span.asset, { x: 0, y: 0, w: fr.w, h: fr.h });
const clipId = span.clip.id;
const panX0 = span.clip.panX ?? 0;
const panY0 = span.clip.panY ?? 0;
@@ -319,7 +342,7 @@ export function Preview() {
)
)}
</div>
- <OverlayPipHandle stage={stage} />
+ <ClipTransformGizmo stage={stage} />
</div>
</div>
</section>
@@ -327,15 +350,23 @@ export function Preview() {
}
/**
- * Direct-manipulation handle for the selected video layer's frame region: drag
- * the box to reposition, drag the corner to resize (both update the clip's
- * `frame` rect). Works for a regioned track-0 clip (split-screen half) or an
- * overlay clip, and only while that clip is live under the playhead so it lines
- * up with the compositor. A full-frame layer needs no handle.
+ * Direct manipulation for the selected video clip, on any track and at any
+ * size: drag the blue box to move it, a corner to scale it whole, a side to
+ * pull one edge, the button above to turn it. Every gesture writes the clip's
+ * own `frame`/`rotation`, and the gizmo shows only while the clip is live under
+ * the playhead so it lines up with what the compositor draws.
+ *
+ * The grey outline beside it is the picture itself — where the source lands
+ * once its box has fitted or cropped it. When it overflows, its corners zoom
+ * and its interior pans, which is how a landscape shot gets framed for a
+ * vertical cut.
*/
/** How close (screen px) a box edge pulls onto a snap line while dragging. */
const SNAP_PX = 8;
+/** The picture's box on screen plus the empty room around it in the pane. */
+type Stage = { w: number; h: number; padX: number; padY: number };
+
/** Snap a moving box along one axis: its leading edge, center, and trailing
* edge each pull to the frame's edges and centerline. Returns the snapped
* position and the frame line it landed on, for the guide. */
@@ -364,13 +395,18 @@ function snapEdge(v: number, tol: number): number | null {
return best;
}
-function OverlayPipHandle({ stage }: { stage: { w: number; h: number } }) {
+function ClipTransformGizmo({ stage }: { stage: Stage }) {
const selection = useEditor((s) => s.selection);
const clips = useEditor((s) => s.clips);
const assets = useEditor((s) => s.assets);
- const aspect = useEditor((s) => s.aspect);
const skimTime = useSkim();
- // The handle only cares whether its clip is on screen, so it subscribes to
+ const layerRef = useRef<HTMLDivElement>(null);
+ const [guides, setGuides] = useState<{ x: number | null; y: number | null }>({
+ x: null,
+ y: null,
+ });
+ const [turning, setTurning] = useState<number | null>(null);
+ // The gizmo only cares whether its clip is on screen, so it subscribes to
// that answer instead of to the clock: one render when the clip comes and
// goes, rather than one per frame while it stays.
const selectedClip = selection?.kind === "clip" ? clips.find((c) => c.id === selection.id) : null;
@@ -380,77 +416,127 @@ function OverlayPipHandle({ stage }: { stage: { w: number; h: number } }) {
const len = Math.max(0.1, (selectedClip.out - selectedClip.in) / speed);
return t >= selectedClip.start && t < selectedClip.start + len;
});
- const [guides, setGuides] = useState<{ x: number | null; y: number | null }>({
- x: null,
- y: null,
- });
// While hover-scrubbing the preview shows the skimmer's frame, where the
- // selected clip may not even be on screen — selection stays, the handle hides.
+ // selected clip may not even be on screen — selection stays, the gizmo hides.
if (skimTime !== null) return null;
+ const clip = selectedClip;
+ if (!clip || clip.hidden || !live) return null;
- // Resolve the selected, live, regioned clip (any track) plus how to patch its
- // rect. A clip's own footprint equals its span length, so one path serves
- // every track.
- let rect: FrameRect | null = null;
- let apply: ((frame: FrameRect) => void) | null = null;
- // A fill clip whose content overflows the box can pan its crop window: how
- // far the scaled picture overhangs the box (frame px), and the pan to start
- // the gesture from.
- let pan: { id: string; ox: number; oy: number; panX0: number; panY0: number } | null = null;
- {
- const clip = selectedClip;
- if (clip && !clip.hidden) {
- if (live) {
- rect = rectOf(clip);
- apply = (frame) => useEditor.getState().updateClipTransient(clip.id, { frame });
- const asset = assets.find((a) => a.id === clip.assetId);
- if (clip.fit === "fill" && asset?.width && asset?.height) {
- const fr = frameOf(aspect);
- const bw = rect.w * fr.w;
- const bh = rect.h * fr.h;
- const sc = Math.max(bw / asset.width, bh / asset.height);
- const ox = asset.width * sc - bw;
- const oy = asset.height * sc - bh;
- if (ox > 1 || oy > 1) {
- pan = { id: clip.id, ox, oy, panX0: clip.panX ?? 0, panY0: clip.panY ?? 0 };
- }
+ const st = () => useEditor.getState();
+ const patch = (p: Partial<VideoClip>) => st().updateClipTransient(clip.id, p);
+ /**
+ * A drag on one of the gizmo's large surfaces — the box body, the picture's
+ * interior. The history checkpoint waits for real travel, and a press that
+ * never travels is a click on the picture, which plays and pauses.
+ */
+ const surfaceDrag = (
+ e: React.PointerEvent,
+ onMove: (dx: number, dy: number) => void,
+ onEnd?: () => void
+ ) => {
+ e.stopPropagation();
+ let began = false;
+ startDrag(e, {
+ onMove: (dx, dy) => {
+ if (!began) {
+ if (Math.abs(dx) <= 3 && Math.abs(dy) <= 3) return;
+ began = true;
+ st().pushHistory();
}
- }
- }
- }
- if (!rect || !apply || isFullRect(rect)) return null;
- const r = rect;
- const patch = apply;
+ onMove(dx, dy);
+ },
+ onUp: (_dx, _dy, moved) => {
+ onEnd?.();
+ if (!moved) togglePlayback();
+ },
+ });
+ };
+ const asset = assets.find((a) => a.id === clip.assetId);
+ const r = rectOf(clip);
+ const rotation = clip.rotation ?? 0;
+ const zoom = clipZoom(clip);
+ // Where the source actually lands inside the box — the same geometry the
+ // compositor draws and the export crops, measured in stage pixels so the two
+ // axes share one scale.
+ const boxPx = { x: r.x * stage.w, y: r.y * stage.h, w: r.w * stage.w, h: r.h * stage.h };
+ const pic =
+ asset?.width && asset?.height
+ ? contentRect(
+ boxPx,
+ asset.width,
+ asset.height,
+ clipCovers(clip),
+ zoom,
+ clip.panX ?? 0,
+ clip.panY ?? 0
+ )
+ : boxPx;
+ const ox = Math.max(0, pic.w - boxPx.w);
+ const oy = Math.max(0, pic.h - boxPx.h);
+ const overflows = ox > 1 || oy > 1;
+ // The outline is worth drawing whenever the picture and its box part ways:
+ // cropped past the edges, or sitting inside them with margins.
+ const showPicture = Math.abs(pic.w - boxPx.w) > 2 || Math.abs(pic.h - boxPx.h) > 2;
// The box may leave the frame — oversize it to focus on an area, or park it
// partly off screen — as long as a sliver stays inside to grab.
- const onMove = (e: React.PointerEvent) => {
- e.stopPropagation();
- useEditor.getState().pushHistory();
- startDrag(e, {
- onMove: (dx, dy) => {
+ const onMoveBox = (e: React.PointerEvent) =>
+ surfaceDrag(
+ e,
+ (dx, dy) => {
const sx = snapAxis(r.x + dx / stage.w, r.w, SNAP_PX / stage.w);
const sy = snapAxis(r.y + dy / stage.h, r.h, SNAP_PX / stage.h);
setGuides({ x: sx.guide, y: sy.guide });
patch({
- ...r,
- x: Math.max(0.05 - r.w, Math.min(0.95, sx.v)),
- y: Math.max(0.05 - r.h, Math.min(0.95, sy.v)),
+ frame: {
+ ...r,
+ x: Math.max(0.05 - r.w, Math.min(0.95, sx.v)),
+ y: Math.max(0.05 - r.h, Math.min(0.95, sy.v)),
+ },
});
},
- onUp: () => setGuides({ x: null, y: null }),
- });
+ () => setGuides({ x: null, y: null })
+ );
+
+ // A turned box is grabbed on screen but sized in the frame's own axes, so a
+ // grip's travel comes back the way the box went out.
+ const unturn = (dx: number, dy: number) => {
+ if (!rotation) return { dx, dy };
+ const a = (-rotation * Math.PI) / 180;
+ return { dx: dx * Math.cos(a) - dy * Math.sin(a), dy: dx * Math.sin(a) + dy * Math.cos(a) };
};
- // A grip drags its own edges and leaves the opposite ones where they are:
- // the box keeps its far corner planted while the grabbed side travels, and
- // a moving edge snaps to the frame the same way a move does.
+ // A corner scales the box whole, the way every editor's corner does: the
+ // shape holds and the opposite corner stays planted. A side grip pulls its
+ // own edge and snaps to the frame.
const onResize = (handle: ResizeHandle, e: React.PointerEvent) => {
e.stopPropagation();
- useEditor.getState().pushHistory();
+ st().pushHistory();
const a = HANDLE_AXIS[handle];
+ const corner = a.x !== 0 && a.y !== 0;
startDrag(e, {
- onMove: (dx, dy) => {
+ onMove: (rawX, rawY) => {
+ const { dx, dy } = unturn(rawX, rawY);
+ if (corner) {
+ const grow =
+ 1 + (a.x * (dx / stage.w) / r.w + a.y * (dy / stage.h) / r.h) / 2;
+ const k = Math.max(
+ 0.05 / Math.min(r.w, r.h),
+ Math.min(REGION_MAX_SCALE / Math.max(r.w, r.h), grow)
+ );
+ const w = r.w * k;
+ const h = r.h * k;
+ setGuides({ x: null, y: null });
+ patch({
+ frame: {
+ x: a.x > 0 ? r.x : r.x + r.w - w,
+ y: a.y > 0 ? r.y : r.y + r.h - h,
+ w,
+ h,
+ },
+ });
+ return;
+ }
// One axis at a time: where the grabbed edge lands, snapped to the
// frame, then the span it leaves against the planted edge.
const pull = (
@@ -478,65 +564,158 @@ function OverlayPipHandle({ stage }: { stage: { w: number; h: number } }) {
const hx = pull(a.x, r.x, r.w, dx, stage.w);
const hy = pull(a.y, r.y, r.h, dy, stage.h);
setGuides({ x: hx.guide, y: hy.guide });
- patch({ ...r, x: hx.pos, w: hx.size, y: hy.pos, h: hy.size });
+ patch({ frame: { x: hx.pos, w: hx.size, y: hy.pos, h: hy.size } });
},
onUp: () => setGuides({ x: null, y: null }),
});
};
- const panContent = pan;
- const onPanContent = (e: React.PointerEvent) => {
- if (!panContent) return;
+ /** Client coordinates of a point given in stage pixels. */
+ const clientOf = (px: number, py: number) => {
+ const at = layerRef.current?.getBoundingClientRect();
+ return { x: (at?.left ?? 0) + stage.padX + px, y: (at?.top ?? 0) + stage.padY + py };
+ };
+
+ const onRotate = (e: React.PointerEvent) => {
e.stopPropagation();
- useEditor.getState().pushHistory();
- const toFrame = frameOf(aspect).w / stage.w; // screen px → frame px
+ st().pushHistory();
+ const c = clientOf(boxPx.x + boxPx.w / 2, boxPx.y + boxPx.h / 2);
+ const angleAt = (x: number, y: number) =>
+ (Math.atan2(y - c.y, x - c.x) * 180) / Math.PI + 90;
+ const grabX = e.clientX;
+ const grabY = e.clientY;
+ const from = angleAt(grabX, grabY);
+ setTurning(rotation);
startDrag(e, {
onMove: (dx, dy) => {
- // Content follows the pointer; pan is the crop-window position.
- useEditor.getState().updateClipTransient(panContent.id, {
- panX:
- panContent.ox > 1
- ? Math.max(-1, Math.min(1, panContent.panX0 - (dx * toFrame) / (panContent.ox / 2)))
- : 0,
- panY:
- panContent.oy > 1
- ? Math.max(-1, Math.min(1, panContent.panY0 - (dy * toFrame) / (panContent.oy / 2)))
- : 0,
- });
+ const raw = rotation + (angleAt(grabX + dx, grabY + dy) - from);
+ const wrapped = ((((raw + 180) % 360) + 360) % 360) - 180;
+ // Square angles pull the turn straight, the way the box edges snap to
+ // the frame.
+ const snapped = [-180, -90, 0, 90, 180].find((t) => Math.abs(wrapped - t) < 4);
+ const deg = Math.round(snapped ?? wrapped);
+ setTurning(deg);
+ patch({ rotation: deg === 0 ? undefined : deg });
},
+ onUp: () => setTurning(null),
+ });
+ };
+
+ // The picture's own grips, live only while something overflows: dragging a
+ // corner out zooms further in, dragging it back settles at the box.
+ const onZoom = (e: React.PointerEvent) => {
+ e.stopPropagation();
+ st().pushHistory();
+ const c = clientOf(pic.x + pic.w / 2, pic.y + pic.h / 2);
+ const reach = (x: number, y: number) => Math.hypot(x - c.x, y - c.y);
+ const grabX = e.clientX;
+ const grabY = e.clientY;
+ const from = Math.max(8, reach(grabX, grabY));
+ startDrag(e, {
+ onMove: (dx, dy) => {
+ const k = reach(grabX + dx, grabY + dy) / from;
+ const next = Math.max(1, Math.min(CLIP_MAX_ZOOM, zoom * k));
+ patch({ zoom: next > 1.001 ? next : undefined });
+ },
+ });
+ };
+
+ const onPanPicture = (e: React.PointerEvent) => {
+ const panX0 = clip.panX ?? 0;
+ const panY0 = clip.panY ?? 0;
+ // Content follows the pointer; pan is the crop window's position.
+ surfaceDrag(e, (rawX, rawY) => {
+ const { dx, dy } = unturn(rawX, rawY);
+ patch({
+ panX: ox > 1 ? Math.max(-1, Math.min(1, panX0 - dx / (ox / 2))) : 0,
+ panY: oy > 1 ? Math.max(-1, Math.min(1, panY0 - dy / (oy / 2))) : 0,
+ });
});
};
- const box = {
- left: r.x * stage.w,
- top: r.y * stage.h,
- width: r.w * stage.w,
- height: r.h * stage.h,
+ // Everything draws in the pane's own space, so an oversized picture's outline
+ // keeps going into the empty room around the frame instead of stopping at its
+ // edge; the layer clips it at the pane so it never reaches another panel.
+ const at = (rect: { x: number; y: number; w: number; h: number }) => ({
+ left: stage.padX + rect.x,
+ top: stage.padY + rect.y,
+ width: rect.w,
+ height: rect.h,
+ });
+ const box = at(boxPx);
+ const picBox = at(pic);
+ const turn = {
+ transform: rotation ? `rotate(${rotation}deg)` : undefined,
+ transformOrigin: `${box.left + box.width / 2}px ${box.top + box.height / 2}px`,
};
- // The dashed box draws the full extent; the frame-clipped solid ring paints
- // over it, so dashes show only where the box leaves the frame.
return (
- <>
+ <div
+ ref={layerRef}
+ className="pointer-events-none absolute overflow-hidden"
+ style={{
+ left: -stage.padX,
+ top: -stage.padY,
+ width: stage.w + 2 * stage.padX,
+ height: stage.h + 2 * stage.padY,
+ }}
+ >
+ <div className="absolute inset-0" style={turn}>
+ {/* The picture as the source really lands it: grey where the frame
+ throws it away, so a crop is something you can see and grab. */}
+ {showPicture && (
+ <div
+ className="absolute rounded-[3px] border border-dashed border-white/70 shadow-[0_0_0_1px_rgba(0,0,0,0.35)]"
+ style={picBox}
+ >
+ {overflows && (
+ <TransformHandles
+ color="#c7c7cc"
+ handles={CORNER_HANDLES}
+ rotation={rotation}
+ onResize={(_h, e) => onZoom(e)}
+ resizeTitle="Drag to zoom the picture"
+ />
+ )}
+ </div>
+ )}
+ {/* The dashed box draws the full extent; the frame-clipped solid ring
+ paints over it, so dashes show only where the box leaves the frame. */}
+ <div
+ className="pointer-events-auto absolute cursor-move rounded-[3px] border-2 border-dashed border-[#0a84ff]"
+ style={box}
+ onPointerDown={onMoveBox}
+ >
+ {/* An overflowing picture pans from the interior; the ring at the
+ border moves the box, the grips resize it. */}
+ {overflows && (
+ <div
+ className="absolute inset-2 cursor-grab active:cursor-grabbing"
+ onPointerDown={onPanPicture}
+ />
+ )}
+ <TransformHandles
+ color="#0a84ff"
+ className="z-20"
+ rotation={rotation}
+ angle={turning}
+ onResize={onResize}
+ onRotate={onRotate}
+ />
+ </div>
+ </div>
+ {/* The solid ring and the snap guides live outside the turn: the ring's
+ frame clipping is the frame's own edges, and a guide is a frame line.
+ A turned box keeps just its dashed outline. */}
<div
- className="absolute cursor-move rounded-[3px] border-2 border-dashed border-[#0a84ff]"
- style={box}
- onPointerDown={onMove}
+ className="pointer-events-none absolute overflow-hidden rounded-xl"
+ style={{ left: stage.padX, top: stage.padY, width: stage.w, height: stage.h }}
>
- {/* Overflowing fill content pans from the interior; the ring at the
- border moves the box, the corner resizes it. */}
- {panContent && (
+ {!rotation && (
<div
- className="absolute inset-2 cursor-grab active:cursor-grabbing"
- onPointerDown={onPanContent}
+ className="absolute rounded-[3px] shadow-[inset_0_0_0_2px_#0a84ff]"
+ style={{ ...box, left: box.left - stage.padX, top: box.top - stage.padY }}
/>
)}
- <TransformHandles color="#0a84ff" className="z-20" onResize={onResize} />
- </div>
- <div className="pointer-events-none absolute inset-0 z-10 overflow-hidden rounded-xl">
- <div
- className="absolute rounded-[3px] shadow-[inset_0_0_0_2px_#0a84ff]"
- style={box}
- />
{guides.x !== null && (
<div className="absolute inset-y-0 w-px bg-[#0a84ff]" style={{ left: guides.x * stage.w }} />
)}
@@ -544,7 +723,7 @@ function OverlayPipHandle({ stage }: { stage: { w: number; h: number } }) {
<div className="absolute inset-x-0 h-px bg-[#0a84ff]" style={{ top: guides.y * stage.h }} />
)}
</div>
- </>
+ </div>
);
}
diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.ts
index 76189a2c..dde58002 100644
--- a/site/src/cut/components/Timeline.tools.ts
+++ b/site/src/cut/components/Timeline.tools.ts
@@ -169,6 +169,9 @@ export const TIMELINE_TOOLS = [
h: num("Height 0..1"),
}, ["x", "y", "w", "h"]),
fit: { type: "string", enum: ["fit", "fill"], description: "How the video meets its region" },
+ zoom: num("Zoom the picture past its fitted size, 1 (none) .. 4; the overflow crops"),
+ rotation: num("Resting turn in degrees clockwise, -180..180 (0 clears)"),
+ opacity: num("Resting opacity 0..1 (1 clears)"),
speed: num("Playback rate (1 = normal, no upper limit)"),
}, ["id"]),
},
diff --git a/site/src/cut/hooks/usePlayback.ts b/site/src/cut/hooks/usePlayback.ts
index aa8401e7..888b997a 100644
--- a/site/src/cut/hooks/usePlayback.ts
+++ b/site/src/cut/hooks/usePlayback.ts
@@ -9,7 +9,7 @@ import {
useEditor,
} from "@/cut/lib/store";
import { playheadAt, previewAt, setPlayhead, subscribePlayhead } from "@/cut/lib/playhead";
-import { isFullRect, projectFadeSeconds, rectOf } from "@/cut/lib/types";
+import { clipCovers, projectFadeSeconds, rectOf } from "@/cut/lib/types";
import type { ClipSpan, MediaAsset, VideoClip } from "@/cut/lib/types";
import { SubjectMaskCompositor } from "@/cut/lib/behindPass";
import { FrameCompositor, MISSING_FRAME, PENDING_FRAME, type Frame } from "@/cut/lib/composite";
@@ -361,9 +361,7 @@ class Engine {
};
const frame = this.frameFor(span, t, playing);
if (frame.kind !== "ready") continue;
- const rect = rectOf(clip);
- const cover = clip.fit === "fill" || (clip.fit == null && isFullRect(rect));
- this.comp.drawIntoRect(frame, rect, cover, alpha, t, zoom, clip);
+ this.comp.drawIntoRect(frame, rectOf(clip), clipCovers(clip), alpha, t, zoom, clip);
}
}
diff --git a/site/src/cut/lib/aiContext.ts b/site/src/cut/lib/aiContext.ts
index 2e3c42f9..a2654c2b 100644
--- a/site/src/cut/lib/aiContext.ts
+++ b/site/src/cut/lib/aiContext.ts
@@ -9,6 +9,7 @@ import { laneCues, subtitleLaneCount } from "./subtitles";
import { watchSweepActive } from "./watch/sweep";
import { libraryFontId, listLibraryFonts } from "./linkedLibrary";
import {
+ clipZoom,
frameOf,
rectOf,
regionLabel,
@@ -39,6 +40,7 @@ function clipEffects(clip: VideoClip) {
? { animOut: { style: clip.animOut.style, seconds: r(clip.animOut.seconds) } }
: {}),
...(clip.look ? { look: clip.look, lookAmount: r(clip.lookAmount ?? 1) } : {}),
+ ...(clip.boxStyle ? { boxStyle: clip.boxStyle } : {}),
};
}
@@ -280,9 +282,15 @@ export function buildAiContext(opts?: { fullCues?: boolean; chatId?: string | nu
const t = transitionToNext(sp, index, spans);
return t ? { transitionToNext: t } : {};
})(),
- ...(sp.clip.fit === "fill"
- ? { panX: r(sp.clip.panX ?? 0), panY: r(sp.clip.panY ?? 0) }
+ ...(sp.clip.fit === "fill" || clipZoom(sp.clip) > 1
+ ? {
+ ...(clipZoom(sp.clip) > 1 ? { zoom: r(clipZoom(sp.clip)) } : {}),
+ panX: r(sp.clip.panX ?? 0),
+ panY: r(sp.clip.panY ?? 0),
+ }
: {}),
+ ...(sp.clip.rotation ? { rotation: sp.clip.rotation } : {}),
+ ...((sp.clip.opacity ?? 1) < 1 ? { opacity: r(sp.clip.opacity ?? 1) } : {}),
...(sp.clip.grade ? { colorGrade: sp.clip.grade } : {}),
...(sp.clip.mask ? { mask: sp.clip.mask } : {}),
...(sp.clip.kf?.length
@@ -407,9 +415,17 @@ function describeOverlayClip(c: VideoClip, assets: Map<string, { name: string }>
layout: regionLabel(rect),
region: { x: r(rect.x), y: r(rect.y), w: r(rect.w), h: r(rect.h) },
fit: c.fit ?? "fit",
+ ...(clipZoom(c) > 1
+ ? { zoom: r(clipZoom(c)), panX: r(c.panX ?? 0), panY: r(c.panY ?? 0) }
+ : c.fit === "fill"
+ ? { panX: r(c.panX ?? 0), panY: r(c.panY ?? 0) }
+ : {}),
+ ...(c.rotation ? { rotation: c.rotation } : {}),
+ ...((c.opacity ?? 1) < 1 ? { opacity: r(c.opacity ?? 1) } : {}),
...(speed !== 1 ? { speed: r(speed) } : {}),
...(c.grade ? { colorGrade: c.grade } : {}),
...(c.mask ? { mask: c.mask } : {}),
+ ...(c.boxStyle ? { boxStyle: c.boxStyle } : {}),
...(c.kf?.length
? { keyframes: c.kf.map((k) => ({ ...k, t: r(k.t), x: r(k.x), y: r(k.y) })) }
: {}),
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index d5605baa..5e673a70 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -108,8 +108,11 @@ import {
allFonts,
ANIM_DEFAULT_SECONDS,
ANIM_STYLE_IDS,
+ CLIP_MAX_ZOOM,
+ clipCovers,
DEFAULT_BACKGROUND,
clampOverlayPos,
+ clipZoom,
frameOf,
IMAGE_CLIP_SECONDS,
isEffectOverlay,
@@ -131,6 +134,7 @@ import {
type AssetWatch,
type CaptionStyleId,
type AudioClip,
+ type ClipShadow,
type ColorGrade,
type FontId,
type MediaAsset,
@@ -1042,6 +1046,18 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
patch.frame = { x: clamp(rg.x, 0, 1 - w), y: clamp(rg.y, 0, 1 - h), w, h };
}
if (input.fit === "fit" || input.fit === "fill") patch.fit = input.fit;
+ if (isNum(input.zoom)) {
+ const z = clamp(input.zoom, 1, CLIP_MAX_ZOOM);
+ patch.zoom = z > 1 ? z : undefined;
+ }
+ if (isNum(input.rotation)) {
+ const deg = Math.round(clamp(input.rotation, -180, 180));
+ patch.rotation = deg === 0 ? undefined : deg;
+ }
+ if (isNum(input.opacity)) {
+ const o = clamp(input.opacity, 0, 1);
+ patch.opacity = o >= 1 ? undefined : o;
+ }
if (isNum(input.speed)) patch.speed = Math.max(SPEED_FLOOR, input.speed);
if (Object.keys(patch).length === 0) throw new ToolError("Nothing to change.");
s.updateClip(c.id, patch);
@@ -1052,6 +1068,9 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
start: round2(next.start),
layout: regionLabel(rectOf(next)),
fit: next.fit ?? "fit",
+ ...(clipZoom(next) > 1 ? { zoom: clipZoom(next) } : {}),
+ ...(next.rotation ? { rotation: next.rotation } : {}),
+ ...((next.opacity ?? 1) < 1 ? { opacity: next.opacity } : {}),
muted: next.muted,
...(next.hidden ? { hidden: true } : {}),
};
@@ -1600,13 +1619,64 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
set_framing: (s, input) => {
const clip = requireItem(s.clips, input.clipId, "video clip");
const mode = input.mode === "fill" ? "fill" : "fit";
+ const zoom = isNum(input.zoom) ? clamp(input.zoom, 1, CLIP_MAX_ZOOM) : 1;
+ // Pan only means something once something overflows the box.
+ const pannable = mode === "fill" || zoom > 1;
s.updateClip(clip.id, {
fit: mode,
- panX: mode === "fill" && isNum(input.panX) ? clamp(input.panX, -1, 1) : 0,
- panY: mode === "fill" && isNum(input.panY) ? clamp(input.panY, -1, 1) : 0,
+ zoom: zoom > 1 ? zoom : undefined,
+ panX: pannable && isNum(input.panX) ? clamp(input.panX, -1, 1) : 0,
+ panY: pannable && isNum(input.panY) ? clamp(input.panY, -1, 1) : 0,
});
const next = useEditor.getState().clips.find((c) => c.id === clip.id)!;
- return { id: next.id, fit: next.fit, panX: next.panX ?? 0, panY: next.panY ?? 0 };
+ return {
+ id: next.id,
+ fit: next.fit,
+ zoom: clipZoom(next),
+ panX: next.panX ?? 0,
+ panY: next.panY ?? 0,
+ };
+ },
+
+ set_clip_style: (s, input) => {
+ const clip = requireItem(s.clips, input.clipId, "video clip");
+ if (input.clear === true) {
+ s.updateClip(clip.id, { boxStyle: undefined });
+ return { id: clip.id, boxStyle: null };
+ }
+ const bs = { ...clip.boxStyle };
+ if (isNum(input.radius)) bs.radius = Math.max(0, input.radius) || undefined;
+ if (isNum(input.border_width)) bs.borderWidth = Math.max(0, input.border_width) || undefined;
+ if (typeof input.border_color === "string") bs.borderColor = input.border_color;
+ const shadowTouched =
+ isNum(input.shadow_blur) ||
+ isNum(input.shadow_x) ||
+ isNum(input.shadow_y) ||
+ typeof input.shadow_color === "string" ||
+ isNum(input.shadow_opacity);
+ if (shadowTouched) {
+ const sh: ClipShadow = {
+ blur: Math.max(0, isNum(input.shadow_blur) ? input.shadow_blur : bs.shadow?.blur ?? 24),
+ ...(isNum(input.shadow_x)
+ ? input.shadow_x !== 0 && { x: input.shadow_x }
+ : bs.shadow?.x !== undefined && { x: bs.shadow.x }),
+ ...(isNum(input.shadow_y)
+ ? input.shadow_y !== 0 && { y: input.shadow_y }
+ : bs.shadow?.y !== undefined && { y: bs.shadow.y }),
+ ...(typeof input.shadow_color === "string"
+ ? { color: input.shadow_color }
+ : bs.shadow?.color !== undefined && { color: bs.shadow.color }),
+ ...(isNum(input.shadow_opacity)
+ ? { opacity: clamp(input.shadow_opacity, 0, 1) }
+ : bs.shadow?.opacity !== undefined && { opacity: bs.shadow.opacity }),
+ };
+ // A shadow with no blur and no throw is no shadow.
+ bs.shadow = sh.blur > 0 || sh.x || sh.y ? sh : undefined;
+ }
+ const boxStyle =
+ bs.radius || bs.borderWidth || bs.shadow ? bs : undefined;
+ s.updateClip(clip.id, { boxStyle });
+ return { id: clip.id, boxStyle: boxStyle ?? null };
},
freeze_frame: async (s, input) => {
@@ -1630,7 +1700,13 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
// sized below to the requested duration, mirroring the engine's
// /freeze clamps (default 1s, 0.5..10s).
const freezeDur = Math.min(10, Math.max(0.5, isNum(input.duration) ? input.duration : 1));
- body = await captureFreezeFrame(projectId, span.asset.url, srcTime, freezeDur).catch(
+ body = await captureFreezeFrame(projectId, span.asset.url, srcTime, freezeDur, {
+ frame: frameOf(s.aspect),
+ cover: clipCovers(span.clip),
+ zoom: clipZoom(span.clip),
+ panX: span.clip.panX ?? 0,
+ panY: span.clip.panY ?? 0,
+ }).catch(
(e) => {
throw new ToolError(
e instanceof Error ? e.message : "Could not render the freeze frame."
@@ -1651,6 +1727,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
frame: frameOf(s.aspect),
framing: {
fit: span.clip.fit ?? "fit",
+ zoom: clipZoom(span.clip),
panX: span.clip.panX ?? 0,
panY: span.clip.panY ?? 0,
},
diff --git a/site/src/cut/lib/composite.ts b/site/src/cut/lib/composite.ts
index 35d4be75..330f930d 100644
--- a/site/src/cut/lib/composite.ts
+++ b/site/src/cut/lib/composite.ts
@@ -21,8 +21,8 @@
import { applyMaskToCanvas, gradeTint, gradeToCssFilter, grainTile, isNeutralGrade, lookCssFilter, lookPost, maskComposite } from "@donkeycut/effects-kit";
import { createRasterCanvas } from "./raster";
-import { clipKeyed, clipPoseAt, DEFAULT_BACKGROUND, isFullRect, rectOf } from "./types";
-import type { FrameRect, TransitionStyle, VideoClip } from "./types";
+import { clipCovers, clipPosed, clipPoseAt, clipZoom, contentRect, DEFAULT_BACKGROUND, isFullRect, rectOf, shadowInk } from "./types";
+import type { ClipShadow, FrameRect, TransitionStyle, VideoClip } from "./types";
/** A clip's picture at some instant, or the reasons there isn't one.
*
@@ -79,6 +79,8 @@ export class FrameCompositor {
private layerScratch: Surface | null = null;
private maskScratch: Surface | null = null;
private poseScratch: Surface | null = null;
+ /** Where a shadow is cast before the picture goes down over it. */
+ private shadowScratch: Surface | null = null;
/** Where a subject-masked clip's person matte comes from: the host hands a
* reader over the canvas as it stands (the layers beneath the clip), so
@@ -126,7 +128,8 @@ export class FrameCompositor {
| "vignetteCanvas"
| "layerScratch"
| "maskScratch"
- | "poseScratch",
+ | "poseScratch"
+ | "shadowScratch",
w: number,
h: number
): { surface: Surface; resized: boolean } {
@@ -339,7 +342,8 @@ export class FrameCompositor {
private needsFx(clip?: VideoClip): boolean {
return (
!!clip &&
- (clipKeyed(clip) ||
+ (clipPosed(clip) ||
+ !!clip.boxStyle?.shadow ||
!!(clip.mask && (clip.mask.kind !== "subject" || this.subjectMatteProvider)))
);
}
@@ -364,7 +368,7 @@ export class FrameCompositor {
const W = this.canvas.width;
const H = this.canvas.height;
const tLocal = Math.max(0, at - clip.start);
- const pose = clipKeyed(clip) ? clipPoseAt(clip, tLocal) : null;
+ const pose = clipPosed(clip) ? clipPoseAt(clip, tLocal) : null;
if (pose && pose.opacity <= 0.001) return;
const mask = clip.mask;
const { surface: layer } = this.scratch("layerScratch", W, H);
@@ -374,7 +378,17 @@ export class FrameCompositor {
const prev = this.canvas;
this.canvas = layer;
try {
- draw({ ...clip, mask: undefined, kf: undefined });
+ // The pose pass owns the turn and the fade; the inner draw takes the
+ // picture plain, or it would apply them a second time (and re-enter
+ // here forever).
+ draw({
+ ...clip,
+ mask: undefined,
+ kf: undefined,
+ rotation: undefined,
+ opacity: undefined,
+ ...(clip.boxStyle ? { boxStyle: { ...clip.boxStyle, shadow: undefined } } : {}),
+ });
} finally {
this.canvas = prev;
}
@@ -437,10 +451,45 @@ export class FrameCompositor {
ctx.save();
ctx.translate(Math.round(fx.dx ?? 0), Math.round(fx.dy ?? 0));
}
+ const shade = this.shadowOf(out, clip.boxStyle?.shadow);
+ if (shade) ctx.drawImage(shade, 0, 0);
ctx.drawImage(out, 0, 0);
if (hasFx) ctx.restore();
}
+ /**
+ * The shadow a finished layer casts: the layer blurred and thrown behind
+ * itself, with the layer's own shape punched back out. What is left is the
+ * shadow alone, so it can be laid down before the picture (here) or over the
+ * frame beside it (the export's painted PNG) and look the same either way.
+ * Null when the clip casts none.
+ */
+ private shadowOf(layer: Surface, sh: ClipShadow | undefined): Surface | null {
+ const ctx = this.ctx();
+ if (!ctx || !sh) return null;
+ const W = this.canvas.width;
+ const H = this.canvas.height;
+ const { surface } = this.scratch("shadowScratch", W, H);
+ const sctx = surface.getContext("2d") as Ctx | null;
+ if (!sctx) return null;
+ sctx.setTransform(1, 0, 0, 1, 0, 0);
+ sctx.globalCompositeOperation = "source-over";
+ sctx.clearRect(0, 0, W, H);
+ // Shadow lengths are design px at the 1080 short side, like masks.
+ const ds = Math.min(W, H) / 1080;
+ sctx.save();
+ sctx.shadowColor = shadowInk(sh);
+ sctx.shadowBlur = Math.max(0, sh.blur) * ds;
+ sctx.shadowOffsetX = (sh.x ?? 0) * ds;
+ sctx.shadowOffsetY = (sh.y ?? 0) * ds;
+ sctx.drawImage(layer as CanvasImageSource, 0, 0);
+ sctx.restore();
+ sctx.globalCompositeOperation = "destination-out";
+ sctx.drawImage(layer as CanvasImageSource, 0, 0);
+ sctx.globalCompositeOperation = "source-over";
+ return surface;
+ }
+
/** Draw a frame into a sub-rectangle of the canvas (a split-screen half, an
* overlay's region), fitted or filled. */
drawIntoRect(
@@ -467,20 +516,22 @@ export class FrameCompositor {
const ry = rect.y * H;
const rw = rect.w * W;
const rh = rect.h * H;
- const vw = frame.width;
- const vh = frame.height;
- const sc = (fill ? Math.max(rw / vw, rh / vh) : Math.min(rw / vw, rh / vh)) * zoom;
- const dw = vw * sc;
- const dh = vh * sc;
- let dx = rx + (rw - dw) / 2;
- let dy = ry + (rh - dh) / 2;
- if (fill) {
- // Pan the crop window across the overflow (matches the export crop).
- const kx = 0.5 + Math.max(-1, Math.min(1, clip?.panX ?? 0)) / 2;
- const ky = 0.5 + Math.max(-1, Math.min(1, clip?.panY ?? 0)) / 2;
- dx = rx - (dw - rw) * kx;
- dy = ry - (dh - rh) * ky;
- }
+ // The clip's own zoom rides on the transition's, so a cross zoom over a
+ // zoomed clip pushes in from where the clip already sits.
+ const {
+ x: dx,
+ y: dy,
+ w: dw,
+ h: dh,
+ } = contentRect(
+ { x: rx, y: ry, w: rw, h: rh },
+ frame.width,
+ frame.height,
+ fill,
+ clipZoom(clip ?? {}) * zoom,
+ clip?.panX ?? 0,
+ clip?.panY ?? 0
+ );
const src = this.gradedSource(frame, clip);
const bs = clip?.boxStyle;
// Box style lengths are design px at the 1080 short side, like masks.
@@ -488,7 +539,7 @@ export class FrameCompositor {
const rad = Math.max(0, (bs?.radius ?? 0) * ds);
const prevAlpha = ctx.globalAlpha;
ctx.globalAlpha = Math.max(0, Math.min(1, alpha));
- if (fill || zoom > 1 || rad > 0) {
+ if (fill || dw > rw + 0.5 || dh > rh + 0.5 || rad > 0) {
ctx.save();
ctx.beginPath();
ctx.roundRect(rx, ry, rw, rh, rad);
@@ -570,25 +621,25 @@ export class FrameCompositor {
// which knows how to draw the style at full frame too.
const rect = rectOf(clip ?? {});
if (!isFullRect(rect) || clip?.boxStyle) {
- this.drawIntoRect(frame, rect, clip?.fit === "fill", alpha, at, zoom, clip);
+ this.drawIntoRect(frame, rect, !!clip && clipCovers(clip), alpha, at, zoom, clip);
if (hasFx) ctx.restore();
return;
}
- const fill = clip?.fit === "fill";
- const vw = frame.width;
- const vh = frame.height;
- const scale = (fill ? Math.max(W / vw, H / vh) : Math.min(W / vw, H / vh)) * zoom;
- const dw = vw * scale;
- const dh = vh * scale;
- let dx = (W - dw) / 2;
- let dy = (H - dh) / 2;
- if (fill) {
- // Pan the crop window across the overflow (matches the export crop).
- const kx = 0.5 + (clip?.panX ?? 0) / 2;
- const ky = 0.5 + (clip?.panY ?? 0) / 2;
- dx = -(dw - W) * kx;
- dy = -(dh - H) * ky;
- }
+ const fill = !!clip && clipCovers(clip);
+ const {
+ x: dx,
+ y: dy,
+ w: dw,
+ h: dh,
+ } = contentRect(
+ { x: 0, y: 0, w: W, h: H },
+ frame.width,
+ frame.height,
+ fill,
+ clipZoom(clip ?? {}) * zoom,
+ clip?.panX ?? 0,
+ clip?.panY ?? 0
+ );
const prevAlpha = ctx.globalAlpha;
ctx.globalAlpha = Math.max(0, Math.min(1, alpha));
ctx.drawImage(this.gradedSource(frame, clip), dx, dy, dw, dh);
diff --git a/site/src/cut/lib/exportClient.ts b/site/src/cut/lib/exportClient.ts
index d27a2dad..b5fbe16b 100644
--- a/site/src/cut/lib/exportClient.ts
+++ b/site/src/cut/lib/exportClient.ts
@@ -17,7 +17,7 @@ import { clipSpeed, getClipSpans, overlayLayers, projectDuration, spanSequence,
import { captionStyle, cueOverlay, cueWordWindows, laneCues, laneHidden, subtitleLaneCount, trackPos } from "./subtitles";
import { isMaskAnimated, isOverlayAnimated, normalizeGrade, paintMaskLuma } from "@donkeycut/effects-kit";
import { renderElementFrames, renderElementPng } from "./textRender";
-import { clipPoseAt, frameOf, isStickerOverlay, isTextOverlay, laneOf, overlayAnimStyle, projectBackground, rectOf, regionPx, subjectMasked } from "./types";
+import { clipCovers, clipKeyed, clipPosed, clipPoseAt, clipZoom, contentRect, frameOf, isStickerOverlay, isTextOverlay, laneOf, overlayAnimStyle, projectBackground, rectOf, regionPx, shadowInk, subjectMasked } from "./types";
import type {
Aspect,
AudioClip,
@@ -283,6 +283,102 @@ async function renderClipMaskPictures(
return { frames };
}
+/**
+ * Paint the shadow a clip casts, frame-sized and transparent everywhere else.
+ * The silhouette is the picture the clip really shows — its box, its rounded
+ * corners, its mask, its pose — thrown behind itself and then punched back
+ * out, so the graph can lay the result over the frame and get what sitting
+ * behind the clip would have looked like. A moving clip samples the shadow
+ * per frame, the way a keyframed mask samples its coverage. Null without a
+ * shadow.
+ */
+async function renderClipShadowPictures(
+ clip: VideoClip,
+ asset: { width?: number; height?: number },
+ W: number,
+ H: number,
+ dur: number,
+ tag: string,
+ pngs: ExportPayload["pngs"]
+): Promise<SpecMask | undefined> {
+ const sh = clip.boxStyle?.shadow;
+ if (!sh) return undefined;
+ const scale = Math.min(W, H) / 1080;
+ const frame = { width: W, height: H, scale };
+ const rect = rectOf(clip);
+ const box = { x: rect.x * W, y: rect.y * H, w: rect.w * W, h: rect.h * H };
+ const radius = Math.max(0, (clip.boxStyle?.radius ?? 0) * scale);
+ const mask = clip.mask && clip.mask.kind !== "subject" ? clip.mask : undefined;
+ // The picture's own footprint inside the box — a fitted clip's shadow follows
+ // the picture, not the empty margins beside it.
+ const pic =
+ asset.width && asset.height
+ ? contentRect(box, asset.width, asset.height, clipCovers(clip), clipZoom(clip), clip.panX ?? 0, clip.panY ?? 0)
+ : box;
+ const shape = createRasterCanvas(W, H);
+ const cover = createRasterCanvas(W, H);
+ const canvas = createRasterCanvas(W, H);
+ const anchor = { x: rect.x + rect.w / 2, y: rect.y + rect.h / 2 };
+ const blobAt = (tLocal: number) => {
+ const shapeCtx = shape.getContext("2d") as CanvasRenderingContext2D;
+ shapeCtx.setTransform(1, 0, 0, 1, 0, 0);
+ shapeCtx.clearRect(0, 0, W, H);
+ const pose = clipPosed(clip) ? clipPoseAt(clip, tLocal) : null;
+ if (pose) {
+ shapeCtx.translate(pose.x * W, pose.y * H);
+ shapeCtx.rotate((pose.rotation * Math.PI) / 180);
+ shapeCtx.scale(pose.scale, pose.scale);
+ shapeCtx.translate(-anchor.x * W, -anchor.y * H);
+ }
+ // The lit shape: the picture, trimmed at the box's rounded corners.
+ shapeCtx.save();
+ shapeCtx.beginPath();
+ shapeCtx.roundRect(box.x, box.y, box.w, box.h, radius);
+ shapeCtx.clip();
+ shapeCtx.fillStyle = "#ffffff";
+ shapeCtx.fillRect(pic.x, pic.y, pic.w, pic.h);
+ shapeCtx.restore();
+ if (mask) {
+ // The mask decides the rest of the shape; its luma becomes coverage.
+ paintMaskLuma(cover, mask, tLocal, frame, anchor, 0, 0);
+ shapeCtx.setTransform(1, 0, 0, 1, 0, 0);
+ shapeCtx.globalCompositeOperation = "destination-in";
+ shapeCtx.drawImage(cover as CanvasImageSource, 0, 0);
+ shapeCtx.globalCompositeOperation = "source-over";
+ }
+ const ctx = canvas.getContext("2d") as CanvasRenderingContext2D;
+ ctx.setTransform(1, 0, 0, 1, 0, 0);
+ ctx.globalCompositeOperation = "source-over";
+ ctx.clearRect(0, 0, W, H);
+ ctx.save();
+ ctx.shadowColor = shadowInk(sh);
+ ctx.shadowBlur = Math.max(0, sh.blur) * scale;
+ ctx.shadowOffsetX = (sh.x ?? 0) * scale;
+ ctx.shadowOffsetY = (sh.y ?? 0) * scale;
+ ctx.drawImage(shape as CanvasImageSource, 0, 0);
+ ctx.restore();
+ ctx.globalCompositeOperation = "destination-out";
+ ctx.drawImage(shape as CanvasImageSource, 0, 0);
+ ctx.globalCompositeOperation = "source-over";
+ return rasterCanvasToPng(canvas);
+ };
+ const moves = (mask && isMaskAnimated(mask)) || clipKeyed(clip);
+ if (!moves) {
+ const name = `${tag}.png`;
+ pngs.push({ name, blob: await blobAt(0) });
+ return { file: name };
+ }
+ const step = 1 / MASK_SAMPLE_FPS;
+ const n = Math.max(1, Math.round(dur * MASK_SAMPLE_FPS));
+ const frames: { file: string; duration: number }[] = [];
+ for (let i = 0; i < n; i++) {
+ const name = `${tag}_f${i}.png`;
+ pngs.push({ name, blob: await blobAt(i * step) });
+ frames.push({ file: name, duration: i === n - 1 ? Math.max(step, dur - (n - 1) * step) : step });
+ }
+ return { frames };
+}
+
/** Paint the clip's border ring — a stroked rounded rect along its box edge,
* transparent everywhere else — sized to the segment the graph frames (the
* full frame for track 0, the region box for overlays). The engine overlays
@@ -351,6 +447,28 @@ async function buildExportPayload(
behindMask = { file: "behind_mask.mp4", from: mask.from };
}
}
+ /** A clip whose resting turn or fade rides as a one-key pose track: the
+ * graph and the mask painter both read the picture's pose off `kf`, and a
+ * single key holds flat, so the export renders the resting transform
+ * through the path the animated one already takes. */
+ const posed = (c: VideoClip): VideoClip => {
+ if (!clipPosed(c) || clipKeyed(c)) return c;
+ const rect = rectOf(c);
+ return {
+ ...c,
+ kf: [
+ {
+ t: 0,
+ x: rect.x + rect.w / 2,
+ y: rect.y + rect.h / 2,
+ scale: 1,
+ rotation: c.rotation ?? 0,
+ opacity: c.opacity ?? 1,
+ },
+ ],
+ };
+ };
+
/** The subject entry a masked item ships, or undefined without a matte. */
const subjectOf = (m: { invert?: boolean; feather?: number } | undefined) =>
behindMask && m
@@ -367,6 +485,7 @@ async function buildExportPayload(
muted: sp.clip.muted,
volume: sp.clip.volume ?? 1,
fit: sp.clip.fit ?? "fit",
+ zoom: clipZoom(sp.clip),
panX: sp.clip.panX ?? 0,
panY: sp.clip.panY ?? 0,
frame: sp.clip.frame,
@@ -386,7 +505,8 @@ async function buildExportPayload(
image: sp.asset.type === "image",
grade: normalizeGrade(sp.clip.grade),
mask: undefined as SpecMask | undefined,
- kf: sp.clip.kf,
+ shadow: undefined as SpecMask | undefined,
+ kf: posed(sp.clip).kf,
border: undefined as string | undefined,
}));
// Track-0 segments render at the full output frame (regioned clips pad out
@@ -394,7 +514,7 @@ async function buildExportPayload(
// matte; painted pictures still travel beside it when the pose track keys
// opacity, since opacity ships as coverage luma.
for (let i = 0; i < spans.length; i++) {
- const c = spans[i].clip;
+ const c = posed(spans[i].clip);
const dur = Math.max(0.1, (c.out - c.in) / clipSpeed(c));
const pictures = await renderClipMaskPictures(
c,
@@ -425,6 +545,15 @@ async function buildExportPayload(
pngs.push({ name, blob: await borderBlob });
clipEntries[i].border = name;
}
+ clipEntries[i].shadow = await renderClipShadowPictures(
+ c,
+ spans[i].asset,
+ settings.width,
+ settings.height,
+ dur,
+ `shadow_c${i}`,
+ pngs
+ );
}
// The server's video graph is a sequential fold, so gaps between the
@@ -437,6 +566,7 @@ async function buildExportPayload(
muted: true,
volume: 0,
fit: "fit" as const,
+ zoom: 1,
panX: 0,
panY: 0,
frame: undefined,
@@ -507,6 +637,7 @@ async function buildExportPayload(
// Pass `fit` through unset so the server's "default full-frame overlay
// covers what's below" branch fires — normalizing to "fit" defeated it.
fit: c.fit,
+ zoom: clipZoom(c),
panX: c.panX ?? 0,
panY: c.panY ?? 0,
muted: c.muted,
@@ -517,11 +648,12 @@ async function buildExportPayload(
look: c.look,
lookAmount: c.lookAmount,
mask: undefined as SpecMask | undefined,
- kf: c.kf,
+ shadow: undefined as SpecMask | undefined,
+ kf: posed(c).kf,
border: undefined as string | undefined,
...ramp,
};
- overlayClipOf.set(entry, c);
+ overlayClipOf.set(entry, posed(c));
return entry;
});
});
@@ -563,6 +695,15 @@ async function buildExportPayload(
pngs.push({ name, blob: await borderBlob });
entry.border = name;
}
+ entry.shadow = await renderClipShadowPictures(
+ c,
+ assetById.get(c.assetId) ?? {},
+ settings.width,
+ settings.height,
+ olen,
+ `shadow_ov${i}`,
+ pngs
+ );
}
const audio = doc.audioClips
diff --git a/site/src/cut/lib/exportRender.ts b/site/src/cut/lib/exportRender.ts
index 497e8922..695f431f 100644
--- a/site/src/cut/lib/exportRender.ts
+++ b/site/src/cut/lib/exportRender.ts
@@ -36,7 +36,7 @@ import { applyEffectToCanvas, evalOverlayFrame, grainTile, isMaskAnimated, isOve
import { hasSubjectOverlays, SubjectMaskCompositor } from "./behindPass";
import { createRasterCanvas, type RasterSurface } from "./raster";
import { renderElementPng } from "./textRender";
-import { behindSubjectOverlay, frameOf, frontSubjectOverlay, isEffectOverlay, isFullRect, isTextOverlay, laneOf, overlayAnimStyle, projectBackground, projectFadeSeconds, rectOf } from "./types";
+import { behindSubjectOverlay, clipCovers, frameOf, frontSubjectOverlay, isEffectOverlay, isTextOverlay, laneOf, overlayAnimStyle, projectBackground, projectFadeSeconds, rectOf } from "./types";
import type { ClipSpan, EffectOverlay, MediaAsset, Overlay, StickerOverlay } from "./types";
import type { ExportDoc, ExportSettings } from "./exportClient";
@@ -890,9 +890,15 @@ export class FramePainter {
const span = this.spanOfClip.get(layer.clip.id);
if (!span) continue;
const frame = await this.readerFor(layer.asset).frameAt(sourceTimeAt(span, t));
- const rect = rectOf(layer.clip);
- const cover = layer.clip.fit === "fill" || (layer.clip.fit == null && isFullRect(rect));
- comp.drawIntoRect(frame, rect, cover, layer.alpha, t, layer.zoom, layer.clip);
+ comp.drawIntoRect(
+ frame,
+ rectOf(layer.clip),
+ clipCovers(layer.clip),
+ layer.alpha,
+ t,
+ layer.zoom,
+ layer.clip
+ );
}
// The subject pass: video → behind elements → segmented person, exactly
diff --git a/site/src/cut/lib/maskVideo.ts b/site/src/cut/lib/maskVideo.ts
index 52ce19c2..37e7423c 100644
--- a/site/src/cut/lib/maskVideo.ts
+++ b/site/src/cut/lib/maskVideo.ts
@@ -26,7 +26,7 @@ import { ClipReader, VIDEO_CODECS } from "./exportRender";
import type { ExportDoc } from "./exportClient";
import { createRasterCanvas } from "./raster";
import { getClipSpans } from "./store";
-import { frameOf, isEffectOverlay, isFullRect, isTextOverlay, rectOf, subjectMasked, type ClipSpan, type MediaAsset } from "./types";
+import { clipCovers, frameOf, isEffectOverlay, isTextOverlay, rectOf, subjectMasked, type ClipSpan, type MediaAsset } from "./types";
/** Encoded mask rate — the server's fps filter duplicates frames up to the
* output rate, and a person moves little in 1/15s. */
@@ -131,9 +131,7 @@ export async function renderSubjectMask(
const span = spanOfClip.get(layer.clip.id);
if (!span) continue;
const f = await readerFor(layer.asset).frameAt(sourceTimeAt(span, t));
- const rect = rectOf(layer.clip);
- const cover = layer.clip.fit === "fill" || (layer.clip.fit == null && isFullRect(rect));
- comp.drawIntoRect(f, rect, cover, layer.alpha, t, layer.zoom, layer.clip);
+ comp.drawIntoRect(f, rectOf(layer.clip), clipCovers(layer.clip), layer.alpha, t, layer.zoom, layer.clip);
}
// Luma mask: black frame, the subject's alpha silhouette drawn white.
mctx.globalCompositeOperation = "source-over";
diff --git a/site/src/cut/lib/media.ts b/site/src/cut/lib/media.ts
index f9cac1d8..dbcf6fe1 100644
--- a/site/src/cut/lib/media.ts
+++ b/site/src/cut/lib/media.ts
@@ -43,7 +43,7 @@ import {
} from "./raster";
import { useEditor } from "./store";
import type { AssetType, AudioClip, MediaAsset, ProjectSummary, StoredAsset, VideoClip, WatchKeepReason } from "./types";
-import { IMAGE_CLIP_SECONDS, mediaUrl } from "./types";
+import { contentRect, IMAGE_CLIP_SECONDS, mediaUrl } from "./types";
import { createDedupSelector, DEDUP_TUNING } from "./watch/dedupSelector";
import { diffCellPct, frameSig } from "./watch/signatures";
import { SIGNATURE_SIZE } from "./watch/types";
@@ -1001,11 +1001,46 @@ export async function captureFreezeFrame(
projectId: string,
sourceUrl: string,
srcTime: number,
- duration = 0
+ duration = 0,
+ /** Bake the still at the project frame, framed as the clip is — the engine's
+ * /freeze route does the same with ffmpeg, so a still captured in the browser
+ * and one captured on the Mac are the same picture. */
+ framed?: {
+ frame: { w: number; h: number };
+ cover: boolean;
+ zoom?: number;
+ panX?: number;
+ panY?: number;
+ }
): Promise<MediaAsset> {
const frame = await frameAt(sourceUrl, srcTime);
if (!frame) throw new Error("Could not render the freeze frame.");
- const blob = await rasterCanvasToBlob(frame.canvas, "image/png");
+ let picture = frame.canvas;
+ if (framed) {
+ const { w, h } = framed.frame;
+ const out = createRasterCanvas(w, h);
+ const ctx = out.getContext("2d") as CanvasRenderingContext2D;
+ ctx.fillStyle = "#000000";
+ ctx.fillRect(0, 0, w, h);
+ const box = { x: 0, y: 0, w, h };
+ const r = contentRect(
+ box,
+ frame.canvas.width,
+ frame.canvas.height,
+ framed.cover,
+ framed.zoom,
+ framed.panX,
+ framed.panY
+ );
+ ctx.save();
+ ctx.beginPath();
+ ctx.rect(0, 0, w, h);
+ ctx.clip();
+ ctx.drawImage(frame.canvas as CanvasImageSource, r.x, r.y, r.w, r.h);
+ ctx.restore();
+ picture = out;
+ }
+ const blob = await rasterCanvasToBlob(picture, "image/png");
const d = new Date();
const pad = (n: number) => String(n).padStart(2, "0");
const fileName = `freeze-${pad(d.getHours())}${pad(d.getMinutes())}${pad(d.getSeconds())}-${uid().slice(0, 4)}.png`;
diff --git a/site/src/cut/lib/store.ts b/site/src/cut/lib/store.ts
index 92a9eb1b..854663cb 100644
--- a/site/src/cut/lib/store.ts
+++ b/site/src/cut/lib/store.ts
@@ -67,6 +67,28 @@ import { captureTimelineFrames } from "./visualFrames";
const uid = () => crypto.randomUUID().slice(0, 8);
+/** How a clip meets its box, for a template to remember. */
+const framingOf = (c: VideoClip) => ({
+ ...(c.frame ? { frame: c.frame } : {}),
+ ...(c.fit ? { fit: c.fit } : {}),
+ ...(c.zoom && c.zoom > 1 ? { zoom: c.zoom } : {}),
+ ...(c.panX ? { panX: c.panX } : {}),
+ ...(c.panY ? { panY: c.panY } : {}),
+ ...(c.rotation ? { rotation: c.rotation } : {}),
+ ...((c.opacity ?? 1) < 1 ? { opacity: c.opacity } : {}),
+});
+
+/** The same framing, back on a clip a template stands up. */
+const templateFraming = (l: TemplateLayer) => ({
+ ...(l.frame ? { frame: l.frame } : {}),
+ ...(l.fit ? { fit: l.fit } : {}),
+ ...(l.zoom && l.zoom > 1 ? { zoom: l.zoom } : {}),
+ ...(l.panX ? { panX: l.panX } : {}),
+ ...(l.panY ? { panY: l.panY } : {}),
+ ...(l.rotation ? { rotation: l.rotation } : {}),
+ ...((l.opacity ?? 1) < 1 ? { opacity: l.opacity } : {}),
+});
+
const MIN_LEN = 0.1;
/** Where a video clip lands when dropped: an existing track or a brand-new
@@ -3241,13 +3263,13 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {
if (mi == null) continue;
// Track-0 clips re-materialize onto track 0 (asClip), so a template
// stands up its own video instead of an empty timeline.
- layers.push({ media: mi, start: sp.start - start0, in: sp.clip.in, out: sp.clip.out, frame: sp.clip.frame, fit: sp.clip.fit, muted: sp.clip.muted, speed: sp.clip.speed, track: 1, asClip: true });
+ layers.push({ media: mi, start: sp.start - start0, in: sp.clip.in, out: sp.clip.out, ...framingOf(sp.clip), muted: sp.clip.muted, speed: sp.clip.speed, track: 1, asClip: true });
} else {
const c = s.clips.find((x) => x.id === sel.id);
if (!c) continue;
const mi = mediaFor(c.assetId);
if (mi == null) continue;
- layers.push({ media: mi, start: c.start - start0, in: c.in, out: c.out, frame: c.frame, fit: c.fit, muted: c.muted, speed: c.speed, track: c.track + 1 });
+ layers.push({ media: mi, start: c.start - start0, in: c.in, out: c.out, ...framingOf(c), muted: c.muted, speed: c.speed, track: c.track + 1 });
}
} else if (sel.kind === "audio") {
const c = s.audioClips.find((x) => x.id === sel.id);
@@ -3346,8 +3368,7 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {
in: l.in,
out: l.out,
muted: l.muted,
- ...(l.frame ? { frame: l.frame } : {}),
- ...(l.fit ? { fit: l.fit } : {}),
+ ...templateFraming(l),
...(l.speed ? { speed: l.speed } : {}),
}));
const topTrack = Math.max(0, ...overlayLayers(get().clips).map((c) => c.track));
@@ -3364,8 +3385,7 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {
in: l.in,
out: l.out,
muted: l.muted,
- ...(l.frame ? { frame: l.frame } : {}),
- ...(l.fit ? { fit: l.fit } : {}),
+ ...templateFraming(l),
...(l.speed ? { speed: l.speed } : {}),
}));
const newAudio: AudioClip[] = template.audio
diff --git a/site/src/cut/lib/types.test.ts b/site/src/cut/lib/types.test.ts
index b61d78d5..61eebdfa 100644
--- a/site/src/cut/lib/types.test.ts
+++ b/site/src/cut/lib/types.test.ts
@@ -1,5 +1,5 @@
import { describe, expect, test } from "bun:test";
-import { migrateLegacyTransitions, normalizeAspect, TRANSITION_STYLE_IDS, type VideoClip } from "./types";
+import { clipCovers, contentRect, migrateLegacyTransitions, normalizeAspect, TRANSITION_STYLE_IDS, type VideoClip } from "./types";
/** A minimal track-0 clip; legacy docs carry retired transitionStyle strings. */
function clip(
@@ -126,3 +126,70 @@ describe("normalizeAspect", () => {
expect(normalizeAspect(null)).toBe(null);
});
});
+
+describe("contentRect", () => {
+ // A vertical frame and a landscape source: the shape every phone shot has to
+ // be framed into.
+ const box = { x: 0, y: 0, w: 720, h: 1280 };
+ const src = { w: 1920, h: 1080 };
+ const near = (a: number, b: number) => expect(Math.abs(a - b)).toBeLessThan(0.01);
+
+ test("a fitted picture is centered, with margins", () => {
+ const r = contentRect(box, src.w, src.h, false);
+ near(r.w, 720);
+ near(r.h, 405);
+ near(r.x, 0);
+ near(r.y, (1280 - 405) / 2);
+ });
+
+ test("a fitted picture ignores the pan on an axis with nothing to spare", () => {
+ const r = contentRect(box, src.w, src.h, false, 1, -1, -1);
+ near(r.x, 0);
+ near(r.y, (1280 - 405) / 2);
+ });
+
+ test("a covering picture fills the box and crops the rest", () => {
+ const r = contentRect(box, src.w, src.h, true);
+ near(r.w, 1280 * (16 / 9));
+ near(r.h, 1280);
+ // Centered: the same slice hangs off each side.
+ near(r.x, (720 - r.w) / 2);
+ });
+
+ test("panning a covering picture slides the box to the picture's edge", () => {
+ const left = contentRect(box, src.w, src.h, true, 1, -1, 0);
+ near(left.x, 0);
+ const right = contentRect(box, src.w, src.h, true, 1, 1, 0);
+ near(right.x, 720 - right.w);
+ });
+
+ test("zoom scales the picture about the box and starts cropping", () => {
+ const r = contentRect(box, src.w, src.h, false, 2);
+ near(r.w, 1440);
+ near(r.h, 810);
+ // Half the picture's width now hangs outside the box, evenly split.
+ near(r.x, -360);
+ // The short axis still fits, so it stays centered.
+ near(r.y, (1280 - 810) / 2);
+ });
+
+ test("a zoomed pan reaches the picture's edge, matching the export crop", () => {
+ const r = contentRect(box, src.w, src.h, false, 2, 1, 0);
+ near(r.x, 720 - 1440);
+ });
+});
+
+describe("clipCovers", () => {
+ const base = { id: "c", assetId: "a", start: 0, in: 0, out: 4, muted: false };
+ test("track 0 letterboxes until it is told to fill", () => {
+ expect(clipCovers({ ...base, track: 0 } as VideoClip)).toBe(false);
+ expect(clipCovers({ ...base, track: 0, fit: "fill" } as VideoClip)).toBe(true);
+ });
+
+ test("a full-frame upper track covers, a regioned one fits", () => {
+ expect(clipCovers({ ...base, track: 1 } as VideoClip)).toBe(true);
+ expect(
+ clipCovers({ ...base, track: 1, frame: { x: 0.1, y: 0.1, w: 0.4, h: 0.4 } } as VideoClip)
+ ).toBe(false);
+ });
+});
diff --git a/site/src/cut/lib/types.ts b/site/src/cut/lib/types.ts
index eca4a4a9..ff15e109 100644
--- a/site/src/cut/lib/types.ts
+++ b/site/src/cut/lib/types.ts
@@ -242,6 +242,32 @@ export interface BoxStyle {
borderWidth?: number;
/** Border stroke color (hex). */
borderColor?: string;
+ /** Drop shadow cast by the clip's own shape — its box, its rounded corners,
+ * its mask. Absent = none. */
+ shadow?: ClipShadow;
+}
+
+/** A drop shadow's ink and throw. Lengths are design px at the 1080 short
+ * side, like the rest of BoxStyle. */
+export interface ClipShadow {
+ /** Blur radius; 0 casts a hard-edged copy. */
+ blur: number;
+ /** Offset; absent = straight down by nothing. */
+ x?: number;
+ y?: number;
+ /** Ink (hex); absent = black. */
+ color?: string;
+ /** 0..1; absent = 0.35. */
+ opacity?: number;
+}
+
+/** The shadow's ink as a CSS color, ready for a canvas shadow. */
+export function shadowInk(sh: ClipShadow): string {
+ const hex = (sh.color ?? "#000000").replace("#", "");
+ const full = hex.length === 3 ? hex.split("").map((c) => c + c).join("") : hex;
+ const n = parseInt(full.slice(0, 6) || "000000", 16);
+ const a = Math.max(0, Math.min(1, sh.opacity ?? 0.35));
+ return `rgba(${(n >> 16) & 255}, ${(n >> 8) & 255}, ${n & 255}, ${a})`;
}
/** How far a region box may oversize the frame on each axis. Oversizing is
@@ -249,6 +275,64 @@ export interface BoxStyle {
* mask canvases within what ffmpeg and browser canvases handle. */
export const REGION_MAX_SCALE = 3;
+/** How far a clip's picture may zoom past the size its box fits it to. The
+ * ceiling keeps export scale targets inside what ffmpeg and browser canvases
+ * handle, the same reason `REGION_MAX_SCALE` has one. */
+export const CLIP_MAX_ZOOM = 4;
+
+/** A clip's zoom, clamped to what every surface renders. */
+export const clipZoom = (c: { zoom?: number }): number =>
+ Math.max(1, Math.min(CLIP_MAX_ZOOM, c.zoom ?? 1));
+
+/**
+ * Where a clip's picture lands inside a box, in the box's own units: fitted
+ * (letterboxed) or covering (cropped), zoomed, and slid by the pan.
+ *
+ * The preview outlines this rect, the compositor draws into it, and the export
+ * graph scales and crops to the same numbers; one geometry keeps the three in
+ * step.
+ */
+export function contentRect(
+ box: { x: number; y: number; w: number; h: number },
+ srcW: number,
+ srcH: number,
+ cover: boolean,
+ zoom = 1,
+ panX = 0,
+ panY = 0
+): { x: number; y: number; w: number; h: number } {
+ // The clip's own zoom arrives already clamped (`clipZoom`); a transition's
+ // push rides on top of it and may pull below 1, so nothing is clamped here.
+ const z = zoom > 0 ? zoom : 1;
+ const base = cover
+ ? Math.max(box.w / srcW, box.h / srcH)
+ : Math.min(box.w / srcW, box.h / srcH);
+ const w = srcW * base * z;
+ const h = srcH * base * z;
+ // Centered, then slid by the pan across whatever overflows the box — an axis
+ // with nothing to spare stays centered, which is the export's
+ // `crop=min(iw,box)` on the same picture.
+ const kx = 0.5 + Math.max(-1, Math.min(1, panX)) / 2;
+ const ky = 0.5 + Math.max(-1, Math.min(1, panY)) / 2;
+ return {
+ x: box.x + (box.w - w) / 2 + Math.max(0, w - box.w) * (0.5 - kx),
+ y: box.y + (box.h - h) / 2 + Math.max(0, h - box.h) * (0.5 - ky),
+ w,
+ h,
+ };
+}
+
+/** Whether a clip's picture covers its box, cropping what hangs over. Track 0
+ * letterboxes until it is told to fill; a clip on an upper track with the whole
+ * frame to itself covers by default, since it stands in for the picture
+ * beneath it. */
+export const clipCovers = (c: {
+ fit?: "fit" | "fill";
+ frame?: FrameRect;
+ track?: number;
+}): boolean =>
+ c.fit === "fill" || (c.fit == null && (c.track ?? 0) > 0 && isFullRect(rectOf(c)));
+
/** A clip's effective region: its own `frame`, or the full frame if unset. */
export function rectOf(clip: { frame?: FrameRect }): FrameRect {
return clip.frame ?? FULL_FRAME;
@@ -292,13 +376,30 @@ export function regionPx(
/** Whether the clip carries pose keys worth evaluating. */
export const clipKeyed = (c: { kf?: OverlayKey[] }): boolean => !!c.kf && c.kf.length > 0;
+/** Whether the clip's picture needs the pose pass at all: a key track, or a
+ * resting turn or fade the compositor has to blit rather than draw. */
+export const clipPosed = (c: {
+ kf?: OverlayKey[];
+ rotation?: number;
+ opacity?: number;
+}): boolean =>
+ clipKeyed(c) || Math.abs(c.rotation ?? 0) > 0.001 || (c.opacity ?? 1) < 0.999;
+
/**
* The clip's pose at `tLocal` seconds into its window: resting at its region
* center, or moving along its key track — the overlay evaluator over the
* clip's own anchor, so clips and elements share one interpolation.
*/
export function clipPoseAt(
- clip: { frame?: FrameRect; in: number; out: number; speed?: number; kf?: OverlayKey[] },
+ clip: {
+ frame?: FrameRect;
+ in: number;
+ out: number;
+ speed?: number;
+ rotation?: number;
+ opacity?: number;
+ kf?: OverlayKey[];
+ },
tLocal: number
): OverlayPose {
const rect = rectOf(clip);
@@ -309,6 +410,8 @@ export function clipPoseAt(
end: len,
x: rect.x + rect.w / 2,
y: rect.y + rect.h / 2,
+ rotation: clip.rotation,
+ opacity: clip.opacity,
kf: clip.kf,
},
tLocal
@@ -369,10 +472,20 @@ export interface VideoClip {
* clip share the frame with another track (e.g. a split-screen half) or float
* small over it (picture-in-picture). */
frame?: FrameRect;
- /** Crop-window pan in fill mode, -1..1 per axis (0 = centered): which part
- * of the oversized video stays visible. */
+ /** How far the picture zooms past the size its box fits it to, 1..
+ * CLIP_MAX_ZOOM; absent = 1. Zooming crops, so the pan below chooses what
+ * stays visible. */
+ zoom?: number;
+ /** Crop-window pan, -1..1 per axis (0 = centered): which part of an
+ * oversized picture stays visible. Live whenever the picture overflows its
+ * box — covering it, zoomed into it, or both. */
panX?: number;
panY?: number;
+ /** Resting turn, degrees clockwise about the box center; absent = upright.
+ * A pose key track overrides it while it plays. */
+ rotation?: number;
+ /** Resting opacity 0..1; absent = opaque. */
+ opacity?: number;
/** Playback rate, default 1 (absent). The source (out-in) seconds play in
* (out-in)/speed timeline seconds, so >1 is faster and shorter. */
speed?: number;
@@ -744,6 +857,12 @@ export interface TemplateLayer {
out: number;
frame?: FrameRect;
fit?: "fit" | "fill";
+ /** The picture's own framing inside that box: zoom, crop pan, turn, fade. */
+ zoom?: number;
+ panX?: number;
+ panY?: number;
+ rotation?: number;
+ opacity?: number;
muted: boolean;
speed?: number;
track: number;
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index c86df172..2f5db3fe 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -83,7 +83,9 @@ Times are in seconds on the shared timeline. The playhead is currentTime; a skim
- Anything can be hidden instead of deleted: it stays on the timeline (grayed) but drops out of playback and export — a hidden video clip plays black and silent. Per item: set_clip_hidden, or the \`hidden\` field on update_overlay_video/update_overlay/update_audio. Whole rows: set_track_hidden (video/soundtrack/text/subtitles) and set_track_muted (video) — the eye/speaker toggles on the track headers.
- detach_audio lifts a video clip's sound to the soundtrack track (and mutes the clip) so audio can be cut independently of video.
- freeze_frame grabs one frame (default: the playhead — what the user currently sees) as a still clip and inserts it, by default at index 0 as a cover/hook frame ("make this the first frame"). The still is baked at the project's current aspect with the clip's framing applied; if the user later switches aspect they should capture a fresh one.
-- set_framing: per-clip Fit (letterbox) vs Fill (crop to cover the project frame). In Fill, panX/panY position the crop window; the user can also drag the video directly in the preview. The control lives in the Inspector under "Framing" when a video clip is selected. Landscape footage usually wants fill + a pan that keeps the subject.
+- set_framing: per-clip Fit (letterbox) vs Fill (crop to cover the box), plus zoom 1..4 to push further in. Whenever the picture overflows, panX/panY position the crop window; the user can also drag the video directly in the preview, where a grey outline shows the part being cropped away. The controls live in the Inspector under "Framing" when a video clip is selected. Landscape footage usually wants fill + a pan that keeps the subject.
+- update_clip also carries a clip's resting transform: rotation and opacity, alongside its region (layout/region) and zoom. Use these for a tilted or faded layer; set_clip_keyframes is for a transform that moves over time.
+- set_clip_style rounds a clip's corners, strokes a border ring, and casts a drop shadow from the clip's shape — mask included, so a circle-masked webcam gets a circular shadow. The floating-cam look is one recipe: put the clip on an upper track, region it small (update_clip region or the preview handles), mask it with kind circle, and give it a soft shadow.
- The user can copy/paste any selected segment (video, overlay video, audio, title) with ⌘C/⌘V; pastes aim for the playhead and slide right to free space.
- Zoom: set_view pxPerSec (12..800) or fit. The timeline panel height: set_view timelineH (170..600).`,
diff --git a/site/src/cut/server/exportPipeline.ts b/site/src/cut/server/exportPipeline.ts
index 16939fb5..9c95ce17 100644
--- a/site/src/cut/server/exportPipeline.ts
+++ b/site/src/cut/server/exportPipeline.ts
@@ -2,7 +2,7 @@ import { spawn, type ChildProcess } from "node:child_process";
import { stat, writeFile } from "node:fs/promises";
import path from "node:path";
import { atempoChain, hasStream, num, videoColorInfo } from "./util";
-import { projectFadeSeconds, regionPx, TRANSITION_XFADE, TRANSITION_ZOOM, type ColorGrade, type TransitionStyle } from "../lib/types";
+import { CLIP_MAX_ZOOM, projectFadeSeconds, regionPx, TRANSITION_XFADE, TRANSITION_ZOOM, type ColorGrade, type TransitionStyle } from "../lib/types";
import { effectFilterLines, gradeToFfmpegFilter, lookFilterLines, shortestTurn, sortedKeys, type OverlayKey } from "@donkeycut/effects-kit";
// The render pipeline itself: spec in, finished mp4 out. Shared by the local
@@ -39,7 +39,10 @@ export interface ExportSpec {
volume?: number;
/** "fit" letterboxes (default); "fill" covers the region and crops. */
fit?: "fit" | "fill";
- panX?: number; // crop-window pan -1..1 (fill mode)
+ /** How far the picture zooms past the size its box fits it to (1 = none).
+ * Whatever the zoom pushes outside the box is cropped. */
+ zoom?: number;
+ panX?: number; // crop-window pan -1..1, across whatever overflows
panY?: number;
/** Region of the frame this clip fills; absent = full frame. */
frame?: { x: number; y: number; w: number; h: number };
@@ -86,6 +89,14 @@ export interface ExportSpec {
* box, transparent elsewhere), full-frame sized; overlaid onto the
* segment before fades, masks and pose so it rides the clip. */
border?: string;
+ /** Client-painted drop shadow: the clip's own silhouette blurred, thrown,
+ * and punched back out, full-frame sized. It goes down last — a mask would
+ * otherwise trim away the very ink that lies outside the shape — and a
+ * clip that moves ships it as a `frames` sequence like a keyframed mask. */
+ shadow?: {
+ file?: string;
+ frames?: { file: string; duration: number }[];
+ };
}[];
/** Video tracks composited over the track-0 `clips`, lowest track first. */
overlayVideos?: {
@@ -97,7 +108,9 @@ export interface ExportSpec {
/** Region of the frame this overlay fills; absent = full frame. */
frame?: { x: number; y: number; w: number; h: number };
fit?: "fit" | "fill";
- panX?: number; // crop-window pan -1..1 (fill mode)
+ /** How far the picture zooms past the size its box fits it to (1 = none). */
+ zoom?: number;
+ panX?: number; // crop-window pan -1..1, across whatever overflows
panY?: number;
muted: boolean;
/** Gain on the clip's own audio, 0..1.5; absent = 1 (unchanged). */
@@ -135,6 +148,12 @@ export interface ExportSpec {
* to its box when bordered); overlaid onto the segment so it rides the
* clip through fades, masks and pose. */
border?: string;
+ /** Client-painted drop shadow, full-frame sized: laid over the frame just
+ * before the clip goes down, so it falls on the tracks beneath it. */
+ shadow?: {
+ file?: string;
+ frames?: { file: string; duration: number }[];
+ };
}[];
audio: {
file: string;
@@ -388,6 +407,34 @@ export async function runExport(
// clip, a gap on track 0, the backdrop behind an edge animation — is the
// project's own background color. ffmpeg takes it as 0xRRGGBB.
const padColor = ffColor(spec.background);
+ /**
+ * The chain that meets a clip's picture with its box: covering it (cropping
+ * the overflow) or fitted inside it, zoomed, with the pan choosing what the
+ * box keeps. The crop takes `min(picture, box)` per axis, so an axis with
+ * room to spare passes through untouched and stays centered — the same
+ * geometry `contentRect` draws on the canvas.
+ */
+ const boxFraming = (
+ bw: number,
+ bh: number,
+ cover: boolean,
+ zoom?: number,
+ panX?: number,
+ panY?: number
+ ): string => {
+ const z = Math.max(1, Math.min(CLIP_MAX_ZOOM, zoom ?? 1));
+ const even = (n: number) => 2 * Math.round(n / 2);
+ const tw = even(bw * z);
+ const th = even(bh * z);
+ const scale = cover
+ ? `scale=${tw}:${th}:force_original_aspect_ratio=increase`
+ : `scale=${tw}:${th}:force_original_aspect_ratio=decrease:force_divisible_by=2`;
+ // A fitted picture at rest overflows nothing, so it needs no crop at all.
+ if (!cover && z <= 1.0001) return scale;
+ const kx = num(0.5 + Math.max(-1, Math.min(1, panX ?? 0)) / 2);
+ const ky = num(0.5 + Math.max(-1, Math.min(1, panY ?? 0)) / 2);
+ return `${scale},crop=min(iw\,${bw}):min(ih\,${bh}):(iw-ow)*${kx}:(ih-oh)*${ky}`;
+ };
// One ffmpeg input per distinct media file (from the project folder),
// plus one per uploaded overlay PNG.
// Still images are excluded here: a plain `-i file` decodes one frame, so
@@ -538,6 +585,27 @@ export async function runExport(
const idx = await maskInput(oc.mask, olen, `mask_ovl_${k}`);
if (idx !== undefined) overlayMaskInput.set(oc, idx);
}
+ // Shadows arrive the same way coverage does — one looped still, or a
+ // slideshow when the clip moves — so they take the same input builder.
+ const clipShadowInput = new Map<number, number>();
+ for (let j = 0; j < spec.clips.length; j++) {
+ const c = spec.clips[j];
+ if (!c.shadow) continue;
+ const dur = c.file ? Math.max(0.1, (c.out - c.in) / clipRate(c)) : Math.max(0, c.out - c.in);
+ const idx = await maskInput(c.shadow, dur, `shadow_clip_${j}`);
+ if (idx !== undefined) clipShadowInput.set(j, idx);
+ }
+ const overlayShadowInput = new Map<(typeof overlayVideos)[number], number>();
+ for (let k = 0; k < overlayVideos.length; k++) {
+ const oc = overlayVideos[k];
+ if (!oc.shadow) continue;
+ const ospeed = oc.speed && oc.speed > 0 ? oc.speed : 1;
+ // An overlay's shadow lands on the timeline beside the clip, so it plays
+ // for the clip's whole window rather than the segment's local clock.
+ const olen = Math.max(0.1, (oc.out - oc.in) / ospeed);
+ const idx = await maskInput(oc.shadow, olen, `shadow_ovl_${k}`);
+ if (idx !== undefined) overlayShadowInput.set(oc, idx);
+ }
// Border rings loop as stills for their segment's length, like mask files.
const borderStill = (file: string, dur: number): number => {
const idx = nInputs++;
@@ -953,23 +1021,17 @@ export async function runExport(
const bw = Math.max(W, rx + rw) - bx;
const bh = Math.max(H, ry + rh) - by;
const win = bw > W || bh > H ? `,crop=${W}:${H}:${-bx}:${-by}` : "";
+ // The framed picture lands centered in its rect: a covering one fills
+ // the rect exactly, a fitted one keeps its margins.
frame =
- c.fit === "fill"
- ? `scale=${rw}:${rh}:force_original_aspect_ratio=increase,` +
- `crop=${rw}:${rh}:(iw-ow)*${num(0.5 + Math.max(-1, Math.min(1, c.panX ?? 0)) / 2)}` +
- `:(ih-oh)*${num(0.5 + Math.max(-1, Math.min(1, c.panY ?? 0)) / 2)},` +
- `pad=${bw}:${bh}:${rx - bx}:${ry - by}:color=${padColor}${win}`
- : `scale=${rw}:${rh}:force_original_aspect_ratio=decrease:force_divisible_by=2,` +
- `pad=${bw}:${bh}:${rx - bx}+(${rw}-iw)/2:${ry - by}+(${rh}-ih)/2:color=${padColor}${win}`;
+ `${boxFraming(rw, rh, c.fit === "fill", c.zoom, c.panX, c.panY)},` +
+ `pad=${bw}:${bh}:${rx - bx}+(${rw}-iw)/2:${ry - by}+(${rh}-ih)/2:color=${padColor}${win}`;
} else {
+ const cover = c.fit === "fill";
frame =
- c.fit === "fill"
- ? // Cover the frame, then crop; the pan chooses the visible window.
- `scale=${W}:${H}:force_original_aspect_ratio=increase,` +
- `crop=${W}:${H}:(iw-ow)*${num(0.5 + Math.max(-1, Math.min(1, c.panX ?? 0)) / 2)}` +
- `:(ih-oh)*${num(0.5 + Math.max(-1, Math.min(1, c.panY ?? 0)) / 2)}`
- : `scale=${W}:${H}:force_original_aspect_ratio=decrease,` +
- `pad=${W}:${H}:(ow-iw)/2:(oh-ih)/2:color=${padColor}`;
+ boxFraming(W, H, cover, c.zoom, c.panX, c.panY) +
+ // A covering picture already spans the frame; a fitted one letterboxes.
+ (cover ? "" : `,pad=${W}:${H}:(ow-iw)/2:(oh-ih)/2:color=${padColor}`);
}
// setpts/speed rescales the clip's duration on the timeline (footage);
// a still just replays its looped input.
@@ -1001,6 +1063,12 @@ export async function runExport(
// split the copies it needs off the pre-backdrop picture.
const nFz = (needFirstFreeze[j] ? 1 : 0) + (needLastFreeze[j] ? 1 : 0);
const segOut = nFz > 0 ? `vseg${j}` : `v${j}`;
+ // The shadow is ink the clip's own mask must not reach, so the segment
+ // finishes first and the shadow goes over it — outside the clip's shape
+ // the segment is bare frame, which is what the shadow would have fallen
+ // on anyway.
+ const shIdx = clipShadowInput.get(j);
+ const segCore = shIdx === undefined ? segOut : `vshi${j}`;
const mkIdx = clipMaskInput.get(j);
const subjMask = subjectActive && matteConsumers > 0 ? c.mask?.subject : undefined;
const keyed = !!c.kf?.length;
@@ -1057,10 +1125,16 @@ export async function runExport(
}
filters.push(`color=c=black:s=${W}x${H}:r=${fps}:d=${num(dur)}[cmb${j}]`);
filters.push(
- `[cmb${j}][${cur}]overlay=0:0:shortest=1,format=${clipFmt},fps=${fps}[${segOut}]`
+ `[cmb${j}][${cur}]overlay=0:0:shortest=1,format=${clipFmt},fps=${fps}[${segCore}]`
);
} else {
- pushEdgeFx(core, dur, headFx, tailFx, W, H, clipFmt, fades, segOut, `c${j}`);
+ pushEdgeFx(core, dur, headFx, tailFx, W, H, clipFmt, fades, segCore, `c${j}`);
+ }
+ if (shIdx !== undefined) {
+ filters.push(`[${shIdx}:v]fps=${fps},scale=${W}:${H},setsar=1,format=rgba[csh${j}]`);
+ filters.push(
+ `[${segCore}][csh${j}]overlay=0:0:eof_action=pass,format=${clipFmt},fps=${fps}[${segOut}]`
+ );
}
if (nFz > 0) {
filters.push(
@@ -1237,23 +1311,14 @@ export async function runExport(
const keyed = !!oc.kf?.length;
const boxW = region ? region.rw : W;
const boxH = region ? region.rh : H;
- let framing: string;
+ // The overlay's picture meets its box the same way a track-0 clip meets
+ // the frame; whatever the box does not swallow sits centered in it.
+ const framing = boxFraming(boxW, boxH, cover, oc.zoom, oc.panX, oc.panY);
let pos: string;
- // A fill overlay's crop window follows the clip's pan, matching the
- // preview compositor.
- const panCrop =
- `:(iw-ow)*${num(0.5 + Math.max(-1, Math.min(1, oc.panX ?? 0)) / 2)}` +
- `:(ih-oh)*${num(0.5 + Math.max(-1, Math.min(1, oc.panY ?? 0)) / 2)}`;
if (!region) {
- framing = cover
- ? `scale=${W}:${H}:force_original_aspect_ratio=increase,crop=${W}:${H}${panCrop}`
- : `scale=${W}:${H}:force_original_aspect_ratio=decrease:force_divisible_by=2`;
pos = cover ? "0:0" : `x=(${W}-w)/2:y=(${H}-h)/2`;
} else {
const { rx, ry, rw, rh } = region;
- framing = cover
- ? `scale=${rw}:${rh}:force_original_aspect_ratio=increase,crop=${rw}:${rh}${panCrop}`
- : `scale=${rw}:${rh}:force_original_aspect_ratio=decrease:force_divisible_by=2`;
pos = cover ? `${rx}:${ry}` : `x=${rx}+(${rw}-w)/2:y=${ry}+(${rh}-h)/2`;
}
// zoompan needs a constant frame size — and a mask trims at the box, so
@@ -1343,6 +1408,19 @@ export async function runExport(
filters.push(`[${masked}]fps=${fps},tpad=start_duration=${num(oc.start)}[${seg}]`);
const next = `vovv${k}`;
const enable = `enable='between(t,${num(oc.start)},${num(end)})'`;
+ // The shadow falls on whatever is already there, then the clip goes down
+ // over it. Its silhouette is punched out, so the two never fight.
+ const oshIdx = overlayShadowInput.get(oc);
+ if (oshIdx !== undefined) {
+ const lit = `ovsh${k}`;
+ filters.push(
+ `[${oshIdx}:v]fps=${fps},scale=${W}:${H},setsar=1,format=rgba,` +
+ `tpad=start_duration=${num(oc.start)}:[email protected][${lit}]`
+ );
+ const shOnto = `vovs${k}`;
+ filters.push(`[${onto}][${lit}]overlay=0:0:${enable}:eof_action=pass[${shOnto}]`);
+ onto = shOnto;
+ }
if (subjMask) {
// Subject-masked overlay clip: the delayed segment lands on the full
// frame at its spot — a static pad, or an expression overlay onto a
diff --git a/site/src/cut/server/frames.ts b/site/src/cut/server/frames.ts
index 292c31c1..5b4c565d 100644
--- a/site/src/cut/server/frames.ts
+++ b/site/src/cut/server/frames.ts
@@ -60,6 +60,8 @@ export function probeDims(file: string): Promise<{ width: number; height: number
export interface FreezeFraming {
fit: "fit" | "fill";
+ /** How far the picture zooms past the size the frame fits it to (1 = none). */
+ zoom?: number;
panX: number;
panY: number;
}
@@ -95,14 +97,20 @@ export async function makeFreezeFrame(
let vf: string | null = null;
if (frame) {
const { w, h } = frame;
- if (framing?.fit === "fill") {
- // Same crop-window math as the preview canvas: pan -1..1 → 0..1.
- const kx = (0.5 + (framing.panX ?? 0) / 2).toFixed(4);
- const ky = (0.5 + (framing.panY ?? 0) / 2).toFixed(4);
- vf = `scale=${w}:${h}:force_original_aspect_ratio=increase,crop=${w}:${h}:(iw-ow)*${kx}:(ih-oh)*${ky}`;
- } else {
- vf = `scale=${w}:${h}:force_original_aspect_ratio=decrease,pad=${w}:${h}:(ow-iw)/2:(oh-ih)/2:black`;
- }
+ // The clip's own framing, baked: the picture meets the frame the way the
+ // preview shows it — covering or fitted, zoomed, panned. Same chain as
+ // the export graph's `boxFraming`.
+ const cover = framing?.fit === "fill";
+ const z = Math.max(1, Math.min(4, framing?.zoom ?? 1));
+ const even = (n: number) => 2 * Math.round(n / 2);
+ const kx = (0.5 + (framing?.panX ?? 0) / 2).toFixed(4);
+ const ky = (0.5 + (framing?.panY ?? 0) / 2).toFixed(4);
+ const crop = `,crop=min(iw\,${w}):min(ih\,${h}):(iw-ow)*${kx}:(ih-oh)*${ky}`;
+ vf = cover
+ ? `scale=${even(w * z)}:${even(h * z)}:force_original_aspect_ratio=increase${crop}`
+ : `scale=${even(w * z)}:${even(h * z)}:force_original_aspect_ratio=decrease` +
+ (z > 1.0001 ? crop : "") +
+ `,pad=${w}:${h}:(ow-iw)/2:(oh-ih)/2:black`;
}
await run("ffmpeg", [
diff --git a/site/src/cut/server/http/projects.ts b/site/src/cut/server/http/projects.ts
index 0e7d05ab..0e6724e2 100644
--- a/site/src/cut/server/http/projects.ts
+++ b/site/src/cut/server/http/projects.ts
@@ -374,7 +374,7 @@ export const projectsApi = {
srcTime?: number;
duration?: number;
frame?: { w?: number; h?: number };
- framing?: { fit?: "fit" | "fill"; panX?: number; panY?: number };
+ framing?: { fit?: "fit" | "fill"; zoom?: number; panX?: number; panY?: number };
};
if (!body.file || typeof body.srcTime !== "number") {
return err("file and srcTime are required.", 400);
@@ -386,6 +386,7 @@ export const projectsApi = {
const framing = frame
? {
fit: body.framing?.fit === "fill" ? ("fill" as const) : ("fit" as const),
+ zoom: typeof body.framing?.zoom === "number" ? body.framing.zoom : 1,
panX: typeof body.framing?.panX === "number" ? body.framing.panX : 0,
panY: typeof body.framing?.panY === "number" ? body.framing.panY : 0,
}
diff --git a/site/src/cut/server/library.ts b/site/src/cut/server/library.ts
index e03b13e4..133a97bc 100644
--- a/site/src/cut/server/library.ts
+++ b/site/src/cut/server/library.ts
@@ -66,6 +66,12 @@ export interface TemplateLayer {
out: number;
frame?: { x: number; y: number; w: number; h: number };
fit?: "fit" | "fill";
+ /** The picture's own framing inside that box: zoom, crop pan, turn, fade. */
+ zoom?: number;
+ panX?: number;
+ panY?: number;
+ rotation?: number;
+ opacity?: number;
muted: boolean;
speed?: number;
track: number;