Skip to content

The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.

updates

Jareerb9f779f96f9c
diff --git a/BUILD.md b/BUILD.md
index 1e2ef08e..75afcc75 100644
--- a/BUILD.md
+++ b/BUILD.md
@@ -74,27 +74,68 @@ Holds an FFmpeg **development** build (headers and import libraries), used only
by the in-progress FFI decoder described below. It is gitignored, roughly
150 MB, and nothing in the normal build path touches it. Delete it freely.
-## FFI decoder — in progress, not yet wired up
+## FFI decoder — optional, off by default
-The engine currently runs FFmpeg as a subprocess
+The engine talks to FFmpeg as a subprocess
(`engine/docs/decisions/0002-ffmpeg-over-a-pipe.md`). That is permanent for
-probing and export, but playback needs real timestamps and frame-accurate
-seeking, which a pipe cannot provide — so a linked decoder is being added
-behind the existing `FrameSource` trait.
-
-Extra prerequisites for that path only:
-
-- **LLVM / libclang** — `bindgen` generates the FFmpeg bindings at build time.
- `winget install LLVM.LLVM`
-- **An FFmpeg dev build** — headers plus `.lib` import libraries, which the
- ordinary runtime download does not include. Get a `*-gpl-shared-*` archive
- from [BtbN/FFmpeg-Builds](https://github.com/BtbN/FFmpeg-Builds/releases) and
- unpack it into `vendor/`.
-- Environment variables pointing at it (see `engine/crates/relay-media/README`
- once this lands).
-
-This section will be replaced with exact, verified steps when the decoder
-actually builds. Until then, ignore it — the app does not need any of it.
+probing and export. Playback is different: a pipe carries no presentation
+timestamps and cannot seek to an exact frame, so there is a second decoder that
+links FFmpeg directly, behind the `ffi` feature.
+
+Nothing in the app enables it yet. **You do not need any of this to build or
+run Relay.**
+
+### Setup, exactly as verified
+
+1. **Get a development build** — headers plus `.lib` import libraries, which
+ the ordinary runtime download does not include. From
+ [BtbN/FFmpeg-Builds](https://github.com/BtbN/FFmpeg-Builds/releases), take a
+ `win64-gpl-**shared**` archive and unpack it into `vendor/`:
+
+ ```sh
+ curl -L -o ff.zip https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n8.1-latest-win64-gpl-shared-8.1.zip
+ unzip ff.zip -d vendor/
+ ```
+
+ **The version has to match the bindings.** `rusty_ffmpeg` ships
+ pre-generated bindings for a specific FFmpeg, currently 8.1 — avcodec 62,
+ avformat 62, avutil 60. A different major version compiles fine and then
+ misreads struct layouts at runtime, which corrupts silently rather than
+ failing. `ffi::tests::links_against_the_expected_ffmpeg` checks this and is
+ the first thing to look at if the decoder behaves strangely.
+
+2. **Nothing else.** No LLVM, no libclang, no bindgen run — the pre-generated
+ bindings are used as-is. `engine/.cargo/config.toml` already points at
+ `vendor/`, so no environment variables to set either.
+
+ If you unpack a differently-named archive, update the paths in that file.
+
+3. **Build and test:**
+
+ ```sh
+ cd engine
+ PATH="../vendor/ffmpeg-n8.1-latest-win64-gpl-shared-8.1/bin:$PATH" \
+ cargo test -p relay-media --features ffi
+ ```
+
+ The DLLs must be on `PATH` at *run* time. Linking uses `lib/` (import
+ libraries); running needs `bin/` (the DLLs themselves). Pointing either at
+ the other is the most likely thing to go wrong — `LNK1181: cannot open input
+ file 'avcodec.lib'` means the linker was aimed at `bin/`.
+
+### What it provides
+
+`FfiDecoder` implements `FrameSource` plus `SeekableSource`:
+
+- **`position()`** — the real presentation timestamp, as an exact rational. The
+ subprocess decoder returns `None` here, honestly: raw video carries no
+ timestamps, which is why variable-frame-rate material desyncs through a pipe.
+- **`seek()`** — frame-accurate, by seeking to the preceding keyframe and
+ decoding forward. Deliberately a separate trait, so the subprocess decoder
+ cannot pretend to offer it.
+
+`tests/ffi_decode.rs` proves both against a generated fixture, and skips
+entirely if FFmpeg is not installed.
## Troubleshooting
diff --git a/desktop/src/App.tsx b/desktop/src/App.tsx
index 749d8cec..54ce1786 100644
--- a/desktop/src/App.tsx
+++ b/desktop/src/App.tsx
@@ -28,6 +28,7 @@ import {
firstFreeTrack,
findMedia,
findTrack,
+ mergeClips,
moveClip,
moveClips,
projectDuration,
@@ -40,6 +41,7 @@ import {
toMediaItem,
updateClip,
trimClip,
+ whyNotMerge,
type MediaItem,
type Project,
} from "./lib/project";
@@ -338,6 +340,14 @@ function Editor({
});
}, [playhead, selectedClipIds]);
+ const mergeSelected = useCallback(() => {
+ setProject((current) => {
+ const { project: next, clipId } = mergeClips(current, selectedClipIds);
+ if (clipId) setSelectedClipIds([clipId]);
+ return next;
+ });
+ }, [selectedClipIds]);
+
const deleteSelected = useCallback(() => {
if (selectedClipIds.length === 0) return;
// Free the audio element first: once the clip is gone the sync pass has
@@ -413,6 +423,9 @@ function Editor({
case "KeyS":
splitAtPlayhead();
break;
+ case "KeyM":
+ mergeSelected();
+ break;
case "Delete":
case "Backspace":
deleteSelected();
@@ -424,7 +437,7 @@ function Editor({
window.addEventListener("keydown", onKey);
return () => window.removeEventListener("keydown", onKey);
- }, [deleteSelected, duration, frameRate, splitAtPlayhead, transport]);
+ }, [deleteSelected, duration, frameRate, mergeSelected, splitAtPlayhead, transport]);
// The exporter works from a flat list: the engine rebuilds a real timeline
// from it, so track *order* has to survive the trip even though track
@@ -671,6 +684,8 @@ function Editor({
setProject((current) => trimClip(current, clipId, edge, delta))
}
onSplitAtPlayhead={splitAtPlayhead}
+ onMergeSelected={mergeSelected}
+ mergeBlockedBecause={whyNotMerge(project, selectedClipIds)}
onDeleteSelected={deleteSelected}
mediaDrag={mediaDrag ? { x: mediaDrag.x, y: mediaDrag.y } : null}
onZoom={zoom}
@@ -717,6 +732,16 @@ function Editor({
hint: "S",
onSelect: splitAtPlayhead,
},
+ ...(whyNotMerge(project, target) === null
+ ? [
+ {
+ label: `Merge ${target.length} clips`,
+ icon: "merge" as const,
+ hint: "M",
+ onSelect: mergeSelected,
+ },
+ ]
+ : []),
{
label: "Move to playhead",
icon: "select",
diff --git a/desktop/src/components/Icon.tsx b/desktop/src/components/Icon.tsx
index 481ad4ad..04a738f3 100644
--- a/desktop/src/components/Icon.tsx
+++ b/desktop/src/components/Icon.tsx
@@ -70,6 +70,22 @@ const GLYPHS = {
<path d="M15 11V6.5a1.5 1.5 0 0 1 3 0V15a6 6 0 0 1-6 6h-1a7 7 0 0 1-7-7v-2a1.5 1.5 0 0 1 3 0v1" />
</>
),
+ // two blocks parted by a cut line
+ split: (
+ <>
+ <path d="M12 3v18" strokeDasharray="3 3" />
+ <path d="M8 7H5a2 2 0 0 0-2 2v6a2 2 0 0 0 2 2h3" />
+ <path d="M16 7h3a2 2 0 0 1 2 2v6a2 2 0 0 1-2 2h-3" />
+ </>
+ ),
+ // two blocks closing on a seam
+ merge: (
+ <>
+ <path d="M12 3v18" />
+ <path d="M3 7h5a2 2 0 0 1 2 2v6a2 2 0 0 1-2 2H3" />
+ <path d="M21 7h-5a2 2 0 0 0-2 2v6a2 2 0 0 0 2 2h5" />
+ </>
+ ),
// horseshoe magnet, for snapping
magnet: (
<>
diff --git a/desktop/src/components/TimelinePanel.tsx b/desktop/src/components/TimelinePanel.tsx
index 37bc3d12..c8945363 100644
--- a/desktop/src/components/TimelinePanel.tsx
+++ b/desktop/src/components/TimelinePanel.tsx
@@ -130,6 +130,8 @@ export function TimelinePanel({
onMoveClips,
onTrimClip,
onSplitAtPlayhead,
+ onMergeSelected,
+ mergeBlockedBecause,
onDeleteSelected,
mediaDrag,
onZoom,
@@ -165,6 +167,9 @@ export function TimelinePanel({
onMoveClips: (moves: ClipMove[]) => void;
onTrimClip: (clipId: string, edge: "start" | "end", delta: number) => void;
onSplitAtPlayhead: () => void;
+ onMergeSelected: () => void;
+ /** Null when the selection can be merged; otherwise why it cannot. */
+ mergeBlockedBecause: string | null;
onDeleteSelected: () => void;
/** A bin item currently being dragged, in client coordinates. */
mediaDrag: { x: number; y: number } | null;
@@ -214,8 +219,8 @@ export function TimelinePanel({
// The draw loop reads everything through this ref, so a prop change never
// tears down and rebuilds the loop.
- const view = useRef({ project, playhead, playing, secondsPerPixel, scrollLeft, trackScroll, frameRate, selected, rows, dropTrack, assets });
- view.current = { project, playhead, playing, secondsPerPixel, scrollLeft, trackScroll, frameRate, selected, rows, dropTrack, assets };
+ const view = useRef({ project, playhead, playing, secondsPerPixel, scrollLeft, trackScroll, frameRate, selected, rows, dropTrack, assets, tool });
+ view.current = { project, playhead, playing, secondsPerPixel, scrollLeft, trackScroll, frameRate, selected, rows, dropTrack, assets, tool };
const timeAt = useCallback(
(clientX: number) => {
@@ -364,6 +369,41 @@ export function TimelinePanel({
);
}
+ // Razor guide: a dashed line at the pointer, but only while it is over a
+ // clip. Showing it in empty space would promise a cut that does nothing.
+ if (state.tool === "razor" && pointer.current && !drag.current) {
+ const canvas = canvasRef.current;
+ const bounds = canvas?.getBoundingClientRect();
+ if (bounds) {
+ const localX = pointer.current.x - bounds.left;
+ const localY = pointer.current.y - bounds.top;
+ const time = localX * state.secondsPerPixel + state.scrollLeft;
+ const rowIndex = Math.floor((localY - RULER_HEIGHT + state.trackScroll) / TRACK_HEIGHT);
+ const track = state.rows[rowIndex];
+
+ const overClip =
+ localY > RULER_HEIGHT &&
+ track !== undefined &&
+ state.project.clips.some(
+ (clip) =>
+ clip.trackId === track.id && time > clip.start && time < clip.start + clip.duration,
+ );
+
+ if (overClip) {
+ const guide = Math.round(localX) + 0.5;
+ context.save();
+ context.strokeStyle = COLORS.playhead;
+ context.lineWidth = 1;
+ context.setLineDash([4, 3]);
+ context.beginPath();
+ context.moveTo(guide, RULER_HEIGHT);
+ context.lineTo(guide, height);
+ context.stroke();
+ context.restore();
+ }
+ }
+ }
+
// The selection band, drawn over the clips it is catching.
if (drag.current?.kind === "marquee") {
const band = normalise(drag.current);
@@ -667,7 +707,15 @@ export function TimelinePanel({
onClick={() => onToolChange("razor")}
/>
<Divider />
- <IconButton icon="minus" label="Split at playhead (S)" onClick={onSplitAtPlayhead} />
+ <IconButton icon="split" label="Split at playhead (S)" onClick={onSplitAtPlayhead} />
+ <IconButton
+ icon="merge"
+ // The reason lives in the tooltip: a button that greys out without
+ // saying why leaves you guessing at the rule.
+ label={mergeBlockedBecause ?? "Merge selected clips (M)"}
+ disabled={mergeBlockedBecause !== null}
+ onClick={onMergeSelected}
+ />
<IconButton
icon="trash"
label={
diff --git a/desktop/src/lib/project.ts b/desktop/src/lib/project.ts
index a3bae5ce..76efeffa 100644
--- a/desktop/src/lib/project.ts
+++ b/desktop/src/lib/project.ts
@@ -411,6 +411,105 @@ export function clipsOnTrack(project: Project, trackId: string): Clip[] {
return project.clips.filter((clip) => clip.trackId === trackId);
}
+/**
+ * How far apart two clips may sit and still count as touching, in seconds.
+ *
+ * Splitting produces edges that meet exactly, but the numbers have been
+ * through float arithmetic on the way, so an exact comparison would
+ * occasionally refuse to rejoin a clip it had just cut.
+ */
+const JOIN_EPSILON = 1e-6;
+
+/**
+ * Why these clips cannot be merged, or `null` if they can.
+ *
+ * Merge is the inverse of split: it rejoins pieces of *one* source that are
+ * still lying end to end. It is not a way to weld two different files into a
+ * single clip - that is a compound clip, which needs nesting and a render of
+ * its own, and pretending otherwise here would produce a clip whose source
+ * range means nothing.
+ *
+ * Returns a sentence, because a disabled button that will not say why is worse
+ * than no button.
+ */
+export function whyNotMerge(project: Project, clipIds: readonly string[]): string | null {
+ if (clipIds.length < 2) return "Select two or more clips to merge.";
+
+ const clips = clipIds.flatMap((id) => {
+ const clip = findClip(project, id);
+ return clip ? [clip] : [];
+ });
+ if (clips.length < 2) return "Select two or more clips to merge.";
+
+ if (clips.some((clip) => clip.trackId !== clips[0].trackId)) {
+ return "Merged clips must be on the same track.";
+ }
+ if (clips.some((clip) => clip.mediaId !== clips[0].mediaId)) {
+ return "Merged clips must come from the same file.";
+ }
+
+ const ordered = [...clips].sort((left, right) => left.start - right.start);
+ for (let index = 1; index < ordered.length; index += 1) {
+ const previous = ordered[index - 1];
+ const current = ordered[index];
+
+ if (Math.abs(current.start - (previous.start + previous.duration)) > JOIN_EPSILON) {
+ return "Merged clips must touch, with no gap or overlap.";
+ }
+ // The source has to be continuous too. Two halves that were rearranged
+ // are adjacent on the timeline but no longer a single run of the file,
+ // and joining them would silently change what plays.
+ if (
+ Math.abs(current.sourceStart - (previous.sourceStart + previous.duration)) > JOIN_EPSILON
+ ) {
+ return "These pieces are no longer in their original order.";
+ }
+ }
+
+ return null;
+}
+
+/**
+ * Rejoins adjacent pieces of one source into a single clip.
+ *
+ * Does nothing unless [`whyNotMerge`] returns null. The surviving clip keeps
+ * the first piece's identity, so a selection referring to it stays valid.
+ */
+export function mergeClips(
+ project: Project,
+ clipIds: readonly string[],
+): { project: Project; clipId: string | null } {
+ if (whyNotMerge(project, clipIds) !== null) return { project, clipId: null };
+
+ const ordered = clipIds
+ .flatMap((id) => {
+ const clip = findClip(project, id);
+ return clip ? [clip] : [];
+ })
+ .sort((left, right) => left.start - right.start);
+
+ const first = ordered[0];
+ const last = ordered[ordered.length - 1];
+ const doomed = new Set(ordered.slice(1).map((clip) => clip.id));
+
+ const merged: Clip = {
+ ...first,
+ duration: last.start + last.duration - first.start,
+ // Gain and fades come from the first piece; the later ones are discarded
+ // along with their clips, which is what "these are one clip again" means.
+ };
+
+ return {
+ project: {
+ ...project,
+ clips: project.clips
+ .filter((clip) => !doomed.has(clip.id))
+ .map((clip) => (clip.id === first.id ? merged : clip)),
+ },
+ clipId: first.id,
+ };
+}
+
export function setTrackFlag(
project: Project,
trackId: string,
diff --git a/engine/crates/relay-media/src/decode.rs b/engine/crates/relay-media/src/decode.rs
index 6960cebf..3655d9e0 100644
--- a/engine/crates/relay-media/src/decode.rs
+++ b/engine/crates/relay-media/src/decode.rs
@@ -28,6 +28,30 @@ pub trait FrameSource {
/// The next frame, or `Ok(None)` at the end of the stream.
fn next_frame(&mut self) -> Result<Option<Frame>>;
+
+ /// When the frame just returned is meant to be shown.
+ ///
+ /// `None` means the source cannot say. The subprocess backend never can -
+ /// raw video carries no timestamps - which is exactly why variable frame
+ /// rate material desyncs through a pipe. A linked decoder reports the real
+ /// presentation timestamp.
+ fn position(&self) -> Option<Rational> {
+ None
+ }
+}
+
+/// A source that can jump to an arbitrary point.
+///
+/// Deliberately separate from [`FrameSource`]. Seeking over a pipe would mean
+/// respawning the process and decoding forward from a keyframe, which is not
+/// the same operation and should not be able to masquerade as one - so the
+/// subprocess decoder simply does not implement this.
+pub trait SeekableSource: FrameSource {
+ /// Moves to `to`, or the nearest decodable point before it.
+ ///
+ /// The next [`FrameSource::next_frame`] returns the first frame at or
+ /// after that point.
+ fn seek(&mut self, to: Rational) -> Result<()>;
}
/// How to open a decoder.
diff --git a/engine/crates/relay-media/src/error.rs b/engine/crates/relay-media/src/error.rs
index 1d149874..a63a371e 100644
--- a/engine/crates/relay-media/src/error.rs
+++ b/engine/crates/relay-media/src/error.rs
@@ -70,6 +70,17 @@ pub enum Error {
want: usize,
},
+ /// A libav* call failed. Only produced by the linked backend.
+ #[error("{operation} failed for {path}: {detail}")]
+ Ffi {
+ /// The libav function that failed.
+ operation: &'static str,
+ /// The file being worked on.
+ path: PathBuf,
+ /// FFmpeg's own description of the error code.
+ detail: String,
+ },
+
/// A frame handed to an encoder was not the size the encoder was opened for.
#[error("expected {want_width}x{want_height} frames, got {got_width}x{got_height}")]
FrameSizeMismatch {
diff --git a/engine/crates/relay-media/src/ffi.rs b/engine/crates/relay-media/src/ffi.rs
index 94f93802..773c4386 100644
--- a/engine/crates/relay-media/src/ffi.rs
+++ b/engine/crates/relay-media/src/ffi.rs
@@ -1,42 +1,446 @@
//! Linked FFmpeg, as opposed to spawned FFmpeg.
//!
//! This backend exists for the two things a pipe fundamentally cannot do:
-//! report a frame's real presentation timestamp, and seek to an exact frame.
+//! report a frame's real presentation timestamp, and seek to an exact point.
//! Both are needed for playback and scrubbing; neither matters for probing or
//! export, which is why the subprocess backend is not going anywhere.
//!
//! Only compiled with the `ffi` feature. See
-//! `docs/decisions/0002-ffmpeg-over-a-pipe.md`.
+//! `docs/decisions/0002-ffmpeg-over-a-pipe.md` and BUILD.md.
+//!
+//! ## On the unsafe in here
+//!
+//! Every raw pointer this module owns is allocated in [`FfiDecoder::open`] and
+//! freed in exactly one place, [`FfiDecoder::drop`]. Nothing escapes: the only
+//! thing callers ever receive is an owned [`Frame`] of copied pixels. That
+//! containment is what keeps the `unsafe` auditable - it is a handful of
+//! functions, not a lifetime that leaks across the crate.
+
+use std::ffi::{CStr, CString};
+use std::path::{Path, PathBuf};
+use std::ptr;
-use std::ffi::CStr;
+use relay_core::frame::Frame;
+use relay_core::time::Rational;
+use rusty_ffmpeg::ffi;
+
+use crate::decode::{FrameSource, SeekableSource};
+use crate::error::{Error, Result};
/// The version of FFmpeg this binary is linked against.
///
-/// Cheap, and the first thing worth checking when something behaves oddly: it
-/// proves which library actually loaded, which is not always the one you think
-/// on a machine with several FFmpeg installs.
+/// The first thing worth checking when something behaves oddly: it proves
+/// which library actually loaded, which is not always the one you expect on a
+/// machine with several FFmpeg installs.
pub fn linked_version() -> String {
- // SAFETY: av_version_info returns a pointer to a static, NUL-terminated
- // string compiled into the library. It is valid for the process lifetime
- // and never freed, so borrowing it here cannot dangle.
- let raw = unsafe { rusty_ffmpeg::ffi::av_version_info() };
+ // SAFETY: av_version_info returns a pointer to a static NUL-terminated
+ // string compiled into the library. Valid for the process lifetime.
+ let raw = unsafe { ffi::av_version_info() };
if raw.is_null() {
return "unknown".to_owned();
}
-
- // SAFETY: checked non-null above, and the contract of av_version_info is a
- // NUL-terminated C string.
+ // SAFETY: checked non-null; contract is a NUL-terminated C string.
unsafe { CStr::from_ptr(raw) }.to_string_lossy().into_owned()
}
/// Major version of the linked `libavcodec`.
-///
-/// Read from the library at runtime rather than from the headers, so a
-/// mismatch between what we compiled against and what actually loaded shows up
-/// as a number rather than as undefined behaviour.
pub fn avcodec_major() -> u32 {
- // SAFETY: a pure accessor returning a packed integer; no pointers involved.
- unsafe { rusty_ffmpeg::ffi::avcodec_version() >> 16 }
+ // SAFETY: pure accessor returning a packed integer, no pointers involved.
+ unsafe { ffi::avcodec_version() >> 16 }
+}
+
+/// Turns a libav error code into the message FFmpeg itself would print.
+fn describe(code: i32) -> String {
+ let mut buffer = [0i8; ffi::AV_ERROR_MAX_STRING_SIZE as usize];
+ // SAFETY: av_strerror writes at most `len` bytes into the buffer and
+ // NUL-terminates. The buffer outlives the call.
+ let written = unsafe { ffi::av_strerror(code, buffer.as_mut_ptr(), buffer.len()) };
+ if written < 0 {
+ return format!("error {code}");
+ }
+ // SAFETY: av_strerror NUL-terminates on success.
+ unsafe { CStr::from_ptr(buffer.as_ptr()) }.to_string_lossy().into_owned()
+}
+
+/// `AVERROR(EAGAIN)`: the decoder wants more input before it can produce output.
+///
+/// Spelled out rather than taken from the bindings because `EAGAIN` comes from
+/// the C library's `errno.h`, which is not part of the FFmpeg headers and is
+/// therefore not always present in generated bindings. 11 on every platform
+/// FFmpeg supports.
+const AVERROR_EAGAIN: i32 = -11;
+
+/// A decoder that links FFmpeg rather than spawning it.
+///
+/// Produces frames as RGBA8 scaled to a fixed size, alongside the real
+/// presentation timestamp of each.
+pub struct FfiDecoder {
+ path: PathBuf,
+ format: *mut ffi::AVFormatContext,
+ codec: *mut ffi::AVCodecContext,
+ scaler: *mut ffi::SwsContext,
+ decoded: *mut ffi::AVFrame,
+ packet: *mut ffi::AVPacket,
+ stream: i32,
+ /// Ticks per second for this stream's timestamps.
+ time_base: ffi::AVRational,
+ width: u32,
+ height: u32,
+ position: Option<Rational>,
+ finished: bool,
+}
+
+// SAFETY: every pointer here is exclusively owned by this struct - none are
+// shared, and `&mut self` is required to touch any of them. FFmpeg contexts
+// may be used from any single thread, just not concurrently, which is exactly
+// what Rust's borrow rules already enforce for the owner.
+unsafe impl Send for FfiDecoder {}
+
+impl FfiDecoder {
+ /// Opens `path`, producing frames scaled to `width` by `height`.
+ pub fn open(path: impl AsRef<Path>, width: u32, height: u32) -> Result<Self> {
+ let path = path.as_ref().to_path_buf();
+ let fail = |operation: &'static str, code: i32| Error::Ffi {
+ operation,
+ path: path.clone(),
+ detail: describe(code),
+ };
+
+ let c_path = CString::new(path.to_string_lossy().as_bytes()).map_err(|_| Error::Ffi {
+ operation: "path conversion",
+ path: path.clone(),
+ detail: "path contains an interior NUL byte".to_owned(),
+ })?;
+
+ // Built up in pieces, and every early return from here on has to undo
+ // what came before it - hence `Guard`, which owns the half-built state
+ // until construction succeeds.
+ let mut guard = Guard::default();
+
+ unsafe {
+ // SAFETY: `format` is null, so avformat_open_input allocates it and
+ // writes the pointer back. On failure it leaves it null.
+ let code = ffi::avformat_open_input(
+ &mut guard.format,
+ c_path.as_ptr(),
+ ptr::null(),
+ ptr::null_mut(),
+ );
+ if code < 0 {
+ return Err(fail("avformat_open_input", code));
+ }
+
+ // SAFETY: format is open; this reads packets to fill in stream info.
+ let code = ffi::avformat_find_stream_info(guard.format, ptr::null_mut());
+ if code < 0 {
+ return Err(fail("avformat_find_stream_info", code));
+ }
+
+ let mut decoder: *const ffi::AVCodec = ptr::null();
+ // SAFETY: format is open; decoder is written on success.
+ let stream = ffi::av_find_best_stream(
+ guard.format,
+ ffi::AVMEDIA_TYPE_VIDEO,
+ -1,
+ -1,
+ &mut decoder,
+ 0,
+ );
+ if stream < 0 {
+ return Err(Error::NoVideoStream { path: path.clone() });
+ }
+
+ // SAFETY: av_find_best_stream returned a valid index into streams.
+ let stream_ptr = *(*guard.format).streams.offset(stream as isize);
+ let time_base = (*stream_ptr).time_base;
+
+ // SAFETY: decoder was written by av_find_best_stream.
+ guard.codec = ffi::avcodec_alloc_context3(decoder);
+ if guard.codec.is_null() {
+ return Err(fail("avcodec_alloc_context3", 0));
+ }
+
+ // SAFETY: both pointers are valid and the codec context is fresh.
+ let code = ffi::avcodec_parameters_to_context(guard.codec, (*stream_ptr).codecpar);
+ if code < 0 {
+ return Err(fail("avcodec_parameters_to_context", code));
+ }
+
+ // SAFETY: context is populated from the stream's parameters.
+ let code = ffi::avcodec_open2(guard.codec, decoder, ptr::null_mut());
+ if code < 0 {
+ return Err(fail("avcodec_open2", code));
+ }
+
+ let source_width = (*guard.codec).width;
+ let source_height = (*guard.codec).height;
+ let source_format = (*guard.codec).pix_fmt;
+
+ // SAFETY: dimensions and format come from the opened codec.
+ guard.scaler = ffi::sws_getContext(
+ source_width,
+ source_height,
+ source_format,
+ width as i32,
+ height as i32,
+ ffi::AV_PIX_FMT_RGBA,
+ ffi::SWS_BILINEAR as i32,
+ ptr::null_mut(),
+ ptr::null_mut(),
+ ptr::null(),
+ );
+ if guard.scaler.is_null() {
+ return Err(fail("sws_getContext", 0));
+ }
+
+ // SAFETY: plain allocations, checked for null.
+ guard.decoded = ffi::av_frame_alloc();
+ guard.packet = ffi::av_packet_alloc();
+ if guard.decoded.is_null() || guard.packet.is_null() {
+ return Err(fail("av_frame_alloc", 0));
+ }
+
+ Ok(Self {
+ path,
+ format: guard.take_format(),
+ codec: guard.take_codec(),
+ scaler: guard.take_scaler(),
+ decoded: guard.take_decoded(),
+ packet: guard.take_packet(),
+ stream,
+ time_base,
+ width,
+ height,
+ position: None,
+ finished: false,
+ })
+ }
+ }
+
+ /// Converts a stream timestamp into seconds, exactly.
+ ///
+ /// This is the whole point of the linked backend: `time_base` is a
+ /// rational, the timestamp is an integer count of those ticks, and
+ /// `relay-core` speaks rationals - so the value survives with no rounding
+ /// anywhere along the way.
+ fn to_seconds(&self, ticks: i64) -> Rational {
+ Rational::new(ticks, 1) * Rational::new(self.time_base.num as i64, self.time_base.den as i64)
+ }
+
+ /// Scales the decoded frame into a fresh RGBA buffer.
+ fn convert(&mut self) -> Frame {
+ let stride = self.width as usize * 4;
+ let mut pixels = vec![0u8; stride * self.height as usize];
+
+ let destination: [*mut u8; 4] = [pixels.as_mut_ptr(), ptr::null_mut(), ptr::null_mut(), ptr::null_mut()];
+ let strides: [i32; 4] = [stride as i32, 0, 0, 0];
+
+ // SAFETY: `decoded` holds a complete frame from the codec, the scaler
+ // was built for exactly that source format and this destination size,
+ // and `pixels` is stride * height bytes - the size sws_scale writes.
+ unsafe {
+ ffi::sws_scale(
+ self.scaler,
+ (*self.decoded).data.as_ptr() as *const *const u8,
+ (*self.decoded).linesize.as_ptr(),
+ 0,
+ (*self.decoded).height,
+ destination.as_ptr(),
+ strides.as_ptr(),
+ );
+ }
+
+ Frame::from_rgba(self.width, self.height, pixels)
+ .expect("buffer is sized from the same width and height")
+ }
+}
+
+impl FrameSource for FfiDecoder {
+ fn width(&self) -> u32 {
+ self.width
+ }
+
+ fn height(&self) -> u32 {
+ self.height
+ }
+
+ fn position(&self) -> Option<Rational> {
+ self.position
+ }
+
+ fn next_frame(&mut self) -> Result<Option<Frame>> {
+ if self.finished {
+ return Ok(None);
+ }
+
+ loop {
+ // SAFETY: codec is open; `decoded` is a valid, reusable frame.
+ let code = unsafe { ffi::avcodec_receive_frame(self.codec, self.decoded) };
+
+ if code == 0 {
+ // SAFETY: receive_frame succeeded, so the frame is populated.
+ let ticks = unsafe { (*self.decoded).best_effort_timestamp };
+ self.position = (ticks != ffi::AV_NOPTS_VALUE).then(|| self.to_seconds(ticks));
+ return Ok(Some(self.convert()));
+ }
+
+ if code == ffi::AVERROR_EOF {
+ self.finished = true;
+ return Ok(None);
+ }
+
+ if code != AVERROR_EAGAIN {
+ return Err(Error::Ffi {
+ operation: "avcodec_receive_frame",
+ path: self.path.clone(),
+ detail: describe(code),
+ });
+ }
+
+ // EAGAIN: the decoder needs another packet before it can produce
+ // anything. Packets for other streams are skipped, not decoded.
+ loop {
+ // SAFETY: format is open; av_read_frame fills the packet, and
+ // we unref it below whatever happens.
+ let code = unsafe { ffi::av_read_frame(self.format, self.packet) };
+
+ if code == ffi::AVERROR_EOF {
+ // Flush: a null packet tells the decoder to emit whatever
+ // it is still holding, which is how the final few frames
+ // of a file come out at all.
+ // SAFETY: passing null is the documented flush signal.
+ unsafe { ffi::avcodec_send_packet(self.codec, ptr::null()) };
+ break;
+ }
+ if code < 0 {
+ return Err(Error::Ffi {
+ operation: "av_read_frame",
+ path: self.path.clone(),
+ detail: describe(code),
+ });
+ }
+
+ // SAFETY: packet is populated by av_read_frame.
+ let belongs = unsafe { (*self.packet).stream_index } == self.stream;
+ if belongs {
+ // SAFETY: sending a valid packet to an open decoder.
+ let sent = unsafe { ffi::avcodec_send_packet(self.codec, self.packet) };
+ // SAFETY: releases the packet's buffer for reuse.
+ unsafe { ffi::av_packet_unref(self.packet) };
+
+ if sent < 0 && sent != AVERROR_EAGAIN {
+ return Err(Error::Ffi {
+ operation: "avcodec_send_packet",
+ path: self.path.clone(),
+ detail: describe(sent),
+ });
+ }
+ break;
+ }
+
+ // SAFETY: not our stream, so drop it and read the next.
+ unsafe { ffi::av_packet_unref(self.packet) };
+ }
+ }
+ }
+}
+
+impl SeekableSource for FfiDecoder {
+ fn seek(&mut self, to: Rational) -> Result<()> {
+ // Convert seconds back into this stream's tick count.
+ let ticks_per_second = Rational::new(self.time_base.den as i64, self.time_base.num as i64);
+ let target = (to * ticks_per_second).floor();
+
+ // BACKWARD lands on the keyframe at or before the target. Decoding
+ // forward from there is what makes the seek frame-accurate rather
+ // than keyframe-accurate - which is precisely what the pipe cannot do.
+ // SAFETY: format is open and the stream index came from it.
+ let code = unsafe {
+ ffi::av_seek_frame(self.format, self.stream, target, ffi::AVSEEK_FLAG_BACKWARD as i32)
+ };
+ if code < 0 {
+ return Err(Error::Ffi {
+ operation: "av_seek_frame",
+ path: self.path.clone(),
+ detail: describe(code),
+ });
+ }
+
+ // The decoder is still holding frames from before the jump.
+ // SAFETY: codec is open.
+ unsafe { ffi::avcodec_flush_buffers(self.codec) };
+
+ self.finished = false;
+ self.position = None;
+ Ok(())
+ }
+}
+
+impl Drop for FfiDecoder {
+ fn drop(&mut self) {
+ // SAFETY: each pointer was allocated in `open` and is freed exactly
+ // once, here. The struct cannot be used after this.
+ unsafe {
+ ffi::av_packet_free(&mut self.packet);
+ ffi::av_frame_free(&mut self.decoded);
+ ffi::sws_freeContext(self.scaler);
+ ffi::avcodec_free_context(&mut self.codec);
+ ffi::avformat_close_input(&mut self.format);
+ }
+ }
+}
+
+/// Owns half-built state so that an early return from `open` still frees it.
+///
+/// Without this, every `?` between the first allocation and the last would
+/// leak whatever had been built up to that point - and there are six of them.
+#[derive(Default)]
+struct Guard {
+ format: *mut ffi::AVFormatContext,
+ codec: *mut ffi::AVCodecContext,
+ scaler: *mut ffi::SwsContext,
+ decoded: *mut ffi::AVFrame,
+ packet: *mut ffi::AVPacket,
+}
+
+macro_rules! take {
+ ($name:ident, $field:ident, $ty:ty) => {
+ fn $name(&mut self) -> $ty {
+ std::mem::replace(&mut self.$field, ptr::null_mut())
+ }
+ };
+}
+
+impl Guard {
+ take!(take_format, format, *mut ffi::AVFormatContext);
+ take!(take_codec, codec, *mut ffi::AVCodecContext);
+ take!(take_scaler, scaler, *mut ffi::SwsContext);
+ take!(take_decoded, decoded, *mut ffi::AVFrame);
+ take!(take_packet, packet, *mut ffi::AVPacket);
+}
+
+impl Drop for Guard {
+ fn drop(&mut self) {
+ // SAFETY: each free is a no-op on null, and a successful `open` has
+ // already taken every pointer out, leaving all fields null.
+ unsafe {
+ if !self.packet.is_null() {
+ ffi::av_packet_free(&mut self.packet);
+ }
+ if !self.decoded.is_null() {
+ ffi::av_frame_free(&mut self.decoded);
+ }
+ if !self.scaler.is_null() {
+ ffi::sws_freeContext(self.scaler);
+ }
+ if !self.codec.is_null() {
+ ffi::avcodec_free_context(&mut self.codec);
+ }
+ if !self.format.is_null() {
+ ffi::avformat_close_input(&mut self.format);
+ }
+ }
+ }
}
#[cfg(test)]
@@ -45,19 +449,23 @@ mod tests {
#[test]
fn links_against_the_expected_ffmpeg() {
- let version = linked_version();
- assert!(!version.is_empty(), "av_version_info returned nothing");
-
- // The bindings rusty_ffmpeg ships are generated for avcodec 62
- // (FFmpeg 8.x). Loading a different major version would mean the
- // struct layouts we compiled against do not match the library that
- // actually loaded - which corrupts silently rather than failing, so
- // it is worth an explicit check.
- assert_eq!(
- avcodec_major(),
- 62,
- "linked libavcodec is {} but the bindings expect 62 - see BUILD.md",
- avcodec_major()
- );
+ assert!(!linked_version().is_empty(), "av_version_info returned nothing");
+
+ // The bindings are generated for avcodec 62 (FFmpeg 8.x). A different
+ // major version means the struct layouts we compiled against do not
+ // match the library that loaded - which corrupts silently rather than
+ // failing, so it is worth an explicit check.
+ assert_eq!(avcodec_major(), 62, "linked libavcodec mismatch - see BUILD.md");
+ }
+
+ #[test]
+ fn describes_error_codes() {
+ // Whatever the wording, it should not be the numeric fallback.
+ assert!(!describe(ffi::AVERROR_EOF).starts_with("error "));
+ }
+
+ #[test]
+ fn a_missing_file_is_an_error_not_a_panic() {
+ assert!(FfiDecoder::open("does-not-exist.mp4", 64, 64).is_err());
}
}
diff --git a/engine/crates/relay-media/src/lib.rs b/engine/crates/relay-media/src/lib.rs
index 49807e3b..93377dee 100644
--- a/engine/crates/relay-media/src/lib.rs
+++ b/engine/crates/relay-media/src/lib.rs
@@ -18,7 +18,7 @@ pub mod error;
pub mod probe;
pub use binaries::{ffmpeg, ffprobe, set_binaries};
-pub use decode::{DecodeOptions, FfmpegDecoder, FrameSource};
+pub use decode::{DecodeOptions, FfmpegDecoder, FrameSource, SeekableSource};
pub use encode::{EncodeOptions, FfmpegEncoder, FrameSink};
pub use error::{Error, Result};
pub use probe::{AudioStream, MediaInfo, VideoStream, probe};
diff --git a/engine/crates/relay-media/tests/ffi_decode.rs b/engine/crates/relay-media/tests/ffi_decode.rs
new file mode 100644
index 00000000..3a6f733f
--- /dev/null
+++ b/engine/crates/relay-media/tests/ffi_decode.rs
@@ -0,0 +1,148 @@
+//! End-to-end checks for the linked decoder.
+//!
+//! These are the claims that justify the FFI backend existing at all - real
+//! presentation timestamps and frame-accurate seeking - so they are tested
+//! against an actual file rather than asserted in a doc comment.
+//!
+//! The fixture is generated with the `ffmpeg` binary at test time. If FFmpeg
+//! is not installed the tests skip rather than fail: a missing tool is a
+//! missing tool, not a broken decoder.
+
+#![cfg(feature = "ffi")]
+
+use std::path::PathBuf;
+use std::process::Command;
+
+use relay_core::time::Rational;
+use relay_media::decode::{FrameSource, SeekableSource};
+use relay_media::ffi::FfiDecoder;
+
+const RATE: i64 = 30;
+const SECONDS: i64 = 4;
+
+/// Renders a deterministic clip and returns its path, or `None` without FFmpeg.
+fn fixture() -> Option<PathBuf> {
+ let path = std::env::temp_dir().join("relay-ffi-fixture.mp4");
+ if path.exists() {
+ return Some(path);
+ }
+
+ let status = Command::new("ffmpeg")
+ .args(["-hide_banner", "-loglevel", "error", "-y"])
+ .args(["-f", "lavfi", "-i", &format!("testsrc2=size=320x240:rate={RATE}")])
+ .args(["-t", &SECONDS.to_string()])
+ // A short keyframe interval, so a seek has something nearby to land on
+ // and the test measures our decode-forward rather than FFmpeg's luck.
+ .args(["-g", "15", "-c:v", "libx264", "-pix_fmt", "yuv420p"])
+ .arg(&path)
+ .status()
+ .ok()?;
+
+ status.success().then_some(path)
+}
+
+/// One frame's duration, as the tolerance for "landed on the right frame".
+fn frame_duration() -> Rational {
+ Rational::new(1, RATE)
+}
+
+#[test]
+fn reports_real_presentation_timestamps() {
+ let Some(path) = fixture() else { return };
+
+ let mut decoder = FfiDecoder::open(&path, 160, 120).expect("open");
+ let mut times = Vec::new();
+
+ for _ in 0..10 {
+ let frame = decoder.next_frame().expect("decode").expect("a frame");
+ assert_eq!((frame.width(), frame.height()), (160, 120), "scaled to the requested size");
+
+ let position = decoder.position().expect("a linked decoder always knows the time");
+ times.push(position);
+ }
+
+ // The whole point: these come from the container, not from counting.
+ assert_eq!(times[0], Rational::ZERO, "first frame starts at zero");
+
+ for (index, time) in times.iter().enumerate() {
+ let expected = frame_duration() * Rational::from_int(index as i64);
+ assert_eq!(*time, expected, "frame {index} should sit exactly on its boundary");
+ }
+
+ assert!(times.windows(2).all(|pair| pair[0] < pair[1]), "timestamps must increase");
+}
+
+#[test]
+fn seeks_to_an_exact_frame() {
+ let Some(path) = fixture() else { return };
+
+ let mut decoder = FfiDecoder::open(&path, 160, 120).expect("open");
+
+ // Deliberately not a keyframe: at -g 15 the keyframes are every half
+ // second, so 2.4s sits between two of them. Landing there is only possible
+ // by decoding forward from the previous one, which is the thing a pipe
+ // cannot do.
+ let target = Rational::new(24, 10);
+ decoder.seek(target).expect("seek");
+
+ let mut landed = None;
+ for _ in 0..64 {
+ decoder.next_frame().expect("decode").expect("a frame");
+ let position = decoder.position().expect("position");
+ if position >= target {
+ landed = Some(position);
+ break;
+ }
+ }
+
+ let landed = landed.expect("never reached the target after a seek");
+ let drift = landed - target;
+ assert!(
+ drift >= Rational::ZERO && drift < frame_duration(),
+ "landed at {landed} for a target of {target}, which is more than one frame out",
+ );
+}
+
+#[test]
+fn seeking_backwards_works_too() {
+ let Some(path) = fixture() else { return };
+
+ let mut decoder = FfiDecoder::open(&path, 64, 64).expect("open");
+
+ // Run well into the file first, so the seek genuinely goes backwards and
+ // has stale decoder state to flush.
+ for _ in 0..60 {
+ decoder.next_frame().expect("decode");
+ }
+ let before = decoder.position().expect("position");
+ assert!(before > Rational::ONE, "should be past a second by now");
+
+ decoder.seek(Rational::new(1, 2)).expect("seek back");
+ decoder.next_frame().expect("decode").expect("a frame");
+
+ let after = decoder.position().expect("position");
+ assert!(after < before, "seeking back should move the position back");
+ assert!(after <= Rational::new(6, 10), "should land near the half-second mark, got {after}");
+}
+
+#[test]
+fn runs_to_the_end_and_stops() {
+ let Some(path) = fixture() else { return };
+
+ let mut decoder = FfiDecoder::open(&path, 64, 64).expect("open");
+ let mut count = 0u32;
+ while decoder.next_frame().expect("decode").is_some() {
+ count += 1;
+ assert!(count < 1000, "decoder never reported end of stream");
+ }
+
+ // Flushing at EOF is what makes the last few frames come out at all; a
+ // decoder that forgets it silently loses them.
+ let expected = (RATE * SECONDS) as u32;
+ assert!(
+ count.abs_diff(expected) <= 1,
+ "decoded {count} frames, expected about {expected}",
+ );
+
+ assert!(decoder.next_frame().expect("decode").is_none(), "end of stream is sticky");
+}
diff --git a/engine/docs/decisions/0002-ffmpeg-over-a-pipe.md b/engine/docs/decisions/0002-ffmpeg-over-a-pipe.md
index ef2033d1..2e55cb77 100644
--- a/engine/docs/decisions/0002-ffmpeg-over-a-pipe.md
+++ b/engine/docs/decisions/0002-ffmpeg-over-a-pipe.md
@@ -66,9 +66,25 @@ Chocolatey - binaries only, no `libavcodec/avcodec.h` and no import libs, and
Budget for the build plumbing, not the Rust. The Rust is the easy half.
+## Status: the FFI half now exists
+
+`relay-media`'s `ffi` feature builds a linked decoder, `FfiDecoder`, which
+implements `FrameSource` and `SeekableSource`. Both correctness gaps above are
+closed in it and covered by `tests/ffi_decode.rs`:
+
+- real presentation timestamps, exact rationals straight from the container
+- frame-accurate seeking, verified by landing between keyframes
+
+It is **off by default and unused by the app.** The subprocess backend is still
+what runs, and probe and export are staying on it permanently.
+
+The seam held: adding it required one new type implementing an existing trait,
+one new trait for the capability the pipe genuinely lacks, and no change to
+`relay-core`, `relay-render`, or any caller.
+
## What would change our mind
-Building playback. Specifically, the first time we need either a real PTS or a
-frame-accurate seek - whichever comes first. Both arrive with the transport
-controls, so that is the deadline.
+Nothing left to decide about *whether*. What remains is wiring playback to the
+linked decoder, at which point scrubbing stops respawning a process per seek.
-Do **not** wait for 4K, and do not migrate probe or export along with it.
+Do **not** migrate probe or export along with it - for those, the tradeoff
+never flips.