The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.
Rust ffmpegfree-video-editormodel-context-protocolnon-linear-editoroffline-firstopen-source-video-editorself-hostedvideo-editingvideo-editing-automationvideo-editing-softwarevideo-editing-toolsvideo-editorvoice-cloningyoutube-automationyoutube-shorts
updates
Jareerb9f779f96f9c
diff --git a/BUILD.md b/BUILD.mdindex 1e2ef08e..75afcc75 100644--- a/BUILD.md+++ b/BUILD.md@@ -74,27 +74,68 @@ Holds an FFmpeg **development** build (headers and import libraries), used onlyby the in-progress FFI decoder described below. It is gitignored, roughly150 MB, and nothing in the normal build path touches it. Delete it freely.-## FFI decoder — in progress, not yet wired up+## FFI decoder — optional, off by default-The engine currently runs FFmpeg as a subprocess+The engine talks to FFmpeg as a subprocess(`engine/docs/decisions/0002-ffmpeg-over-a-pipe.md`). That is permanent for-probing and export, but playback needs real timestamps and frame-accurate-seeking, which a pipe cannot provide — so a linked decoder is being added-behind the existing `FrameSource` trait.--Extra prerequisites for that path only:--- **LLVM / libclang** — `bindgen` generates the FFmpeg bindings at build time.- `winget install LLVM.LLVM`-- **An FFmpeg dev build** — headers plus `.lib` import libraries, which the- ordinary runtime download does not include. Get a `*-gpl-shared-*` archive- from [BtbN/FFmpeg-Builds](https://github.com/BtbN/FFmpeg-Builds/releases) and- unpack it into `vendor/`.-- Environment variables pointing at it (see `engine/crates/relay-media/README`- once this lands).--This section will be replaced with exact, verified steps when the decoder-actually builds. Until then, ignore it — the app does not need any of it.+probing and export. Playback is different: a pipe carries no presentation+timestamps and cannot seek to an exact frame, so there is a second decoder that+links FFmpeg directly, behind the `ffi` feature.++Nothing in the app enables it yet. **You do not need any of this to build or+run Relay.**++### Setup, exactly as verified++1. **Get a development build** — headers plus `.lib` import libraries, which+ the ordinary runtime download does not include. From+ [BtbN/FFmpeg-Builds](https://github.com/BtbN/FFmpeg-Builds/releases), take a+ `win64-gpl-**shared**` archive and unpack it into `vendor/`:++ ```sh+ curl -L -o ff.zip https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n8.1-latest-win64-gpl-shared-8.1.zip+ unzip ff.zip -d vendor/+ ```++ **The version has to match the bindings.** `rusty_ffmpeg` ships+ pre-generated bindings for a specific FFmpeg, currently 8.1 — avcodec 62,+ avformat 62, avutil 60. A different major version compiles fine and then+ misreads struct layouts at runtime, which corrupts silently rather than+ failing. `ffi::tests::links_against_the_expected_ffmpeg` checks this and is+ the first thing to look at if the decoder behaves strangely.++2. **Nothing else.** No LLVM, no libclang, no bindgen run — the pre-generated+ bindings are used as-is. `engine/.cargo/config.toml` already points at+ `vendor/`, so no environment variables to set either.++ If you unpack a differently-named archive, update the paths in that file.++3. **Build and test:**++ ```sh+ cd engine+ PATH="../vendor/ffmpeg-n8.1-latest-win64-gpl-shared-8.1/bin:$PATH" \+ cargo test -p relay-media --features ffi+ ```++ The DLLs must be on `PATH` at *run* time. Linking uses `lib/` (import+ libraries); running needs `bin/` (the DLLs themselves). Pointing either at+ the other is the most likely thing to go wrong — `LNK1181: cannot open input+ file 'avcodec.lib'` means the linker was aimed at `bin/`.++### What it provides++`FfiDecoder` implements `FrameSource` plus `SeekableSource`:++- **`position()`** — the real presentation timestamp, as an exact rational. The+ subprocess decoder returns `None` here, honestly: raw video carries no+ timestamps, which is why variable-frame-rate material desyncs through a pipe.+- **`seek()`** — frame-accurate, by seeking to the preceding keyframe and+ decoding forward. Deliberately a separate trait, so the subprocess decoder+ cannot pretend to offer it.++`tests/ffi_decode.rs` proves both against a generated fixture, and skips+entirely if FFmpeg is not installed.## Troubleshootingdiff --git a/desktop/src/App.tsx b/desktop/src/App.tsxindex 749d8cec..54ce1786 100644--- a/desktop/src/App.tsx+++ b/desktop/src/App.tsx@@ -28,6 +28,7 @@ import {firstFreeTrack,findMedia,findTrack,+ mergeClips,moveClip,moveClips,projectDuration,@@ -40,6 +41,7 @@ import {toMediaItem,updateClip,trimClip,+ whyNotMerge,type MediaItem,type Project,} from "./lib/project";@@ -338,6 +340,14 @@ function Editor({});}, [playhead, selectedClipIds]);+ const mergeSelected = useCallback(() => {+ setProject((current) => {+ const { project: next, clipId } = mergeClips(current, selectedClipIds);+ if (clipId) setSelectedClipIds([clipId]);+ return next;+ });+ }, [selectedClipIds]);+const deleteSelected = useCallback(() => {if (selectedClipIds.length === 0) return;// Free the audio element first: once the clip is gone the sync pass has@@ -413,6 +423,9 @@ function Editor({case "KeyS":splitAtPlayhead();break;+ case "KeyM":+ mergeSelected();+ break;case "Delete":case "Backspace":deleteSelected();@@ -424,7 +437,7 @@ function Editor({window.addEventListener("keydown", onKey);return () => window.removeEventListener("keydown", onKey);- }, [deleteSelected, duration, frameRate, splitAtPlayhead, transport]);+ }, [deleteSelected, duration, frameRate, mergeSelected, splitAtPlayhead, transport]);// The exporter works from a flat list: the engine rebuilds a real timeline// from it, so track *order* has to survive the trip even though track@@ -671,6 +684,8 @@ function Editor({setProject((current) => trimClip(current, clipId, edge, delta))}onSplitAtPlayhead={splitAtPlayhead}+ onMergeSelected={mergeSelected}+ mergeBlockedBecause={whyNotMerge(project, selectedClipIds)}onDeleteSelected={deleteSelected}mediaDrag={mediaDrag ? { x: mediaDrag.x, y: mediaDrag.y } : null}onZoom={zoom}@@ -717,6 +732,16 @@ function Editor({hint: "S",onSelect: splitAtPlayhead,},+ ...(whyNotMerge(project, target) === null+ ? [+ {+ label: `Merge ${target.length} clips`,+ icon: "merge" as const,+ hint: "M",+ onSelect: mergeSelected,+ },+ ]+ : []),{label: "Move to playhead",icon: "select",diff --git a/desktop/src/components/Icon.tsx b/desktop/src/components/Icon.tsxindex 481ad4ad..04a738f3 100644--- a/desktop/src/components/Icon.tsx+++ b/desktop/src/components/Icon.tsx@@ -70,6 +70,22 @@ const GLYPHS = {<path d="M15 11V6.5a1.5 1.5 0 0 1 3 0V15a6 6 0 0 1-6 6h-1a7 7 0 0 1-7-7v-2a1.5 1.5 0 0 1 3 0v1" /></>),+ // two blocks parted by a cut line+ split: (+ <>+ <path d="M12 3v18" strokeDasharray="3 3" />+ <path d="M8 7H5a2 2 0 0 0-2 2v6a2 2 0 0 0 2 2h3" />+ <path d="M16 7h3a2 2 0 0 1 2 2v6a2 2 0 0 1-2 2h-3" />+ </>+ ),+ // two blocks closing on a seam+ merge: (+ <>+ <path d="M12 3v18" />+ <path d="M3 7h5a2 2 0 0 1 2 2v6a2 2 0 0 1-2 2H3" />+ <path d="M21 7h-5a2 2 0 0 0-2 2v6a2 2 0 0 0 2 2h5" />+ </>+ ),// horseshoe magnet, for snappingmagnet: (<>diff --git a/desktop/src/components/TimelinePanel.tsx b/desktop/src/components/TimelinePanel.tsxindex 37bc3d12..c8945363 100644--- a/desktop/src/components/TimelinePanel.tsx+++ b/desktop/src/components/TimelinePanel.tsx@@ -130,6 +130,8 @@ export function TimelinePanel({onMoveClips,onTrimClip,onSplitAtPlayhead,+ onMergeSelected,+ mergeBlockedBecause,onDeleteSelected,mediaDrag,onZoom,@@ -165,6 +167,9 @@ export function TimelinePanel({onMoveClips: (moves: ClipMove[]) => void;onTrimClip: (clipId: string, edge: "start" | "end", delta: number) => void;onSplitAtPlayhead: () => void;+ onMergeSelected: () => void;+ /** Null when the selection can be merged; otherwise why it cannot. */+ mergeBlockedBecause: string | null;onDeleteSelected: () => void;/** A bin item currently being dragged, in client coordinates. */mediaDrag: { x: number; y: number } | null;@@ -214,8 +219,8 @@ export function TimelinePanel({// The draw loop reads everything through this ref, so a prop change never// tears down and rebuilds the loop.- const view = useRef({ project, playhead, playing, secondsPerPixel, scrollLeft, trackScroll, frameRate, selected, rows, dropTrack, assets });- view.current = { project, playhead, playing, secondsPerPixel, scrollLeft, trackScroll, frameRate, selected, rows, dropTrack, assets };+ const view = useRef({ project, playhead, playing, secondsPerPixel, scrollLeft, trackScroll, frameRate, selected, rows, dropTrack, assets, tool });+ view.current = { project, playhead, playing, secondsPerPixel, scrollLeft, trackScroll, frameRate, selected, rows, dropTrack, assets, tool };const timeAt = useCallback((clientX: number) => {@@ -364,6 +369,41 @@ export function TimelinePanel({);}+ // Razor guide: a dashed line at the pointer, but only while it is over a+ // clip. Showing it in empty space would promise a cut that does nothing.+ if (state.tool === "razor" && pointer.current && !drag.current) {+ const canvas = canvasRef.current;+ const bounds = canvas?.getBoundingClientRect();+ if (bounds) {+ const localX = pointer.current.x - bounds.left;+ const localY = pointer.current.y - bounds.top;+ const time = localX * state.secondsPerPixel + state.scrollLeft;+ const rowIndex = Math.floor((localY - RULER_HEIGHT + state.trackScroll) / TRACK_HEIGHT);+ const track = state.rows[rowIndex];++ const overClip =+ localY > RULER_HEIGHT &&+ track !== undefined &&+ state.project.clips.some(+ (clip) =>+ clip.trackId === track.id && time > clip.start && time < clip.start + clip.duration,+ );++ if (overClip) {+ const guide = Math.round(localX) + 0.5;+ context.save();+ context.strokeStyle = COLORS.playhead;+ context.lineWidth = 1;+ context.setLineDash([4, 3]);+ context.beginPath();+ context.moveTo(guide, RULER_HEIGHT);+ context.lineTo(guide, height);+ context.stroke();+ context.restore();+ }+ }+ }+// The selection band, drawn over the clips it is catching.if (drag.current?.kind === "marquee") {const band = normalise(drag.current);@@ -667,7 +707,15 @@ export function TimelinePanel({onClick={() => onToolChange("razor")}/><Divider />- <IconButton icon="minus" label="Split at playhead (S)" onClick={onSplitAtPlayhead} />+ <IconButton icon="split" label="Split at playhead (S)" onClick={onSplitAtPlayhead} />+ <IconButton+ icon="merge"+ // The reason lives in the tooltip: a button that greys out without+ // saying why leaves you guessing at the rule.+ label={mergeBlockedBecause ?? "Merge selected clips (M)"}+ disabled={mergeBlockedBecause !== null}+ onClick={onMergeSelected}+ /><IconButtonicon="trash"label={diff --git a/desktop/src/lib/project.ts b/desktop/src/lib/project.tsindex a3bae5ce..76efeffa 100644--- a/desktop/src/lib/project.ts+++ b/desktop/src/lib/project.ts@@ -411,6 +411,105 @@ export function clipsOnTrack(project: Project, trackId: string): Clip[] {return project.clips.filter((clip) => clip.trackId === trackId);}+/**+ * How far apart two clips may sit and still count as touching, in seconds.+ *+ * Splitting produces edges that meet exactly, but the numbers have been+ * through float arithmetic on the way, so an exact comparison would+ * occasionally refuse to rejoin a clip it had just cut.+ */+const JOIN_EPSILON = 1e-6;++/**+ * Why these clips cannot be merged, or `null` if they can.+ *+ * Merge is the inverse of split: it rejoins pieces of *one* source that are+ * still lying end to end. It is not a way to weld two different files into a+ * single clip - that is a compound clip, which needs nesting and a render of+ * its own, and pretending otherwise here would produce a clip whose source+ * range means nothing.+ *+ * Returns a sentence, because a disabled button that will not say why is worse+ * than no button.+ */+export function whyNotMerge(project: Project, clipIds: readonly string[]): string | null {+ if (clipIds.length < 2) return "Select two or more clips to merge.";++ const clips = clipIds.flatMap((id) => {+ const clip = findClip(project, id);+ return clip ? [clip] : [];+ });+ if (clips.length < 2) return "Select two or more clips to merge.";++ if (clips.some((clip) => clip.trackId !== clips[0].trackId)) {+ return "Merged clips must be on the same track.";+ }+ if (clips.some((clip) => clip.mediaId !== clips[0].mediaId)) {+ return "Merged clips must come from the same file.";+ }++ const ordered = [...clips].sort((left, right) => left.start - right.start);+ for (let index = 1; index < ordered.length; index += 1) {+ const previous = ordered[index - 1];+ const current = ordered[index];++ if (Math.abs(current.start - (previous.start + previous.duration)) > JOIN_EPSILON) {+ return "Merged clips must touch, with no gap or overlap.";+ }+ // The source has to be continuous too. Two halves that were rearranged+ // are adjacent on the timeline but no longer a single run of the file,+ // and joining them would silently change what plays.+ if (+ Math.abs(current.sourceStart - (previous.sourceStart + previous.duration)) > JOIN_EPSILON+ ) {+ return "These pieces are no longer in their original order.";+ }+ }++ return null;+}++/**+ * Rejoins adjacent pieces of one source into a single clip.+ *+ * Does nothing unless [`whyNotMerge`] returns null. The surviving clip keeps+ * the first piece's identity, so a selection referring to it stays valid.+ */+export function mergeClips(+ project: Project,+ clipIds: readonly string[],+): { project: Project; clipId: string | null } {+ if (whyNotMerge(project, clipIds) !== null) return { project, clipId: null };++ const ordered = clipIds+ .flatMap((id) => {+ const clip = findClip(project, id);+ return clip ? [clip] : [];+ })+ .sort((left, right) => left.start - right.start);++ const first = ordered[0];+ const last = ordered[ordered.length - 1];+ const doomed = new Set(ordered.slice(1).map((clip) => clip.id));++ const merged: Clip = {+ ...first,+ duration: last.start + last.duration - first.start,+ // Gain and fades come from the first piece; the later ones are discarded+ // along with their clips, which is what "these are one clip again" means.+ };++ return {+ project: {+ ...project,+ clips: project.clips+ .filter((clip) => !doomed.has(clip.id))+ .map((clip) => (clip.id === first.id ? merged : clip)),+ },+ clipId: first.id,+ };+}+export function setTrackFlag(project: Project,trackId: string,diff --git a/engine/crates/relay-media/src/decode.rs b/engine/crates/relay-media/src/decode.rsindex 6960cebf..3655d9e0 100644--- a/engine/crates/relay-media/src/decode.rs+++ b/engine/crates/relay-media/src/decode.rs@@ -28,6 +28,30 @@ pub trait FrameSource {/// The next frame, or `Ok(None)` at the end of the stream.fn next_frame(&mut self) -> Result<Option<Frame>>;++ /// When the frame just returned is meant to be shown.+ ///+ /// `None` means the source cannot say. The subprocess backend never can -+ /// raw video carries no timestamps - which is exactly why variable frame+ /// rate material desyncs through a pipe. A linked decoder reports the real+ /// presentation timestamp.+ fn position(&self) -> Option<Rational> {+ None+ }+}++/// A source that can jump to an arbitrary point.+///+/// Deliberately separate from [`FrameSource`]. Seeking over a pipe would mean+/// respawning the process and decoding forward from a keyframe, which is not+/// the same operation and should not be able to masquerade as one - so the+/// subprocess decoder simply does not implement this.+pub trait SeekableSource: FrameSource {+ /// Moves to `to`, or the nearest decodable point before it.+ ///+ /// The next [`FrameSource::next_frame`] returns the first frame at or+ /// after that point.+ fn seek(&mut self, to: Rational) -> Result<()>;}/// How to open a decoder.diff --git a/engine/crates/relay-media/src/error.rs b/engine/crates/relay-media/src/error.rsindex 1d149874..a63a371e 100644--- a/engine/crates/relay-media/src/error.rs+++ b/engine/crates/relay-media/src/error.rs@@ -70,6 +70,17 @@ pub enum Error {want: usize,},+ /// A libav* call failed. Only produced by the linked backend.+ #[error("{operation} failed for {path}: {detail}")]+ Ffi {+ /// The libav function that failed.+ operation: &'static str,+ /// The file being worked on.+ path: PathBuf,+ /// FFmpeg's own description of the error code.+ detail: String,+ },+/// A frame handed to an encoder was not the size the encoder was opened for.#[error("expected {want_width}x{want_height} frames, got {got_width}x{got_height}")]FrameSizeMismatch {diff --git a/engine/crates/relay-media/src/ffi.rs b/engine/crates/relay-media/src/ffi.rsindex 94f93802..773c4386 100644--- a/engine/crates/relay-media/src/ffi.rs+++ b/engine/crates/relay-media/src/ffi.rs@@ -1,42 +1,446 @@//! Linked FFmpeg, as opposed to spawned FFmpeg.//!//! This backend exists for the two things a pipe fundamentally cannot do:-//! report a frame's real presentation timestamp, and seek to an exact frame.+//! report a frame's real presentation timestamp, and seek to an exact point.//! Both are needed for playback and scrubbing; neither matters for probing or//! export, which is why the subprocess backend is not going anywhere.//!//! Only compiled with the `ffi` feature. See-//! `docs/decisions/0002-ffmpeg-over-a-pipe.md`.+//! `docs/decisions/0002-ffmpeg-over-a-pipe.md` and BUILD.md.+//!+//! ## On the unsafe in here+//!+//! Every raw pointer this module owns is allocated in [`FfiDecoder::open`] and+//! freed in exactly one place, [`FfiDecoder::drop`]. Nothing escapes: the only+//! thing callers ever receive is an owned [`Frame`] of copied pixels. That+//! containment is what keeps the `unsafe` auditable - it is a handful of+//! functions, not a lifetime that leaks across the crate.++use std::ffi::{CStr, CString};+use std::path::{Path, PathBuf};+use std::ptr;-use std::ffi::CStr;+use relay_core::frame::Frame;+use relay_core::time::Rational;+use rusty_ffmpeg::ffi;++use crate::decode::{FrameSource, SeekableSource};+use crate::error::{Error, Result};/// The version of FFmpeg this binary is linked against.///-/// Cheap, and the first thing worth checking when something behaves oddly: it-/// proves which library actually loaded, which is not always the one you think-/// on a machine with several FFmpeg installs.+/// The first thing worth checking when something behaves oddly: it proves+/// which library actually loaded, which is not always the one you expect on a+/// machine with several FFmpeg installs.pub fn linked_version() -> String {- // SAFETY: av_version_info returns a pointer to a static, NUL-terminated- // string compiled into the library. It is valid for the process lifetime- // and never freed, so borrowing it here cannot dangle.- let raw = unsafe { rusty_ffmpeg::ffi::av_version_info() };+ // SAFETY: av_version_info returns a pointer to a static NUL-terminated+ // string compiled into the library. Valid for the process lifetime.+ let raw = unsafe { ffi::av_version_info() };if raw.is_null() {return "unknown".to_owned();}-- // SAFETY: checked non-null above, and the contract of av_version_info is a- // NUL-terminated C string.+ // SAFETY: checked non-null; contract is a NUL-terminated C string.unsafe { CStr::from_ptr(raw) }.to_string_lossy().into_owned()}/// Major version of the linked `libavcodec`.-///-/// Read from the library at runtime rather than from the headers, so a-/// mismatch between what we compiled against and what actually loaded shows up-/// as a number rather than as undefined behaviour.pub fn avcodec_major() -> u32 {- // SAFETY: a pure accessor returning a packed integer; no pointers involved.- unsafe { rusty_ffmpeg::ffi::avcodec_version() >> 16 }+ // SAFETY: pure accessor returning a packed integer, no pointers involved.+ unsafe { ffi::avcodec_version() >> 16 }+}++/// Turns a libav error code into the message FFmpeg itself would print.+fn describe(code: i32) -> String {+ let mut buffer = [0i8; ffi::AV_ERROR_MAX_STRING_SIZE as usize];+ // SAFETY: av_strerror writes at most `len` bytes into the buffer and+ // NUL-terminates. The buffer outlives the call.+ let written = unsafe { ffi::av_strerror(code, buffer.as_mut_ptr(), buffer.len()) };+ if written < 0 {+ return format!("error {code}");+ }+ // SAFETY: av_strerror NUL-terminates on success.+ unsafe { CStr::from_ptr(buffer.as_ptr()) }.to_string_lossy().into_owned()+}++/// `AVERROR(EAGAIN)`: the decoder wants more input before it can produce output.+///+/// Spelled out rather than taken from the bindings because `EAGAIN` comes from+/// the C library's `errno.h`, which is not part of the FFmpeg headers and is+/// therefore not always present in generated bindings. 11 on every platform+/// FFmpeg supports.+const AVERROR_EAGAIN: i32 = -11;++/// A decoder that links FFmpeg rather than spawning it.+///+/// Produces frames as RGBA8 scaled to a fixed size, alongside the real+/// presentation timestamp of each.+pub struct FfiDecoder {+ path: PathBuf,+ format: *mut ffi::AVFormatContext,+ codec: *mut ffi::AVCodecContext,+ scaler: *mut ffi::SwsContext,+ decoded: *mut ffi::AVFrame,+ packet: *mut ffi::AVPacket,+ stream: i32,+ /// Ticks per second for this stream's timestamps.+ time_base: ffi::AVRational,+ width: u32,+ height: u32,+ position: Option<Rational>,+ finished: bool,+}++// SAFETY: every pointer here is exclusively owned by this struct - none are+// shared, and `&mut self` is required to touch any of them. FFmpeg contexts+// may be used from any single thread, just not concurrently, which is exactly+// what Rust's borrow rules already enforce for the owner.+unsafe impl Send for FfiDecoder {}++impl FfiDecoder {+ /// Opens `path`, producing frames scaled to `width` by `height`.+ pub fn open(path: impl AsRef<Path>, width: u32, height: u32) -> Result<Self> {+ let path = path.as_ref().to_path_buf();+ let fail = |operation: &'static str, code: i32| Error::Ffi {+ operation,+ path: path.clone(),+ detail: describe(code),+ };++ let c_path = CString::new(path.to_string_lossy().as_bytes()).map_err(|_| Error::Ffi {+ operation: "path conversion",+ path: path.clone(),+ detail: "path contains an interior NUL byte".to_owned(),+ })?;++ // Built up in pieces, and every early return from here on has to undo+ // what came before it - hence `Guard`, which owns the half-built state+ // until construction succeeds.+ let mut guard = Guard::default();++ unsafe {+ // SAFETY: `format` is null, so avformat_open_input allocates it and+ // writes the pointer back. On failure it leaves it null.+ let code = ffi::avformat_open_input(+ &mut guard.format,+ c_path.as_ptr(),+ ptr::null(),+ ptr::null_mut(),+ );+ if code < 0 {+ return Err(fail("avformat_open_input", code));+ }++ // SAFETY: format is open; this reads packets to fill in stream info.+ let code = ffi::avformat_find_stream_info(guard.format, ptr::null_mut());+ if code < 0 {+ return Err(fail("avformat_find_stream_info", code));+ }++ let mut decoder: *const ffi::AVCodec = ptr::null();+ // SAFETY: format is open; decoder is written on success.+ let stream = ffi::av_find_best_stream(+ guard.format,+ ffi::AVMEDIA_TYPE_VIDEO,+ -1,+ -1,+ &mut decoder,+ 0,+ );+ if stream < 0 {+ return Err(Error::NoVideoStream { path: path.clone() });+ }++ // SAFETY: av_find_best_stream returned a valid index into streams.+ let stream_ptr = *(*guard.format).streams.offset(stream as isize);+ let time_base = (*stream_ptr).time_base;++ // SAFETY: decoder was written by av_find_best_stream.+ guard.codec = ffi::avcodec_alloc_context3(decoder);+ if guard.codec.is_null() {+ return Err(fail("avcodec_alloc_context3", 0));+ }++ // SAFETY: both pointers are valid and the codec context is fresh.+ let code = ffi::avcodec_parameters_to_context(guard.codec, (*stream_ptr).codecpar);+ if code < 0 {+ return Err(fail("avcodec_parameters_to_context", code));+ }++ // SAFETY: context is populated from the stream's parameters.+ let code = ffi::avcodec_open2(guard.codec, decoder, ptr::null_mut());+ if code < 0 {+ return Err(fail("avcodec_open2", code));+ }++ let source_width = (*guard.codec).width;+ let source_height = (*guard.codec).height;+ let source_format = (*guard.codec).pix_fmt;++ // SAFETY: dimensions and format come from the opened codec.+ guard.scaler = ffi::sws_getContext(+ source_width,+ source_height,+ source_format,+ width as i32,+ height as i32,+ ffi::AV_PIX_FMT_RGBA,+ ffi::SWS_BILINEAR as i32,+ ptr::null_mut(),+ ptr::null_mut(),+ ptr::null(),+ );+ if guard.scaler.is_null() {+ return Err(fail("sws_getContext", 0));+ }++ // SAFETY: plain allocations, checked for null.+ guard.decoded = ffi::av_frame_alloc();+ guard.packet = ffi::av_packet_alloc();+ if guard.decoded.is_null() || guard.packet.is_null() {+ return Err(fail("av_frame_alloc", 0));+ }++ Ok(Self {+ path,+ format: guard.take_format(),+ codec: guard.take_codec(),+ scaler: guard.take_scaler(),+ decoded: guard.take_decoded(),+ packet: guard.take_packet(),+ stream,+ time_base,+ width,+ height,+ position: None,+ finished: false,+ })+ }+ }++ /// Converts a stream timestamp into seconds, exactly.+ ///+ /// This is the whole point of the linked backend: `time_base` is a+ /// rational, the timestamp is an integer count of those ticks, and+ /// `relay-core` speaks rationals - so the value survives with no rounding+ /// anywhere along the way.+ fn to_seconds(&self, ticks: i64) -> Rational {+ Rational::new(ticks, 1) * Rational::new(self.time_base.num as i64, self.time_base.den as i64)+ }++ /// Scales the decoded frame into a fresh RGBA buffer.+ fn convert(&mut self) -> Frame {+ let stride = self.width as usize * 4;+ let mut pixels = vec![0u8; stride * self.height as usize];++ let destination: [*mut u8; 4] = [pixels.as_mut_ptr(), ptr::null_mut(), ptr::null_mut(), ptr::null_mut()];+ let strides: [i32; 4] = [stride as i32, 0, 0, 0];++ // SAFETY: `decoded` holds a complete frame from the codec, the scaler+ // was built for exactly that source format and this destination size,+ // and `pixels` is stride * height bytes - the size sws_scale writes.+ unsafe {+ ffi::sws_scale(+ self.scaler,+ (*self.decoded).data.as_ptr() as *const *const u8,+ (*self.decoded).linesize.as_ptr(),+ 0,+ (*self.decoded).height,+ destination.as_ptr(),+ strides.as_ptr(),+ );+ }++ Frame::from_rgba(self.width, self.height, pixels)+ .expect("buffer is sized from the same width and height")+ }+}++impl FrameSource for FfiDecoder {+ fn width(&self) -> u32 {+ self.width+ }++ fn height(&self) -> u32 {+ self.height+ }++ fn position(&self) -> Option<Rational> {+ self.position+ }++ fn next_frame(&mut self) -> Result<Option<Frame>> {+ if self.finished {+ return Ok(None);+ }++ loop {+ // SAFETY: codec is open; `decoded` is a valid, reusable frame.+ let code = unsafe { ffi::avcodec_receive_frame(self.codec, self.decoded) };++ if code == 0 {+ // SAFETY: receive_frame succeeded, so the frame is populated.+ let ticks = unsafe { (*self.decoded).best_effort_timestamp };+ self.position = (ticks != ffi::AV_NOPTS_VALUE).then(|| self.to_seconds(ticks));+ return Ok(Some(self.convert()));+ }++ if code == ffi::AVERROR_EOF {+ self.finished = true;+ return Ok(None);+ }++ if code != AVERROR_EAGAIN {+ return Err(Error::Ffi {+ operation: "avcodec_receive_frame",+ path: self.path.clone(),+ detail: describe(code),+ });+ }++ // EAGAIN: the decoder needs another packet before it can produce+ // anything. Packets for other streams are skipped, not decoded.+ loop {+ // SAFETY: format is open; av_read_frame fills the packet, and+ // we unref it below whatever happens.+ let code = unsafe { ffi::av_read_frame(self.format, self.packet) };++ if code == ffi::AVERROR_EOF {+ // Flush: a null packet tells the decoder to emit whatever+ // it is still holding, which is how the final few frames+ // of a file come out at all.+ // SAFETY: passing null is the documented flush signal.+ unsafe { ffi::avcodec_send_packet(self.codec, ptr::null()) };+ break;+ }+ if code < 0 {+ return Err(Error::Ffi {+ operation: "av_read_frame",+ path: self.path.clone(),+ detail: describe(code),+ });+ }++ // SAFETY: packet is populated by av_read_frame.+ let belongs = unsafe { (*self.packet).stream_index } == self.stream;+ if belongs {+ // SAFETY: sending a valid packet to an open decoder.+ let sent = unsafe { ffi::avcodec_send_packet(self.codec, self.packet) };+ // SAFETY: releases the packet's buffer for reuse.+ unsafe { ffi::av_packet_unref(self.packet) };++ if sent < 0 && sent != AVERROR_EAGAIN {+ return Err(Error::Ffi {+ operation: "avcodec_send_packet",+ path: self.path.clone(),+ detail: describe(sent),+ });+ }+ break;+ }++ // SAFETY: not our stream, so drop it and read the next.+ unsafe { ffi::av_packet_unref(self.packet) };+ }+ }+ }+}++impl SeekableSource for FfiDecoder {+ fn seek(&mut self, to: Rational) -> Result<()> {+ // Convert seconds back into this stream's tick count.+ let ticks_per_second = Rational::new(self.time_base.den as i64, self.time_base.num as i64);+ let target = (to * ticks_per_second).floor();++ // BACKWARD lands on the keyframe at or before the target. Decoding+ // forward from there is what makes the seek frame-accurate rather+ // than keyframe-accurate - which is precisely what the pipe cannot do.+ // SAFETY: format is open and the stream index came from it.+ let code = unsafe {+ ffi::av_seek_frame(self.format, self.stream, target, ffi::AVSEEK_FLAG_BACKWARD as i32)+ };+ if code < 0 {+ return Err(Error::Ffi {+ operation: "av_seek_frame",+ path: self.path.clone(),+ detail: describe(code),+ });+ }++ // The decoder is still holding frames from before the jump.+ // SAFETY: codec is open.+ unsafe { ffi::avcodec_flush_buffers(self.codec) };++ self.finished = false;+ self.position = None;+ Ok(())+ }+}++impl Drop for FfiDecoder {+ fn drop(&mut self) {+ // SAFETY: each pointer was allocated in `open` and is freed exactly+ // once, here. The struct cannot be used after this.+ unsafe {+ ffi::av_packet_free(&mut self.packet);+ ffi::av_frame_free(&mut self.decoded);+ ffi::sws_freeContext(self.scaler);+ ffi::avcodec_free_context(&mut self.codec);+ ffi::avformat_close_input(&mut self.format);+ }+ }+}++/// Owns half-built state so that an early return from `open` still frees it.+///+/// Without this, every `?` between the first allocation and the last would+/// leak whatever had been built up to that point - and there are six of them.+#[derive(Default)]+struct Guard {+ format: *mut ffi::AVFormatContext,+ codec: *mut ffi::AVCodecContext,+ scaler: *mut ffi::SwsContext,+ decoded: *mut ffi::AVFrame,+ packet: *mut ffi::AVPacket,+}++macro_rules! take {+ ($name:ident, $field:ident, $ty:ty) => {+ fn $name(&mut self) -> $ty {+ std::mem::replace(&mut self.$field, ptr::null_mut())+ }+ };+}++impl Guard {+ take!(take_format, format, *mut ffi::AVFormatContext);+ take!(take_codec, codec, *mut ffi::AVCodecContext);+ take!(take_scaler, scaler, *mut ffi::SwsContext);+ take!(take_decoded, decoded, *mut ffi::AVFrame);+ take!(take_packet, packet, *mut ffi::AVPacket);+}++impl Drop for Guard {+ fn drop(&mut self) {+ // SAFETY: each free is a no-op on null, and a successful `open` has+ // already taken every pointer out, leaving all fields null.+ unsafe {+ if !self.packet.is_null() {+ ffi::av_packet_free(&mut self.packet);+ }+ if !self.decoded.is_null() {+ ffi::av_frame_free(&mut self.decoded);+ }+ if !self.scaler.is_null() {+ ffi::sws_freeContext(self.scaler);+ }+ if !self.codec.is_null() {+ ffi::avcodec_free_context(&mut self.codec);+ }+ if !self.format.is_null() {+ ffi::avformat_close_input(&mut self.format);+ }+ }+ }}#[cfg(test)]@@ -45,19 +449,23 @@ mod tests {#[test]fn links_against_the_expected_ffmpeg() {- let version = linked_version();- assert!(!version.is_empty(), "av_version_info returned nothing");-- // The bindings rusty_ffmpeg ships are generated for avcodec 62- // (FFmpeg 8.x). Loading a different major version would mean the- // struct layouts we compiled against do not match the library that- // actually loaded - which corrupts silently rather than failing, so- // it is worth an explicit check.- assert_eq!(- avcodec_major(),- 62,- "linked libavcodec is {} but the bindings expect 62 - see BUILD.md",- avcodec_major()- );+ assert!(!linked_version().is_empty(), "av_version_info returned nothing");++ // The bindings are generated for avcodec 62 (FFmpeg 8.x). A different+ // major version means the struct layouts we compiled against do not+ // match the library that loaded - which corrupts silently rather than+ // failing, so it is worth an explicit check.+ assert_eq!(avcodec_major(), 62, "linked libavcodec mismatch - see BUILD.md");+ }++ #[test]+ fn describes_error_codes() {+ // Whatever the wording, it should not be the numeric fallback.+ assert!(!describe(ffi::AVERROR_EOF).starts_with("error "));+ }++ #[test]+ fn a_missing_file_is_an_error_not_a_panic() {+ assert!(FfiDecoder::open("does-not-exist.mp4", 64, 64).is_err());}}diff --git a/engine/crates/relay-media/src/lib.rs b/engine/crates/relay-media/src/lib.rsindex 49807e3b..93377dee 100644--- a/engine/crates/relay-media/src/lib.rs+++ b/engine/crates/relay-media/src/lib.rs@@ -18,7 +18,7 @@ pub mod error;pub mod probe;pub use binaries::{ffmpeg, ffprobe, set_binaries};-pub use decode::{DecodeOptions, FfmpegDecoder, FrameSource};+pub use decode::{DecodeOptions, FfmpegDecoder, FrameSource, SeekableSource};pub use encode::{EncodeOptions, FfmpegEncoder, FrameSink};pub use error::{Error, Result};pub use probe::{AudioStream, MediaInfo, VideoStream, probe};diff --git a/engine/crates/relay-media/tests/ffi_decode.rs b/engine/crates/relay-media/tests/ffi_decode.rsnew file mode 100644index 00000000..3a6f733f--- /dev/null+++ b/engine/crates/relay-media/tests/ffi_decode.rs@@ -0,0 +1,148 @@+//! End-to-end checks for the linked decoder.+//!+//! These are the claims that justify the FFI backend existing at all - real+//! presentation timestamps and frame-accurate seeking - so they are tested+//! against an actual file rather than asserted in a doc comment.+//!+//! The fixture is generated with the `ffmpeg` binary at test time. If FFmpeg+//! is not installed the tests skip rather than fail: a missing tool is a+//! missing tool, not a broken decoder.++#![cfg(feature = "ffi")]++use std::path::PathBuf;+use std::process::Command;++use relay_core::time::Rational;+use relay_media::decode::{FrameSource, SeekableSource};+use relay_media::ffi::FfiDecoder;++const RATE: i64 = 30;+const SECONDS: i64 = 4;++/// Renders a deterministic clip and returns its path, or `None` without FFmpeg.+fn fixture() -> Option<PathBuf> {+ let path = std::env::temp_dir().join("relay-ffi-fixture.mp4");+ if path.exists() {+ return Some(path);+ }++ let status = Command::new("ffmpeg")+ .args(["-hide_banner", "-loglevel", "error", "-y"])+ .args(["-f", "lavfi", "-i", &format!("testsrc2=size=320x240:rate={RATE}")])+ .args(["-t", &SECONDS.to_string()])+ // A short keyframe interval, so a seek has something nearby to land on+ // and the test measures our decode-forward rather than FFmpeg's luck.+ .args(["-g", "15", "-c:v", "libx264", "-pix_fmt", "yuv420p"])+ .arg(&path)+ .status()+ .ok()?;++ status.success().then_some(path)+}++/// One frame's duration, as the tolerance for "landed on the right frame".+fn frame_duration() -> Rational {+ Rational::new(1, RATE)+}++#[test]+fn reports_real_presentation_timestamps() {+ let Some(path) = fixture() else { return };++ let mut decoder = FfiDecoder::open(&path, 160, 120).expect("open");+ let mut times = Vec::new();++ for _ in 0..10 {+ let frame = decoder.next_frame().expect("decode").expect("a frame");+ assert_eq!((frame.width(), frame.height()), (160, 120), "scaled to the requested size");++ let position = decoder.position().expect("a linked decoder always knows the time");+ times.push(position);+ }++ // The whole point: these come from the container, not from counting.+ assert_eq!(times[0], Rational::ZERO, "first frame starts at zero");++ for (index, time) in times.iter().enumerate() {+ let expected = frame_duration() * Rational::from_int(index as i64);+ assert_eq!(*time, expected, "frame {index} should sit exactly on its boundary");+ }++ assert!(times.windows(2).all(|pair| pair[0] < pair[1]), "timestamps must increase");+}++#[test]+fn seeks_to_an_exact_frame() {+ let Some(path) = fixture() else { return };++ let mut decoder = FfiDecoder::open(&path, 160, 120).expect("open");++ // Deliberately not a keyframe: at -g 15 the keyframes are every half+ // second, so 2.4s sits between two of them. Landing there is only possible+ // by decoding forward from the previous one, which is the thing a pipe+ // cannot do.+ let target = Rational::new(24, 10);+ decoder.seek(target).expect("seek");++ let mut landed = None;+ for _ in 0..64 {+ decoder.next_frame().expect("decode").expect("a frame");+ let position = decoder.position().expect("position");+ if position >= target {+ landed = Some(position);+ break;+ }+ }++ let landed = landed.expect("never reached the target after a seek");+ let drift = landed - target;+ assert!(+ drift >= Rational::ZERO && drift < frame_duration(),+ "landed at {landed} for a target of {target}, which is more than one frame out",+ );+}++#[test]+fn seeking_backwards_works_too() {+ let Some(path) = fixture() else { return };++ let mut decoder = FfiDecoder::open(&path, 64, 64).expect("open");++ // Run well into the file first, so the seek genuinely goes backwards and+ // has stale decoder state to flush.+ for _ in 0..60 {+ decoder.next_frame().expect("decode");+ }+ let before = decoder.position().expect("position");+ assert!(before > Rational::ONE, "should be past a second by now");++ decoder.seek(Rational::new(1, 2)).expect("seek back");+ decoder.next_frame().expect("decode").expect("a frame");++ let after = decoder.position().expect("position");+ assert!(after < before, "seeking back should move the position back");+ assert!(after <= Rational::new(6, 10), "should land near the half-second mark, got {after}");+}++#[test]+fn runs_to_the_end_and_stops() {+ let Some(path) = fixture() else { return };++ let mut decoder = FfiDecoder::open(&path, 64, 64).expect("open");+ let mut count = 0u32;+ while decoder.next_frame().expect("decode").is_some() {+ count += 1;+ assert!(count < 1000, "decoder never reported end of stream");+ }++ // Flushing at EOF is what makes the last few frames come out at all; a+ // decoder that forgets it silently loses them.+ let expected = (RATE * SECONDS) as u32;+ assert!(+ count.abs_diff(expected) <= 1,+ "decoded {count} frames, expected about {expected}",+ );++ assert!(decoder.next_frame().expect("decode").is_none(), "end of stream is sticky");+}diff --git a/engine/docs/decisions/0002-ffmpeg-over-a-pipe.md b/engine/docs/decisions/0002-ffmpeg-over-a-pipe.mdindex ef2033d1..2e55cb77 100644--- a/engine/docs/decisions/0002-ffmpeg-over-a-pipe.md+++ b/engine/docs/decisions/0002-ffmpeg-over-a-pipe.md@@ -66,9 +66,25 @@ Chocolatey - binaries only, no `libavcodec/avcodec.h` and no import libs, andBudget for the build plumbing, not the Rust. The Rust is the easy half.+## Status: the FFI half now exists++`relay-media`'s `ffi` feature builds a linked decoder, `FfiDecoder`, which+implements `FrameSource` and `SeekableSource`. Both correctness gaps above are+closed in it and covered by `tests/ffi_decode.rs`:++- real presentation timestamps, exact rationals straight from the container+- frame-accurate seeking, verified by landing between keyframes++It is **off by default and unused by the app.** The subprocess backend is still+what runs, and probe and export are staying on it permanently.++The seam held: adding it required one new type implementing an existing trait,+one new trait for the capability the pipe genuinely lacks, and no change to+`relay-core`, `relay-render`, or any caller.+## What would change our mind-Building playback. Specifically, the first time we need either a real PTS or a-frame-accurate seek - whichever comes first. Both arrive with the transport-controls, so that is the deadline.+Nothing left to decide about *whether*. What remains is wiring playback to the+linked decoder, at which point scrubbing stops respawning a process per seek.-Do **not** wait for 4K, and do not migrate probe or export along with it.+Do **not** migrate probe or export along with it - for those, the tradeoff+never flips.