Skip to content

The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.

updates

Jareer984f101056d1
diff --git a/.github/workflows/ffmpeg.yml b/.github/workflows/ffmpeg.yml
index fa4142f0..4445e2c0 100644
--- a/.github/workflows/ffmpeg.yml
+++ b/.github/workflows/ffmpeg.yml
@@ -180,3 +180,108 @@ jobs:
out/bin/ffprobe.exe
SOURCE-COMMIT.txt
if-no-files-found: error
+
+ # The macOS half of the trimmed pair, so the build can one day replace the
+ # full static downloads in build.yml on both platforms at once. Same
+ # configure list as Windows; the differences are only what the platform
+ # forces: no `-static` (macOS cannot fully static-link against libSystem),
+ # x264 from Homebrew, `otool` instead of `ldd` for the portability check.
+ macos:
+ runs-on: macos-latest
+ steps:
+ - name: Tools
+ run: brew install nasm x264
+
+ - name: Fetch FFmpeg source
+ run: |
+ git clone --depth 1 --branch "${{ inputs.version }}" \
+ https://github.com/FFmpeg/FFmpeg.git ffmpeg
+ cd ffmpeg && git rev-parse HEAD > ../SOURCE-COMMIT.txt
+
+ - name: Configure
+ run: |
+ cd ffmpeg
+ ./configure \
+ --prefix=../out \
+ --pkg-config-flags=--static \
+ --disable-everything \
+ --disable-doc \
+ --disable-debug \
+ --disable-network \
+ --disable-autodetect \
+ --disable-ffplay \
+ --disable-shared \
+ --enable-static \
+ --enable-small \
+ --enable-gpl \
+ --enable-libx264 \
+ --enable-zlib \
+ --enable-decoder=rawvideo,h264,hevc,vp8,vp9,av1,mpeg4,mjpeg,prores,dnxhd,png,webp,bmp,tiff,gif,aac,mp3,flac,opus,vorbis,alac,pcm_s16le,pcm_s24le,pcm_s32le,pcm_f32le \
+ --enable-encoder=rawvideo,libx264,aac,mjpeg,png,pcm_s16le \
+ --enable-demuxer=mov,matroska,avi,mp3,wav,flac,ogg,aac,image2,gif,rawvideo,pcm_s16le,concat \
+ --enable-muxer=mp4,mov,mp3,wav,ipod,mjpeg,image2,rawvideo,pcm_s16le,null \
+ --enable-parser=h264,hevc,vp8,vp9,av1,aac,mpegaudio,flac,vorbis,png,mjpeg \
+ --enable-bsf=h264_mp4toannexb,hevc_mp4toannexb,aac_adtstoasc \
+ --enable-protocol=file,pipe \
+ --enable-filter=scale,tile,fps,format,null,copy,setpts,trim,transpose,crop,overlay \
+ --enable-filter=fade,hue,colorchannelmixer,negate,unsharp,gblur,boxblur,colortemperature \
+ --enable-filter=vibrance,eq,vignette,noise,lutrgb,pixelize,lenscorrection \
+ --enable-filter=split,blend,hstack,hflip \
+ --enable-filter=aresample,asetrate,atempo,aformat,anull,atrim,asetpts,adelay,amix,apad,afade,volume \
+ --enable-filter=highpass,lowpass,equalizer,bass,treble,compand,deesser,aecho,alimiter,aphaser,flanger \
+ --enable-filter=abuffer,abuffersink,buffer,buffersink
+
+ - name: Build
+ run: cd ffmpeg && make -j"$(sysctl -n hw.ncpu)" && make install
+
+ - name: Verify it can do the work
+ run: |
+ set -e
+ BIN=out/bin
+ "$BIN/ffmpeg" -hide_banner -version | head -1
+ "$BIN/ffprobe" -hide_banner -version | head -1
+
+ head -c 9216000 /dev/zero > frames.raw
+ head -c 384000 /dev/zero > tone.pcm
+
+ "$BIN/ffmpeg" -hide_banner -loglevel error -y \
+ -f rawvideo -pix_fmt rgba -s 320x240 -r 30 -i frames.raw \
+ -c:v libx264 -preset veryfast -crf 20 -pix_fmt yuv420p \
+ -movflags +faststart probe.mp4
+ "$BIN/ffprobe" -v error -show_streams -show_format -of json probe.mp4 > /dev/null
+ "$BIN/ffmpeg" -hide_banner -loglevel error -y -i probe.mp4 \
+ -vf "fps=4,scale=-2:72,tile=4x1" -frames:v 1 -f mjpeg strip.jpg
+ "$BIN/ffmpeg" -hide_banner -loglevel error -y -i probe.mp4 \
+ -f rawvideo -pix_fmt rgba - > /dev/null
+ "$BIN/ffmpeg" -hide_banner -loglevel error -y -i probe.mp4 -frames:v 1 still.png
+ "$BIN/ffmpeg" -hide_banner -loglevel error -y -i still.png \
+ -frames:v 1 -f rawvideo -pix_fmt rgba - > /dev/null
+ "$BIN/ffmpeg" -hide_banner -loglevel error -y \
+ -f s16le -ar 48000 -ac 2 -i tone.pcm -c:a aac tone.m4a
+ "$BIN/ffmpeg" -hide_banner -loglevel error -y -ss 0 -t 1 -i tone.m4a -vn \
+ -af "aresample=48000,aformat=sample_fmts=s16:channel_layouts=stereo" \
+ -f s16le span.pcm
+ "$BIN/ffmpeg" -hide_banner -loglevel error -y -i tone.m4a -i tone.m4a \
+ -filter_complex "[0:a]atrim=start=0:duration=1,asetpts=PTS-STARTPTS,volume=1.5,afade=t=in:st=0:d=0.2,adelay=100:all=1[a0];[1:a]atrim=start=0:duration=1,asetpts=PTS-STARTPTS[a1];[a0][a1]amix=inputs=2:normalize=0,apad,atrim=duration=2[out]" \
+ -map "[out]" -c:a aac -b:a 192k mix.m4a
+ "$BIN/ffmpeg" -hide_banner -loglevel error -y -i probe.mp4 -i mix.m4a \
+ -c:v copy -c:a copy -shortest -movflags +faststart final.mp4
+
+ # Portable means system frameworks only - a Homebrew dylib in this
+ # list would bind the binary to the machine that built it.
+ if otool -L "$BIN/ffmpeg" | tail -n +2 | grep -v "/usr/lib\|/System/Library"; then
+ echo "not self-contained (see above)"; exit 1
+ fi
+ echo "every path the app uses is present, and the build is self-contained"
+
+ - name: Report size
+ run: ls -l out/bin && du -sh out/bin
+
+ - uses: actions/upload-artifact@v4
+ with:
+ name: ffmpeg-${{ inputs.version }}-macos-arm64
+ path: |
+ out/bin/ffmpeg
+ out/bin/ffprobe
+ SOURCE-COMMIT.txt
+ if-no-files-found: error
diff --git a/desktop/docs/decisions/0007-preview-approximation-layer.md b/desktop/docs/decisions/0007-preview-approximation-layer.md
index 19e29c56..a933afa8 100644
--- a/desktop/docs/decisions/0007-preview-approximation-layer.md
+++ b/desktop/docs/decisions/0007-preview-approximation-layer.md
@@ -27,7 +27,15 @@ the architecture doctrine warns about. Containment:
- Every effect's chain and preview live in **one catalogue entry**, so adding
or changing an effect touches one place.
+## Status: the retirement clock is running
+The first step of the engine presentation path shipped: the reader pool and
+frame cache exist (`relay_media::pool`), and the paused monitor already shows
+the engine's true composite through `preview_frame` - real multi-track
+stacking and exact effect chains, with the approximation layer doing motion
+and the truth taking over on dwell. What remains for full retirement is
+streaming presentation against the transport clock.
+
## What would change our mind
-The engine presentation path (frame cache + reader pool + native surface).
-When the engine can show its own frames at interactive rates, this layer
-retires bottom-up: composite first, effects last.
+The engine presentation path completing. When the engine can show its own
+frames at interactive rates, this layer retires bottom-up: composite first,
+effects last.
diff --git a/desktop/src-tauri/src/export.rs b/desktop/src-tauri/src/export.rs
index 71f4eb45..2fe4364e 100644
--- a/desktop/src-tauri/src/export.rs
+++ b/desktop/src-tauri/src/export.rs
@@ -181,7 +181,7 @@ pub fn run(
/// against every other track is preserved, and nothing else occupies odd
/// lanes. Cuts are collected before anything mutates, so resolving one
/// transition cannot unhook the adjacency test of the next.
-fn resolve_transitions(clips: &mut [ExportClip], rate: FrameRate) {
+fn resolve_transitions(clips: &mut [ExportClip], rate: FrameRate, bake_fades: bool) {
for clip in clips.iter_mut() {
clip.track *= 2;
}
@@ -253,7 +253,7 @@ fn resolve_transitions(clips: &mut [ExportClip], rate: FrameRate) {
b.fade_in = b.fade_in.max(d);
b.track = a_track + 1;
}
- "fade-black" | "fade-white" => {
+ "fade-black" | "fade-white" if bake_fades => {
// Half the duration on each side of the cut, as fade filters
// at decode. Frame-based, because the decoder emits exactly
// one frame per output frame - so the fade lands on the same
@@ -311,7 +311,7 @@ pub fn render(request: &ExportRequest, mut reporter: Reporter<'_>) -> Result<Str
// else reads the clip list, so the picture and sound paths below never
// know transitions exist.
let mut resolved = request.clips.clone();
- resolve_transitions(&mut resolved, rate);
+ resolve_transitions(&mut resolved, rate, true);
// Stills composite exactly like footage; they only differ in how they are
// decoded, which is handled where the decoder is opened.
@@ -636,6 +636,107 @@ fn quantise(seconds: f64, rate: FrameRate) -> Rational {
rate.time_of_frame((seconds * rate.fps().as_f64()).round().max(0.0) as i64)
}
+/// One paused-monitor frame: the same clip list the exporter takes, one
+/// timestamp, a preview resolution.
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct PreviewFrameRequest {
+ pub time: f64,
+ pub width: u32,
+ pub height: u32,
+ pub rate_num: i64,
+ pub rate_den: i64,
+ pub clips: Vec<ExportClip>,
+}
+
+/// Composites the true frame at one instant, for the paused monitor.
+///
+/// This is the engine presentation path's first step (desktop decision 0007):
+/// the identical plan/composite the exporter runs, fed from the reader pool
+/// so scrubbing revisits are cache hits. Fade-to-colour transitions are NOT
+/// baked here - their filter frame numbers assume decode-from-clip-start,
+/// which pooled seeks break - so the UI keeps drawing its veil, whose shape
+/// already matches the exporter's. Returns raw RGBA, exactly
+/// `width * height * 4` bytes.
+pub fn preview_frame(
+ pool: &mut relay_media::ReaderPool,
+ request: &PreviewFrameRequest,
+) -> Result<Vec<u8>, String> {
+ let rate = FrameRate::new(Rational::new(request.rate_num, request.rate_den));
+
+ let mut resolved = request.clips.clone();
+ resolve_transitions(&mut resolved, rate, false);
+ let visible: Vec<&ExportClip> = resolved
+ .iter()
+ .filter(|clip| (clip.kind == "video" || clip.kind == "image") && !clip.hidden)
+ .collect();
+
+ // build_timeline reads only the output format off the request; the shim
+ // keeps one conversion path rather than a preview-flavoured copy of it.
+ let shim = ExportRequest {
+ output: String::new(),
+ width: request.width,
+ height: request.height,
+ rate_num: request.rate_num,
+ rate_den: request.rate_den,
+ crf: 18,
+ preset: String::new(),
+ clips: Vec::new(),
+ };
+ let (timeline, stills, decode_sizes, filter_chains) = build_timeline(&shim, rate, &visible);
+ let plan = plan_frame(&timeline, quantise(request.time, rate));
+
+ let mut sources: Vec<(std::sync::Arc<Frame>, f32, Transform)> =
+ Vec::with_capacity(plan.layers.len());
+ for layer in &plan.layers {
+ let (decode_width, decode_height) = decode_sizes
+ .get(&layer.clip)
+ .copied()
+ .unwrap_or((request.width, request.height));
+ let chain = filter_chains.get(&layer.clip).map(String::as_str);
+ // A source that fails to decode contributes nothing rather than
+ // blanking the monitor - same grace the exporter extends.
+ if let Ok(frame) = pool.frame_at(
+ std::path::Path::new(&layer.media),
+ layer.source_time,
+ decode_width,
+ decode_height,
+ stills.contains(&layer.clip),
+ chain,
+ ) {
+ sources.push((frame, layer.opacity, layer.transform));
+ }
+ }
+
+ // The same fraction-to-pixel placement the exporter applies.
+ let layers: Vec<Layer<'_>> = sources
+ .iter()
+ .map(|(frame, opacity, transform)| {
+ let x = (i64::from(request.width) - i64::from(frame.width())) / 2;
+ let y = (i64::from(request.height) - i64::from(frame.height())) / 2;
+ let placement = if transform.is_identity() {
+ Placement::IDENTITY
+ } else {
+ Placement {
+ scale: transform.scale as f32,
+ rotation: transform.rotation.to_radians() as f32,
+ translate_x: (transform.offset_x * f64::from(request.width)) as f32,
+ translate_y: (transform.offset_y * f64::from(request.height)) as f32,
+ }
+ };
+ Layer::new(frame)
+ .at(x as i32, y as i32)
+ .with_opacity(*opacity)
+ .with_placement(placement)
+ })
+ .collect();
+
+ // CPU on purpose: one frame at preview size is milliseconds, and holding
+ // a GPU context alive for occasional scrubs is not worth its memory.
+ let composed = CpuCompositor.composite(request.width, request.height, &layers);
+ Ok(composed.into_pixels())
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -677,7 +778,7 @@ mod tests {
fn a_cross_fade_overlaps_the_incoming_clip_on_its_own_lane() {
let mut clips = vec![clip("video", 0, 0.0, 4.0, 0.0), clip("video", 0, 4.0, 4.0, 2.0)];
clips[1].transition = spec("cross-fade", 1.0);
- resolve_transitions(&mut clips, FrameRate::THIRTY);
+ resolve_transitions(&mut clips, FrameRate::THIRTY, true);
let b = &clips[1];
assert_eq!(b.start, 3.0, "extends backwards over the cut");
@@ -693,7 +794,7 @@ mod tests {
fn a_cross_fade_clamps_to_the_available_handle() {
let mut clips = vec![clip("video", 0, 0.0, 4.0, 0.0), clip("video", 0, 4.0, 4.0, 0.25)];
clips[1].transition = spec("cross-fade", 2.0);
- resolve_transitions(&mut clips, FrameRate::THIRTY);
+ resolve_transitions(&mut clips, FrameRate::THIRTY, true);
// Only a quarter second of source exists before the in-point, so that
// is the whole dissolve.
@@ -706,7 +807,7 @@ mod tests {
fn a_still_needs_no_handle_to_dissolve() {
let mut clips = vec![clip("video", 0, 0.0, 4.0, 0.0), clip("image", 0, 4.0, 4.0, 0.0)];
clips[1].transition = spec("cross-fade", 1.0);
- resolve_transitions(&mut clips, FrameRate::THIRTY);
+ resolve_transitions(&mut clips, FrameRate::THIRTY, true);
assert_eq!(clips[1].video_fade_in, 1.0);
assert_eq!(clips[1].source_start, 0.0, "a still has no source clock to rewind");
}
@@ -715,7 +816,7 @@ mod tests {
fn a_fade_to_black_splits_across_the_cut_as_fade_filters() {
let mut clips = vec![clip("video", 0, 0.0, 4.0, 0.0), clip("video", 0, 4.0, 4.0, 0.0)];
clips[1].transition = spec("fade-black", 1.0);
- resolve_transitions(&mut clips, FrameRate::THIRTY);
+ resolve_transitions(&mut clips, FrameRate::THIRTY, true);
// Half a second each side at 30fps is 15 frames.
assert_eq!(clips[0].video_filter_chain, "fade=t=out:start_frame=105:nb_frames=15");
@@ -727,7 +828,7 @@ mod tests {
fn a_fade_to_white_names_its_colour() {
let mut clips = vec![clip("video", 0, 0.0, 2.0, 0.0), clip("video", 0, 2.0, 2.0, 0.0)];
clips[1].transition = spec("fade-white", 0.5);
- resolve_transitions(&mut clips, FrameRate::THIRTY);
+ resolve_transitions(&mut clips, FrameRate::THIRTY, true);
assert!(clips[0].video_filter_chain.ends_with(":color=white"));
assert!(clips[1].video_filter_chain.contains("t=in"));
}
@@ -737,7 +838,7 @@ mod tests {
let mut clips = vec![clip("video", 0, 0.0, 2.0, 0.0), clip("video", 0, 2.0, 2.0, 0.0)];
clips[1].video_filter_chain = "hue=s=0".to_owned();
clips[1].transition = spec("fade-black", 0.5);
- resolve_transitions(&mut clips, FrameRate::THIRTY);
+ resolve_transitions(&mut clips, FrameRate::THIRTY, true);
assert!(
clips[1].video_filter_chain.starts_with("hue=s=0,fade=t=in"),
"was: {}",
@@ -749,7 +850,7 @@ mod tests {
fn a_transition_with_no_adjacent_clip_is_orphaned_not_fatal() {
let mut clips = vec![clip("video", 0, 0.0, 2.0, 0.0), clip("video", 0, 5.0, 2.0, 0.0)];
clips[1].transition = spec("cross-fade", 1.0);
- resolve_transitions(&mut clips, FrameRate::THIRTY);
+ resolve_transitions(&mut clips, FrameRate::THIRTY, true);
assert_eq!(clips[1].start, 5.0);
assert_eq!(clips[1].video_fade_in, 0.0);
}
@@ -757,7 +858,7 @@ mod tests {
#[test]
fn track_indices_are_doubled_for_everyone() {
let mut clips = vec![clip("video", 0, 0.0, 2.0, 0.0), clip("audio", 3, 0.0, 2.0, 0.0)];
- resolve_transitions(&mut clips, FrameRate::THIRTY);
+ resolve_transitions(&mut clips, FrameRate::THIRTY, true);
assert_eq!(clips[0].track, 0);
assert_eq!(clips[1].track, 6);
}
@@ -766,7 +867,7 @@ mod tests {
fn an_unknown_transition_kind_renders_as_a_plain_cut() {
let mut clips = vec![clip("video", 0, 0.0, 2.0, 0.0), clip("video", 0, 2.0, 2.0, 0.0)];
clips[1].transition = spec("wipe-left", 1.0);
- resolve_transitions(&mut clips, FrameRate::THIRTY);
+ resolve_transitions(&mut clips, FrameRate::THIRTY, true);
assert_eq!(clips[1].start, 2.0);
assert!(clips[1].video_filter_chain.is_empty());
}
diff --git a/desktop/src-tauri/src/lib.rs b/desktop/src-tauri/src/lib.rs
index 1553cc42..f06f4b35 100644
--- a/desktop/src-tauri/src/lib.rs
+++ b/desktop/src-tauri/src/lib.rs
@@ -448,6 +448,30 @@ fn config_dir(app: &tauri::AppHandle) -> Result<std::path::PathBuf, String> {
/// The stop flag for the one export that can run at a time.
struct ExportState(std::sync::Arc<std::sync::atomic::AtomicBool>);
+/// The reader pool behind the paused monitor's true frames. One pool for the
+/// app's lifetime: its whole value is what stays warm between scrubs.
+struct PoolState(std::sync::Arc<std::sync::Mutex<relay_media::ReaderPool>>);
+
+/// The engine-composited frame at one instant, as raw RGBA bytes.
+///
+/// Serialised through the pool's mutex on a blocking thread: one scrub at a
+/// time, in order, off the main thread. The UI debounces and drops stale
+/// responses, so a slow decode never wedges anything but itself.
+#[tauri::command]
+async fn preview_frame(
+ state: tauri::State<'_, PoolState>,
+ request: export::PreviewFrameRequest,
+) -> Result<tauri::ipc::Response, String> {
+ let pool = std::sync::Arc::clone(&state.0);
+ tauri::async_runtime::spawn_blocking(move || {
+ let mut pool = pool.lock().map_err(|_| "reader pool poisoned".to_owned())?;
+ export::preview_frame(&mut pool, &request)
+ })
+ .await
+ .map_err(|error| format!("preview task failed: {error}"))?
+ .map(tauri::ipc::Response::new)
+}
+
/// Renders the timeline to a file.
///
/// Runs on a blocking thread and reports progress through the
@@ -549,6 +573,9 @@ pub fn run() {
std::sync::Mutex::new(None),
)));
app.manage(editor_api::EditorState(std::sync::Mutex::new(None)));
+ app.manage(PoolState(std::sync::Arc::new(std::sync::Mutex::new(
+ relay_media::ReaderPool::with_defaults(),
+ ))));
Ok(())
})
.plugin(tauri_plugin_opener::init())
@@ -573,6 +600,7 @@ pub fn run() {
forget_project,
export_project,
cancel_export,
+ preview_frame,
editor_api::editor_open,
editor_api::editor_apply,
editor_api::editor_undo,
diff --git a/desktop/src/App.tsx b/desktop/src/App.tsx
index c5790b9a..e1b8fe7c 100644
--- a/desktop/src/App.tsx
+++ b/desktop/src/App.tsx
@@ -51,6 +51,7 @@ import {
editorSave,
editorUndo,
engineVersion,
+ previewFrame,
probeMedia,
readMediaBytes,
transcribeClip,
@@ -1178,6 +1179,48 @@ function Editor({
[project, timeline],
);
+ /**
+ * The engine's true frame for the paused monitor: the exporter's own plan,
+ * compositor and effect chains, via the host's reader pool. Fetched when
+ * the playhead settles; cleared the moment it moves again, so scrubbing
+ * shows the live approximation and dwelling shows the truth.
+ */
+ const [engineStill, setEngineStill] = useState<{
+ bytes: ArrayBuffer;
+ width: number;
+ height: number;
+ } | null>(null);
+ const stillToken = useRef(0);
+ useEffect(() => {
+ const token = ++stillToken.current;
+ setEngineStill(null);
+ if (playing || !loaded || exportClips.length === 0) return;
+
+ const timer = window.setTimeout(() => {
+ // Preview resolution: the frame's aspect, capped for IPC weight. Even
+ // dimensions, because decoders round odd ones themselves.
+ const cap = 960;
+ const scale = Math.min(1, cap / Math.max(frame.width, frame.height));
+ const width = Math.max(2, Math.round((frame.width * scale) / 2) * 2);
+ const height = Math.max(2, Math.round((frame.height * scale) / 2) * 2);
+ previewFrame({
+ time: latest.current.playhead,
+ width,
+ height,
+ rateNum: session.rateNum,
+ rateDen: session.rateDen,
+ clips: exportClips,
+ })
+ .then((bytes) => {
+ if (stillToken.current === token) setEngineStill({ bytes, width, height });
+ })
+ .catch(() => {
+ if (stillToken.current === token) setEngineStill(null);
+ });
+ }, 160);
+ return () => window.clearTimeout(timer);
+ }, [playing, loaded, playhead, exportClips, frame, session.rateNum, session.rateDen]);
+
const selectedClip =
selectedClipIds.length === 1 ? findClip(project, selectedClipIds[0]) : null;
const inspectorMedia = selectedClip
@@ -1385,6 +1428,7 @@ function Editor({
opacity={previewClip?.opacity ?? 1}
effects={previewClip?.videoEffects ?? null}
ghost={previewGhost}
+ engineStill={engineStill}
veil={previewVeil}
mediaSize={
previewMedia && previewMedia.width && previewMedia.height
diff --git a/desktop/src/components/Preview.tsx b/desktop/src/components/Preview.tsx
index 6efe3be5..761664c8 100644
--- a/desktop/src/components/Preview.tsx
+++ b/desktop/src/components/Preview.tsx
@@ -212,6 +212,7 @@ export function Preview({
opacity,
effects,
ghost,
+ engineStill,
veil,
mediaSize,
selectedClipId,
@@ -243,6 +244,9 @@ export function Preview({
/** A cross-fade in progress: the incoming clip's pre-roll, faded in over
* the picture. Null outside a dissolve window. */
ghost: { clipId: string; path: string; time: number; speed: number; opacity: number } | null;
+ /** The engine's true composite for the paused playhead - the exporter's own
+ * plan and compositor. Drawn over the approximation while it holds. */
+ engineStill: { bytes: ArrayBuffer; width: number; height: number } | null;
/** A fade-to-colour transition passing over the playhead: a coloured wash
* whose opacity the app computes per frame. Null when no fade is live. */
veil: { color: string; opacity: number } | null;
@@ -541,6 +545,12 @@ export function Preview({
<GhostVideo key={ghost.clipId} ghost={ghost} playing={playing} />
)}
+ {/* The engine's true frame, once the paused playhead settles:
+ real multi-track compositing, exact effect chains. It covers
+ the approximation layers beneath; the veil and titles still
+ draw above, exactly as the exporter stacks them. */}
+ {engineStill && <EngineStillLayer still={engineStill} />}
+
{/* A fade-to-black/white transition washing over the cut. Above
the picture, below titles - titles ride through a fade the
same way the exporter composites them, on top. */}
@@ -724,6 +734,43 @@ export function Preview({
);
}
+/**
+ * The engine's paused-frame composite, blitted into the frame box.
+ *
+ * The bytes arrive as raw RGBA at the request's own resolution; the canvas
+ * holds them at that resolution and CSS scales it to the frame, which shares
+ * its aspect by construction.
+ */
+function EngineStillLayer({
+ still,
+}: {
+ still: { bytes: ArrayBuffer; width: number; height: number };
+}) {
+ const canvasRef = useRef<HTMLCanvasElement>(null);
+
+ useEffect(() => {
+ const canvas = canvasRef.current;
+ const context = canvas?.getContext("2d");
+ if (!canvas || !context) return;
+ if (still.bytes.byteLength !== still.width * still.height * 4) return;
+ canvas.width = still.width;
+ canvas.height = still.height;
+ context.putImageData(
+ new ImageData(new Uint8ClampedArray(still.bytes), still.width, still.height),
+ 0,
+ 0,
+ );
+ }, [still]);
+
+ return (
+ <canvas
+ ref={canvasRef}
+ className="pointer-events-none absolute inset-0 h-full w-full"
+ aria-hidden
+ />
+ );
+}
+
/**
* The canvas pass for geometry effects the CSS filter pipeline cannot do.
*
diff --git a/desktop/src/lib/engine.ts b/desktop/src/lib/engine.ts
index 76795593..e107e731 100644
--- a/desktop/src/lib/engine.ts
+++ b/desktop/src/lib/engine.ts
@@ -235,6 +235,23 @@ export async function cancelExport(): Promise<void> {
return invoke<void>("cancel_export");
}
+/**
+ * The engine-composited frame at one instant: the exporter's own plan,
+ * compositor and effects, fed from the host's reader pool. Raw RGBA bytes,
+ * exactly `width * height * 4` of them. This is what the paused monitor
+ * shows - the true frame, not the approximation.
+ */
+export async function previewFrame(request: {
+ time: number;
+ width: number;
+ height: number;
+ rateNum: number;
+ rateDen: number;
+ clips: ExportClip[];
+}): Promise<ArrayBuffer> {
+ return invoke<ArrayBuffer>("preview_frame", { request });
+}
+
/**
* Writes one file into the project's cache folder and returns its path.
*
diff --git a/engine/crates/relay-cli/src/main.rs b/engine/crates/relay-cli/src/main.rs
index 36e169c2..851e732a 100644
--- a/engine/crates/relay-cli/src/main.rs
+++ b/engine/crates/relay-cli/src/main.rs
@@ -10,9 +10,7 @@ use std::path::PathBuf;
use clap::{Parser, Subcommand};
use relay_core::time::{FrameRate, Rational};
use relay_core::timeline::{Clip, MediaRef, Timeline, Track, TrackKind};
-use relay_media::{
- DecodeOptions, EncodeOptions, FfmpegDecoder, FfmpegEncoder, FrameSink, FrameSource,
-};
+use relay_media::{EncodeOptions, FfmpegEncoder, FrameSink, ReaderPool};
use relay_render::{Compositor, CpuCompositor, Layer, plan_frame};
#[derive(Parser)]
@@ -89,12 +87,11 @@ fn probe(path: &PathBuf) -> Result<(), Box<dyn Error>> {
/// The vertical slice.
///
-/// Note the deliberate shortcut: this pulls frames from one decoder in order
-/// and assumes the plan asks for them in the same order. That holds for a
-/// single clip played start to finish, and it is exactly what breaks the
-/// moment there are two overlapping clips or the playhead jumps. The fix is a
-/// per-media reader pool with a frame cache, which is the next real piece of
-/// work - not something to bodge in here.
+/// Frames come from the reader pool by (media, source time), which is what
+/// makes this loop correct for *any* plan - overlapping clips, gaps, jumps -
+/// not just one clip played start to finish. The shortcut this function
+/// carried for its whole early life is gone; the pool it was waiting for
+/// exists (`relay_media::pool`).
fn render(
input: &PathBuf,
output: &PathBuf,
@@ -108,10 +105,7 @@ fn render(
let timeline = single_clip_timeline(input, width, height, rate, frames);
- let mut decoder = FfmpegDecoder::open(
- input,
- &DecodeOptions::default().limited_to(frames).at_rate(rate),
- )?;
+ let mut pool = ReaderPool::with_defaults();
let mut encoder =
FfmpegEncoder::create(output, width, height, rate, &EncodeOptions::default())?;
let mut compositor = CpuCompositor;
@@ -125,12 +119,19 @@ fn render(
// A gap in the timeline is black, not an error.
compositor.composite(width, height, &[])
} else {
- let Some(decoded) = decoder.next_frame()? else {
- println!("source ended after {index} frames");
- break;
- };
- let opacity = plan.layers[0].opacity * fade_in(index, fade);
- compositor.composite(width, height, &[Layer::new(&decoded).with_opacity(opacity)])
+ let mut sources = Vec::with_capacity(plan.layers.len());
+ for layer in &plan.layers {
+ let frame =
+ pool.frame_at(&layer.media, layer.source_time, width, height, false, None)?;
+ sources.push((frame, layer.opacity));
+ }
+ let layers: Vec<Layer<'_>> = sources
+ .iter()
+ .map(|(frame, opacity)| {
+ Layer::new(frame).with_opacity(opacity * fade_in(index, fade))
+ })
+ .collect();
+ compositor.composite(width, height, &layers)
};
encoder.write_frame(&composed)?;
diff --git a/engine/crates/relay-media/src/lib.rs b/engine/crates/relay-media/src/lib.rs
index 34f78d79..76d638c9 100644
--- a/engine/crates/relay-media/src/lib.rs
+++ b/engine/crates/relay-media/src/lib.rs
@@ -16,6 +16,7 @@ pub mod encode;
#[cfg(feature = "ffi")]
pub mod ffi;
pub mod error;
+pub mod pool;
pub mod probe;
mod process;
@@ -24,4 +25,5 @@ pub use binaries::{ffmpeg, ffprobe, set_binaries};
pub use decode::{DecodeOptions, FfmpegDecoder, FrameSource, SeekableSource};
pub use encode::{EncodeOptions, FfmpegEncoder, FrameSink};
pub use error::{Error, Result};
+pub use pool::{FrameCache, ReaderPool};
pub use probe::{AudioStream, MediaInfo, VideoStream, probe};
diff --git a/engine/crates/relay-media/src/pool.rs b/engine/crates/relay-media/src/pool.rs
new file mode 100644
index 00000000..a7744d28
--- /dev/null
+++ b/engine/crates/relay-media/src/pool.rs
@@ -0,0 +1,529 @@
+//! The reader pool and frame cache: random access to any frame of any file.
+//!
+//! Everything before this decoded *forward*: open a decoder, pull frames in
+//! order, drop it. That is exactly right for export and exactly wrong for
+//! interactive use - scrubbing asks for arbitrary (media, time) pairs, and
+//! respawning a process per request is why playback could not exist. This
+//! module is the piece the decision log has called "the next real piece of
+//! work" since the first CLI commit:
+//!
+//! - **One reader per (file, size)**, kept warm between requests. A request
+//! near the reader's current position rolls forward (cheap, exact); a
+//! request elsewhere seeks - frame-accurately through the FFI decoder when
+//! the `ffi` feature is on, by respawning the subprocess decoder at the
+//! target otherwise.
+//! - **A byte-budgeted LRU frame cache** in front of the readers. Scrubbing
+//! back over ground just covered, or dwelling on one frame, costs a hash
+//! lookup. Frames roll into the cache as decoding passes them, so rolling
+//! forward to frame N caches N-1 frames of the path there.
+//!
+//! The pool is deliberately not used by export: export decodes every frame
+//! exactly once in order, and a cache in that path is pure overhead. This is
+//! playback infrastructure - see `docs/decisions/0002`.
+
+use std::collections::HashMap;
+use std::path::{Path, PathBuf};
+use std::sync::Arc;
+
+use relay_core::frame::Frame;
+use relay_core::time::{FrameRate, Rational};
+
+use crate::decode::{DecodeOptions, FfmpegDecoder, FrameSource};
+use crate::error::Result;
+use crate::probe;
+
+/// How far ahead of a reader's position a request may be and still be worth
+/// decoding forward to, rather than seeking. Two seconds of 30fps material is
+/// sixty decodes - cheaper than a subprocess respawn, comparable to a seek.
+const ROLL_FORWARD_FRAMES: i64 = 60;
+
+/// What one pooled frame request asks for.
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+struct FrameKey {
+ path: PathBuf,
+ width: u32,
+ height: u32,
+ /// The clip's effect chain, part of the frame's identity: the same source
+ /// frame through two different chains is two different pictures.
+ chain: Option<String>,
+ index: i64,
+}
+
+/// A byte-budgeted LRU of decoded frames.
+///
+/// Plain and measurable on purpose: a `HashMap` plus a logical clock, evicting
+/// the least-recently-touched frame until the budget holds. At preview sizes a
+/// frame is ~2-4 MB, so the default budget holds a few hundred frames - many
+/// seconds of scrub history.
+pub struct FrameCache {
+ budget: usize,
+ held: usize,
+ tick: u64,
+ frames: HashMap<FrameKey, (Arc<Frame>, u64)>,
+}
+
+impl FrameCache {
+ /// An empty cache holding at most `budget` bytes of frames.
+ pub fn new(budget: usize) -> Self {
+ Self { budget, held: 0, tick: 0, frames: HashMap::new() }
+ }
+
+ fn get(&mut self, key: &FrameKey) -> Option<Arc<Frame>> {
+ self.tick += 1;
+ let tick = self.tick;
+ self.frames.get_mut(key).map(|(frame, touched)| {
+ *touched = tick;
+ Arc::clone(frame)
+ })
+ }
+
+ fn insert(&mut self, key: FrameKey, frame: Arc<Frame>) {
+ let bytes = frame.pixels().len();
+ // A frame larger than the whole budget would evict everything and
+ // still not fit; hold it once without caching it.
+ if bytes > self.budget {
+ return;
+ }
+ self.tick += 1;
+ if let Some((previous, _)) = self.frames.insert(key, (frame, self.tick)) {
+ self.held -= previous.pixels().len();
+ }
+ self.held += bytes;
+ while self.held > self.budget {
+ let Some(oldest) = self
+ .frames
+ .iter()
+ .min_by_key(|(_, (_, touched))| *touched)
+ .map(|(key, _)| key.clone())
+ else {
+ break;
+ };
+ if let Some((evicted, _)) = self.frames.remove(&oldest) {
+ self.held -= evicted.pixels().len();
+ }
+ }
+ }
+
+ /// Bytes currently held.
+ pub fn held(&self) -> usize {
+ self.held
+ }
+
+ /// Frames currently held.
+ pub fn len(&self) -> usize {
+ self.frames.len()
+ }
+
+ /// True when nothing is cached.
+ pub fn is_empty(&self) -> bool {
+ self.frames.is_empty()
+ }
+}
+
+/// How a reader should satisfy a request for `target`, given where it is.
+///
+/// Pure, so the seek-versus-roll policy is testable without a decoder: the
+/// bug this class of code grows is "scrubbing backwards is mysteriously slow",
+/// and that is a policy bug, not a decoder bug.
+#[derive(Clone, Copy, PartialEq, Eq, Debug)]
+enum Access {
+ /// Decode forward this many frames from the current position.
+ Roll(i64),
+ /// Jump: the target is behind, or too far ahead to roll to.
+ Seek,
+}
+
+fn plan_access(current_next: Option<i64>, target: i64) -> Access {
+ match current_next {
+ Some(next) if target >= next && target - next <= ROLL_FORWARD_FRAMES => {
+ Access::Roll(target - next)
+ }
+ _ => Access::Seek,
+ }
+}
+
+/// One warm reader: a decoder plus where its *next* frame will land.
+struct Reader {
+ backend: Backend,
+ /// The frame index `next_frame` will produce, in the media's own rate.
+ next_index: i64,
+}
+
+enum Backend {
+ /// Frame-accurate seeks, real timestamps. Only with the `ffi` feature.
+ #[cfg(feature = "ffi")]
+ Linked(crate::ffi::FfiDecoder),
+ /// The subprocess pipe: no seeking, so "seek" means respawn at the
+ /// target's keyframe-fast `-ss` and count ordinally from there. Exact for
+ /// constant-frame-rate material, the pipe's known limit otherwise.
+ Pipe { decoder: FfmpegDecoder, path: PathBuf, width: u32, height: u32, chain: Option<String> },
+}
+
+impl Reader {
+ fn open(
+ path: &Path,
+ width: u32,
+ height: u32,
+ chain: Option<&str>,
+ rate: FrameRate,
+ index: i64,
+ ) -> Result<Self> {
+ #[cfg(feature = "ffi")]
+ // The linked decoder produces raw frames; a clip with an effect chain
+ // needs FFmpeg's filters, so it stays on the pipe.
+ if chain.is_none() {
+ if let Ok(mut linked) = crate::ffi::FfiDecoder::open(path, width, height) {
+ use crate::decode::SeekableSource;
+ linked.seek(rate.time_of_frame(index))?;
+ // The linked decoder lands on the keyframe at or before the
+ // target; rolling forward from there is what `frame_at` does,
+ // guided by real timestamps. Report where decoding resumes.
+ return Ok(Self { backend: Backend::Linked(linked), next_index: i64::MIN });
+ }
+ // A file the linked build cannot open falls through to the pipe.
+ }
+
+ let mut options = DecodeOptions::default()
+ .starting_at(rate.time_of_frame(index))
+ .scaled_to(width, height)
+ .at_rate(rate);
+ if let Some(chain) = chain {
+ options = options.filtered(chain);
+ }
+ let decoder = FfmpegDecoder::open(path, &options)?;
+ Ok(Self {
+ backend: Backend::Pipe {
+ decoder,
+ path: path.to_path_buf(),
+ width,
+ height,
+ chain: chain.map(str::to_owned),
+ },
+ next_index: index,
+ })
+ }
+
+ #[cfg_attr(not(feature = "ffi"), allow(unused_variables))]
+ fn next_frame(&mut self, rate: FrameRate) -> Result<Option<(i64, Frame)>> {
+ match &mut self.backend {
+ #[cfg(feature = "ffi")]
+ Backend::Linked(decoder) => {
+ let Some(frame) = decoder.next_frame()? else { return Ok(None) };
+ // The linked decoder tells us where the frame really sits; a
+ // half-frame nudge keeps exact boundary timestamps from
+ // rounding down a frame.
+ let index = decoder
+ .position()
+ .map(|position| rate.frame_at(position + rate.frame_duration() / Rational::from_int(2)))
+ .unwrap_or(self.next_index.max(0));
+ self.next_index = index + 1;
+ Ok(Some((index, frame)))
+ }
+ Backend::Pipe { decoder, .. } => {
+ let Some(frame) = decoder.next_frame()? else { return Ok(None) };
+ let index = self.next_index;
+ self.next_index = index + 1;
+ Ok(Some((index, frame)))
+ }
+ }
+ }
+
+ fn seek(&mut self, rate: FrameRate, index: i64) -> Result<()> {
+ match &mut self.backend {
+ #[cfg(feature = "ffi")]
+ Backend::Linked(decoder) => {
+ use crate::decode::SeekableSource;
+ decoder.seek(rate.time_of_frame(index))?;
+ self.next_index = i64::MIN;
+ Ok(())
+ }
+ Backend::Pipe { decoder, path, width, height, chain } => {
+ // The pipe cannot seek; a new process at the target is the
+ // seek. This is the cost the FFI backend exists to remove.
+ let mut options = DecodeOptions::default()
+ .starting_at(rate.time_of_frame(index))
+ .scaled_to(*width, *height)
+ .at_rate(rate);
+ if let Some(chain) = chain {
+ options = options.filtered(chain.clone());
+ }
+ *decoder = FfmpegDecoder::open(path.as_path(), &options)?;
+ self.next_index = index;
+ Ok(())
+ }
+ }
+ }
+}
+
+/// The media facts the pool needs per file, probed once.
+struct MediaFacts {
+ rate: FrameRate,
+ /// A still image: one frame, served for every requested time.
+ still: bool,
+}
+
+/// Random access to frames across many files, cache in front, warm readers
+/// behind. `&mut self` throughout: callers serialise access, which is also
+/// the useful property - one scrub request at a time, in order.
+pub struct ReaderPool {
+ cache: FrameCache,
+ readers: HashMap<(PathBuf, u32, u32, Option<String>), Reader>,
+ facts: HashMap<PathBuf, MediaFacts>,
+ /// Readers kept warm before the least-recently-used is dropped.
+ max_readers: usize,
+ /// Recency order for reader eviction, oldest first.
+ reader_order: Vec<(PathBuf, u32, u32, Option<String>)>,
+}
+
+impl ReaderPool {
+ /// A pool with `cache_bytes` of frame cache and up to `max_readers` warm
+ /// decoders.
+ pub fn new(cache_bytes: usize, max_readers: usize) -> Self {
+ Self {
+ cache: FrameCache::new(cache_bytes),
+ readers: HashMap::new(),
+ facts: HashMap::new(),
+ max_readers: max_readers.max(1),
+ reader_order: Vec::new(),
+ }
+ }
+
+ /// 512 MB of frames, eight warm readers - enough for a busy timeline.
+ pub fn with_defaults() -> Self {
+ Self::new(512 * 1024 * 1024, 8)
+ }
+
+ /// The frame of `path` on screen at `time` (in the media's own clock),
+ /// scaled to `width` x `height`.
+ ///
+ /// `still` marks image files: a one-frame stream served for every time,
+ /// which the caller knows (from its own model) and the pool must not
+ /// guess from probing.
+ pub fn frame_at(
+ &mut self,
+ path: &Path,
+ time: Rational,
+ width: u32,
+ height: u32,
+ still: bool,
+ chain: Option<&str>,
+ ) -> Result<Arc<Frame>> {
+ let facts = self.facts_for(path, still)?;
+ let rate = facts.rate;
+ let target = if facts.still { 0 } else { rate.frame_at(time).max(0) };
+ let chain_key = chain.map(str::to_owned);
+
+ let key = FrameKey {
+ path: path.to_path_buf(),
+ width,
+ height,
+ chain: chain_key.clone(),
+ index: target,
+ };
+ if let Some(frame) = self.cache.get(&key) {
+ return Ok(frame);
+ }
+
+ self.touch_reader(path, width, height, chain, rate, target)?;
+ let reader_key = (path.to_path_buf(), width, height, chain_key.clone());
+ let reader = self.readers.get_mut(&reader_key).expect("just ensured");
+
+ // Roll forward to the target, caching everything passed on the way -
+ // the next scrub over this span is then free.
+ let mut latest: Option<Arc<Frame>> = None;
+ loop {
+ match reader.next_frame(rate)? {
+ Some((index, frame)) => {
+ let frame = Arc::new(frame);
+ self.cache.insert(
+ FrameKey {
+ path: path.to_path_buf(),
+ width,
+ height,
+ chain: chain_key.clone(),
+ index,
+ },
+ Arc::clone(&frame),
+ );
+ latest = Some(frame);
+ if index >= target {
+ break;
+ }
+ }
+ // End of stream before the target: a clip trimmed past its
+ // media's end. The last real frame is the honest answer.
+ None => break,
+ }
+ }
+
+ latest.ok_or_else(|| crate::error::Error::PartialFrame {
+ path: path.to_path_buf(),
+ got: 0,
+ want: Frame::byte_len(width, height),
+ })
+ }
+
+ fn facts_for(&mut self, path: &Path, still: bool) -> Result<&MediaFacts> {
+ if !self.facts.contains_key(path) {
+ let facts = if still {
+ MediaFacts { rate: FrameRate::THIRTY, still: true }
+ } else {
+ let info = probe::probe(path)?;
+ let rate = info
+ .video
+ .as_ref()
+ .map(|video| video.frame_rate)
+ .unwrap_or(FrameRate::THIRTY);
+ MediaFacts { rate, still: false }
+ };
+ self.facts.insert(path.to_path_buf(), facts);
+ }
+ Ok(self.facts.get(path).expect("just inserted"))
+ }
+
+ /// Ensures a reader positioned to reach `target`, evicting the coldest
+ /// reader when the pool is full.
+ fn touch_reader(
+ &mut self,
+ path: &Path,
+ width: u32,
+ height: u32,
+ chain: Option<&str>,
+ rate: FrameRate,
+ target: i64,
+ ) -> Result<()> {
+ let key = (path.to_path_buf(), width, height, chain.map(str::to_owned));
+
+ self.reader_order.retain(|entry| entry != &key);
+ self.reader_order.push(key.clone());
+
+ if let Some(reader) = self.readers.get_mut(&key) {
+ let next = (reader.next_index != i64::MIN).then_some(reader.next_index);
+ if plan_access(next, target) == Access::Seek {
+ reader.seek(rate, target)?;
+ }
+ return Ok(());
+ }
+
+ while self.readers.len() >= self.max_readers && !self.reader_order.is_empty() {
+ let coldest = self.reader_order.remove(0);
+ if coldest == key {
+ self.reader_order.push(coldest);
+ break;
+ }
+ self.readers.remove(&coldest);
+ }
+
+ let reader = Reader::open(path, width, height, chain, rate, target)?;
+ self.readers.insert(key, reader);
+ Ok(())
+ }
+
+ /// Drops every warm reader and cached frame - for when the media set
+ /// changes wholesale, like closing a project.
+ pub fn clear(&mut self) {
+ self.readers.clear();
+ self.reader_order.clear();
+ self.facts.clear();
+ self.cache = FrameCache::new(self.cache.budget);
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::encode::{EncodeOptions, FfmpegEncoder, FrameSink};
+
+ #[test]
+ fn the_access_policy_rolls_forward_and_seeks_backward() {
+ assert_eq!(plan_access(Some(10), 10), Access::Roll(0), "the very next frame");
+ assert_eq!(plan_access(Some(10), 30), Access::Roll(20), "a short hop ahead");
+ assert_eq!(plan_access(Some(10), 9), Access::Seek, "behind means seek");
+ assert_eq!(
+ plan_access(Some(10), 10 + ROLL_FORWARD_FRAMES + 1),
+ Access::Seek,
+ "too far ahead means seek"
+ );
+ assert_eq!(plan_access(None, 5), Access::Seek, "unknown position means seek");
+ }
+
+ #[test]
+ fn the_cache_evicts_least_recently_used_within_budget() {
+ // Frames are 16 bytes each (2x2); budget holds exactly two.
+ let mut cache = FrameCache::new(32);
+ let key = |index: i64| FrameKey {
+ path: PathBuf::from("a.mp4"),
+ width: 2,
+ height: 2,
+ chain: None,
+ index,
+ };
+ cache.insert(key(0), Arc::new(Frame::black(2, 2)));
+ cache.insert(key(1), Arc::new(Frame::black(2, 2)));
+ assert_eq!(cache.len(), 2);
+
+ // Touch 0 so 1 is the coldest, then overflow.
+ assert!(cache.get(&key(0)).is_some());
+ cache.insert(key(2), Arc::new(Frame::black(2, 2)));
+
+ assert_eq!(cache.len(), 2);
+ assert!(cache.get(&key(0)).is_some(), "recently touched survives");
+ assert!(cache.get(&key(1)).is_none(), "coldest was evicted");
+ assert!(cache.get(&key(2)).is_some());
+ assert_eq!(cache.held(), 32);
+ }
+
+ #[test]
+ fn an_oversized_frame_is_served_but_never_cached() {
+ let mut cache = FrameCache::new(8);
+ cache.insert(
+ FrameKey { path: PathBuf::from("a"), width: 2, height: 2, chain: None, index: 0 },
+ Arc::new(Frame::black(2, 2)),
+ );
+ assert!(cache.is_empty(), "16 bytes cannot fit an 8 byte budget");
+ }
+
+ /// End to end against a real FFmpeg: encode a tiny video whose frames are
+ /// identifiable by colour, then read arbitrary frames back out of order.
+ /// Skips silently on machines without FFmpeg, like the encoder's own test.
+ #[test]
+ fn random_access_returns_the_right_frames() {
+ let path = std::env::temp_dir().join("relay-pool-test.mp4");
+ let Ok(mut encoder) =
+ FfmpegEncoder::create(&path, 64, 64, FrameRate::THIRTY, &EncodeOptions::default())
+ else {
+ return; // no ffmpeg here
+ };
+ // 90 frames; the red channel encodes the frame index (x2 to survive
+ // compression rounding).
+ for index in 0..90u32 {
+ let mut frame = Frame::black(64, 64);
+ frame.fill([(index * 2).min(255) as u8, 40, 40, 255]);
+ encoder.write_frame(&frame).expect("writes");
+ }
+ encoder.finish().expect("finishes");
+
+ let mut pool = ReaderPool::new(64 * 1024 * 1024, 4);
+ let rate = FrameRate::THIRTY;
+ let red_at = |pool: &mut ReaderPool, index: i64| -> i64 {
+ let frame = pool
+ .frame_at(&path, rate.time_of_frame(index), 64, 64, false, None)
+ .expect("frame decodes");
+ i64::from(frame.pixel(32, 32).expect("in bounds")[0])
+ };
+
+ // Forward, backward, far jump, and revisit - the scrub shapes.
+ let near = |got: i64, index: i64| (got - index * 2).abs() <= 8;
+ let at_10 = red_at(&mut pool, 10);
+ assert!(near(at_10, 10), "frame 10 read {at_10}");
+ let at_50 = red_at(&mut pool, 50);
+ assert!(near(at_50, 50), "frame 50 read {at_50}");
+ let back_at_20 = red_at(&mut pool, 20);
+ assert!(near(back_at_20, 20), "backward to 20 read {back_at_20}");
+ let again_at_50 = red_at(&mut pool, 50);
+ assert_eq!(again_at_50, at_50, "revisit must come from cache, identical");
+ assert!(pool.cache.len() > 0, "rolling forward populated the cache");
+
+ let _ = std::fs::remove_file(&path);
+ }
+}
diff --git a/engine/docs/decisions/0002-ffmpeg-over-a-pipe.md b/engine/docs/decisions/0002-ffmpeg-over-a-pipe.md
index f5873b7c..c204a409 100644
--- a/engine/docs/decisions/0002-ffmpeg-over-a-pipe.md
+++ b/engine/docs/decisions/0002-ffmpeg-over-a-pipe.md
@@ -82,9 +82,20 @@ The seam held: adding it required one new type implementing an existing trait,
one new trait for the capability the pipe genuinely lacks, and no change to
`relay-core`, `relay-render`, or any caller.
+## Status: the reader pool exists too
+
+`relay_media::pool` is the per-media reader pool with a byte-budgeted LRU
+frame cache that playback was waiting on: warm readers roll forward for near
+requests and seek for far ones - frame-accurately through `FfiDecoder` when
+the `ffi` feature is on, by respawn on the pipe otherwise. Its first two
+consumers are live: `relay-cli render` (whose founding "deliberate shortcut"
+comment is finally deleted) and the host's `preview_frame` command, which
+composites the true frame for the paused monitor.
+
## What would change our mind
-Nothing left to decide about *whether*. What remains is wiring playback to the
-linked decoder, at which point scrubbing stops respawning a process per seek.
+Nothing left to decide about *whether*. What remains is streaming playback -
+presenting pooled frames continuously against the transport clock rather than
+one paused frame at a time.
Do **not** migrate probe or export along with it - for those, the tradeoff
never flips.