Skip to content

The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.

updates

Jareerce8a64c4b904
diff --git a/.gitignore b/.gitignore
index 3ef258a6..f4d43db9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -58,3 +58,7 @@ desktop.ini
# Local environment
.env
.env.local
+
+# FFmpeg development build for the FFI decoder (headers + import libs).
+# Large, machine-specific, and not needed for a normal build - see BUILD.md.
+/vendor/
diff --git a/BUILD.md b/BUILD.md
new file mode 100644
index 00000000..1e2ef08e
--- /dev/null
+++ b/BUILD.md
@@ -0,0 +1,110 @@
+# Building Relay
+
+Two independent pieces:
+
+- **`engine/`** — the Rust video engine. Builds and tests on its own, with no UI
+ and no Node involved.
+- **`desktop/`** — the Tauri + React editor. Depends on the engine by path.
+
+You can work on the engine without ever touching the desktop app.
+
+## What you need
+
+| | Version | Why | Check |
+|---|---|---|---|
+| **Rust** | 1.93 | Pinned by `engine/rust-toolchain.toml`; rustup installs it automatically | `rustc --version` |
+| **Node** | 22+ | Vite 7 and the Tauri CLI | `node --version` |
+| **FFmpeg** | 7+ | Runtime dependency — the engine shells out to it | `ffmpeg -version` |
+| **MSVC Build Tools** | 2022 | Windows only; Rust's default toolchain links with it | — |
+
+FFmpeg must be on `PATH` as both `ffmpeg` **and** `ffprobe`. Relay does not
+bundle it. Without it the app starts fine and then fails on first import with
+"could not run `ffprobe` - is FFmpeg installed and on PATH?".
+
+```
+winget install Gyan.FFmpeg # Windows
+brew install ffmpeg # macOS
+sudo apt install ffmpeg # Debian/Ubuntu
+```
+
+## Engine
+
+```sh
+cd engine
+cargo test --workspace # ~60 tests, none of them need FFmpeg
+cargo clippy --workspace --all-targets
+cargo run -p relay-cli -- probe some-video.mp4
+cargo run -p relay-cli -- render in.mp4 out.mp4 --frames 120 --fade 30
+```
+
+The tests are pure — they exercise the timeline model, rational time and the
+compositor, and pass on a machine with no FFmpeg at all. The CLI is how you
+exercise the parts that do need it.
+
+## Desktop app
+
+```sh
+cd desktop
+npm install
+npm run app # dev: builds the Rust host, opens the window, HMR for the UI
+npm run typecheck # tsc --noEmit
+npm run build # UI only, into dist/
+npm run app:build # production installers
+```
+
+**First `npm run app` takes several minutes** — it compiles ~360 Rust crates.
+After that only your own crates rebuild, so it is seconds. Editing anything
+under `src/` hot-reloads without a Rust rebuild at all.
+
+`npm run app:build` produces, under `desktop/src-tauri/target/release/`:
+
+```
+relay-desktop.exe standalone, no install needed
+bundle/nsis/Relay_0.1.0_x64-setup.exe installer
+bundle/msi/Relay_0.1.0_x64_en-US.msi installer
+```
+
+Release builds use `lto = true` and `codegen-units = 1`, so expect them to be
+several times slower than dev. That is deliberate; the final link is
+single-threaded and is most of the wait.
+
+## `vendor/` — not required
+
+Holds an FFmpeg **development** build (headers and import libraries), used only
+by the in-progress FFI decoder described below. It is gitignored, roughly
+150 MB, and nothing in the normal build path touches it. Delete it freely.
+
+## FFI decoder — in progress, not yet wired up
+
+The engine currently runs FFmpeg as a subprocess
+(`engine/docs/decisions/0002-ffmpeg-over-a-pipe.md`). That is permanent for
+probing and export, but playback needs real timestamps and frame-accurate
+seeking, which a pipe cannot provide — so a linked decoder is being added
+behind the existing `FrameSource` trait.
+
+Extra prerequisites for that path only:
+
+- **LLVM / libclang** — `bindgen` generates the FFmpeg bindings at build time.
+ `winget install LLVM.LLVM`
+- **An FFmpeg dev build** — headers plus `.lib` import libraries, which the
+ ordinary runtime download does not include. Get a `*-gpl-shared-*` archive
+ from [BtbN/FFmpeg-Builds](https://github.com/BtbN/FFmpeg-Builds/releases) and
+ unpack it into `vendor/`.
+- Environment variables pointing at it (see `engine/crates/relay-media/README`
+ once this lands).
+
+This section will be replaced with exact, verified steps when the decoder
+actually builds. Until then, ignore it — the app does not need any of it.
+
+## Troubleshooting
+
+**"could not run `ffprobe`"** — FFmpeg is not on `PATH`. Restart the terminal
+after installing; a running shell keeps its old `PATH`.
+
+**Rust rebuilds everything on each `npm run app`** — something is changing
+`engine/`'s files. The desktop app depends on the engine by path, so a touched
+engine file invalidates the host too.
+
+**The window opens white** — the Vite dev server did not start. Check that port
+1420 is free; Tauri is configured with `strictPort`, so it will not silently
+pick another.
diff --git a/desktop/src-tauri/src/export.rs b/desktop/src-tauri/src/export.rs
index a6c78f3f..c0fb9f4f 100644
--- a/desktop/src-tauri/src/export.rs
+++ b/desktop/src-tauri/src/export.rs
@@ -282,7 +282,7 @@ fn quantise(seconds: f64, rate: FrameRate) -> Rational {
/// Trims, delays and mixes every audible clip into one track.
fn mix_audio(clips: &[&ExportClip], duration: f64, destination: &Path) -> Result<(), String> {
- let mut command = std::process::Command::new("ffmpeg");
+ let mut command = std::process::Command::new(relay_media::ffmpeg());
command.args(["-hide_banner", "-nostdin", "-loglevel", "error", "-y"]);
for clip in clips {
@@ -339,7 +339,7 @@ fn mix_audio(clips: &[&ExportClip], duration: f64, destination: &Path) -> Result
}
fn mux(video: &Path, audio: &Path, output: &Path) -> Result<(), String> {
- let mut command = std::process::Command::new("ffmpeg");
+ let mut command = std::process::Command::new(relay_media::ffmpeg());
command
.args(["-hide_banner", "-nostdin", "-loglevel", "error", "-y"])
.arg("-i")
diff --git a/desktop/src-tauri/src/lib.rs b/desktop/src-tauri/src/lib.rs
index f3803b04..9180b523 100644
--- a/desktop/src-tauri/src/lib.rs
+++ b/desktop/src-tauri/src/lib.rs
@@ -169,7 +169,7 @@ fn filmstrip(path: &str, count: u32, height: u32) -> Result<Vec<u8>, String> {
.filter(|seconds| *seconds > 0.0)
.ok_or_else(|| format!("{path} reports no duration"))?;
- let output = std::process::Command::new("ffmpeg")
+ let output = std::process::Command::new(relay_media::ffmpeg())
.args(["-hide_banner", "-nostdin", "-loglevel", "error"])
.args(["-i", path])
.args([
@@ -285,9 +285,33 @@ fn describe(error: relay_media::Error) -> String {
message
}
+/// Points the engine at the bundled FFmpeg, if there is one.
+///
+/// Tauri drops sidecar binaries next to the app executable with the target
+/// triple stripped, so this looks for them there and silently leaves the
+/// engine on its `PATH` default otherwise. That fallback is what keeps
+/// `cargo run` working without copying binaries into the build directory.
+///
+/// Both must be present to switch: a bundled decoder paired with whatever
+/// `ffprobe` happens to be on PATH is a version mismatch waiting to happen.
+fn use_bundled_ffmpeg() {
+ let Ok(executable) = std::env::current_exe() else { return };
+ let Some(directory) = executable.parent() else { return };
+
+ let suffix = if cfg!(windows) { ".exe" } else { "" };
+ let ffmpeg = directory.join(format!("ffmpeg{suffix}"));
+ let ffprobe = directory.join(format!("ffprobe{suffix}"));
+
+ if ffmpeg.is_file() && ffprobe.is_file() {
+ relay_media::set_binaries(ffmpeg, ffprobe);
+ }
+}
+
/// Builds and runs the editor window.
#[cfg_attr(mobile, tauri::mobile_entry_point)]
pub fn run() {
+ use_bundled_ffmpeg();
+
tauri::Builder::default()
.plugin(tauri_plugin_opener::init())
.plugin(tauri_plugin_dialog::init())
diff --git a/desktop/src/App.tsx b/desktop/src/App.tsx
index 6546a85b..749d8cec 100644
--- a/desktop/src/App.tsx
+++ b/desktop/src/App.tsx
@@ -219,9 +219,9 @@ function Editor({
clipId: clip.id,
path: media.path,
time: clip.sourceStart + (playhead - clip.start),
- // Media elements cannot play above unity, so a boost is audible
- // only in the export. The Adjust panel says so where it matters.
- volume: Math.min(1, clipGainAt(clip, playhead)),
+ // Uncapped: the preview routes through a gain node, so what you
+ // hear matches what the exporter will render.
+ volume: clipGainAt(clip, playhead),
},
];
});
@@ -248,6 +248,7 @@ function Editor({
path: media.path,
time: top.sourceStart + (playhead - top.start),
muted: false,
+ volume: clipGainAt(top, playhead),
isStill: top.kind === "image",
};
}, [project, playhead]);
diff --git a/desktop/src/components/AdjustPanel.tsx b/desktop/src/components/AdjustPanel.tsx
index dd4b7506..5e626cb3 100644
--- a/desktop/src/components/AdjustPanel.tsx
+++ b/desktop/src/components/AdjustPanel.tsx
@@ -3,24 +3,47 @@ import { Group, Slider } from "./controls";
import { Icon } from "./Icon";
import { Empty } from "./Panel";
-/** Above unity the preview cannot follow; see the note in the panel. */
-const MAX_GAIN = 2;
-/** A fade longer than the clip is meaningless, and this keeps it obvious. */
-const MAX_FADE = 5;
+/**
+ * Level range, in decibels.
+ *
+ * The bottom is silence, not -60: a fader that cannot reach zero is a fader
+ * you cannot use to mute. +24 at the top because quiet material genuinely
+ * needs it - a distant lavalier or a phone recording is routinely 15-20 dB
+ * under, and a limit that cannot fix those is a limit in the wrong place.
+ */
+const MIN_DB = -60;
+const MAX_DB = 24;
+/** A fade longer than half the clip would overlap the other one. */
+const MAX_FADE = 10;
+
+/** Linear gain to decibels. Zero maps to the bottom of the fader, not -inf. */
+function toDecibels(gain: number): number {
+ if (gain <= 0) return MIN_DB;
+ return Math.max(MIN_DB, Math.min(MAX_DB, 20 * Math.log10(gain)));
+}
+
+function fromDecibels(decibels: number): number {
+ // At the very bottom the fader means silence, so snap to exactly zero
+ // rather than leaving an inaudible-but-nonzero gain in the project.
+ return decibels <= MIN_DB ? 0 : 10 ** (decibels / 20);
+}
-/** Linear gain as decibels, which is the unit anyone mixing audio thinks in. */
-function decibels(gain: number): string {
- if (gain <= 0.0001) return "-inf dB";
- const value = 20 * Math.log10(gain);
- return `${value > 0 ? "+" : ""}${value.toFixed(1)} dB`;
+function formatDecibels(decibels: number): string {
+ if (decibels <= MIN_DB) return "silent";
+ if (Math.abs(decibels) < 0.05) return "0.0 dB";
+ return `${decibels > 0 ? "+" : ""}${decibels.toFixed(1)} dB`;
}
/**
* The Adjust tab: properties of the selected clip that change how it plays.
*
- * Everything here is applied in two places - the preview, so you can hear it
- * now, and the exporter, so the file matches. Both read the same numbers off
- * the clip, which is the only way those two can be guaranteed to agree.
+ * Everything here applies in two places - the preview, so you hear it now, and
+ * the exporter, so the file matches. Both read the same numbers off the clip,
+ * which is the only way those two can be guaranteed to agree.
+ *
+ * The fader works in decibels rather than linear gain. Linear puts every
+ * useful setting in the bottom sixth of the travel; decibels is both how the
+ * ear works and how anyone mixing thinks.
*/
export function AdjustPanel({
clip,
@@ -37,62 +60,58 @@ export function AdjustPanel({
);
}
- const hasSound = clip.kind !== "image";
- // A fade cannot be longer than half the clip without the two overlapping.
- const fadeLimit = Math.min(MAX_FADE, clip.duration / 2);
+ if (clip.kind === "image") {
+ return (
+ <Empty icon={<Icon name="image" size={26} strokeWidth={1.5} />}>
+ A still has nothing to adjust yet. Opacity and transform land with the
+ compositor.
+ </Empty>
+ );
+ }
+
+ const fadeLimit = Math.max(0.1, Math.min(MAX_FADE, clip.duration / 2));
return (
<div className="px-3 py-3">
- {hasSound ? (
- <>
- <Group title="Volume">
- <Slider
- label="Level"
- value={clip.volume}
- min={0}
- max={MAX_GAIN}
- step={0.01}
- format={decibels}
- onReset={() => onChange({ volume: 1 })}
- onChange={(volume) => onChange({ volume })}
- />
- {clip.volume > 1 && (
- <p className="-mt-1 mb-1 text-[11px] leading-snug text-tertiary">
- Boost above 0 dB applies on export. The preview cannot play louder than
- unity, so it will sound quieter than the exported file.
- </p>
- )}
- </Group>
+ <Group title="Volume">
+ <Slider
+ label="Level"
+ value={toDecibels(clip.volume)}
+ min={MIN_DB}
+ max={MAX_DB}
+ step={0.5}
+ format={formatDecibels}
+ onReset={() => onChange({ volume: 1 })}
+ onChange={(decibels) => onChange({ volume: fromDecibels(decibels) })}
+ />
+ </Group>
+
+ <Group title="Fades">
+ <Slider
+ label="Fade in"
+ value={Math.min(clip.fadeIn, fadeLimit)}
+ min={0}
+ max={fadeLimit}
+ step={0.05}
+ format={(value) => (value === 0 ? "none" : `${value.toFixed(2)}s`)}
+ onReset={() => onChange({ fadeIn: 0 })}
+ onChange={(fadeIn) => onChange({ fadeIn })}
+ />
+ <Slider
+ label="Fade out"
+ value={Math.min(clip.fadeOut, fadeLimit)}
+ min={0}
+ max={fadeLimit}
+ step={0.05}
+ format={(value) => (value === 0 ? "none" : `${value.toFixed(2)}s`)}
+ onReset={() => onChange({ fadeOut: 0 })}
+ onChange={(fadeOut) => onChange({ fadeOut })}
+ />
+ </Group>
- <Group title="Fades">
- <Slider
- label="Fade in"
- value={Math.min(clip.fadeIn, fadeLimit)}
- min={0}
- max={fadeLimit}
- step={0.05}
- format={(value) => (value === 0 ? "none" : `${value.toFixed(2)}s`)}
- onReset={() => onChange({ fadeIn: 0 })}
- onChange={(fadeIn) => onChange({ fadeIn })}
- />
- <Slider
- label="Fade out"
- value={Math.min(clip.fadeOut, fadeLimit)}
- min={0}
- max={fadeLimit}
- step={0.05}
- format={(value) => (value === 0 ? "none" : `${value.toFixed(2)}s`)}
- onReset={() => onChange({ fadeOut: 0 })}
- onChange={(fadeOut) => onChange({ fadeOut })}
- />
- </Group>
- </>
- ) : (
- <Empty icon={<Icon name="image" size={26} strokeWidth={1.5} />}>
- A still has nothing to adjust yet. Opacity and transform land with the
- compositor.
- </Empty>
- )}
+ <p className="text-[11px] leading-relaxed text-tertiary">
+ Double-click a label to reset it.
+ </p>
</div>
);
}
diff --git a/desktop/src/components/Preview.tsx b/desktop/src/components/Preview.tsx
index 75aadf26..b67ce9b4 100644
--- a/desktop/src/components/Preview.tsx
+++ b/desktop/src/components/Preview.tsx
@@ -1,6 +1,7 @@
import { useEffect, useRef } from "react";
import { convertFileSrc } from "@tauri-apps/api/core";
+import { connectElement } from "../lib/audio";
import { timecode } from "../lib/time";
import { Icon, IconButton } from "./Icon";
import { PANEL_SHELL } from "./Panel";
@@ -12,6 +13,8 @@ export interface PreviewSource {
/** Where in the source file the playhead sits, in seconds. */
time: number;
muted: boolean;
+ /** Linear gain with fades applied. May exceed 1; see lib/audio.ts. */
+ volume: number;
/** A still is shown as an image; there is nothing to seek or play. */
isStill: boolean;
}
@@ -82,6 +85,15 @@ export function Preview({
element.muted = source.muted;
+ // Through a gain node, not element.volume, which browsers cap at 1.0.
+ // A video clip's audio obeys the same clip gain as an audio clip's.
+ try {
+ connectElement(element).gain.value = Math.max(0, source.volume);
+ } catch {
+ // Routing can only fail if the element was already claimed by another
+ // graph, which cannot happen here - but silence is not worth a crash.
+ }
+
const tolerance = playing ? 0.3 : 0.03;
if (element.readyState > 0 && Math.abs(element.currentTime - source.time) > tolerance) {
try {
diff --git a/desktop/src/components/TimelinePanel.tsx b/desktop/src/components/TimelinePanel.tsx
index ec0b8245..37bc3d12 100644
--- a/desktop/src/components/TimelinePanel.tsx
+++ b/desktop/src/components/TimelinePanel.tsx
@@ -1097,6 +1097,10 @@ function drawClip(
}
}
+ if (clip.fadeIn > 0 || clip.fadeOut > 0) {
+ drawFades(context, clip, x, bodyY, drawWidth, bodyHeight, secondsPerPixel);
+ }
+
context.fillStyle = palette.header;
context.fillRect(x, y, drawWidth, CLIP_HEADER);
@@ -1117,6 +1121,64 @@ function drawClip(
context.stroke();
}
+/**
+ * Draws the fade ramps as wedges over the clip body.
+ *
+ * The shaded area is the part being attenuated and the bright line is the
+ * envelope itself - the same shape every editor draws, because it reads as
+ * "this much is being taken away" at a glance.
+ *
+ * On a video clip the wedge is confined to a band at the bottom. The fade is
+ * an *audio* property, and shading the picture would say it fades to black.
+ */
+function drawFades(
+ context: CanvasRenderingContext2D,
+ clip: Clip,
+ x: number,
+ bodyY: number,
+ width: number,
+ bodyHeight: number,
+ secondsPerPixel: number,
+) {
+ const band = clip.kind === "audio" ? bodyHeight : Math.min(10, bodyHeight);
+ const top = bodyY + bodyHeight - band;
+ const bottom = bodyY + bodyHeight;
+ if (band <= 1) return;
+
+ context.save();
+ context.beginPath();
+ context.rect(x, top, width, band);
+ context.clip();
+
+ const ramp = (seconds: number, fromLeft: boolean) => {
+ const span = Math.min(seconds / secondsPerPixel, width);
+ if (span < 1) return;
+
+ const originX = fromLeft ? x : x + width;
+ const endX = fromLeft ? x + span : x + width - span;
+
+ context.fillStyle = "rgba(0,0,0,0.42)";
+ context.beginPath();
+ context.moveTo(originX, top);
+ context.lineTo(endX, top);
+ context.lineTo(originX, bottom);
+ context.closePath();
+ context.fill();
+
+ context.strokeStyle = "rgba(255,255,255,0.75)";
+ context.lineWidth = 1.5;
+ context.beginPath();
+ context.moveTo(originX, bottom);
+ context.lineTo(endX, top);
+ context.stroke();
+ };
+
+ if (clip.fadeIn > 0) ramp(clip.fadeIn, true);
+ if (clip.fadeOut > 0) ramp(clip.fadeOut, false);
+
+ context.restore();
+}
+
/**
* Tiles frames from a single strip image across the clip body.
*
@@ -1175,6 +1237,10 @@ function drawWaveform(
) {
const centre = y + height / 2;
const half = height / 2 - 1;
+ // The drawn amplitude follows the clip's gain, so turning a clip up makes
+ // its waveform visibly taller. Clamped, because a boosted waveform that
+ // overflowed its clip would bleed into the lane above.
+ const gain = Math.max(0, clip.volume);
const perPixel = Math.max(1, Math.round(secondsPerPixel * peaks.bucketsPerSecond));
const stride = Math.max(1, Math.floor(perPixel / 8));
@@ -1193,8 +1259,9 @@ function drawWaveform(
if (peaks.max[bucket] > high) high = peaks.max[bucket];
}
- const top = centre - high * half;
- context.rect(x + column, top, 1, Math.max(1, centre - low * half - top));
+ const top = centre - Math.min(1, high * gain) * half;
+ const bottom = centre - Math.max(-1, low * gain) * half;
+ context.rect(x + column, top, 1, Math.max(1, bottom - top));
}
context.fill();
diff --git a/desktop/src/lib/audio.ts b/desktop/src/lib/audio.ts
index 9bea1e19..f6e60399 100644
--- a/desktop/src/lib/audio.ts
+++ b/desktop/src/lib/audio.ts
@@ -1,28 +1,28 @@
/**
* Audio preview.
*
- * This is what makes an mp3 on the timeline actually audible. It is a
- * *preview* path, not the engine's mixer: one `HTMLAudioElement` per audio
- * clip, kept roughly in step with the transport clock.
+ * Each audible clip gets a media element for decoding, routed through a Web
+ * Audio `GainNode` for level. The gain node is the whole reason this is not
+ * just `element.volume`: that property is hard-capped at 1.0 by every browser,
+ * so a boost could not be previewed at all - it could only be promised and
+ * applied later at export, which is a useless thing to tell someone mixing.
+ * A `GainNode` has no ceiling, so what you hear is what gets rendered.
*
- * Why the webview and not Rust: real-time audio needs a callback-driven output
- * device (cpal), a resampler, and a mixer that can survive the UI thread
- * stalling. That is a genuine piece of engineering and it belongs in the
- * engine. Until it exists, the webview already has a decoder and an output
- * device, and using them is the difference between hearing your edit and not.
+ * Audio is loaded as a blob rather than through the asset protocol, and that
+ * is deliberate too. `createMediaElementSource` on a cross-origin element
+ * yields *silence* with no error - the element plays, the graph outputs
+ * nothing. A blob URL is same-origin, so the graph is guaranteed to work. The
+ * cost is holding the file in memory, which we already do for the waveform.
*
- * The known limits, so nobody mistakes this for the finished thing:
- * - Sync is corrective, not sample-accurate. Drift beyond a tolerance is
- * fixed by reseeking, which is audible if it happens often.
- * - No mixing. Overlapping clips play simultaneously and clip if loud.
- * - No effects, no fades, no per-clip gain automation.
- * - Scrubbing does not scrub audio, it repositions it.
+ * The known limits, so nobody mistakes this for a finished mixer:
+ * - Sync is corrective, not sample-accurate. Drift past a tolerance is
+ * fixed by reseeking, which is audible when it happens.
+ * - No effects and no automation beyond the clip's constant gain and fades.
+ * - Scrubbing repositions audio, it does not scrub it.
*
* See docs/decisions/0004-audio-preview-in-the-webview.md.
*/
-import { convertFileSrc } from "@tauri-apps/api/core";
-
import { readMediaBytes } from "./engine";
/** One audio clip that should be sounding right now. */
@@ -31,6 +31,7 @@ export interface Voice {
path: string;
/** Where in the source file the playhead currently sits, in seconds. */
time: number;
+ /** Linear gain, fades already applied. May exceed 1. */
volume: number;
}
@@ -39,18 +40,55 @@ export interface Voice {
*
* Generous while playing: a reseek is an audible click, and browsers do not
* keep a media element perfectly locked to an external clock anyway. Tight
- * while paused, where "seek" is the entire operation and there is no click to
- * worry about.
+ * while paused, where "seek" is the entire operation.
*/
const TOLERANCE_PLAYING = 0.3;
const TOLERANCE_PAUSED = 0.02;
+let sharedContext: AudioContext | null = null;
+
+/** The one AudioContext. Browsers cap how many a page may create. */
+export function audioContext(): AudioContext {
+ sharedContext ??= new AudioContext();
+ return sharedContext;
+}
+
+/**
+ * A context created before any user interaction starts suspended. Resuming is
+ * safe to call repeatedly and only does anything the first time.
+ */
+function wake(): void {
+ const context = audioContext();
+ if (context.state === "suspended") void context.resume().catch(() => undefined);
+}
+
+// One source node per element, forever: calling createMediaElementSource twice
+// on the same element throws, and there is no way to undo the first call.
+const routed = new WeakMap<HTMLMediaElement, GainNode>();
+
+/**
+ * Routes a media element through a gain node and returns it.
+ *
+ * Shared with the video preview, so picture and sound obey the same clip gain
+ * through the same graph.
+ */
+export function connectElement(element: HTMLMediaElement): GainNode {
+ const existing = routed.get(element);
+ if (existing) return existing;
+
+ const context = audioContext();
+ const gain = context.createGain();
+ context.createMediaElementSource(element).connect(gain);
+ gain.connect(context.destination);
+ routed.set(element, gain);
+ return gain;
+}
+
interface Loaded {
element: HTMLAudioElement;
- /** Set when we fell back to a blob, so it can be revoked on dispose. */
+ gain: GainNode;
objectUrl?: string;
- /** Guards against retrying the blob fallback in a loop. */
- triedFallback: boolean;
+ loading: boolean;
}
export class AudioPreview {
@@ -59,18 +97,15 @@ export class AudioPreview {
/**
* Brings the audio output in line with the transport.
*
- * Call this whenever the playhead, the play state, or the set of audible
- * clips changes. It is idempotent - calling it every animation frame is fine
- * and is in fact how drift gets corrected.
+ * Idempotent - calling it every animation frame is fine, and is in fact how
+ * drift gets corrected.
*/
sync(active: Voice[], playing: boolean): void {
- const wanted = new Set(active.map((voice) => voice.clipId));
+ if (playing) wake();
- // Silence anything that has fallen out from under the playhead.
+ const wanted = new Set(active.map((voice) => voice.clipId));
for (const [clipId, loaded] of this.voices) {
- if (!wanted.has(clipId) && !loaded.element.paused) {
- loaded.element.pause();
- }
+ if (!wanted.has(clipId) && !loaded.element.paused) loaded.element.pause();
}
const tolerance = playing ? TOLERANCE_PLAYING : TOLERANCE_PAUSED;
@@ -78,32 +113,28 @@ export class AudioPreview {
for (const voice of active) {
const loaded = this.load(voice.clipId, voice.path);
const element = loaded.element;
- element.volume = Math.max(0, Math.min(1, voice.volume));
- // readyState 0 means metadata has not arrived; setting currentTime now
- // throws in some engines and is silently discarded in others.
+ // No clamp. A boost is meant to be audible.
+ loaded.gain.gain.value = Math.max(0, voice.volume);
+
+ // readyState 0 means metadata has not arrived; setting currentTime then
+ // throws in some engines and is discarded in others.
if (element.readyState > 0 && Math.abs(element.currentTime - voice.time) > tolerance) {
try {
element.currentTime = Math.max(0, voice.time);
} catch {
- // Seeking an element that is still opening is not an error worth
- // surfacing; the next sync will land it.
+ // Still opening; the next sync lands it.
}
}
if (playing) {
- if (element.paused) {
- // Rejects when the element is not ready or autoplay is refused.
- // Either way the next sync retries, so there is nothing to handle.
- void element.play().catch(() => undefined);
- }
+ if (element.paused) void element.play().catch(() => undefined);
} else if (!element.paused) {
element.pause();
}
}
}
- /** Stops everything, e.g. when the transport stops. */
stopAll(): void {
for (const loaded of this.voices.values()) loaded.element.pause();
}
@@ -112,9 +143,11 @@ export class AudioPreview {
release(clipId: string): void {
const loaded = this.voices.get(clipId);
if (!loaded) return;
+
loaded.element.pause();
loaded.element.removeAttribute("src");
loaded.element.load();
+ loaded.gain.disconnect();
if (loaded.objectUrl) URL.revokeObjectURL(loaded.objectUrl);
this.voices.delete(clipId);
}
@@ -129,35 +162,24 @@ export class AudioPreview {
const element = new Audio();
element.preload = "auto";
- // Nothing here is a user-facing media player; the element exists only as a
- // decoder and an output. Keep it out of the document.
- element.src = convertFileSrc(path);
- const loaded: Loaded = { element, triedFallback: false };
+ const loaded: Loaded = { element, gain: connectElement(element), loading: true };
this.voices.set(clipId, loaded);
- // The asset protocol is the good path: it streams and it seeks. If it is
- // unavailable - scope misconfigured, protocol disabled - fall back to
- // pulling the bytes through the engine and playing a blob. Slower and
- // memory-hungry, but it means audio works rather than silently not.
- element.addEventListener("error", () => {
- if (loaded.triedFallback) return;
- loaded.triedFallback = true;
- void this.loadViaBlob(loaded, path);
- });
+ // Source arrives asynchronously; until then the element is silent and the
+ // sync pass above simply has nothing to play, which is correct.
+ void readMediaBytes(path)
+ .then((bytes) => {
+ const url = URL.createObjectURL(new Blob([bytes]));
+ loaded.objectUrl = url;
+ loaded.element.src = url;
+ loaded.element.load();
+ })
+ .catch((cause) => console.error(`Relay: could not load audio for ${path}`, cause))
+ .finally(() => {
+ loaded.loading = false;
+ });
return loaded;
}
-
- private async loadViaBlob(loaded: Loaded, path: string): Promise<void> {
- try {
- const bytes = await readMediaBytes(path);
- const url = URL.createObjectURL(new Blob([bytes]));
- loaded.objectUrl = url;
- loaded.element.src = url;
- loaded.element.load();
- } catch (cause) {
- console.error(`Relay: could not load audio for ${path}`, cause);
- }
- }
}
diff --git a/engine/.cargo/config.toml b/engine/.cargo/config.toml
new file mode 100644
index 00000000..f725608a
--- /dev/null
+++ b/engine/.cargo/config.toml
@@ -0,0 +1,17 @@
+# Where the FFmpeg development build lives.
+#
+# Only the optional `ffi` feature of relay-media reads these. A normal build -
+# which talks to FFmpeg as a subprocess - ignores them entirely, so setting
+# them here is harmless even on a machine with no `vendor/` directory.
+#
+# `relative = true` resolves against this file's parent, so the paths follow
+# the repository rather than whoever checked it out. See BUILD.md.
+[env]
+# Note the deliberate absence of FFMPEG_DLL_PATH. rusty_ffmpeg turns that one
+# into the linker's /LIBPATH, which sends it hunting for `avcodec.lib` in
+# `bin/` - where only the DLLs live. The import libraries are in `lib/`, so
+# FFMPEG_LIBS_DIR is the one that matters at link time.
+#
+# The DLLs still have to be findable at *run* time; see BUILD.md.
+FFMPEG_INCLUDE_DIR = { value = "../vendor/ffmpeg-n8.1-latest-win64-gpl-shared-8.1/include", relative = true }
+FFMPEG_LIBS_DIR = { value = "../vendor/ffmpeg-n8.1-latest-win64-gpl-shared-8.1/lib", relative = true }
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index 5b3de5cb..021a8f0d 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -2,6 +2,15 @@
# It is not intended for manual editing.
version = 4
+[[package]]
+name = "aho-corasick"
+version = "1.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba"
+dependencies = [
+ "memchr",
+]
+
[[package]]
name = "anstream"
version = "1.0.0"
@@ -52,6 +61,64 @@ dependencies = [
"windows-sys",
]
+[[package]]
+name = "bindgen"
+version = "0.72.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895"
+dependencies = [
+ "bitflags",
+ "cexpr",
+ "clang-sys",
+ "itertools",
+ "log",
+ "prettyplease",
+ "proc-macro2",
+ "quote",
+ "regex",
+ "rustc-hash",
+ "shlex",
+ "syn 2.0.119",
+]
+
+[[package]]
+name = "bitflags"
+version = "2.13.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da"
+
+[[package]]
+name = "camino"
+version = "1.2.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d"
+
+[[package]]
+name = "cexpr"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766"
+dependencies = [
+ "nom",
+]
+
+[[package]]
+name = "cfg-if"
+version = "1.0.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
+
+[[package]]
+name = "clang-sys"
+version = "1.9.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "157a8ba7b480713b56f4c09fd13fc3e0a22a5dfab8097ba61cbc5feef950788a"
+dependencies = [
+ "glob",
+ "libc",
+ "libloading",
+]
+
[[package]]
name = "clap"
version = "4.6.6"
@@ -83,7 +150,7 @@ dependencies = [
"heck",
"proc-macro2",
"quote",
- "syn",
+ "syn 3.0.4",
]
[[package]]
@@ -98,6 +165,18 @@ version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
+[[package]]
+name = "either"
+version = "1.18.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "252afb9ae5eaa683babdc6a068b3f5726eb19e05070c731f9b2a23a7c3e8ed34"
+
+[[package]]
+name = "glob"
+version = "0.3.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e4eba85ea1d0a966a983acd07deee566e67395d2d96b6fb39e62b5a833f1eb0b"
+
[[package]]
name = "heck"
version = "0.5.0"
@@ -110,24 +189,93 @@ version = "1.70.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695"
+[[package]]
+name = "itertools"
+version = "0.13.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186"
+dependencies = [
+ "either",
+]
+
[[package]]
name = "itoa"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
+[[package]]
+name = "libc"
+version = "0.2.189"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
+
+[[package]]
+name = "libloading"
+version = "0.8.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55"
+dependencies = [
+ "cfg-if",
+ "windows-link",
+]
+
+[[package]]
+name = "log"
+version = "0.4.34"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
+
[[package]]
name = "memchr"
version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
+[[package]]
+name = "minimal-lexical"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
+
+[[package]]
+name = "nom"
+version = "7.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
+dependencies = [
+ "memchr",
+ "minimal-lexical",
+]
+
+[[package]]
+name = "once_cell"
+version = "1.21.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
+
[[package]]
name = "once_cell_polyfill"
version = "1.70.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
+[[package]]
+name = "pkg-config"
+version = "0.3.34"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
+
+[[package]]
+name = "prettyplease"
+version = "0.2.37"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b"
+dependencies = [
+ "proc-macro2",
+ "syn 2.0.119",
+]
+
[[package]]
name = "proc-macro2"
version = "1.0.107"
@@ -146,6 +294,35 @@ dependencies = [
"proc-macro2",
]
+[[package]]
+name = "regex"
+version = "1.13.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-automata",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-automata"
+version = "0.4.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-syntax"
+version = "0.8.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
+
[[package]]
name = "relay-cli"
version = "0.1.0"
@@ -165,6 +342,7 @@ name = "relay-media"
version = "0.1.0"
dependencies = [
"relay-core",
+ "rusty_ffmpeg",
"serde_json",
"thiserror",
]
@@ -176,6 +354,24 @@ dependencies = [
"relay-core",
]
+[[package]]
+name = "rustc-hash"
+version = "2.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"
+
+[[package]]
+name = "rusty_ffmpeg"
+version = "0.17.0+ffmpeg.8.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f2af9fcc780549d620ddedd4d9f26d62c523f221c0db882ea75e8a5ffe2fff9a"
+dependencies = [
+ "bindgen",
+ "camino",
+ "once_cell",
+ "pkg-config",
+]
+
[[package]]
name = "serde"
version = "1.0.229"
@@ -202,7 +398,7 @@ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
- "syn",
+ "syn 3.0.4",
]
[[package]]
@@ -218,12 +414,29 @@ dependencies = [
"zmij",
]
+[[package]]
+name = "shlex"
+version = "1.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"
+
[[package]]
name = "strsim"
version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
+[[package]]
+name = "syn"
+version = "2.0.119"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "unicode-ident",
+]
+
[[package]]
name = "syn"
version = "3.0.4"
@@ -252,7 +465,7 @@ checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af"
dependencies = [
"proc-macro2",
"quote",
- "syn",
+ "syn 3.0.4",
]
[[package]]
diff --git a/engine/crates/relay-media/Cargo.toml b/engine/crates/relay-media/Cargo.toml
index 208adb7d..3d808b00 100644
--- a/engine/crates/relay-media/Cargo.toml
+++ b/engine/crates/relay-media/Cargo.toml
@@ -7,8 +7,19 @@ description = "Getting pixels in and out of media files. The only crate that kno
[dependencies]
relay-core.workspace = true
+rusty_ffmpeg = { version = "0.17.0", default-features = false, features = ["ffmpeg8", "use_prebuilt_binding"], optional = true }
serde_json.workspace = true
thiserror.workspace = true
[lints]
workspace = true
+
+[features]
+# Links FFmpeg directly instead of only spawning it.
+#
+# Off by default, and deliberately so: the subprocess backend needs nothing but
+# `ffmpeg` on PATH, while this one needs a development build with headers and
+# import libraries. Building the engine must not require that.
+#
+# See BUILD.md and docs/decisions/0002-ffmpeg-over-a-pipe.md.
+ffi = ["dep:rusty_ffmpeg"]
diff --git a/engine/crates/relay-media/src/binaries.rs b/engine/crates/relay-media/src/binaries.rs
new file mode 100644
index 00000000..9f6c442a
--- /dev/null
+++ b/engine/crates/relay-media/src/binaries.rs
@@ -0,0 +1,51 @@
+//! Where the FFmpeg binaries are.
+//!
+//! By default they are looked up on `PATH`, which is what a developer with
+//! FFmpeg installed expects. An application that ships its own copy calls
+//! [`set_binaries`] once at startup and everything downstream picks it up,
+//! because every process this crate spawns goes through [`ffmpeg`] or
+//! [`ffprobe`] rather than naming the program itself.
+//!
+//! Set once, at startup, before any decoding. Later calls are ignored rather
+//! than racing: a half-switched pair - old encoder, new decoder - would be a
+//! genuinely confusing thing to debug.
+
+use std::path::{Path, PathBuf};
+use std::sync::OnceLock;
+
+static FFMPEG: OnceLock<PathBuf> = OnceLock::new();
+static FFPROBE: OnceLock<PathBuf> = OnceLock::new();
+
+/// Points the crate at a specific pair of binaries.
+///
+/// Returns `false` if they were already resolved, in which case nothing
+/// changed. Call this before opening any media.
+pub fn set_binaries(ffmpeg: impl Into<PathBuf>, ffprobe: impl Into<PathBuf>) -> bool {
+ let first = FFMPEG.set(ffmpeg.into()).is_ok();
+ let second = FFPROBE.set(ffprobe.into()).is_ok();
+ first && second
+}
+
+/// The `ffmpeg` binary to run. Defaults to whatever `PATH` finds.
+pub fn ffmpeg() -> &'static Path {
+ FFMPEG.get_or_init(|| PathBuf::from("ffmpeg"))
+}
+
+/// The `ffprobe` binary to run. Defaults to whatever `PATH` finds.
+pub fn ffprobe() -> &'static Path {
+ FFPROBE.get_or_init(|| PathBuf::from("ffprobe"))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn defaults_to_the_bare_name_so_path_resolves_it() {
+ // Deliberately does not call set_binaries: these statics are global,
+ // and a test that set them would change the behaviour of every other
+ // test in the binary depending on the order they happened to run in.
+ assert_eq!(ffmpeg().as_os_str(), "ffmpeg");
+ assert_eq!(ffprobe().as_os_str(), "ffprobe");
+ }
+}
diff --git a/engine/crates/relay-media/src/decode.rs b/engine/crates/relay-media/src/decode.rs
index d7df0e85..6960cebf 100644
--- a/engine/crates/relay-media/src/decode.rs
+++ b/engine/crates/relay-media/src/decode.rs
@@ -10,6 +10,8 @@ use relay_core::time::{FrameRate, Rational};
use crate::error::{Error, Result};
use crate::probe;
+/// Name used in error messages. The binary actually run comes from
+/// [`crate::binaries::ffmpeg`], which may be a bundled copy.
const FFMPEG: &str = "ffmpeg";
/// Anything that yields frames in order.
@@ -116,7 +118,7 @@ impl FfmpegDecoder {
}
};
- let mut command = Command::new(FFMPEG);
+ let mut command = Command::new(crate::binaries::ffmpeg());
command.args(["-hide_banner", "-nostdin", "-loglevel", "error"]);
// Both of these are *input* options and have to precede -i.
diff --git a/engine/crates/relay-media/src/encode.rs b/engine/crates/relay-media/src/encode.rs
index 8099c808..9f3c78a3 100644
--- a/engine/crates/relay-media/src/encode.rs
+++ b/engine/crates/relay-media/src/encode.rs
@@ -9,6 +9,8 @@ use relay_core::time::FrameRate;
use crate::error::{Error, Result};
+/// Name used in error messages. The binary actually run comes from
+/// [`crate::binaries::ffmpeg`], which may be a bundled copy.
const FFMPEG: &str = "ffmpeg";
/// Anything that accepts finished frames.
@@ -72,7 +74,7 @@ impl FfmpegEncoder {
let path = path.as_ref();
let fps = frame_rate.fps();
- let mut command = Command::new(FFMPEG);
+ let mut command = Command::new(crate::binaries::ffmpeg());
command
.args(["-hide_banner", "-nostdin", "-loglevel", "error", "-y"])
// Input: what is coming down the pipe.
diff --git a/engine/crates/relay-media/src/ffi.rs b/engine/crates/relay-media/src/ffi.rs
new file mode 100644
index 00000000..94f93802
--- /dev/null
+++ b/engine/crates/relay-media/src/ffi.rs
@@ -0,0 +1,63 @@
+//! Linked FFmpeg, as opposed to spawned FFmpeg.
+//!
+//! This backend exists for the two things a pipe fundamentally cannot do:
+//! report a frame's real presentation timestamp, and seek to an exact frame.
+//! Both are needed for playback and scrubbing; neither matters for probing or
+//! export, which is why the subprocess backend is not going anywhere.
+//!
+//! Only compiled with the `ffi` feature. See
+//! `docs/decisions/0002-ffmpeg-over-a-pipe.md`.
+
+use std::ffi::CStr;
+
+/// The version of FFmpeg this binary is linked against.
+///
+/// Cheap, and the first thing worth checking when something behaves oddly: it
+/// proves which library actually loaded, which is not always the one you think
+/// on a machine with several FFmpeg installs.
+pub fn linked_version() -> String {
+ // SAFETY: av_version_info returns a pointer to a static, NUL-terminated
+ // string compiled into the library. It is valid for the process lifetime
+ // and never freed, so borrowing it here cannot dangle.
+ let raw = unsafe { rusty_ffmpeg::ffi::av_version_info() };
+ if raw.is_null() {
+ return "unknown".to_owned();
+ }
+
+ // SAFETY: checked non-null above, and the contract of av_version_info is a
+ // NUL-terminated C string.
+ unsafe { CStr::from_ptr(raw) }.to_string_lossy().into_owned()
+}
+
+/// Major version of the linked `libavcodec`.
+///
+/// Read from the library at runtime rather than from the headers, so a
+/// mismatch between what we compiled against and what actually loaded shows up
+/// as a number rather than as undefined behaviour.
+pub fn avcodec_major() -> u32 {
+ // SAFETY: a pure accessor returning a packed integer; no pointers involved.
+ unsafe { rusty_ffmpeg::ffi::avcodec_version() >> 16 }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn links_against_the_expected_ffmpeg() {
+ let version = linked_version();
+ assert!(!version.is_empty(), "av_version_info returned nothing");
+
+ // The bindings rusty_ffmpeg ships are generated for avcodec 62
+ // (FFmpeg 8.x). Loading a different major version would mean the
+ // struct layouts we compiled against do not match the library that
+ // actually loaded - which corrupts silently rather than failing, so
+ // it is worth an explicit check.
+ assert_eq!(
+ avcodec_major(),
+ 62,
+ "linked libavcodec is {} but the bindings expect 62 - see BUILD.md",
+ avcodec_major()
+ );
+ }
+}
diff --git a/engine/crates/relay-media/src/lib.rs b/engine/crates/relay-media/src/lib.rs
index 10cc7a94..49807e3b 100644
--- a/engine/crates/relay-media/src/lib.rs
+++ b/engine/crates/relay-media/src/lib.rs
@@ -9,11 +9,15 @@
//! [`FrameSink`]. A future GPU-decode or FFI backend implements those two and
//! nothing else in the workspace changes.
+pub mod binaries;
pub mod decode;
pub mod encode;
+#[cfg(feature = "ffi")]
+pub mod ffi;
pub mod error;
pub mod probe;
+pub use binaries::{ffmpeg, ffprobe, set_binaries};
pub use decode::{DecodeOptions, FfmpegDecoder, FrameSource};
pub use encode::{EncodeOptions, FfmpegEncoder, FrameSink};
pub use error::{Error, Result};
diff --git a/engine/crates/relay-media/src/probe.rs b/engine/crates/relay-media/src/probe.rs
index d68fe8be..db70da57 100644
--- a/engine/crates/relay-media/src/probe.rs
+++ b/engine/crates/relay-media/src/probe.rs
@@ -8,6 +8,7 @@ use serde_json::Value;
use crate::error::{Error, Result};
+/// Name used in error messages; see [`crate::binaries::ffprobe`].
const FFPROBE: &str = "ffprobe";
/// What a video stream looks like.
@@ -65,7 +66,7 @@ impl MediaInfo {
/// Runs `ffprobe` and summarises what it found.
pub fn probe(path: impl AsRef<Path>) -> Result<MediaInfo> {
let path = path.as_ref();
- let output = Command::new(FFPROBE)
+ let output = Command::new(crate::binaries::ffprobe())
.args(["-v", "error", "-show_streams", "-show_format", "-of", "json"])
.arg(path)
.output()