The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.
Rust ffmpegfree-video-editormodel-context-protocolnon-linear-editoroffline-firstopen-source-video-editorself-hostedvideo-editingvideo-editing-automationvideo-editing-softwarevideo-editing-toolsvideo-editorvoice-cloningyoutube-automationyoutube-shorts
updates
Jareerce8a64c4b904
diff --git a/.gitignore b/.gitignoreindex 3ef258a6..f4d43db9 100644--- a/.gitignore+++ b/.gitignore@@ -58,3 +58,7 @@ desktop.ini# Local environment.env.env.local++# FFmpeg development build for the FFI decoder (headers + import libs).+# Large, machine-specific, and not needed for a normal build - see BUILD.md.+/vendor/diff --git a/BUILD.md b/BUILD.mdnew file mode 100644index 00000000..1e2ef08e--- /dev/null+++ b/BUILD.md@@ -0,0 +1,110 @@+# Building Relay++Two independent pieces:++- **`engine/`** — the Rust video engine. Builds and tests on its own, with no UI+ and no Node involved.+- **`desktop/`** — the Tauri + React editor. Depends on the engine by path.++You can work on the engine without ever touching the desktop app.++## What you need++| | Version | Why | Check |+|---|---|---|---|+| **Rust** | 1.93 | Pinned by `engine/rust-toolchain.toml`; rustup installs it automatically | `rustc --version` |+| **Node** | 22+ | Vite 7 and the Tauri CLI | `node --version` |+| **FFmpeg** | 7+ | Runtime dependency — the engine shells out to it | `ffmpeg -version` |+| **MSVC Build Tools** | 2022 | Windows only; Rust's default toolchain links with it | — |++FFmpeg must be on `PATH` as both `ffmpeg` **and** `ffprobe`. Relay does not+bundle it. Without it the app starts fine and then fails on first import with+"could not run `ffprobe` - is FFmpeg installed and on PATH?".++```+winget install Gyan.FFmpeg # Windows+brew install ffmpeg # macOS+sudo apt install ffmpeg # Debian/Ubuntu+```++## Engine++```sh+cd engine+cargo test --workspace # ~60 tests, none of them need FFmpeg+cargo clippy --workspace --all-targets+cargo run -p relay-cli -- probe some-video.mp4+cargo run -p relay-cli -- render in.mp4 out.mp4 --frames 120 --fade 30+```++The tests are pure — they exercise the timeline model, rational time and the+compositor, and pass on a machine with no FFmpeg at all. The CLI is how you+exercise the parts that do need it.++## Desktop app++```sh+cd desktop+npm install+npm run app # dev: builds the Rust host, opens the window, HMR for the UI+npm run typecheck # tsc --noEmit+npm run build # UI only, into dist/+npm run app:build # production installers+```++**First `npm run app` takes several minutes** — it compiles ~360 Rust crates.+After that only your own crates rebuild, so it is seconds. Editing anything+under `src/` hot-reloads without a Rust rebuild at all.++`npm run app:build` produces, under `desktop/src-tauri/target/release/`:++```+relay-desktop.exe standalone, no install needed+bundle/nsis/Relay_0.1.0_x64-setup.exe installer+bundle/msi/Relay_0.1.0_x64_en-US.msi installer+```++Release builds use `lto = true` and `codegen-units = 1`, so expect them to be+several times slower than dev. That is deliberate; the final link is+single-threaded and is most of the wait.++## `vendor/` — not required++Holds an FFmpeg **development** build (headers and import libraries), used only+by the in-progress FFI decoder described below. It is gitignored, roughly+150 MB, and nothing in the normal build path touches it. Delete it freely.++## FFI decoder — in progress, not yet wired up++The engine currently runs FFmpeg as a subprocess+(`engine/docs/decisions/0002-ffmpeg-over-a-pipe.md`). That is permanent for+probing and export, but playback needs real timestamps and frame-accurate+seeking, which a pipe cannot provide — so a linked decoder is being added+behind the existing `FrameSource` trait.++Extra prerequisites for that path only:++- **LLVM / libclang** — `bindgen` generates the FFmpeg bindings at build time.+ `winget install LLVM.LLVM`+- **An FFmpeg dev build** — headers plus `.lib` import libraries, which the+ ordinary runtime download does not include. Get a `*-gpl-shared-*` archive+ from [BtbN/FFmpeg-Builds](https://github.com/BtbN/FFmpeg-Builds/releases) and+ unpack it into `vendor/`.+- Environment variables pointing at it (see `engine/crates/relay-media/README`+ once this lands).++This section will be replaced with exact, verified steps when the decoder+actually builds. Until then, ignore it — the app does not need any of it.++## Troubleshooting++**"could not run `ffprobe`"** — FFmpeg is not on `PATH`. Restart the terminal+after installing; a running shell keeps its old `PATH`.++**Rust rebuilds everything on each `npm run app`** — something is changing+`engine/`'s files. The desktop app depends on the engine by path, so a touched+engine file invalidates the host too.++**The window opens white** — the Vite dev server did not start. Check that port+1420 is free; Tauri is configured with `strictPort`, so it will not silently+pick another.diff --git a/desktop/src-tauri/src/export.rs b/desktop/src-tauri/src/export.rsindex a6c78f3f..c0fb9f4f 100644--- a/desktop/src-tauri/src/export.rs+++ b/desktop/src-tauri/src/export.rs@@ -282,7 +282,7 @@ fn quantise(seconds: f64, rate: FrameRate) -> Rational {/// Trims, delays and mixes every audible clip into one track.fn mix_audio(clips: &[&ExportClip], duration: f64, destination: &Path) -> Result<(), String> {- let mut command = std::process::Command::new("ffmpeg");+ let mut command = std::process::Command::new(relay_media::ffmpeg());command.args(["-hide_banner", "-nostdin", "-loglevel", "error", "-y"]);for clip in clips {@@ -339,7 +339,7 @@ fn mix_audio(clips: &[&ExportClip], duration: f64, destination: &Path) -> Result}fn mux(video: &Path, audio: &Path, output: &Path) -> Result<(), String> {- let mut command = std::process::Command::new("ffmpeg");+ let mut command = std::process::Command::new(relay_media::ffmpeg());command.args(["-hide_banner", "-nostdin", "-loglevel", "error", "-y"]).arg("-i")diff --git a/desktop/src-tauri/src/lib.rs b/desktop/src-tauri/src/lib.rsindex f3803b04..9180b523 100644--- a/desktop/src-tauri/src/lib.rs+++ b/desktop/src-tauri/src/lib.rs@@ -169,7 +169,7 @@ fn filmstrip(path: &str, count: u32, height: u32) -> Result<Vec<u8>, String> {.filter(|seconds| *seconds > 0.0).ok_or_else(|| format!("{path} reports no duration"))?;- let output = std::process::Command::new("ffmpeg")+ let output = std::process::Command::new(relay_media::ffmpeg()).args(["-hide_banner", "-nostdin", "-loglevel", "error"]).args(["-i", path]).args([@@ -285,9 +285,33 @@ fn describe(error: relay_media::Error) -> String {message}+/// Points the engine at the bundled FFmpeg, if there is one.+///+/// Tauri drops sidecar binaries next to the app executable with the target+/// triple stripped, so this looks for them there and silently leaves the+/// engine on its `PATH` default otherwise. That fallback is what keeps+/// `cargo run` working without copying binaries into the build directory.+///+/// Both must be present to switch: a bundled decoder paired with whatever+/// `ffprobe` happens to be on PATH is a version mismatch waiting to happen.+fn use_bundled_ffmpeg() {+ let Ok(executable) = std::env::current_exe() else { return };+ let Some(directory) = executable.parent() else { return };++ let suffix = if cfg!(windows) { ".exe" } else { "" };+ let ffmpeg = directory.join(format!("ffmpeg{suffix}"));+ let ffprobe = directory.join(format!("ffprobe{suffix}"));++ if ffmpeg.is_file() && ffprobe.is_file() {+ relay_media::set_binaries(ffmpeg, ffprobe);+ }+}+/// Builds and runs the editor window.#[cfg_attr(mobile, tauri::mobile_entry_point)]pub fn run() {+ use_bundled_ffmpeg();+tauri::Builder::default().plugin(tauri_plugin_opener::init()).plugin(tauri_plugin_dialog::init())diff --git a/desktop/src/App.tsx b/desktop/src/App.tsxindex 6546a85b..749d8cec 100644--- a/desktop/src/App.tsx+++ b/desktop/src/App.tsx@@ -219,9 +219,9 @@ function Editor({clipId: clip.id,path: media.path,time: clip.sourceStart + (playhead - clip.start),- // Media elements cannot play above unity, so a boost is audible- // only in the export. The Adjust panel says so where it matters.- volume: Math.min(1, clipGainAt(clip, playhead)),+ // Uncapped: the preview routes through a gain node, so what you+ // hear matches what the exporter will render.+ volume: clipGainAt(clip, playhead),},];});@@ -248,6 +248,7 @@ function Editor({path: media.path,time: top.sourceStart + (playhead - top.start),muted: false,+ volume: clipGainAt(top, playhead),isStill: top.kind === "image",};}, [project, playhead]);diff --git a/desktop/src/components/AdjustPanel.tsx b/desktop/src/components/AdjustPanel.tsxindex dd4b7506..5e626cb3 100644--- a/desktop/src/components/AdjustPanel.tsx+++ b/desktop/src/components/AdjustPanel.tsx@@ -3,24 +3,47 @@ import { Group, Slider } from "./controls";import { Icon } from "./Icon";import { Empty } from "./Panel";-/** Above unity the preview cannot follow; see the note in the panel. */-const MAX_GAIN = 2;-/** A fade longer than the clip is meaningless, and this keeps it obvious. */-const MAX_FADE = 5;+/**+ * Level range, in decibels.+ *+ * The bottom is silence, not -60: a fader that cannot reach zero is a fader+ * you cannot use to mute. +24 at the top because quiet material genuinely+ * needs it - a distant lavalier or a phone recording is routinely 15-20 dB+ * under, and a limit that cannot fix those is a limit in the wrong place.+ */+const MIN_DB = -60;+const MAX_DB = 24;+/** A fade longer than half the clip would overlap the other one. */+const MAX_FADE = 10;++/** Linear gain to decibels. Zero maps to the bottom of the fader, not -inf. */+function toDecibels(gain: number): number {+ if (gain <= 0) return MIN_DB;+ return Math.max(MIN_DB, Math.min(MAX_DB, 20 * Math.log10(gain)));+}++function fromDecibels(decibels: number): number {+ // At the very bottom the fader means silence, so snap to exactly zero+ // rather than leaving an inaudible-but-nonzero gain in the project.+ return decibels <= MIN_DB ? 0 : 10 ** (decibels / 20);+}-/** Linear gain as decibels, which is the unit anyone mixing audio thinks in. */-function decibels(gain: number): string {- if (gain <= 0.0001) return "-inf dB";- const value = 20 * Math.log10(gain);- return `${value > 0 ? "+" : ""}${value.toFixed(1)} dB`;+function formatDecibels(decibels: number): string {+ if (decibels <= MIN_DB) return "silent";+ if (Math.abs(decibels) < 0.05) return "0.0 dB";+ return `${decibels > 0 ? "+" : ""}${decibels.toFixed(1)} dB`;}/*** The Adjust tab: properties of the selected clip that change how it plays.*- * Everything here is applied in two places - the preview, so you can hear it- * now, and the exporter, so the file matches. Both read the same numbers off- * the clip, which is the only way those two can be guaranteed to agree.+ * Everything here applies in two places - the preview, so you hear it now, and+ * the exporter, so the file matches. Both read the same numbers off the clip,+ * which is the only way those two can be guaranteed to agree.+ *+ * The fader works in decibels rather than linear gain. Linear puts every+ * useful setting in the bottom sixth of the travel; decibels is both how the+ * ear works and how anyone mixing thinks.*/export function AdjustPanel({clip,@@ -37,62 +60,58 @@ export function AdjustPanel({);}- const hasSound = clip.kind !== "image";- // A fade cannot be longer than half the clip without the two overlapping.- const fadeLimit = Math.min(MAX_FADE, clip.duration / 2);+ if (clip.kind === "image") {+ return (+ <Empty icon={<Icon name="image" size={26} strokeWidth={1.5} />}>+ A still has nothing to adjust yet. Opacity and transform land with the+ compositor.+ </Empty>+ );+ }++ const fadeLimit = Math.max(0.1, Math.min(MAX_FADE, clip.duration / 2));return (<div className="px-3 py-3">- {hasSound ? (- <>- <Group title="Volume">- <Slider- label="Level"- value={clip.volume}- min={0}- max={MAX_GAIN}- step={0.01}- format={decibels}- onReset={() => onChange({ volume: 1 })}- onChange={(volume) => onChange({ volume })}- />- {clip.volume > 1 && (- <p className="-mt-1 mb-1 text-[11px] leading-snug text-tertiary">- Boost above 0 dB applies on export. The preview cannot play louder than- unity, so it will sound quieter than the exported file.- </p>- )}- </Group>+ <Group title="Volume">+ <Slider+ label="Level"+ value={toDecibels(clip.volume)}+ min={MIN_DB}+ max={MAX_DB}+ step={0.5}+ format={formatDecibels}+ onReset={() => onChange({ volume: 1 })}+ onChange={(decibels) => onChange({ volume: fromDecibels(decibels) })}+ />+ </Group>++ <Group title="Fades">+ <Slider+ label="Fade in"+ value={Math.min(clip.fadeIn, fadeLimit)}+ min={0}+ max={fadeLimit}+ step={0.05}+ format={(value) => (value === 0 ? "none" : `${value.toFixed(2)}s`)}+ onReset={() => onChange({ fadeIn: 0 })}+ onChange={(fadeIn) => onChange({ fadeIn })}+ />+ <Slider+ label="Fade out"+ value={Math.min(clip.fadeOut, fadeLimit)}+ min={0}+ max={fadeLimit}+ step={0.05}+ format={(value) => (value === 0 ? "none" : `${value.toFixed(2)}s`)}+ onReset={() => onChange({ fadeOut: 0 })}+ onChange={(fadeOut) => onChange({ fadeOut })}+ />+ </Group>- <Group title="Fades">- <Slider- label="Fade in"- value={Math.min(clip.fadeIn, fadeLimit)}- min={0}- max={fadeLimit}- step={0.05}- format={(value) => (value === 0 ? "none" : `${value.toFixed(2)}s`)}- onReset={() => onChange({ fadeIn: 0 })}- onChange={(fadeIn) => onChange({ fadeIn })}- />- <Slider- label="Fade out"- value={Math.min(clip.fadeOut, fadeLimit)}- min={0}- max={fadeLimit}- step={0.05}- format={(value) => (value === 0 ? "none" : `${value.toFixed(2)}s`)}- onReset={() => onChange({ fadeOut: 0 })}- onChange={(fadeOut) => onChange({ fadeOut })}- />- </Group>- </>- ) : (- <Empty icon={<Icon name="image" size={26} strokeWidth={1.5} />}>- A still has nothing to adjust yet. Opacity and transform land with the- compositor.- </Empty>- )}+ <p className="text-[11px] leading-relaxed text-tertiary">+ Double-click a label to reset it.+ </p></div>);}diff --git a/desktop/src/components/Preview.tsx b/desktop/src/components/Preview.tsxindex 75aadf26..b67ce9b4 100644--- a/desktop/src/components/Preview.tsx+++ b/desktop/src/components/Preview.tsx@@ -1,6 +1,7 @@import { useEffect, useRef } from "react";import { convertFileSrc } from "@tauri-apps/api/core";+import { connectElement } from "../lib/audio";import { timecode } from "../lib/time";import { Icon, IconButton } from "./Icon";import { PANEL_SHELL } from "./Panel";@@ -12,6 +13,8 @@ export interface PreviewSource {/** Where in the source file the playhead sits, in seconds. */time: number;muted: boolean;+ /** Linear gain with fades applied. May exceed 1; see lib/audio.ts. */+ volume: number;/** A still is shown as an image; there is nothing to seek or play. */isStill: boolean;}@@ -82,6 +85,15 @@ export function Preview({element.muted = source.muted;+ // Through a gain node, not element.volume, which browsers cap at 1.0.+ // A video clip's audio obeys the same clip gain as an audio clip's.+ try {+ connectElement(element).gain.value = Math.max(0, source.volume);+ } catch {+ // Routing can only fail if the element was already claimed by another+ // graph, which cannot happen here - but silence is not worth a crash.+ }+const tolerance = playing ? 0.3 : 0.03;if (element.readyState > 0 && Math.abs(element.currentTime - source.time) > tolerance) {try {diff --git a/desktop/src/components/TimelinePanel.tsx b/desktop/src/components/TimelinePanel.tsxindex ec0b8245..37bc3d12 100644--- a/desktop/src/components/TimelinePanel.tsx+++ b/desktop/src/components/TimelinePanel.tsx@@ -1097,6 +1097,10 @@ function drawClip(}}+ if (clip.fadeIn > 0 || clip.fadeOut > 0) {+ drawFades(context, clip, x, bodyY, drawWidth, bodyHeight, secondsPerPixel);+ }+context.fillStyle = palette.header;context.fillRect(x, y, drawWidth, CLIP_HEADER);@@ -1117,6 +1121,64 @@ function drawClip(context.stroke();}+/**+ * Draws the fade ramps as wedges over the clip body.+ *+ * The shaded area is the part being attenuated and the bright line is the+ * envelope itself - the same shape every editor draws, because it reads as+ * "this much is being taken away" at a glance.+ *+ * On a video clip the wedge is confined to a band at the bottom. The fade is+ * an *audio* property, and shading the picture would say it fades to black.+ */+function drawFades(+ context: CanvasRenderingContext2D,+ clip: Clip,+ x: number,+ bodyY: number,+ width: number,+ bodyHeight: number,+ secondsPerPixel: number,+) {+ const band = clip.kind === "audio" ? bodyHeight : Math.min(10, bodyHeight);+ const top = bodyY + bodyHeight - band;+ const bottom = bodyY + bodyHeight;+ if (band <= 1) return;++ context.save();+ context.beginPath();+ context.rect(x, top, width, band);+ context.clip();++ const ramp = (seconds: number, fromLeft: boolean) => {+ const span = Math.min(seconds / secondsPerPixel, width);+ if (span < 1) return;++ const originX = fromLeft ? x : x + width;+ const endX = fromLeft ? x + span : x + width - span;++ context.fillStyle = "rgba(0,0,0,0.42)";+ context.beginPath();+ context.moveTo(originX, top);+ context.lineTo(endX, top);+ context.lineTo(originX, bottom);+ context.closePath();+ context.fill();++ context.strokeStyle = "rgba(255,255,255,0.75)";+ context.lineWidth = 1.5;+ context.beginPath();+ context.moveTo(originX, bottom);+ context.lineTo(endX, top);+ context.stroke();+ };++ if (clip.fadeIn > 0) ramp(clip.fadeIn, true);+ if (clip.fadeOut > 0) ramp(clip.fadeOut, false);++ context.restore();+}+/*** Tiles frames from a single strip image across the clip body.*@@ -1175,6 +1237,10 @@ function drawWaveform() {const centre = y + height / 2;const half = height / 2 - 1;+ // The drawn amplitude follows the clip's gain, so turning a clip up makes+ // its waveform visibly taller. Clamped, because a boosted waveform that+ // overflowed its clip would bleed into the lane above.+ const gain = Math.max(0, clip.volume);const perPixel = Math.max(1, Math.round(secondsPerPixel * peaks.bucketsPerSecond));const stride = Math.max(1, Math.floor(perPixel / 8));@@ -1193,8 +1259,9 @@ function drawWaveform(if (peaks.max[bucket] > high) high = peaks.max[bucket];}- const top = centre - high * half;- context.rect(x + column, top, 1, Math.max(1, centre - low * half - top));+ const top = centre - Math.min(1, high * gain) * half;+ const bottom = centre - Math.max(-1, low * gain) * half;+ context.rect(x + column, top, 1, Math.max(1, bottom - top));}context.fill();diff --git a/desktop/src/lib/audio.ts b/desktop/src/lib/audio.tsindex 9bea1e19..f6e60399 100644--- a/desktop/src/lib/audio.ts+++ b/desktop/src/lib/audio.ts@@ -1,28 +1,28 @@/*** Audio preview.*- * This is what makes an mp3 on the timeline actually audible. It is a- * *preview* path, not the engine's mixer: one `HTMLAudioElement` per audio- * clip, kept roughly in step with the transport clock.+ * Each audible clip gets a media element for decoding, routed through a Web+ * Audio `GainNode` for level. The gain node is the whole reason this is not+ * just `element.volume`: that property is hard-capped at 1.0 by every browser,+ * so a boost could not be previewed at all - it could only be promised and+ * applied later at export, which is a useless thing to tell someone mixing.+ * A `GainNode` has no ceiling, so what you hear is what gets rendered.*- * Why the webview and not Rust: real-time audio needs a callback-driven output- * device (cpal), a resampler, and a mixer that can survive the UI thread- * stalling. That is a genuine piece of engineering and it belongs in the- * engine. Until it exists, the webview already has a decoder and an output- * device, and using them is the difference between hearing your edit and not.+ * Audio is loaded as a blob rather than through the asset protocol, and that+ * is deliberate too. `createMediaElementSource` on a cross-origin element+ * yields *silence* with no error - the element plays, the graph outputs+ * nothing. A blob URL is same-origin, so the graph is guaranteed to work. The+ * cost is holding the file in memory, which we already do for the waveform.*- * The known limits, so nobody mistakes this for the finished thing:- * - Sync is corrective, not sample-accurate. Drift beyond a tolerance is- * fixed by reseeking, which is audible if it happens often.- * - No mixing. Overlapping clips play simultaneously and clip if loud.- * - No effects, no fades, no per-clip gain automation.- * - Scrubbing does not scrub audio, it repositions it.+ * The known limits, so nobody mistakes this for a finished mixer:+ * - Sync is corrective, not sample-accurate. Drift past a tolerance is+ * fixed by reseeking, which is audible when it happens.+ * - No effects and no automation beyond the clip's constant gain and fades.+ * - Scrubbing repositions audio, it does not scrub it.** See docs/decisions/0004-audio-preview-in-the-webview.md.*/-import { convertFileSrc } from "@tauri-apps/api/core";-import { readMediaBytes } from "./engine";/** One audio clip that should be sounding right now. */@@ -31,6 +31,7 @@ export interface Voice {path: string;/** Where in the source file the playhead currently sits, in seconds. */time: number;+ /** Linear gain, fades already applied. May exceed 1. */volume: number;}@@ -39,18 +40,55 @@ export interface Voice {** Generous while playing: a reseek is an audible click, and browsers do not* keep a media element perfectly locked to an external clock anyway. Tight- * while paused, where "seek" is the entire operation and there is no click to- * worry about.+ * while paused, where "seek" is the entire operation.*/const TOLERANCE_PLAYING = 0.3;const TOLERANCE_PAUSED = 0.02;+let sharedContext: AudioContext | null = null;++/** The one AudioContext. Browsers cap how many a page may create. */+export function audioContext(): AudioContext {+ sharedContext ??= new AudioContext();+ return sharedContext;+}++/**+ * A context created before any user interaction starts suspended. Resuming is+ * safe to call repeatedly and only does anything the first time.+ */+function wake(): void {+ const context = audioContext();+ if (context.state === "suspended") void context.resume().catch(() => undefined);+}++// One source node per element, forever: calling createMediaElementSource twice+// on the same element throws, and there is no way to undo the first call.+const routed = new WeakMap<HTMLMediaElement, GainNode>();++/**+ * Routes a media element through a gain node and returns it.+ *+ * Shared with the video preview, so picture and sound obey the same clip gain+ * through the same graph.+ */+export function connectElement(element: HTMLMediaElement): GainNode {+ const existing = routed.get(element);+ if (existing) return existing;++ const context = audioContext();+ const gain = context.createGain();+ context.createMediaElementSource(element).connect(gain);+ gain.connect(context.destination);+ routed.set(element, gain);+ return gain;+}+interface Loaded {element: HTMLAudioElement;- /** Set when we fell back to a blob, so it can be revoked on dispose. */+ gain: GainNode;objectUrl?: string;- /** Guards against retrying the blob fallback in a loop. */- triedFallback: boolean;+ loading: boolean;}export class AudioPreview {@@ -59,18 +97,15 @@ export class AudioPreview {/*** Brings the audio output in line with the transport.*- * Call this whenever the playhead, the play state, or the set of audible- * clips changes. It is idempotent - calling it every animation frame is fine- * and is in fact how drift gets corrected.+ * Idempotent - calling it every animation frame is fine, and is in fact how+ * drift gets corrected.*/sync(active: Voice[], playing: boolean): void {- const wanted = new Set(active.map((voice) => voice.clipId));+ if (playing) wake();- // Silence anything that has fallen out from under the playhead.+ const wanted = new Set(active.map((voice) => voice.clipId));for (const [clipId, loaded] of this.voices) {- if (!wanted.has(clipId) && !loaded.element.paused) {- loaded.element.pause();- }+ if (!wanted.has(clipId) && !loaded.element.paused) loaded.element.pause();}const tolerance = playing ? TOLERANCE_PLAYING : TOLERANCE_PAUSED;@@ -78,32 +113,28 @@ export class AudioPreview {for (const voice of active) {const loaded = this.load(voice.clipId, voice.path);const element = loaded.element;- element.volume = Math.max(0, Math.min(1, voice.volume));- // readyState 0 means metadata has not arrived; setting currentTime now- // throws in some engines and is silently discarded in others.+ // No clamp. A boost is meant to be audible.+ loaded.gain.gain.value = Math.max(0, voice.volume);++ // readyState 0 means metadata has not arrived; setting currentTime then+ // throws in some engines and is discarded in others.if (element.readyState > 0 && Math.abs(element.currentTime - voice.time) > tolerance) {try {element.currentTime = Math.max(0, voice.time);} catch {- // Seeking an element that is still opening is not an error worth- // surfacing; the next sync will land it.+ // Still opening; the next sync lands it.}}if (playing) {- if (element.paused) {- // Rejects when the element is not ready or autoplay is refused.- // Either way the next sync retries, so there is nothing to handle.- void element.play().catch(() => undefined);- }+ if (element.paused) void element.play().catch(() => undefined);} else if (!element.paused) {element.pause();}}}- /** Stops everything, e.g. when the transport stops. */stopAll(): void {for (const loaded of this.voices.values()) loaded.element.pause();}@@ -112,9 +143,11 @@ export class AudioPreview {release(clipId: string): void {const loaded = this.voices.get(clipId);if (!loaded) return;+loaded.element.pause();loaded.element.removeAttribute("src");loaded.element.load();+ loaded.gain.disconnect();if (loaded.objectUrl) URL.revokeObjectURL(loaded.objectUrl);this.voices.delete(clipId);}@@ -129,35 +162,24 @@ export class AudioPreview {const element = new Audio();element.preload = "auto";- // Nothing here is a user-facing media player; the element exists only as a- // decoder and an output. Keep it out of the document.- element.src = convertFileSrc(path);- const loaded: Loaded = { element, triedFallback: false };+ const loaded: Loaded = { element, gain: connectElement(element), loading: true };this.voices.set(clipId, loaded);- // The asset protocol is the good path: it streams and it seeks. If it is- // unavailable - scope misconfigured, protocol disabled - fall back to- // pulling the bytes through the engine and playing a blob. Slower and- // memory-hungry, but it means audio works rather than silently not.- element.addEventListener("error", () => {- if (loaded.triedFallback) return;- loaded.triedFallback = true;- void this.loadViaBlob(loaded, path);- });+ // Source arrives asynchronously; until then the element is silent and the+ // sync pass above simply has nothing to play, which is correct.+ void readMediaBytes(path)+ .then((bytes) => {+ const url = URL.createObjectURL(new Blob([bytes]));+ loaded.objectUrl = url;+ loaded.element.src = url;+ loaded.element.load();+ })+ .catch((cause) => console.error(`Relay: could not load audio for ${path}`, cause))+ .finally(() => {+ loaded.loading = false;+ });return loaded;}-- private async loadViaBlob(loaded: Loaded, path: string): Promise<void> {- try {- const bytes = await readMediaBytes(path);- const url = URL.createObjectURL(new Blob([bytes]));- loaded.objectUrl = url;- loaded.element.src = url;- loaded.element.load();- } catch (cause) {- console.error(`Relay: could not load audio for ${path}`, cause);- }- }}diff --git a/engine/.cargo/config.toml b/engine/.cargo/config.tomlnew file mode 100644index 00000000..f725608a--- /dev/null+++ b/engine/.cargo/config.toml@@ -0,0 +1,17 @@+# Where the FFmpeg development build lives.+#+# Only the optional `ffi` feature of relay-media reads these. A normal build -+# which talks to FFmpeg as a subprocess - ignores them entirely, so setting+# them here is harmless even on a machine with no `vendor/` directory.+#+# `relative = true` resolves against this file's parent, so the paths follow+# the repository rather than whoever checked it out. See BUILD.md.+[env]+# Note the deliberate absence of FFMPEG_DLL_PATH. rusty_ffmpeg turns that one+# into the linker's /LIBPATH, which sends it hunting for `avcodec.lib` in+# `bin/` - where only the DLLs live. The import libraries are in `lib/`, so+# FFMPEG_LIBS_DIR is the one that matters at link time.+#+# The DLLs still have to be findable at *run* time; see BUILD.md.+FFMPEG_INCLUDE_DIR = { value = "../vendor/ffmpeg-n8.1-latest-win64-gpl-shared-8.1/include", relative = true }+FFMPEG_LIBS_DIR = { value = "../vendor/ffmpeg-n8.1-latest-win64-gpl-shared-8.1/lib", relative = true }diff --git a/engine/Cargo.lock b/engine/Cargo.lockindex 5b3de5cb..021a8f0d 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -2,6 +2,15 @@# It is not intended for manual editing.version = 4+[[package]]+name = "aho-corasick"+version = "1.1.5"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba"+dependencies = [+ "memchr",+]+[[package]]name = "anstream"version = "1.0.0"@@ -52,6 +61,64 @@ dependencies = ["windows-sys",]+[[package]]+name = "bindgen"+version = "0.72.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895"+dependencies = [+ "bitflags",+ "cexpr",+ "clang-sys",+ "itertools",+ "log",+ "prettyplease",+ "proc-macro2",+ "quote",+ "regex",+ "rustc-hash",+ "shlex",+ "syn 2.0.119",+]++[[package]]+name = "bitflags"+version = "2.13.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da"++[[package]]+name = "camino"+version = "1.2.5"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d"++[[package]]+name = "cexpr"+version = "0.6.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766"+dependencies = [+ "nom",+]++[[package]]+name = "cfg-if"+version = "1.0.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"++[[package]]+name = "clang-sys"+version = "1.9.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "157a8ba7b480713b56f4c09fd13fc3e0a22a5dfab8097ba61cbc5feef950788a"+dependencies = [+ "glob",+ "libc",+ "libloading",+]+[[package]]name = "clap"version = "4.6.6"@@ -83,7 +150,7 @@ dependencies = ["heck","proc-macro2","quote",- "syn",+ "syn 3.0.4",][[package]]@@ -98,6 +165,18 @@ version = "1.0.5"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"+[[package]]+name = "either"+version = "1.18.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "252afb9ae5eaa683babdc6a068b3f5726eb19e05070c731f9b2a23a7c3e8ed34"++[[package]]+name = "glob"+version = "0.3.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e4eba85ea1d0a966a983acd07deee566e67395d2d96b6fb39e62b5a833f1eb0b"+[[package]]name = "heck"version = "0.5.0"@@ -110,24 +189,93 @@ version = "1.70.2"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695"+[[package]]+name = "itertools"+version = "0.13.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186"+dependencies = [+ "either",+]+[[package]]name = "itoa"version = "1.0.18"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"+[[package]]+name = "libc"+version = "0.2.189"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"++[[package]]+name = "libloading"+version = "0.8.9"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55"+dependencies = [+ "cfg-if",+ "windows-link",+]++[[package]]+name = "log"+version = "0.4.34"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"+[[package]]name = "memchr"version = "2.8.3"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"+[[package]]+name = "minimal-lexical"+version = "0.2.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"++[[package]]+name = "nom"+version = "7.1.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"+dependencies = [+ "memchr",+ "minimal-lexical",+]++[[package]]+name = "once_cell"+version = "1.21.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"+[[package]]name = "once_cell_polyfill"version = "1.70.2"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"+[[package]]+name = "pkg-config"+version = "0.3.34"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"++[[package]]+name = "prettyplease"+version = "0.2.37"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b"+dependencies = [+ "proc-macro2",+ "syn 2.0.119",+]+[[package]]name = "proc-macro2"version = "1.0.107"@@ -146,6 +294,35 @@ dependencies = ["proc-macro2",]+[[package]]+name = "regex"+version = "1.13.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"+dependencies = [+ "aho-corasick",+ "memchr",+ "regex-automata",+ "regex-syntax",+]++[[package]]+name = "regex-automata"+version = "0.4.18"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2"+dependencies = [+ "aho-corasick",+ "memchr",+ "regex-syntax",+]++[[package]]+name = "regex-syntax"+version = "0.8.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"+[[package]]name = "relay-cli"version = "0.1.0"@@ -165,6 +342,7 @@ name = "relay-media"version = "0.1.0"dependencies = ["relay-core",+ "rusty_ffmpeg","serde_json","thiserror",]@@ -176,6 +354,24 @@ dependencies = ["relay-core",]+[[package]]+name = "rustc-hash"+version = "2.1.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"++[[package]]+name = "rusty_ffmpeg"+version = "0.17.0+ffmpeg.8.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "f2af9fcc780549d620ddedd4d9f26d62c523f221c0db882ea75e8a5ffe2fff9a"+dependencies = [+ "bindgen",+ "camino",+ "once_cell",+ "pkg-config",+]+[[package]]name = "serde"version = "1.0.229"@@ -202,7 +398,7 @@ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"dependencies = ["proc-macro2","quote",- "syn",+ "syn 3.0.4",][[package]]@@ -218,12 +414,29 @@ dependencies = ["zmij",]+[[package]]+name = "shlex"+version = "1.3.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"+[[package]]name = "strsim"version = "0.11.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"+[[package]]+name = "syn"+version = "2.0.119"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297"+dependencies = [+ "proc-macro2",+ "quote",+ "unicode-ident",+]+[[package]]name = "syn"version = "3.0.4"@@ -252,7 +465,7 @@ checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af"dependencies = ["proc-macro2","quote",- "syn",+ "syn 3.0.4",][[package]]diff --git a/engine/crates/relay-media/Cargo.toml b/engine/crates/relay-media/Cargo.tomlindex 208adb7d..3d808b00 100644--- a/engine/crates/relay-media/Cargo.toml+++ b/engine/crates/relay-media/Cargo.toml@@ -7,8 +7,19 @@ description = "Getting pixels in and out of media files. The only crate that kno[dependencies]relay-core.workspace = true+rusty_ffmpeg = { version = "0.17.0", default-features = false, features = ["ffmpeg8", "use_prebuilt_binding"], optional = true }serde_json.workspace = truethiserror.workspace = true[lints]workspace = true++[features]+# Links FFmpeg directly instead of only spawning it.+#+# Off by default, and deliberately so: the subprocess backend needs nothing but+# `ffmpeg` on PATH, while this one needs a development build with headers and+# import libraries. Building the engine must not require that.+#+# See BUILD.md and docs/decisions/0002-ffmpeg-over-a-pipe.md.+ffi = ["dep:rusty_ffmpeg"]diff --git a/engine/crates/relay-media/src/binaries.rs b/engine/crates/relay-media/src/binaries.rsnew file mode 100644index 00000000..9f6c442a--- /dev/null+++ b/engine/crates/relay-media/src/binaries.rs@@ -0,0 +1,51 @@+//! Where the FFmpeg binaries are.+//!+//! By default they are looked up on `PATH`, which is what a developer with+//! FFmpeg installed expects. An application that ships its own copy calls+//! [`set_binaries`] once at startup and everything downstream picks it up,+//! because every process this crate spawns goes through [`ffmpeg`] or+//! [`ffprobe`] rather than naming the program itself.+//!+//! Set once, at startup, before any decoding. Later calls are ignored rather+//! than racing: a half-switched pair - old encoder, new decoder - would be a+//! genuinely confusing thing to debug.++use std::path::{Path, PathBuf};+use std::sync::OnceLock;++static FFMPEG: OnceLock<PathBuf> = OnceLock::new();+static FFPROBE: OnceLock<PathBuf> = OnceLock::new();++/// Points the crate at a specific pair of binaries.+///+/// Returns `false` if they were already resolved, in which case nothing+/// changed. Call this before opening any media.+pub fn set_binaries(ffmpeg: impl Into<PathBuf>, ffprobe: impl Into<PathBuf>) -> bool {+ let first = FFMPEG.set(ffmpeg.into()).is_ok();+ let second = FFPROBE.set(ffprobe.into()).is_ok();+ first && second+}++/// The `ffmpeg` binary to run. Defaults to whatever `PATH` finds.+pub fn ffmpeg() -> &'static Path {+ FFMPEG.get_or_init(|| PathBuf::from("ffmpeg"))+}++/// The `ffprobe` binary to run. Defaults to whatever `PATH` finds.+pub fn ffprobe() -> &'static Path {+ FFPROBE.get_or_init(|| PathBuf::from("ffprobe"))+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn defaults_to_the_bare_name_so_path_resolves_it() {+ // Deliberately does not call set_binaries: these statics are global,+ // and a test that set them would change the behaviour of every other+ // test in the binary depending on the order they happened to run in.+ assert_eq!(ffmpeg().as_os_str(), "ffmpeg");+ assert_eq!(ffprobe().as_os_str(), "ffprobe");+ }+}diff --git a/engine/crates/relay-media/src/decode.rs b/engine/crates/relay-media/src/decode.rsindex d7df0e85..6960cebf 100644--- a/engine/crates/relay-media/src/decode.rs+++ b/engine/crates/relay-media/src/decode.rs@@ -10,6 +10,8 @@ use relay_core::time::{FrameRate, Rational};use crate::error::{Error, Result};use crate::probe;+/// Name used in error messages. The binary actually run comes from+/// [`crate::binaries::ffmpeg`], which may be a bundled copy.const FFMPEG: &str = "ffmpeg";/// Anything that yields frames in order.@@ -116,7 +118,7 @@ impl FfmpegDecoder {}};- let mut command = Command::new(FFMPEG);+ let mut command = Command::new(crate::binaries::ffmpeg());command.args(["-hide_banner", "-nostdin", "-loglevel", "error"]);// Both of these are *input* options and have to precede -i.diff --git a/engine/crates/relay-media/src/encode.rs b/engine/crates/relay-media/src/encode.rsindex 8099c808..9f3c78a3 100644--- a/engine/crates/relay-media/src/encode.rs+++ b/engine/crates/relay-media/src/encode.rs@@ -9,6 +9,8 @@ use relay_core::time::FrameRate;use crate::error::{Error, Result};+/// Name used in error messages. The binary actually run comes from+/// [`crate::binaries::ffmpeg`], which may be a bundled copy.const FFMPEG: &str = "ffmpeg";/// Anything that accepts finished frames.@@ -72,7 +74,7 @@ impl FfmpegEncoder {let path = path.as_ref();let fps = frame_rate.fps();- let mut command = Command::new(FFMPEG);+ let mut command = Command::new(crate::binaries::ffmpeg());command.args(["-hide_banner", "-nostdin", "-loglevel", "error", "-y"])// Input: what is coming down the pipe.diff --git a/engine/crates/relay-media/src/ffi.rs b/engine/crates/relay-media/src/ffi.rsnew file mode 100644index 00000000..94f93802--- /dev/null+++ b/engine/crates/relay-media/src/ffi.rs@@ -0,0 +1,63 @@+//! Linked FFmpeg, as opposed to spawned FFmpeg.+//!+//! This backend exists for the two things a pipe fundamentally cannot do:+//! report a frame's real presentation timestamp, and seek to an exact frame.+//! Both are needed for playback and scrubbing; neither matters for probing or+//! export, which is why the subprocess backend is not going anywhere.+//!+//! Only compiled with the `ffi` feature. See+//! `docs/decisions/0002-ffmpeg-over-a-pipe.md`.++use std::ffi::CStr;++/// The version of FFmpeg this binary is linked against.+///+/// Cheap, and the first thing worth checking when something behaves oddly: it+/// proves which library actually loaded, which is not always the one you think+/// on a machine with several FFmpeg installs.+pub fn linked_version() -> String {+ // SAFETY: av_version_info returns a pointer to a static, NUL-terminated+ // string compiled into the library. It is valid for the process lifetime+ // and never freed, so borrowing it here cannot dangle.+ let raw = unsafe { rusty_ffmpeg::ffi::av_version_info() };+ if raw.is_null() {+ return "unknown".to_owned();+ }++ // SAFETY: checked non-null above, and the contract of av_version_info is a+ // NUL-terminated C string.+ unsafe { CStr::from_ptr(raw) }.to_string_lossy().into_owned()+}++/// Major version of the linked `libavcodec`.+///+/// Read from the library at runtime rather than from the headers, so a+/// mismatch between what we compiled against and what actually loaded shows up+/// as a number rather than as undefined behaviour.+pub fn avcodec_major() -> u32 {+ // SAFETY: a pure accessor returning a packed integer; no pointers involved.+ unsafe { rusty_ffmpeg::ffi::avcodec_version() >> 16 }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn links_against_the_expected_ffmpeg() {+ let version = linked_version();+ assert!(!version.is_empty(), "av_version_info returned nothing");++ // The bindings rusty_ffmpeg ships are generated for avcodec 62+ // (FFmpeg 8.x). Loading a different major version would mean the+ // struct layouts we compiled against do not match the library that+ // actually loaded - which corrupts silently rather than failing, so+ // it is worth an explicit check.+ assert_eq!(+ avcodec_major(),+ 62,+ "linked libavcodec is {} but the bindings expect 62 - see BUILD.md",+ avcodec_major()+ );+ }+}diff --git a/engine/crates/relay-media/src/lib.rs b/engine/crates/relay-media/src/lib.rsindex 10cc7a94..49807e3b 100644--- a/engine/crates/relay-media/src/lib.rs+++ b/engine/crates/relay-media/src/lib.rs@@ -9,11 +9,15 @@//! [`FrameSink`]. A future GPU-decode or FFI backend implements those two and//! nothing else in the workspace changes.+pub mod binaries;pub mod decode;pub mod encode;+#[cfg(feature = "ffi")]+pub mod ffi;pub mod error;pub mod probe;+pub use binaries::{ffmpeg, ffprobe, set_binaries};pub use decode::{DecodeOptions, FfmpegDecoder, FrameSource};pub use encode::{EncodeOptions, FfmpegEncoder, FrameSink};pub use error::{Error, Result};diff --git a/engine/crates/relay-media/src/probe.rs b/engine/crates/relay-media/src/probe.rsindex d68fe8be..db70da57 100644--- a/engine/crates/relay-media/src/probe.rs+++ b/engine/crates/relay-media/src/probe.rs@@ -8,6 +8,7 @@ use serde_json::Value;use crate::error::{Error, Result};+/// Name used in error messages; see [`crate::binaries::ffprobe`].const FFPROBE: &str = "ffprobe";/// What a video stream looks like.@@ -65,7 +66,7 @@ impl MediaInfo {/// Runs `ffprobe` and summarises what it found.pub fn probe(path: impl AsRef<Path>) -> Result<MediaInfo> {let path = path.as_ref();- let output = Command::new(FFPROBE)+ let output = Command::new(crate::binaries::ffprobe()).args(["-v", "error", "-show_streams", "-show_format", "-of", "json"]).arg(path).output()