Skip to content

The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.

updates

Jareer866b801df2e8
diff --git a/.github/workflows/ffmpeg.yml b/.github/workflows/ffmpeg.yml
index a84733aa..23df4e46 100644
--- a/.github/workflows/ffmpeg.yml
+++ b/.github/workflows/ffmpeg.yml
@@ -92,7 +92,7 @@ jobs:
--enable-decoder=rawvideo,h264,hevc,vp8,vp9,av1,mpeg4,mjpeg,prores,dnxhd,png,webp,bmp,tiff,gif,aac,mp3,flac,opus,vorbis,alac,pcm_s16le,pcm_s24le,pcm_s32le,pcm_f32le \
--enable-encoder=rawvideo,libx264,aac,mjpeg,png,pcm_s16le \
--enable-demuxer=mov,matroska,avi,mp3,wav,flac,ogg,aac,image2,gif,rawvideo,pcm_s16le,concat \
- --enable-muxer=mp4,mov,mp3,wav,ipod,mjpeg,image2,rawvideo,null \
+ --enable-muxer=mp4,mov,mp3,wav,ipod,mjpeg,image2,rawvideo,pcm_s16le,null \
--enable-parser=h264,hevc,vp8,vp9,av1,aac,mpegaudio,flac,vorbis,png,mjpeg \
--enable-bsf=h264_mp4toannexb,hevc_mp4toannexb,aac_adtstoasc \
--enable-protocol=file,pipe \
@@ -141,6 +141,14 @@ jobs:
# Sound: the preview render, then the full export mix and mux.
"$BIN/ffmpeg.exe" -hide_banner -loglevel error -y \
-f s16le -ar 48000 -ac 2 -i tone.pcm -c:a aac tone.m4a
+
+ # The playback engine's decode: a clip span straight to raw PCM.
+ # See desktop/src-tauri/src/playback.rs - this needs the pcm_s16le
+ # *muxer*, easy to forget because the demuxer of the same name is
+ # enabled for a different reason.
+ "$BIN/ffmpeg.exe" -hide_banner -loglevel error -y -ss 0 -t 1 -i tone.m4a -vn \
+ -af "aresample=48000,aformat=sample_fmts=s16:channel_layouts=stereo" \
+ -f s16le span.pcm
"$BIN/ffmpeg.exe" -hide_banner -loglevel error -y -i tone.m4a -vn \
-af "aresample=48000,asetrate=52000,aresample=48000,atempo=0.92,highpass=f=85,equalizer=f=4200:t=q:w=0.8:g=3,compand=attacks=0.01:decays=0.18:points=-80/-80|0/-1.5,deesser=i=0.3,aecho=0.8:0.82:38:0.02,alimiter=limit=0.94,aphaser=type=t:speed=1.2,flanger=delay=2:depth=4,bass=g=5,treble=g=4" \
-ac 2 -ar 48000 -c:a pcm_s16le -f wav filtered.wav
diff --git a/desktop/docs/decisions/0004-audio-preview-in-the-webview.md b/desktop/docs/decisions/0004-audio-preview-in-the-webview.md
index 152fab3a..87c424aa 100644
--- a/desktop/docs/decisions/0004-audio-preview-in-the-webview.md
+++ b/desktop/docs/decisions/0004-audio-preview-in-the-webview.md
@@ -1,5 +1,12 @@
# 0004 - Audio and video preview run in the webview, for now
+> **Superseded in part by [0005](0005-audio-playback-in-the-engine.md):**
+> audio playback now lives in the engine - decoded, mixed and clocked in
+> `src-tauri/src/playback.rs`, with the UI following the engine's position
+> events. What remains true here is the *video* half: the monitor picture is
+> still the webview's `<video>` element, muted, until the engine can present
+> frames.
+
## Decision
Playback preview - both the picture in the monitor and the sound - uses the
webview's own `<video>` and `<audio>` elements, fed through Tauri's asset
diff --git a/desktop/docs/decisions/0005-audio-playback-in-the-engine.md b/desktop/docs/decisions/0005-audio-playback-in-the-engine.md
new file mode 100644
index 00000000..72d70a4c
--- /dev/null
+++ b/desktop/docs/decisions/0005-audio-playback-in-the-engine.md
@@ -0,0 +1,51 @@
+# 0005 - Audio playback lives in the engine
+
+## Decision
+
+All audible playback is decoded, mixed and clocked by the Rust host
+(`src-tauri/src/playback.rs`). The webview no longer owns any part of the
+audio path: it describes the audible clip set (`audio_set_clips`), drives the
+transport (`transport_play` / `transport_pause` / `transport_seek`), and
+follows the engine's `transport` position events. The monitor's `<video>`
+element is permanently muted - it is a picture surface, nothing more.
+
+## Why
+
+The webview mixer (decision 0004) was built on `HTMLMediaElement` routed
+through `createMediaElementSource`, corrected against a wall-clock transport.
+That design hit its ceiling in practice, not in theory: WebKit's
+element-through-graph pipeline drops out audibly even on a three-second file,
+corrective reseeks are audible clicks, filtered clips required rendering a
+whole WAV per slider adjustment, and there were three clocks - wall clock,
+media element, audio context - that could only ever agree approximately.
+
+The engine path has exactly one clock: the audio device's sample counter.
+
+## How it works
+
+- **Decode once, mix forever.** Each audible clip's source span is decoded by
+ FFmpeg to raw 48 kHz stereo PCM, with the clip's filter chain and speed
+ baked in using the same filters the exporter uses - preview and export
+ cannot disagree. Volume and fades apply at mix time (the gain law mirrors
+ `clipGainAt`), so gain edits never re-decode.
+- **Disk-backed, memory-mapped.** PCM lands in `<project>/cache/audio/` keyed
+ by everything that changes the samples, and is memory-mapped rather than
+ loaded - memory stays bounded on long material, and a reopened project
+ replays instantly.
+- **A lock-free mix callback.** The cpal callback owns its state and drains a
+ message channel; no locks are taken on the audio thread. Clip-set changes,
+ play, pause and seek are messages.
+- **The clock flows outward.** The callback publishes its position; a 30 Hz
+ event stream carries it to the UI, which interpolates on the wall clock
+ between events. The UI never tells the engine what time it is while playing.
+
+## What this is not, yet
+
+- **Not video presentation.** The picture is still the muted webview element
+ showing the top-most clip (decision 0004's video half stands). When the
+ engine can decode and present frames, the same transport already drives it.
+- **Not streaming decode.** A clip's span is decoded whole before it joins
+ the mix. New material is silent for the decode's duration (cached
+ thereafter). Fine for editing; a capture monitor would need a ring buffer.
+- **Not an undo boundary.** The mixer is a pure function of the clip set it
+ is sent; it holds no editing state at all.
diff --git a/desktop/src-tauri/Cargo.lock b/desktop/src-tauri/Cargo.lock
index 6d02793e..e684ea61 100644
--- a/desktop/src-tauri/Cargo.lock
+++ b/desktop/src-tauri/Cargo.lock
@@ -32,6 +32,28 @@ dependencies = [
"alloc-no-stdlib",
]
+[[package]]
+name = "alsa"
+version = "0.9.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ed7572b7ba83a31e20d1b48970ee402d2e3e0537dcfe0a3ff4d6eb7508617d43"
+dependencies = [
+ "alsa-sys",
+ "bitflags 2.13.1",
+ "cfg-if",
+ "libc",
+]
+
+[[package]]
+name = "alsa-sys"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "db8fee663d06c4e303404ef5f40488a53e062f89ba8bfed81f42325aafad1527"
+dependencies = [
+ "libc",
+ "pkg-config",
+]
+
[[package]]
name = "android_system_properties"
version = "0.1.6"
@@ -534,6 +556,46 @@ dependencies = [
"libc",
]
+[[package]]
+name = "coreaudio-rs"
+version = "0.13.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1aae284fbaf7d27aa0e292f7677dfbe26503b0d555026f702940805a630eac17"
+dependencies = [
+ "bitflags 1.3.2",
+ "libc",
+ "objc2-audio-toolbox",
+ "objc2-core-audio",
+ "objc2-core-audio-types",
+ "objc2-core-foundation",
+]
+
+[[package]]
+name = "cpal"
+version = "0.16.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cbd307f43cc2a697e2d1f8bc7a1d824b5269e052209e28883e5bc04d095aaa3f"
+dependencies = [
+ "alsa",
+ "coreaudio-rs",
+ "dasp_sample",
+ "jni",
+ "js-sys",
+ "libc",
+ "mach2",
+ "ndk",
+ "ndk-context",
+ "num-derive",
+ "num-traits",
+ "objc2-audio-toolbox",
+ "objc2-core-audio",
+ "objc2-core-audio-types",
+ "wasm-bindgen",
+ "wasm-bindgen-futures",
+ "web-sys",
+ "windows 0.54.0",
+]
+
[[package]]
name = "cpufeatures"
version = "0.2.17"
@@ -650,6 +712,12 @@ dependencies = [
"syn 2.0.119",
]
+[[package]]
+name = "dasp_sample"
+version = "0.11.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0c87e182de0887fd5361989c677c4e8f5000cd9491d6d563161a8f3a5519fc7f"
+
[[package]]
name = "dbus"
version = "0.9.12"
@@ -2057,6 +2125,15 @@ version = "0.4.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
+[[package]]
+name = "mach2"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d640282b302c0bb0a2a8e0233ead9035e3bed871f0b7e81fe4a1ec829765db44"
+dependencies = [
+ "libc",
+]
+
[[package]]
name = "markup5ever"
version = "0.38.0"
@@ -2074,6 +2151,15 @@ version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
+[[package]]
+name = "memmap2"
+version = "0.9.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d1219ed1b7f229ee7104d281dd01d6802fe28bb6e95d292942c4daacdeb798c0"
+dependencies = [
+ "libc",
+]
+
[[package]]
name = "memoffset"
version = "0.9.1"
@@ -2146,6 +2232,12 @@ dependencies = [
"thiserror 1.0.69",
]
+[[package]]
+name = "ndk-context"
+version = "0.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "27b02d87554356db9e9a873add8782d4ea6e3e58ea071a9adb9a2e8ddb884a8b"
+
[[package]]
name = "ndk-sys"
version = "0.6.0+11769913"
@@ -2167,6 +2259,17 @@ version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441"
+[[package]]
+name = "num-derive"
+version = "0.4.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ed3955f1a9c7c0c15e092f9c887db08b1fc683305fdf6eb6684f22555355e202"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.119",
+]
+
[[package]]
name = "num-traits"
version = "0.2.19"
@@ -2221,6 +2324,21 @@ dependencies = [
"objc2-foundation",
]
+[[package]]
+name = "objc2-audio-toolbox"
+version = "0.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6948501a91121d6399b79abaa33a8aa4ea7857fe019f341b8c23ad6e81b79b08"
+dependencies = [
+ "bitflags 2.13.1",
+ "libc",
+ "objc2",
+ "objc2-core-audio",
+ "objc2-core-audio-types",
+ "objc2-core-foundation",
+ "objc2-foundation",
+]
+
[[package]]
name = "objc2-cloud-kit"
version = "0.3.2"
@@ -2232,6 +2350,28 @@ dependencies = [
"objc2-foundation",
]
+[[package]]
+name = "objc2-core-audio"
+version = "0.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e1eebcea8b0dbff5f7c8504f3107c68fc061a3eb44932051c8cf8a68d969c3b2"
+dependencies = [
+ "dispatch2",
+ "objc2",
+ "objc2-core-audio-types",
+ "objc2-core-foundation",
+]
+
+[[package]]
+name = "objc2-core-audio-types"
+version = "0.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5a89f2ec274a0cf4a32642b2991e8b351a404d290da87bb6a9a9d8632490bd1c"
+dependencies = [
+ "bitflags 2.13.1",
+ "objc2",
+]
+
[[package]]
name = "objc2-core-data"
version = "0.3.2"
@@ -2827,6 +2967,8 @@ version = "0.1.0"
name = "relay-desktop"
version = "0.1.0"
dependencies = [
+ "cpal",
+ "memmap2",
"relay-core",
"relay-media",
"relay-render",
@@ -3493,7 +3635,7 @@ dependencies = [
"tao-macros",
"unicode-segmentation",
"url",
- "windows",
+ "windows 0.61.3",
"windows-core 0.61.2",
"windows-version",
"x11-dl",
@@ -3565,7 +3707,7 @@ dependencies = [
"webkit2gtk",
"webview2-com",
"window-vibrancy",
- "windows",
+ "windows 0.61.3",
]
[[package]]
@@ -3706,7 +3848,7 @@ dependencies = [
"tauri-plugin",
"thiserror 2.0.20",
"url",
- "windows",
+ "windows 0.61.3",
"zbus",
]
@@ -3732,7 +3874,7 @@ dependencies = [
"url",
"webkit2gtk",
"webview2-com",
- "windows",
+ "windows 0.61.3",
]
[[package]]
@@ -3757,7 +3899,7 @@ dependencies = [
"url",
"webkit2gtk",
"webview2-com",
- "windows",
+ "windows 0.61.3",
"wry",
]
@@ -4511,7 +4653,7 @@ checksum = "7130243a7a5b33c54a444e54842e6a9e133de08b5ad7b5861cd8ed9a6a5bc96a"
dependencies = [
"webview2-com-macros",
"webview2-com-sys",
- "windows",
+ "windows 0.61.3",
"windows-core 0.61.2",
"windows-implement",
"windows-interface",
@@ -4535,7 +4677,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "381336cfffd772377d291702245447a5251a2ffa5bad679c99e61bc48bacbf9c"
dependencies = [
"thiserror 2.0.20",
- "windows",
+ "windows 0.61.3",
"windows-core 0.61.2",
]
@@ -4585,6 +4727,16 @@ dependencies = [
"windows-version",
]
+[[package]]
+name = "windows"
+version = "0.54.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9252e5725dbed82865af151df558e754e4a3c2c30818359eb17465f1346a1b49"
+dependencies = [
+ "windows-core 0.54.0",
+ "windows-targets 0.52.6",
+]
+
[[package]]
name = "windows"
version = "0.61.3"
@@ -4607,6 +4759,16 @@ dependencies = [
"windows-core 0.61.2",
]
+[[package]]
+name = "windows-core"
+version = "0.54.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "12661b9c89351d684a50a8a643ce5f608e20243b9fb84687800163429f161d65"
+dependencies = [
+ "windows-result 0.1.2",
+ "windows-targets 0.52.6",
+]
+
[[package]]
name = "windows-core"
version = "0.61.2"
@@ -4688,6 +4850,15 @@ dependencies = [
"windows-link 0.1.3",
]
+[[package]]
+name = "windows-result"
+version = "0.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5e383302e8ec8515204254685643de10811af0ed97ea37210dc26fb0032647f8"
+dependencies = [
+ "windows-targets 0.52.6",
+]
+
[[package]]
name = "windows-result"
version = "0.3.4"
@@ -5048,7 +5219,7 @@ dependencies = [
"webkit2gtk",
"webkit2gtk-sys",
"webview2-com",
- "windows",
+ "windows 0.61.3",
"windows-core 0.61.2",
"windows-version",
"x11-dl",
diff --git a/desktop/src-tauri/Cargo.toml b/desktop/src-tauri/Cargo.toml
index 8586e52f..0a0b9867 100644
--- a/desktop/src-tauri/Cargo.toml
+++ b/desktop/src-tauri/Cargo.toml
@@ -28,6 +28,11 @@ relay-media = { path = "../../engine/crates/relay-media" }
relay-render = { path = "../../engine/crates/relay-render" }
rusqlite = { version = "0.40.2", features = ["bundled"] }
+# Engine-owned audio playback: cpal talks to the output device, memmap2 keeps
+# decoded PCM disk-backed instead of resident.
+cpal = "0.16"
+memmap2 = "0.9"
+
[profile.release]
opt-level = 3
lto = true
diff --git a/desktop/src-tauri/src/export.rs b/desktop/src-tauri/src/export.rs
index 43046830..3951d9ce 100644
--- a/desktop/src-tauri/src/export.rs
+++ b/desktop/src-tauri/src/export.rs
@@ -17,11 +17,11 @@ use std::path::{Path, PathBuf};
use relay_core::frame::Frame;
use relay_core::time::{FrameRate, Rational};
-use relay_core::timeline::{Clip, ClipId, MediaRef, Timeline, Track, TrackKind};
+use relay_core::timeline::{Clip, ClipId, MediaRef, Timeline, Track, TrackKind, Transform};
use relay_media::{
DecodeOptions, EncodeOptions, FfmpegDecoder, FfmpegEncoder, FrameSink, FrameSource,
};
-use relay_render::{Compositor, CpuCompositor, Layer, plan_frame};
+use relay_render::{Compositor, CpuCompositor, Layer, Placement, plan_frame};
use serde::{Deserialize, Serialize};
use tauri::Emitter;
@@ -57,6 +57,25 @@ pub struct ExportClip {
/// False lets pitch rise with the rate, like tape.
#[serde(default = "yes")]
pub preserve_pitch: bool,
+ /// Multiplier over the fitted size. 1 fills the frame, preserving aspect.
+ #[serde(default = "unity")]
+ pub scale: f64,
+ /// Offset of the picture's centre from frame centre, frame-width fraction.
+ #[serde(default)]
+ pub offset_x: f64,
+ /// Offset as a frame-height fraction.
+ #[serde(default)]
+ pub offset_y: f64,
+ /// Clockwise rotation in degrees.
+ #[serde(default)]
+ pub rotation: f64,
+ /// The source's pixel size, when the UI knows it. What makes an
+ /// aspect-correct decode possible - absent, the frame is filled edge to
+ /// edge the way it always was.
+ #[serde(default)]
+ pub media_width: Option<u32>,
+ #[serde(default)]
+ pub media_height: Option<u32>,
}
fn yes() -> bool {
@@ -170,7 +189,7 @@ fn render_picture(
visible: &[&ExportClip],
destination: &Path,
) -> Result<(), String> {
- let (timeline, stills) = build_timeline(request, rate, visible);
+ let (timeline, stills, decode_sizes) = build_timeline(request, rate, visible);
let mut encoder = FfmpegEncoder::create(
destination,
@@ -197,14 +216,21 @@ fn render_picture(
let time = rate.time_of_frame(index);
let plan = plan_frame(&timeline, time);
- let mut sources: Vec<(Frame, f32)> = Vec::with_capacity(plan.layers.len());
+ let mut sources: Vec<(Frame, f32, Transform)> = Vec::with_capacity(plan.layers.len());
for layer in &plan.layers {
let decoder = match decoders.entry(layer.clip) {
std::collections::hash_map::Entry::Occupied(entry) => entry.into_mut(),
std::collections::hash_map::Entry::Vacant(entry) => {
+ // Aspect-correct: decode at the source's fitted size and
+ // let the compositor place it, rather than stretching to
+ // the frame and losing the picture's shape.
+ let (decode_width, decode_height) = decode_sizes
+ .get(&layer.clip)
+ .copied()
+ .unwrap_or((request.width, request.height));
let mut options = DecodeOptions::default()
.starting_at(layer.source_time)
- .scaled_to(request.width, request.height)
+ .scaled_to(decode_width, decode_height)
.at_rate(rate);
// A still is a one-frame stream. Without looping it would
@@ -224,13 +250,33 @@ fn render_picture(
// aborting the export - a clip trimmed past its media's end is a
// mistake in the edit, not a failure of the renderer.
if let Some(frame) = decoder.next_frame().map_err(|error| error.to_string())? {
- sources.push((frame, layer.opacity));
+ sources.push((frame, layer.opacity, layer.transform));
}
}
let layers: Vec<Layer<'_>> = sources
.iter()
- .map(|(frame, opacity)| Layer::new(frame).with_opacity(*opacity))
+ .map(|(frame, opacity, transform)| {
+ // Fitted and centred is the base; the clip's transform moves
+ // it from there. The fraction-to-pixel conversion happens
+ // here and nowhere else.
+ let x = (i64::from(request.width) - i64::from(frame.width())) / 2;
+ let y = (i64::from(request.height) - i64::from(frame.height())) / 2;
+ let placement = if transform.is_identity() {
+ Placement::IDENTITY
+ } else {
+ Placement {
+ scale: transform.scale as f32,
+ rotation: transform.rotation.to_radians() as f32,
+ translate_x: (transform.offset_x * f64::from(request.width)) as f32,
+ translate_y: (transform.offset_y * f64::from(request.height)) as f32,
+ }
+ };
+ Layer::new(frame)
+ .at(x as i32, y as i32)
+ .with_opacity(*opacity)
+ .with_placement(placement)
+ })
.collect();
let composed = compositor.composite(request.width, request.height, &layers);
@@ -257,9 +303,14 @@ fn build_timeline(
request: &ExportRequest,
rate: FrameRate,
visible: &[&ExportClip],
-) -> (Timeline, std::collections::HashSet<ClipId>) {
+) -> (
+ Timeline,
+ std::collections::HashSet<ClipId>,
+ HashMap<ClipId, (u32, u32)>,
+) {
let mut timeline = Timeline::new(request.width, request.height, rate);
let mut stills = std::collections::HashSet::new();
+ let mut decode_sizes: HashMap<ClipId, (u32, u32)> = HashMap::new();
let lanes = visible.iter().map(|clip| clip.track).max().unwrap_or(0) + 1;
let tracks: Vec<_> = (0..lanes)
@@ -278,15 +329,39 @@ fn build_timeline(
let mut engine_clip = Clip::new(MediaRef::new(&clip.path), start, duration);
engine_clip.source_start = quantise(clip.source_start, rate);
+ engine_clip.transform = Transform {
+ scale: clip.scale,
+ offset_x: clip.offset_x,
+ offset_y: clip.offset_y,
+ rotation: clip.rotation,
+ };
if let Some(id) = timeline.add_clip(tracks[clip.track], engine_clip) {
if clip.kind == "image" {
stills.insert(id);
}
+ if let Some(size) = fitted_size(request, clip) {
+ decode_sizes.insert(id, size);
+ }
}
}
- (timeline, stills)
+ (timeline, stills, decode_sizes)
+}
+
+/// The source's contain-fitted size inside the output frame, or `None` when
+/// the UI never learnt the source's dimensions.
+fn fitted_size(request: &ExportRequest, clip: &ExportClip) -> Option<(u32, u32)> {
+ let media_width = clip.media_width.filter(|value| *value > 0)?;
+ let media_height = clip.media_height.filter(|value| *value > 0)?;
+
+ let fit = (f64::from(request.width) / f64::from(media_width))
+ .min(f64::from(request.height) / f64::from(media_height));
+ let width = ((f64::from(media_width) * fit).round() as u32).max(2);
+ let height = ((f64::from(media_height) * fit).round() as u32).max(2);
+ // Even, because a decoder asked for an odd width may round it itself and
+ // then every frame read is misaligned by a pixel's worth of bytes.
+ Some((width & !1, height & !1))
}
fn quantise(seconds: f64, rate: FrameRate) -> Rational {
diff --git a/desktop/src-tauri/src/lib.rs b/desktop/src-tauri/src/lib.rs
index f137aa6f..f435b4ea 100644
--- a/desktop/src-tauri/src/lib.rs
+++ b/desktop/src-tauri/src/lib.rs
@@ -7,6 +7,7 @@
//! `relay-render` where it can be unit-tested without a window.
mod export;
+mod playback;
mod projects;
use serde::Serialize;
@@ -140,6 +141,49 @@ fn read_media_bytes(path: String) -> Result<tauri::ipc::Response, String> {
.map_err(|error| format!("could not read {path}: {error}"))
}
+/// Where one artwork file lives inside a project's cache.
+///
+/// The cache sits in the project folder so it travels with the project and
+/// vanishes with it. The key is confined to a single flat filename - anything
+/// that could walk out of the folder is refused rather than sanitised,
+/// because the only caller is our own frontend and a strange key is a bug.
+fn artwork_file(project: &str, key: &str) -> Result<std::path::PathBuf, String> {
+ if key.is_empty()
+ || !key
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-'))
+ || key.starts_with('.')
+ {
+ return Err(format!("refusing artwork key {key:?}"));
+ }
+ let root = std::path::Path::new(project);
+ if !root.is_dir() {
+ return Err(format!("{project} is not a project folder"));
+ }
+ Ok(root.join("cache").join(key))
+}
+
+/// Returns one cached artwork file, or an error the frontend treats as a miss.
+#[tauri::command]
+fn read_artwork(project: String, key: String) -> Result<tauri::ipc::Response, String> {
+ let file = artwork_file(&project, &key)?;
+ std::fs::read(&file)
+ .map(tauri::ipc::Response::new)
+ .map_err(|error| format!("no cached artwork {key}: {error}"))
+}
+
+/// Stores one artwork file in the project's cache. Best-effort: the caller
+/// fires and forgets, and a failed write only means regenerating next launch.
+#[tauri::command]
+fn write_artwork(project: String, key: String, bytes: Vec<u8>) -> Result<(), String> {
+ let file = artwork_file(&project, &key)?;
+ if let Some(parent) = file.parent() {
+ std::fs::create_dir_all(parent)
+ .map_err(|error| format!("could not create {}: {error}", parent.display()))?;
+ }
+ std::fs::write(&file, bytes).map_err(|error| format!("could not write {key}: {error}"))
+}
+
/// Renders a strip of evenly spaced frames from a video as one JPEG.
///
/// One image rather than N, because the timeline draws the frames as slices of
@@ -176,12 +220,15 @@ fn filmstrip(path: &str, count: u32, height: u32) -> Result<Vec<u8>, String> {
"-vf",
// scale=-2 keeps the aspect ratio and an even width, which the
// JPEG encoder requires.
+ // lanczos because the default bilinear leaves 4K sources visibly
+ // mushy at thumbnail sizes, and the strip is rendered once but
+ // looked at constantly.
&format!(
- "fps={:.6},scale=-2:{height},tile={count}x1",
+ "fps={:.6},scale=-2:{height}:flags=lanczos,tile={count}x1",
f64::from(count) / duration
),
])
- .args(["-frames:v", "1", "-q:v", "6", "-f", "mjpeg", "-"])
+ .args(["-frames:v", "1", "-q:v", "3", "-f", "mjpeg", "-"])
.output()
.map_err(|error| format!("could not run ffmpeg: {error}"))?;
@@ -195,61 +242,33 @@ fn filmstrip(path: &str, count: u32, height: u32) -> Result<Vec<u8>, String> {
Ok(output.stdout)
}
-/// Renders one clip's audio through a filter chain and returns it as WAV.
-///
-/// This is what lets the preview play filtered audio *exactly* as the export
-/// will render it. The alternative - reimplementing pitch shifting, companding
-/// and de-essing in Web Audio - would be a second implementation to keep in
-/// step with the first, and it would be wrong in ways nobody would notice
-/// until the exported file sounded different from the edit.
+/// The audible clip set, replaced wholesale whenever the timeline changes.
///
-/// WAV because it decodes instantly in the browser and needs no container
-/// seeking; the result is short-lived and never written to disk.
+/// Decoding, mixing and the playback clock all live in the engine - see
+/// `playback.rs`. The UI describes what should be audible and follows the
+/// engine's `transport` position events.
#[tauri::command]
-async fn render_filtered_audio(
- path: String,
- source_start: f64,
- duration: f64,
- chain: String,
-) -> Result<tauri::ipc::Response, String> {
- let bytes = tauri::async_runtime::spawn_blocking(move || {
- // A filter chain arrives as text and goes straight to FFmpeg, so it
- // must not be able to smuggle in another argument.
- if chain.contains('\n') || chain.contains('\r') {
- return Err("filter chain contains a line break".to_owned());
- }
+fn audio_set_clips(
+ state: tauri::State<'_, std::sync::Arc<playback::Playback>>,
+ project: String,
+ clips: Vec<playback::ClipSpec>,
+) {
+ state.set_clips(std::path::PathBuf::from(project), clips);
+}
- let mut command = std::process::Command::new(relay_media::ffmpeg());
- command
- .args(["-hide_banner", "-nostdin", "-loglevel", "error"])
- .args(["-ss", &format!("{source_start:.6}")])
- .args(["-t", &format!("{duration:.6}")])
- .args(["-i", &path])
- .args(["-vn", "-af", &chain])
- // 16-bit stereo at 48k: what the filters run at anyway, and small
- // enough to move over the bridge without a second thought.
- .args(["-ac", "2", "-ar", "48000", "-c:a", "pcm_s16le"])
- .args(["-f", "wav", "-"]);
-
- let output = command
- .output()
- .map_err(|error| format!("could not run ffmpeg: {error}"))?;
-
- if !output.status.success() {
- let detail = String::from_utf8_lossy(&output.stderr);
- let tail = detail.lines().rev().take(2).collect::<Vec<_>>().join(" / ");
- return Err(format!("filter failed: {tail}"));
- }
- if output.stdout.is_empty() {
- return Err("filter produced no audio".to_owned());
- }
+#[tauri::command]
+fn transport_play(state: tauri::State<'_, std::sync::Arc<playback::Playback>>, position: f64) {
+ state.play(position);
+}
- Ok(output.stdout)
- })
- .await
- .map_err(|error| format!("filter task failed: {error}"))??;
+#[tauri::command]
+fn transport_pause(state: tauri::State<'_, std::sync::Arc<playback::Playback>>) {
+ state.pause();
+}
- Ok(tauri::ipc::Response::new(bytes))
+#[tauri::command]
+fn transport_seek(state: tauri::State<'_, std::sync::Arc<playback::Playback>>, position: f64) {
+ state.seek(position);
}
/// Creates a project folder, writes its manifest and records it as recent.
@@ -406,7 +425,9 @@ pub fn run() {
.setup(|app| {
// Resolved here rather than before the builder, because finding
// the resource directory needs the app to exist.
+ use tauri::Manager;
use_bundled_ffmpeg(app);
+ app.manage(playback::Playback::start(app.handle().clone()));
Ok(())
})
.plugin(tauri_plugin_opener::init())
@@ -415,8 +436,13 @@ pub fn run() {
probe_media,
engine_version,
read_media_bytes,
- render_filtered_audio,
+ audio_set_clips,
+ transport_play,
+ transport_pause,
+ transport_seek,
extract_filmstrip,
+ read_artwork,
+ write_artwork,
create_project,
open_project,
save_project,
diff --git a/desktop/src-tauri/src/playback.rs b/desktop/src-tauri/src/playback.rs
new file mode 100644
index 00000000..2ce8fd9d
--- /dev/null
+++ b/desktop/src-tauri/src/playback.rs
@@ -0,0 +1,475 @@
+//! Engine-owned audio playback.
+//!
+//! This replaces the webview audio preview (see the superseded
+//! docs/decisions/0004): the engine decodes, mixes and clocks all audible
+//! clips itself, and the UI is a controller that follows.
+//!
+//! The shape of it:
+//!
+//! - Every audible clip's span is decoded **once** by FFmpeg straight to raw
+//! PCM - the clip's filter chain and speed baked in with the same filters
+//! the exporter uses, so the preview and the export cannot disagree. The
+//! PCM lands as a file in the project's cache and is memory-mapped, so
+//! memory stays bounded however long the material is, and a reopened
+//! project pays nothing.
+//!
+//! - One cpal output stream mixes the mapped clips sample by sample, with
+//! volume and fades applied at mix time (they mirror `clipGainAt` in the
+//! frontend exactly). Gain-only edits therefore never re-decode.
+//!
+//! - The playback clock is the audio device's own sample counter. The UI
+//! receives `transport` position events and interpolates between them;
+//! there is no second clock to drift against.
+//!
+//! The mix callback owns its state and receives changes as messages, so it
+//! takes no locks on the audio thread. Decodes run on worker threads and the
+//! clip set is re-sent to the callback as each one lands.
+
+use std::collections::{HashMap, HashSet};
+use std::hash::{Hash, Hasher};
+use std::path::{Path, PathBuf};
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::sync::{mpsc, Arc, Mutex};
+
+use serde::Deserialize;
+use tauri::Emitter;
+
+/// Every clip is decoded to this rate; the callback resamples to the device.
+const PCM_RATE: u32 = 48_000;
+
+/// One audible clip, as the frontend describes it.
+#[derive(Clone, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct ClipSpec {
+ pub path: String,
+ /// Timeline seconds.
+ pub start: f64,
+ pub duration: f64,
+ /// Seconds into the source file.
+ pub source_start: f64,
+ pub volume: f32,
+ pub fade_in: f64,
+ pub fade_out: f64,
+ pub speed: f64,
+ pub preserve_pitch: bool,
+ /// FFmpeg audio filter chain, or empty.
+ #[serde(default)]
+ pub chain: String,
+}
+
+/// Decoded audio: 16-bit little-endian stereo at [`PCM_RATE`], memory-mapped
+/// from the cache file so the OS decides what stays resident.
+struct Pcm {
+ map: memmap2::Mmap,
+ frames: u64,
+}
+
+impl Pcm {
+ fn open(path: &Path) -> Result<Pcm, String> {
+ let file = std::fs::File::open(path)
+ .map_err(|error| format!("could not open {}: {error}", path.display()))?;
+ let map = unsafe { memmap2::Mmap::map(&file) }
+ .map_err(|error| format!("could not map {}: {error}", path.display()))?;
+ let frames = (map.len() / 4) as u64;
+ Ok(Pcm { map, frames })
+ }
+
+ /// One sample as a float in [-1, 1]. Out of range reads are silence.
+ fn sample(&self, frame: u64, channel: usize) -> f32 {
+ if frame >= self.frames {
+ return 0.0;
+ }
+ let at = (frame as usize * 2 + channel) * 2;
+ let raw = i16::from_le_bytes([self.map[at], self.map[at + 1]]);
+ f32::from(raw) / 32768.0
+ }
+}
+
+/// A clip the mix callback can play: spec fields it needs, plus the audio.
+struct ActiveClip {
+ start: f64,
+ duration: f64,
+ volume: f32,
+ fade_in: f64,
+ fade_out: f64,
+ pcm: Arc<Pcm>,
+}
+
+/// Mirror of the frontend's `clipGainAt`, so preview loudness has exactly one
+/// definition even though it is computed in two languages.
+fn gain_at(clip: &ActiveClip, local: f64) -> f32 {
+ if local < 0.0 || local > clip.duration {
+ return 0.0;
+ }
+ let mut gain = f64::from(clip.volume);
+ if clip.fade_in > 0.0 {
+ gain *= (local / clip.fade_in).min(1.0);
+ }
+ if clip.fade_out > 0.0 {
+ gain *= ((clip.duration - local) / clip.fade_out).min(1.0);
+ }
+ gain.max(0.0) as f32
+}
+
+enum Msg {
+ SetClips(Vec<ActiveClip>),
+ Play(f64),
+ Pause,
+ Seek(f64),
+}
+
+struct Shared {
+ /// Timeline position in microseconds, written by the mix callback.
+ position_micros: AtomicU64,
+ playing: AtomicBool,
+}
+
+pub struct Playback {
+ tx: mpsc::Sender<Msg>,
+ shared: Arc<Shared>,
+ /// Decoded clips by decode key, for the lifetime of the app.
+ cache: Mutex<HashMap<String, Arc<Pcm>>>,
+ decoding: Mutex<HashSet<String>>,
+ /// What the timeline currently wants audible.
+ specs: Mutex<Vec<ClipSpec>>,
+}
+
+impl Playback {
+ pub fn start(app: tauri::AppHandle) -> Arc<Playback> {
+ let (tx, rx) = mpsc::channel::<Msg>();
+ let shared = Arc::new(Shared {
+ position_micros: AtomicU64::new(0),
+ playing: AtomicBool::new(false),
+ });
+
+ {
+ let shared = Arc::clone(&shared);
+ std::thread::Builder::new()
+ .name("audio-output".into())
+ .spawn(move || audio_thread(rx, shared))
+ .expect("could not spawn the audio thread");
+ }
+
+ // Position events at ~30Hz while playing. The UI interpolates between
+ // them, so this cadence bounds correction error, not smoothness.
+ {
+ let shared = Arc::clone(&shared);
+ std::thread::Builder::new()
+ .name("transport-events".into())
+ .spawn(move || loop {
+ std::thread::sleep(std::time::Duration::from_millis(33));
+ if shared.playing.load(Ordering::Relaxed) {
+ let position =
+ shared.position_micros.load(Ordering::Relaxed) as f64 / 1_000_000.0;
+ let _ = app.emit("transport", position);
+ }
+ })
+ .expect("could not spawn the transport event thread");
+ }
+
+ Arc::new(Playback {
+ tx,
+ shared,
+ cache: Mutex::new(HashMap::new()),
+ decoding: Mutex::new(HashSet::new()),
+ specs: Mutex::new(Vec::new()),
+ })
+ }
+
+ pub fn play(&self, position: f64) {
+ self.shared.playing.store(true, Ordering::Relaxed);
+ self.shared
+ .position_micros
+ .store((position.max(0.0) * 1_000_000.0) as u64, Ordering::Relaxed);
+ let _ = self.tx.send(Msg::Play(position.max(0.0)));
+ }
+
+ pub fn pause(&self) {
+ self.shared.playing.store(false, Ordering::Relaxed);
+ let _ = self.tx.send(Msg::Pause);
+ }
+
+ pub fn seek(&self, position: f64) {
+ let _ = self.tx.send(Msg::Seek(position.max(0.0)));
+ }
+
+ /// Replaces the audible clip set.
+ ///
+ /// Anything already decoded plays immediately; anything new decodes on a
+ /// worker and joins the mix when it lands. `project` is the project
+ /// folder, whose cache holds the PCM files.
+ pub fn set_clips(self: &Arc<Self>, project: PathBuf, specs: Vec<ClipSpec>) {
+ *self.specs.lock().unwrap() = specs.clone();
+
+ for spec in specs {
+ let key = decode_key(&spec);
+ if self.cache.lock().unwrap().contains_key(&key) {
+ continue;
+ }
+ if !self.decoding.lock().unwrap().insert(key.clone()) {
+ continue;
+ }
+
+ let this = Arc::clone(self);
+ let project = project.clone();
+ std::thread::Builder::new()
+ .name("audio-decode".into())
+ .spawn(move || {
+ match decode(&spec, &project, &key) {
+ Ok(pcm) => {
+ this.cache.lock().unwrap().insert(key.clone(), Arc::new(pcm));
+ }
+ Err(error) => eprintln!("audio decode failed for {}: {error}", spec.path),
+ }
+ this.decoding.lock().unwrap().remove(&key);
+ this.resync();
+ })
+ .expect("could not spawn a decode thread");
+ }
+
+ self.resync();
+ }
+
+ /// Sends the mix callback everything currently wanted *and* decoded.
+ fn resync(&self) {
+ let specs = self.specs.lock().unwrap();
+ let cache = self.cache.lock().unwrap();
+ let active = specs
+ .iter()
+ .filter_map(|spec| {
+ let pcm = cache.get(&decode_key(spec))?;
+ Some(ActiveClip {
+ start: spec.start,
+ duration: spec.duration,
+ volume: spec.volume,
+ fade_in: spec.fade_in,
+ fade_out: spec.fade_out,
+ pcm: Arc::clone(pcm),
+ })
+ })
+ .collect();
+ let _ = self.tx.send(Msg::SetClips(active));
+ }
+}
+
+/// Identity of one decoded span: everything that changes the samples.
+/// Volume, fades and timeline position are deliberately absent - they apply
+/// at mix time.
+fn decode_key(spec: &ClipSpec) -> String {
+ let mut hasher = std::collections::hash_map::DefaultHasher::new();
+ spec.path.hash(&mut hasher);
+ spec.source_start.to_bits().hash(&mut hasher);
+ spec.duration.to_bits().hash(&mut hasher);
+ spec.speed.to_bits().hash(&mut hasher);
+ spec.preserve_pitch.hash(&mut hasher);
+ spec.chain.hash(&mut hasher);
+ format!("{:016x}", hasher.finish())
+}
+
+/// Decodes one clip span to the cache and maps it.
+///
+/// The FFmpeg invocation mirrors the exporter's treatment of filters and
+/// speed, so what the preview mixes is what the export renders. A file
+/// already in the cache is reused as is - that is the point of the cache.
+fn decode(spec: &ClipSpec, project: &Path, key: &str) -> Result<Pcm, String> {
+ if spec.chain.contains('\n') || spec.chain.contains('\r') {
+ return Err("filter chain contains a line break".to_owned());
+ }
+
+ let directory = project.join("cache").join("audio");
+ let destination = directory.join(format!("{key}.pcm"));
+ if destination.is_file() {
+ return Pcm::open(&destination);
+ }
+ std::fs::create_dir_all(&directory)
+ .map_err(|error| format!("could not create {}: {error}", directory.display()))?;
+
+ let mut filters: Vec<String> = Vec::new();
+ if !spec.chain.is_empty() {
+ filters.push(spec.chain.clone());
+ }
+ if (spec.speed - 1.0).abs() > 1e-9 {
+ if spec.preserve_pitch {
+ // atempo only accepts [0.5, 2]; factors outside compose.
+ let mut factor = spec.speed.clamp(0.0625, 16.0);
+ while factor > 2.0 {
+ filters.push("atempo=2.0".to_owned());
+ factor /= 2.0;
+ }
+ while factor < 0.5 {
+ filters.push("atempo=0.5".to_owned());
+ factor /= 0.5;
+ }
+ filters.push(format!("atempo={factor:.6}"));
+ } else {
+ // Tape behaviour: resample to the common rate, shift it, and land
+ // back on the common rate. Pitch rides the speed.
+ filters.push(format!(
+ "aresample={PCM_RATE},asetrate={:.0},aresample={PCM_RATE}",
+ f64::from(PCM_RATE) * spec.speed
+ ));
+ }
+ }
+ filters.push(format!("aresample={PCM_RATE}"));
+ filters.push("aformat=sample_fmts=s16:channel_layouts=stereo".to_owned());
+
+ // A sped-up clip reads further into the file than it occupies on the
+ // timeline: the source window is `duration * speed`, exactly the window
+ // the exporter trims.
+ let window = spec.duration * spec.speed.clamp(0.0625, 16.0);
+
+ let temporary = directory.join(format!("{key}.pcm.decoding"));
+ let file = std::fs::File::create(&temporary)
+ .map_err(|error| format!("could not create {}: {error}", temporary.display()))?;
+
+ let status = std::process::Command::new(relay_media::ffmpeg())
+ .args(["-hide_banner", "-nostdin", "-loglevel", "error"])
+ .args(["-ss", &format!("{:.6}", spec.source_start)])
+ .args(["-t", &format!("{window:.6}")])
+ .args(["-i", &spec.path])
+ .args(["-vn", "-af", &filters.join(",")])
+ .args(["-f", "s16le", "-"])
+ .stdout(file)
+ .status()
+ .map_err(|error| format!("could not run ffmpeg: {error}"))?;
+
+ if !status.success() {
+ let _ = std::fs::remove_file(&temporary);
+ return Err(format!("ffmpeg exited with {status}"));
+ }
+
+ std::fs::rename(&temporary, &destination)
+ .map_err(|error| format!("could not commit {}: {error}", destination.display()))?;
+ Pcm::open(&destination)
+}
+
+/// Owns the output stream for the life of the app.
+///
+/// The stream's callback owns all mix state and drains the message channel
+/// itself, so no lock is ever taken on the audio thread.
+fn audio_thread(rx: mpsc::Receiver<Msg>, shared: Arc<Shared>) {
+ use cpal::traits::{DeviceTrait, HostTrait};
+
+ let host = cpal::default_host();
+ let Some(device) = host.default_output_device() else {
+ eprintln!("no audio output device; preview will be silent");
+ return;
+ };
+ let Ok(config) = device.default_output_config() else {
+ eprintln!("no default output config; preview will be silent");
+ return;
+ };
+
+ let sample_format = config.sample_format();
+ let config: cpal::StreamConfig = config.into();
+
+ let result = match sample_format {
+ cpal::SampleFormat::F32 => run_stream::<f32>(&device, &config, rx, shared),
+ cpal::SampleFormat::I16 => run_stream::<i16>(&device, &config, rx, shared),
+ cpal::SampleFormat::U16 => run_stream::<u16>(&device, &config, rx, shared),
+ other => Err(format!("unsupported sample format {other:?}")),
+ };
+ if let Err(error) = result {
+ eprintln!("audio output failed: {error}");
+ }
+}
+
+fn run_stream<T>(
+ device: &cpal::Device,
+ config: &cpal::StreamConfig,
+ rx: mpsc::Receiver<Msg>,
+ shared: Arc<Shared>,
+) -> Result<(), String>
+where
+ T: cpal::SizedSample + cpal::FromSample<f32>,
+{
+ use cpal::traits::{DeviceTrait, StreamTrait};
+
+ let channels = config.channels as usize;
+ let step = 1.0 / f64::from(config.sample_rate.0);
+
+ let mut clips: Vec<ActiveClip> = Vec::new();
+ let mut playing = false;
+ let mut position = 0.0f64;
+
+ let stream = device
+ .build_output_stream(
+ config,
+ move |data: &mut [T], _| {
+ while let Ok(message) = rx.try_recv() {
+ match message {
+ Msg::SetClips(next) => clips = next,
+ Msg::Play(at) => {
+ position = at;
+ playing = true;
+ }
+ Msg::Pause => playing = false,
+ Msg::Seek(at) => position = at,
+ }
+ }
+
+ if !playing {
+ data.fill(T::from_sample(0.0));
+ return;
+ }
+
+ for frame in data.chunks_mut(channels) {
+ let mut left = 0.0f32;
+ let mut right = 0.0f32;
+
+ for clip in &clips {
+ let local = position - clip.start;
+ if local < 0.0 || local >= clip.duration {
+ continue;
+ }
+ let gain = gain_at(clip, local);
+ if gain <= 0.0 {
+ continue;
+ }
+
+ // Linear interpolation across the 48k source grid -
+ // exact when the device runs at 48k, inaudibly close
+ // when it does not.
+ let read = local * f64::from(PCM_RATE);
+ let index = read.floor() as u64;
+ let fraction = (read - read.floor()) as f32;
+ let pcm = &clip.pcm;
+ left += gain
+ * (pcm.sample(index, 0) * (1.0 - fraction)
+ + pcm.sample(index + 1, 0) * fraction);
+ right += gain
+ * (pcm.sample(index, 1) * (1.0 - fraction)
+ + pcm.sample(index + 1, 1) * fraction);
+ }
+
+ // Hard limit. A mix of boosted clips can exceed full
+ // scale; wrapping would be far worse than flattening.
+ frame[0] = T::from_sample(left.clamp(-1.0, 1.0));
+ if channels > 1 {
+ frame[1] = T::from_sample(right.clamp(-1.0, 1.0));
+ }
+ for extra in frame.iter_mut().skip(2) {
+ *extra = T::from_sample(0.0);
+ }
+
+ position += step;
+ }
+
+ shared
+ .position_micros
+ .store((position * 1_000_000.0) as u64, Ordering::Relaxed);
+ },
+ |error| eprintln!("audio stream error: {error}"),
+ None,
+ )
+ .map_err(|error| format!("could not build the output stream: {error}"))?;
+
+ stream
+ .play()
+ .map_err(|error| format!("could not start the output stream: {error}"))?;
+
+ // The stream lives exactly as long as this thread.
+ loop {
+ std::thread::sleep(std::time::Duration::from_secs(3600));
+ }
+}
diff --git a/desktop/src-tauri/tauri.macos.conf.json b/desktop/src-tauri/tauri.macos.conf.json
new file mode 100644
index 00000000..17f6198f
--- /dev/null
+++ b/desktop/src-tauri/tauri.macos.conf.json
@@ -0,0 +1,20 @@
+{
+ "$schema": "https://schema.tauri.app/config/2",
+ "app": {
+ "windows": [
+ {
+ "title": "WolfCut",
+ "width": 1440,
+ "height": 900,
+ "minWidth": 1024,
+ "minHeight": 640,
+ "resizable": true,
+ "decorations": true,
+ "titleBarStyle": "Overlay",
+ "hiddenTitle": true,
+ "backgroundColor": "#07080c",
+ "dragDropEnabled": true
+ }
+ ]
+ }
+}
diff --git a/desktop/src/App.tsx b/desktop/src/App.tsx
index 25e85b79..a6a36ebf 100644
--- a/desktop/src/App.tsx
+++ b/desktop/src/App.tsx
@@ -1,5 +1,5 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
-import { isTauri } from "@tauri-apps/api/core";
+import { invoke, isTauri } from "@tauri-apps/api/core";
import { open as openDialog } from "@tauri-apps/plugin-dialog";
import { getCurrentWebview } from "@tauri-apps/api/webview";
import { getCurrentWindow } from "@tauri-apps/api/window";
@@ -14,8 +14,7 @@ import { RightPanel, type RightTab } from "./components/RightPanel";
import { StartScreen, type ProjectSession } from "./components/StartScreen";
import { TitleBar } from "./components/TitleBar";
import { TimelinePanel, resolveDrop, type Tool } from "./components/TimelinePanel";
-import { createAssets, requestAssets } from "./lib/assets";
-import { AudioPreview } from "./lib/audio";
+import { createAssets, requestAssets, requestVideoPeaks } from "./lib/assets";
import {
engineVersion,
loadProject,
@@ -28,10 +27,10 @@ import {
addClip,
addMedia,
addTrack,
- clipGainAt,
clipsAt,
- clipsOnTrack,
createProject,
+ detachAudio,
+ detachedAudioOf,
findClip,
firstFreeTrack,
findMedia,
@@ -42,6 +41,7 @@ import {
moveClip,
moveClips,
projectDuration,
+ reattachAudio,
removeClips,
removeFont,
removeTrack,
@@ -54,11 +54,10 @@ import {
updateClip,
trimClip,
whyNotMerge,
- type Clip,
type MediaItem,
type Project,
} from "./lib/project";
-import { buildChain, chainKey } from "./lib/filters";
+import { buildChain } from "./lib/filters";
import { familyForPath, registerFont } from "./lib/text";
import { fromDocument, toDocument } from "./lib/persist";
import { useTheme, type Theme } from "./lib/theme";
@@ -71,7 +70,7 @@ const MAX_SECONDS_PER_PIXEL = 2;
* The window: either the launch screen or an open editor.
*
* The editor is a separate component rather than a branch inside one, so that
- * every hook it owns - transport, audio, asset cache - mounts when a project
+ * every hook it owns - transport, asset cache - mounts when a project
* opens and unmounts when it closes. Closing a project therefore cannot leave
* a stale audio element or a half-finished waveform behind.
*/
@@ -144,6 +143,11 @@ function Editor({
const [saveState, setSaveState] = useState<"idle" | "saving" | "saved" | "failed">("idle");
/** Blocks autosave until the project on disk has been read in. */
const [loaded, setLoaded] = useState(false);
+ /**
+ * The output frame. Starts from the project settings but is editable from
+ * the preview footer, and the edited value is what the document keeps.
+ */
+ const [frame, setFrame] = useState({ width: session.width, height: session.height });
// Panel geometry.
const [leftWidth, setLeftWidth] = useState(340);
@@ -163,8 +167,8 @@ function Editor({
const { playhead, playing } = transport;
// Read by event listeners that outlive the render that installed them.
- const latest = useRef({ project, secondsPerPixel, scrollLeft, trackScroll, snap, playhead });
- latest.current = { project, secondsPerPixel, scrollLeft, trackScroll, snap, playhead };
+ const latest = useRef({ project, secondsPerPixel, scrollLeft, trackScroll, snap, playhead, frame });
+ latest.current = { project, secondsPerPixel, scrollLeft, trackScroll, snap, playhead, frame };
// The project's rate is authoritative, not the first clip's. Timecode that
// changes meaning when you import a 25fps file is worse than useless.
@@ -177,6 +181,18 @@ function Editor({
.then((document) => {
const restored = fromDocument(document);
if (!cancelled && restored) setProject(restored);
+ // The document's frame wins over the manifest: it is where an edited
+ // output size was saved.
+ const video = (document as { video?: { width?: unknown; height?: unknown } }).video;
+ if (
+ !cancelled &&
+ typeof video?.width === "number" &&
+ typeof video?.height === "number" &&
+ video.width > 0 &&
+ video.height > 0
+ ) {
+ setFrame({ width: Math.round(video.width), height: Math.round(video.height) });
+ }
})
.catch(() => undefined)
.finally(() => {
@@ -283,21 +299,50 @@ function Editor({
const save = useCallback(async () => {
setSaveState("saving");
try {
- await saveProject(session.path, toDocument(session, latest.current.project));
+ await saveProject(
+ session.path,
+ toDocument(
+ { ...session, width: latest.current.frame.width, height: latest.current.frame.height },
+ latest.current.project,
+ ),
+ );
setSaveState("saved");
+ return true;
} catch (cause) {
setSaveState("failed");
setError(String(cause));
+ return false;
}
}, [session]);
+ // A deliberate save deserves an acknowledgement; the autosave stays silent
+ // because a toast every 1.5 seconds of editing would be noise, not news.
+ const [toast, setToast] = useState<{ id: number; message: string; failed: boolean } | null>(
+ null,
+ );
+ const saveAndNotify = useCallback(() => {
+ void save().then((saved) =>
+ setToast({
+ id: Date.now(),
+ message: saved ? "Project saved" : "Save failed",
+ failed: !saved,
+ }),
+ );
+ }, [save]);
+
+ useEffect(() => {
+ if (!toast) return;
+ const timer = window.setTimeout(() => setToast(null), 2200);
+ return () => window.clearTimeout(timer);
+ }, [toast]);
+
// Autosave, debounced. The edit is the thing people lose; making them
// remember a keystroke to keep it is a design decision nobody wants.
useEffect(() => {
if (!loaded) return;
const timer = window.setTimeout(() => void save(), 1500);
return () => clearTimeout(timer);
- }, [project, loaded, save]);
+ }, [project, frame, loaded, save]);
useEffect(() => {
engineVersion()
@@ -319,14 +364,32 @@ function Editor({
setProject((current) => addMedia(current, item));
setSelectedMediaId(item.id);
// Fire and forget: the clip draws flat until the artwork lands.
- requestAssets(assets.current, item);
+ requestAssets(assets.current, item, session.path);
} catch (cause) {
// The engine's errors already name the file and quote FFmpeg.
setError(String(cause));
}
}
setBusy(false);
- }, []);
+ }, [session.path]);
+
+ // Artwork for everything already in the project: a reopened project should
+ // get its thumbnails and waveforms back without re-importing anything. The
+ // disk cache makes the second launch cheap; requestAssets dedupes the rest.
+ useEffect(() => {
+ if (!loaded) return;
+ // Audio clips cut from a video (detached, or split out) draw a waveform,
+ // so those videos - and only those - need peaks decoded.
+ const wantsPeaks = new Set(
+ project.clips.filter((clip) => clip.kind === "audio").map((clip) => clip.mediaId),
+ );
+ for (const item of project.media) {
+ requestAssets(assets.current, item, session.path);
+ if (item.kind === "video" && wantsPeaks.has(item.id)) {
+ requestVideoPeaks(assets.current, item, session.path);
+ }
+ }
+ }, [loaded, project.media, project.clips, session.path]);
// Files dropped from the OS. Tauri intercepts these before the webview sees
// them, which is why this is an event subscription and not an HTML5 handler.
@@ -349,47 +412,49 @@ function Editor({
return () => stop?.();
}, [importPaths]);
- // ── audio preview ────────────────────────────────────────────────────────
- const audio = useRef<AudioPreview | null>(null);
- if (audio.current === null) audio.current = new AudioPreview();
-
- const [renderingClips, setRenderingClips] = useState<ReadonlySet<string>>(new Set());
-
- useEffect(() => {
- const preview = audio.current;
- if (preview) {
- preview.onRenderChange = () => setRenderingClips(new Set(preview.pending()));
- }
- return () => preview?.dispose();
- }, []);
-
- useEffect(() => {
- const voices = clipsAt(project, playhead)
- .filter((clip) => clip.kind === "audio")
- .flatMap((clip) => {
+ // ── audio playback ───────────────────────────────────────────────────────
+ // The engine mixes everything audible - see src-tauri/src/playback.rs. The
+ // UI's whole job is to describe the audible clip set whenever it changes:
+ // audio clips, plus video clips that still own their sound. The engine
+ // decodes what is new (cached across sessions) and remixes the rest.
+ const audibleClips = useMemo(
+ () =>
+ project.clips.flatMap((clip) => {
+ if (clip.kind !== "audio" && clip.kind !== "video") return [];
+ if (clip.kind === "video" && clip.muted) return [];
+ const track = findTrack(project, clip.trackId);
+ if (!track || track.muted) return [];
const media = findMedia(project, clip.mediaId);
- if (!media) return [];
+ if (!media || !media.hasAudio) return [];
return [
{
- clipId: clip.id,
path: media.path,
- // Source advances at `speed`, so the same playhead maps further
- // into the file on a sped-up clip.
- time: clip.sourceStart + (playhead - clip.start) * clip.speed,
+ start: clip.start,
+ duration: clip.duration,
+ sourceStart: clip.sourceStart,
+ volume: clip.volume,
+ fadeIn: clip.fadeIn,
+ fadeOut: clip.fadeOut,
speed: clip.speed,
preservePitch: clip.preservePitch,
- // A filtered clip plays a render of itself, produced by the same
- // ffmpeg chain the exporter will use.
- filter: chainFor(clip),
- // Uncapped: the preview routes through a gain node, so what you
- // hear matches what the exporter will render.
- volume: clipGainAt(clip, playhead),
+ chain: buildChain(clip.filters) ?? "",
},
];
- });
+ }),
+ [project],
+ );
- audio.current?.sync(voices, playing);
- }, [project, playhead, playing]);
+ useEffect(() => {
+ if (!isTauri()) return;
+ // Debounced so a slider drag settles before its chain re-decodes; gain
+ // and position changes are cheap remixes either way.
+ const timer = window.setTimeout(() => {
+ void invoke("audio_set_clips", { project: session.path, clips: audibleClips }).catch(
+ (cause: unknown) => console.error("WolfCut: could not update the mix", cause),
+ );
+ }, 150);
+ return () => window.clearTimeout(timer);
+ }, [audibleClips, session.path]);
/**
* Every title live at the playhead, bottom track first.
@@ -435,8 +500,6 @@ function Editor({
clipId: top.id,
path: media.path,
time: top.sourceStart + (playhead - top.start),
- muted: false,
- volume: clipGainAt(top, playhead),
isStill: top.kind === "image",
};
}, [project, playhead]);
@@ -536,9 +599,6 @@ function Editor({
const deleteSelected = useCallback(() => {
if (selectedClipIds.length === 0) return;
- // Free the audio element first: once the clip is gone the sync pass has
- // nothing to match it against and it would keep sounding.
- for (const clipId of selectedClipIds) audio.current?.release(clipId);
setProject((current) => removeClips(current, selectedClipIds));
setSelectedClipIds([]);
}, [selectedClipIds]);
@@ -590,7 +650,7 @@ function Editor({
switch (event.code) {
case "KeyS":
event.preventDefault();
- void save();
+ saveAndNotify();
break;
case "KeyB":
event.preventDefault();
@@ -653,29 +713,7 @@ function Editor({
window.addEventListener("keydown", onKey);
return () => window.removeEventListener("keydown", onKey);
- }, [deleteSelected, duration, frameRate, mergeSelected, save, splitAtPlayhead, transport]);
-
- // Filters are rendered per clip, keyed so that returning a slider to a value
- // you already heard reuses the render instead of producing it again.
- const chainFor = useCallback((clip: Clip) => {
- const chain = buildChain(clip.filters);
- if (!chain) return undefined;
-
- // A sped-up clip reads further into the file than it occupies on the
- // timeline, so the render has to cover `duration * speed` of source - the
- // same window the exporter trims. Rendering only `duration` left the
- // preview seeking past the end of a short file, which is heard as the
- // clip going silent partway through rather than as a filter problem.
- // The window is part of the key, so changing speed re-renders.
- const window = clip.duration * Math.min(8, Math.max(0.1, clip.speed));
-
- return {
- key: chainKey(clip.mediaId, clip.sourceStart, window, clip.filters),
- chain,
- sourceStart: clip.sourceStart,
- duration: window,
- };
- }, []);
+ }, [deleteSelected, duration, frameRate, mergeSelected, saveAndNotify, splitAtPlayhead, transport]);
// The exporter works from a flat list: the engine rebuilds a real timeline
// from it, so track *order* has to survive the trip even though track
@@ -700,7 +738,7 @@ function Editor({
sourceStart: clip.sourceStart,
track: index,
hidden: !track.visible,
- muted: track.muted,
+ muted: track.muted || clip.muted === true,
volume: clip.volume,
fadeIn: clip.fadeIn,
fadeOut: clip.fadeOut,
@@ -766,7 +804,7 @@ function Editor({
label: "Save",
icon: "folder",
hint: "Ctrl+S",
- onSelect: () => void save(),
+ onSelect: () => saveAndNotify(),
},
{
label: "Export...",
@@ -889,6 +927,8 @@ function Editor({
playhead={playhead}
duration={duration}
frameRate={frameRate}
+ frame={frame}
+ onFrameChange={(width, height) => setFrame({ width, height })}
onTogglePlay={transport.toggle}
onStep={(frames) => transport.step(frames, frameRate)}
onSeek={transport.seek}
@@ -909,10 +949,13 @@ function Editor({
clip={selectedClip}
media={inspectorMedia}
project={project}
+ projectName={session.name}
+ projectPath={session.path}
+ frame={frame}
+ duration={duration}
frameRate={frameRate}
onAddFont={() => void pickFont()}
onRemoveFont={(family) => setProject((current) => removeFont(current, family))}
- rendering={selectedClipIds.some((id) => renderingClips.has(id))}
onChangeClip={(patch) => {
if (selectedClipIds.length !== 1) return;
setProject((current) => updateClip(current, selectedClipIds[0], patch));
@@ -972,11 +1015,6 @@ function Editor({
}
onRemoveTrack={(trackId) =>
setProject((current) => {
- // Free the audio for anything about to disappear, or it keeps
- // sounding with no clip left to match it against.
- for (const clip of clipsOnTrack(current, trackId)) {
- audio.current?.release(clip.id);
- }
setSelectedClipIds((ids) =>
ids.filter((id) => findClip(current, id)?.trackId !== trackId),
);
@@ -1013,6 +1051,38 @@ function Editor({
},
]
: []),
+ // "Detach audio" and its reverse, offered only where they
+ // mean something: one clip, with a link to make or unmake.
+ ...(!many &&
+ clip?.kind === "video" &&
+ !clip.muted &&
+ findMedia(project, clip.mediaId)?.hasAudio &&
+ detachedAudioOf(project, clip.id).length === 0
+ ? [
+ {
+ label: "Detach audio",
+ icon: "waveform" as const,
+ onSelect: () => setProject((current) => detachAudio(current, clipId)),
+ },
+ ]
+ : []),
+ ...(!many &&
+ clip &&
+ ((clip.kind === "video" && detachedAudioOf(project, clip.id).length > 0) ||
+ (clip.kind === "audio" &&
+ clip.detachedFrom &&
+ findClip(project, clip.detachedFrom)))
+ ? [
+ {
+ label: "Reattach audio",
+ icon: "merge" as const,
+ onSelect: () => {
+ setProject((current) => reattachAudio(current, clipId));
+ setSelectedClipIds([]);
+ },
+ },
+ ]
+ : []),
{
label: "Move to playhead",
icon: "select",
@@ -1030,7 +1100,6 @@ function Editor({
hint: "Del",
danger: true,
onSelect: () => {
- for (const id of target) audio.current?.release(id);
setProject((current) => removeClips(current, target));
setSelectedClipIds([]);
},
@@ -1044,12 +1113,28 @@ function Editor({
{context && <ContextMenu target={context} onClose={() => setContext(null)} />}
+ {toast && (
+ <div
+ key={toast.id}
+ role="status"
+ className={`surface fixed bottom-4 right-4 z-50 flex items-center gap-2 rounded-xl
+ px-3 py-2 text-xs ${toast.failed ? "text-danger" : "text-primary"}`}
+ >
+ <Icon
+ name={toast.failed ? "close" : "check"}
+ size={13}
+ className={toast.failed ? "" : "text-success"}
+ />
+ {toast.message}
+ </div>
+ )}
+
{exporting && (
<ExportDialog
projectName={session.name}
projectPath={session.path}
- width={session.width}
- height={session.height}
+ width={frame.width}
+ height={frame.height}
rateNum={session.rateNum}
rateDen={session.rateDen}
duration={duration}
diff --git a/desktop/src/assets/wolfcut-logo.png b/desktop/src/assets/wolfcut-logo.png
new file mode 100644
index 00000000..e740347a
Binary files /dev/null and b/desktop/src/assets/wolfcut-logo.png differ
diff --git a/desktop/src/components/FiltersPanel.tsx b/desktop/src/components/FiltersPanel.tsx
index afd2ab68..f6d397b1 100644
--- a/desktop/src/components/FiltersPanel.tsx
+++ b/desktop/src/components/FiltersPanel.tsx
@@ -26,12 +26,9 @@ import { Empty } from "./Panel";
*/
export function FiltersPanel({
clip,
- rendering,
onChange,
}: {
clip: Clip | null;
- /** True while the preview is re-rendering this clip's audio. */
- rendering: boolean;
onChange: (filters: ClipFilter[]) => void;
}) {
const [category, setCategory] = useState<FilterCategory>("voice");
@@ -76,11 +73,6 @@ export function FiltersPanel({
<section className="mb-5">
<h3 className="mb-2 flex items-center gap-2 text-[11px] font-semibold uppercase tracking-wider text-tertiary">
Applied
- {rendering && (
- <span className="font-normal normal-case tracking-normal text-accent">
- rendering...
- </span>
- )}
</h3>
<ul className="flex flex-col gap-2">
diff --git a/desktop/src/components/Inspector.tsx b/desktop/src/components/Inspector.tsx
index 08f172f8..26736b73 100644
--- a/desktop/src/components/Inspector.tsx
+++ b/desktop/src/components/Inspector.tsx
@@ -1,13 +1,11 @@
import type { ReactNode } from "react";
-import type { Clip, MediaItem } from "../lib/project";
+import type { Clip, MediaItem, Project } from "../lib/project";
import { shortDuration, timecode } from "../lib/time";
-import { Icon } from "./Icon";
-import { Empty } from "./Panel";
/**
* Properties of the current selection - a clip if one is selected, otherwise
- * the highlighted bin item.
+ * the highlighted bin item, otherwise the project itself.
*
* Read-only for now. When these become editable the edits go to the engine as
* commands and come back as new state; the panel stays dumb. That is what
@@ -17,17 +15,40 @@ export function Inspector({
clip,
media,
frameRate,
+ project,
+ projectName,
+ projectPath,
+ frame,
+ duration,
}: {
clip: Clip | null;
media: MediaItem | null;
frameRate: number;
+ project: Project;
+ projectName: string;
+ projectPath: string;
+ /** The output size, as currently set in the preview footer. */
+ frame: { width: number; height: number };
+ /** Timeline length in seconds. */
+ duration: number;
}) {
return (
<div>
{!clip && !media ? (
- <Empty icon={<Icon name="info" size={26} strokeWidth={1.5} />}>
- Select a clip or a bin item.
- </Empty>
+ <div className="selectable px-3 py-2">
+ <Section title="Project">
+ <Row label="Name" value={projectName} />
+ <Row label="Folder" value={projectPath} mono wrap />
+ <Row label="Output" value={`${frame.width} x ${frame.height}`} mono />
+ <Row label="Rate" value={`${frameRate.toFixed(2)} fps`} mono />
+ <Row label="Duration" value={timecode(duration, frameRate)} mono />
+ </Section>
+ <Section title="Contents">
+ <Row label="Media" value={`${project.media.length}`} mono />
+ <Row label="Tracks" value={`${project.tracks.length}`} mono />
+ <Row label="Clips" value={`${project.clips.length}`} mono />
+ </Section>
+ </div>
) : (
<div className="selectable px-3 py-2">
{clip && (
diff --git a/desktop/src/components/MediaBin.tsx b/desktop/src/components/MediaBin.tsx
index 8dfbd937..757a325e 100644
--- a/desktop/src/components/MediaBin.tsx
+++ b/desktop/src/components/MediaBin.tsx
@@ -335,6 +335,7 @@ function MediaThumb({ item, assets }: { item: MediaItem; assets: MediaAssets })
// most of it, and a crop reads better than bars at this size.
const tile = strip.width / frames;
const scale = Math.max(width / tile, height / strip.height);
+ context.imageSmoothingQuality = "high";
const drawWidth = tile * scale;
const drawHeight = strip.height * scale;
context.drawImage(
diff --git a/desktop/src/components/Menu.tsx b/desktop/src/components/Menu.tsx
index 3aba2046..586650d9 100644
--- a/desktop/src/components/Menu.tsx
+++ b/desktop/src/components/Menu.tsx
@@ -31,10 +31,13 @@ export function Menu({
groups,
trigger,
align = "left",
+ direction = "down",
}: {
groups: MenuOption[][];
trigger: (open: boolean) => ReactNode;
align?: "left" | "right";
+ /** "up" hangs the menu above the trigger, for triggers near the bottom edge. */
+ direction?: "down" | "up";
}) {
const [open, setOpen] = useState(false);
const root = useRef<HTMLDivElement>(null);
@@ -73,7 +76,8 @@ export function Menu({
{open && (
<div
role="menu"
- className={`surface absolute z-50 mt-1.5 min-w-52 rounded-xl px-1.5 py-1.5 text-primary
+ className={`surface absolute z-50 min-w-52 rounded-xl px-1.5 py-1.5 text-primary
+ ${direction === "up" ? "bottom-full mb-1.5" : "mt-1.5"}
${align === "right" ? "right-0" : "left-0"}`}
>
{groups.map((group, groupIndex) => (
diff --git a/desktop/src/components/Preview.tsx b/desktop/src/components/Preview.tsx
index 31fa71df..e7501c58 100644
--- a/desktop/src/components/Preview.tsx
+++ b/desktop/src/components/Preview.tsx
@@ -1,21 +1,42 @@
import { useEffect, useLayoutEffect, useRef, useState } from "react";
import { convertFileSrc } from "@tauri-apps/api/core";
-import { connectElement } from "../lib/audio";
import { timecode } from "../lib/time";
import { textCss, type TextStyle } from "../lib/text";
import { Icon, IconButton } from "./Icon";
+import { Menu } from "./Menu";
import { PANEL_SHELL } from "./Panel";
+/**
+ * Output frames people actually deliver to. Dimensions rather than bare
+ * ratios, because a ratio alone does not say how many pixels the export has.
+ */
+const FRAME_PRESETS = [
+ { label: "16:9", width: 1920, height: 1080 },
+ { label: "16:9 · 4K", width: 3840, height: 2160 },
+ { label: "9:16", width: 1080, height: 1920 },
+ { label: "1:1", width: 1080, height: 1080 },
+ { label: "4:3", width: 1440, height: 1080 },
+ { label: "21:9", width: 2560, height: 1080 },
+] as const;
+
+/** "16:9" for a preset size, the reduced fraction for anything else. */
+function ratioLabel(width: number, height: number): string {
+ const preset = FRAME_PRESETS.find(
+ (candidate) => candidate.width === width && candidate.height === height,
+ );
+ if (preset) return preset.label.split(" ")[0];
+ const gcd = (a: number, b: number): number => (b === 0 ? a : gcd(b, a % b));
+ const divisor = gcd(width, height) || 1;
+ return `${width / divisor}:${height / divisor}`;
+}
+
/** The video clip that should be on screen right now, if any. */
export interface PreviewSource {
clipId: string;
path: string;
/** Where in the source file the playhead sits, in seconds. */
time: number;
- muted: boolean;
- /** Linear gain with fades applied. May exceed 1; see lib/audio.ts. */
- volume: number;
/** A still is shown as an image; there is nothing to seek or play. */
isStill: boolean;
}
@@ -49,6 +70,8 @@ export function Preview({
playhead,
duration,
frameRate,
+ frame,
+ onFrameChange,
onTogglePlay,
onStep,
onSeek,
@@ -60,6 +83,9 @@ export function Preview({
playhead: number;
duration: number;
frameRate: number;
+ /** The project's output size, shown and changed in the footer. */
+ frame: { width: number; height: number };
+ onFrameChange: (width: number, height: number) => void;
onTogglePlay: () => void;
onStep: (frames: number) => void;
onSeek: (seconds: number) => void;
@@ -118,17 +144,6 @@ export function Preview({
const element = video.current;
if (!element || !source || source.isStill) return;
- element.muted = source.muted;
-
- // Through a gain node, not element.volume, which browsers cap at 1.0.
- // A video clip's audio obeys the same clip gain as an audio clip's.
- try {
- connectElement(element).gain.value = Math.max(0, source.volume);
- } catch {
- // Routing can only fail if the element was already claimed by another
- // graph, which cannot happen here - but silence is not worth a crash.
- }
-
const tolerance = playing ? 0.3 : 0.03;
if (element.readyState > 0 && Math.abs(element.currentTime - source.time) > tolerance) {
try {
@@ -162,8 +177,12 @@ export function Preview({
*/}
<div className="min-h-0 min-w-0 flex-1 overflow-hidden bg-stage p-4">
<div ref={stage} data-preview-surface className="relative h-full w-full">
+ {/* Picture only. Every drop of audio - a video clip's included -
+ comes from the engine's mixer, so there is exactly one clock and
+ one gain law. */}
<video
ref={video}
+ muted
playsInline
className={`absolute inset-0 h-full w-full object-contain ${
source && !source.isStill ? "block" : "hidden"
@@ -275,8 +294,38 @@ export function Preview({
/>
</div>
- <span className="min-w-0 truncate justify-self-end font-technical text-[10px] text-tertiary">
- {frameRate.toFixed(2)} fps
+ <span className="flex min-w-0 items-center gap-2 justify-self-end">
+ <Menu
+ align="right"
+ direction="up"
+ groups={[
+ FRAME_PRESETS.map((preset) => ({
+ label: `${preset.label} — ${preset.width} x ${preset.height}`,
+ icon:
+ preset.width === frame.width && preset.height === frame.height
+ ? ("check" as const)
+ : undefined,
+ onSelect: () => onFrameChange(preset.width, preset.height),
+ })),
+ ]}
+ trigger={(open) => (
+ <span
+ title="Output size"
+ className={`flex items-center gap-1 rounded-md px-1.5 py-0.5 font-technical text-[10px]
+ transition-colors ${
+ open
+ ? "bg-active text-primary"
+ : "text-tertiary hover:bg-hover hover:text-secondary"
+ }`}
+ >
+ {ratioLabel(frame.width, frame.height)} · {frame.width}x{frame.height}
+ <Icon name="chevronDown" size={10} />
+ </span>
+ )}
+ />
+ <span className="truncate font-technical text-[10px] text-tertiary">
+ {frameRate.toFixed(2)} fps
+ </span>
</span>
</div>
</div>
diff --git a/desktop/src/components/RightPanel.tsx b/desktop/src/components/RightPanel.tsx
index 081a2e44..131fe4fd 100644
--- a/desktop/src/components/RightPanel.tsx
+++ b/desktop/src/components/RightPanel.tsx
@@ -8,24 +8,25 @@ import { TextPanel } from "./TextPanel";
export type RightTab = "details" | "adjust" | "filters" | "text";
-const TABS: { id: RightTab; label: string }[] = [
- { id: "details", label: "Details" },
- { id: "adjust", label: "Adjust" },
- { id: "filters", label: "Filters" },
-];
-
/**
- * Text replaces Adjust and Filters when a title is selected.
+ * The tab strip follows the selection.
*
- * A text clip has no volume, no speed and no audio filters, so offering those
- * tabs would be offering three panels that can only say "nothing here". The
- * tab strip therefore depends on what is selected rather than being fixed.
+ * Nothing selected: only Details, showing the project itself - there is
+ * nothing to edit, so no editing tabs. A media clip selected: Adjust and
+ * Filters, the things that can actually be changed. A title: just Text,
+ * because a text clip has no volume, no speed and no audio filters, and
+ * offering those tabs would be offering panels that can only say "nothing
+ * here".
*/
-const TEXT_TABS: { id: RightTab; label: string }[] = [
- { id: "details", label: "Details" },
- { id: "text", label: "Text" },
+const DETAILS_TABS: { id: RightTab; label: string }[] = [{ id: "details", label: "Details" }];
+
+const CLIP_TABS: { id: RightTab; label: string }[] = [
+ { id: "adjust", label: "Adjust" },
+ { id: "filters", label: "Filters" },
];
+const TEXT_TABS: { id: RightTab; label: string }[] = [{ id: "text", label: "Text" }];
+
/**
* The right-hand panel.
*
@@ -40,33 +41,37 @@ export function RightPanel({
clip,
media,
project,
+ projectName,
+ projectPath,
+ frame,
+ duration,
frameRate,
onChangeClip,
onSpeedChange,
onAddFont,
onRemoveFont,
- rendering,
}: {
tab: RightTab;
onTab: (tab: RightTab) => void;
clip: Clip | null;
media: MediaItem | null;
project: Project;
+ projectName: string;
+ projectPath: string;
+ frame: { width: number; height: number };
+ duration: number;
frameRate: number;
onChangeClip: (patch: Partial<Clip>) => void;
onSpeedChange: (speed: number) => void;
onAddFont: () => void;
onRemoveFont: (family: string) => void;
- /** True while the selected clip's filtered audio is being rendered. */
- rendering: boolean;
}) {
const isText = clip?.kind === "text";
- const tabs = isText ? TEXT_TABS : TABS;
+ const tabs = clip ? (isText ? TEXT_TABS : CLIP_TABS) : DETAILS_TABS;
- // Selecting a title while sitting on Filters would otherwise leave the strip
- // with nothing highlighted and the panel showing a tab that is no longer
- // offered, so the selection falls back to one that exists.
- const active = tabs.some((entry) => entry.id === tab) ? tab : "details";
+ // A selection change can leave the strip showing a tab that is no longer
+ // offered, so the active tab falls back to the first one that exists.
+ const active = tabs.some((entry) => entry.id === tab) ? tab : tabs[0].id;
return (
<Panel>
@@ -90,7 +95,18 @@ export function RightPanel({
</div>
</div>
- {active === "details" && <Inspector clip={clip} media={media} frameRate={frameRate} />}
+ {active === "details" && (
+ <Inspector
+ clip={clip}
+ media={media}
+ frameRate={frameRate}
+ project={project}
+ projectName={projectName}
+ projectPath={projectPath}
+ frame={frame}
+ duration={duration}
+ />
+ )}
{active === "text" && (
<TextPanel
clip={clip}
@@ -106,7 +122,6 @@ export function RightPanel({
{active === "filters" && (
<FiltersPanel
clip={clip}
- rendering={rendering}
onChange={(filters: ClipFilter[]) => onChangeClip({ filters })}
/>
)}
diff --git a/desktop/src/components/TimelinePanel.tsx b/desktop/src/components/TimelinePanel.tsx
index a790c708..c2e0f3c4 100644
--- a/desktop/src/components/TimelinePanel.tsx
+++ b/desktop/src/components/TimelinePanel.tsx
@@ -1272,6 +1272,8 @@ function drawFilmstrip(
const drawWidth = height * (tileWidth / strip.height);
if (!Number.isFinite(drawWidth) || drawWidth <= 0) return;
+ context.imageSmoothingQuality = "high";
+
const columns = Math.ceil(width / drawWidth);
for (let column = 0; column < columns; column += 1) {
const fraction = columns > 1 ? column / (columns - 1) : 0;
diff --git a/desktop/src/components/TitleBar.tsx b/desktop/src/components/TitleBar.tsx
index ef9ba5b5..7756187d 100644
--- a/desktop/src/components/TitleBar.tsx
+++ b/desktop/src/components/TitleBar.tsx
@@ -3,6 +3,7 @@ import type { ReactNode } from "react";
import { isTauri } from "@tauri-apps/api/core";
import { getCurrentWindow } from "@tauri-apps/api/window";
+import logo from "../assets/wolfcut-logo.png";
import type { Theme } from "../lib/theme";
import { Icon } from "./Icon";
import { Menu, type MenuOption } from "./Menu";
@@ -10,10 +11,16 @@ import { Menu, type MenuOption } from "./Menu";
/**
* The custom title bar.
*
- * The window is undecorated (`decorations: false`), so this strip *is* the
- * title bar: it carries the app menu, the project name and the window
+ * On Windows the window is undecorated (`decorations: false`), so this strip
+ * *is* the title bar: it carries the app menu, the project name and the window
* controls, and everything not otherwise interactive is a drag region.
*
+ * On macOS the native traffic lights are overlaid instead
+ * (`tauri.macos.conf.json` sets `titleBarStyle: "Overlay"`), because
+ * Windows-style controls on a Mac read as "ported, carelessly". The strip
+ * keeps everything else but drops its own window buttons and leaves room on
+ * the left for the traffic lights.
+ *
* `data-tauri-drag-region` is what makes an area draggable, and Tauri also
* gives it double-click-to-maximise for free. It has to be on the actual
* element under the pointer - a parent with the attribute does not delegate -
@@ -40,6 +47,7 @@ export function TitleBar({
}) {
const [maximized, setMaximized] = useState(false);
const native = isTauri();
+ const macOS = navigator.userAgent.includes("Mac");
useEffect(() => {
if (!native) return;
@@ -69,12 +77,12 @@ export function TitleBar({
return (
<header
data-tauri-drag-region
- className="flex h-8 shrink-0 items-center gap-1 border-b border-hairline bg-panel pl-2"
+ className={`flex h-8 shrink-0 items-center gap-1 border-b border-hairline bg-panel ${
+ macOS ? "pl-20 pr-1" : "pl-2"
+ }`}
>
<span className="flex items-center gap-1.5 pr-1" data-tauri-drag-region>
- <span className="flex h-4 w-4 items-center justify-center rounded bg-accent text-on-accent">
- <Icon name="film" size={11} strokeWidth={2.5} />
- </span>
+ <img src={logo} alt="" className="pointer-events-none h-4 w-4" draggable={false} />
<span className="font-display text-[13px] font-bold tracking-tight text-primary">WolfCut</span>
</span>
@@ -121,7 +129,7 @@ export function TitleBar({
{actions && <div className="flex items-center gap-1 pr-2">{actions}</div>}
- {native && (
+ {native && !macOS && (
<div className="flex h-full items-stretch">
<WindowButton label="Minimise" onClick={() => void control("minimize")}>
<Icon name="winMinimize" size={14} strokeWidth={1.5} />
diff --git a/desktop/src/lib/assets.ts b/desktop/src/lib/assets.ts
index dc144dcf..22380db6 100644
--- a/desktop/src/lib/assets.ts
+++ b/desktop/src/lib/assets.ts
@@ -71,7 +71,13 @@ const BUCKETS_PER_SECOND = 200;
/** Frames per filmstrip. Enough to read the shot, few enough to stay cheap. */
const STRIP_FRAMES = 24;
-const STRIP_HEIGHT = 72;
+/**
+ * Strip frames are drawn at ~32-72 CSS pixels tall, so they need the
+ * display's real pixel density or a Retina screen upscales them into mush.
+ * Capped at the engine's 240 limit; the ratio is read once because a strip
+ * cached at one density is not worth regenerating on a monitor change.
+ */
+const STRIP_HEIGHT = Math.min(240, Math.round(72 * (window.devicePixelRatio || 1)));
/**
* Starts producing artwork for one media item if it is not already cached.
@@ -86,19 +92,34 @@ export function requestAssets(
path: string;
kind: "video" | "audio" | "image";
duration: number | null;
+ hasAudio: boolean;
},
+ /** Project folder holding the on-disk artwork cache; null skips caching. */
+ projectPath: string | null = null,
): void {
if (assets.pending.has(media.id)) return;
- if (assets.peaks.has(media.id) || assets.strips.has(media.id)) return;
+
+ // Video peaks are deliberately NOT loaded here. Decoding a video's audio
+ // means reading the whole file and crunching every sample, which stalls the
+ // main thread hard enough to make playback stutter - so it happens only on
+ // demand, via requestVideoPeaks, when a detached audio clip needs drawing.
+ const wantsPeaks = media.kind === "audio" && !assets.peaks.has(media.id);
+ const wantsStrip = media.kind !== "audio" && !assets.strips.has(media.id);
+ if (!wantsPeaks && !wantsStrip) return;
assets.pending.add(media.id);
- const work =
- media.kind === "audio"
- ? loadPeaks(media.path).then((peaks) => {
- if (peaks) assets.peaks.set(media.id, peaks);
- })
- : media.kind === "image"
+ const jobs: Promise<unknown>[] = [];
+ if (wantsPeaks) {
+ jobs.push(
+ loadPeaks(media.path, projectPath).then((peaks) => {
+ if (peaks) assets.peaks.set(media.id, peaks);
+ }),
+ );
+ }
+ if (wantsStrip) {
+ jobs.push(
+ media.kind === "image"
? // A still is a one-frame filmstrip. Storing it that way means the
// timeline and the bin draw it with the code they already have, and
// a long still clip tiles the same frame instead of stretching it.
@@ -108,21 +129,127 @@ export function requestAssets(
assets.stripFrames.set(media.id, 1);
}
})
- : loadStrip(media.path, media.duration).then((strip) => {
+ : loadStrip(media.path, media.duration, projectPath).then((strip) => {
if (strip) {
assets.strips.set(media.id, strip);
assets.stripFrames.set(media.id, STRIP_FRAMES);
}
- });
+ }),
+ );
+ }
- void work.finally(() => {
+ void Promise.allSettled(jobs).then(() => {
assets.pending.delete(media.id);
announce(assets);
});
}
+/**
+ * Waveform for a video's audio track, on demand.
+ *
+ * Called when a detached audio clip exists (or is being created) for a video,
+ * which is the only time a video needs peaks. Expensive on first run - the
+ * whole file is read and decoded - but the result lands in the disk cache, so
+ * a reopened project pays only a file read.
+ */
+export function requestVideoPeaks(
+ assets: MediaAssets,
+ media: { id: string; path: string; kind: "video" | "audio" | "image"; hasAudio: boolean },
+ projectPath: string | null = null,
+): void {
+ if (media.kind !== "video" || !media.hasAudio) return;
+ if (assets.peaks.has(media.id)) return;
+
+ const pendingKey = `${media.id}:peaks`;
+ if (assets.pending.has(pendingKey)) return;
+ assets.pending.add(pendingKey);
+
+ void loadPeaks(media.path, projectPath)
+ .then((peaks) => {
+ if (peaks) assets.peaks.set(media.id, peaks);
+ })
+ .finally(() => {
+ assets.pending.delete(pendingKey);
+ announce(assets);
+ });
+}
+
+/**
+ * The on-disk artwork cache, kept in the project's folder.
+ *
+ * Strips and peaks survive a relaunch there, which is the difference between
+ * a reopened project showing its artwork immediately and every launch paying
+ * for ffmpeg runs and full audio decodes again. Reads and writes both fail
+ * soft: a miss or a failed write only means generating now, caching later.
+ */
+function artworkKey(path: string): string {
+ // FNV-1a over the absolute path. Collisions are astronomically unlikely at
+ // bin sizes, and a collision's worst case is a wrong thumbnail.
+ let hash = 0x811c9dc5;
+ for (let index = 0; index < path.length; index += 1) {
+ hash ^= path.charCodeAt(index);
+ hash = Math.imul(hash, 0x01000193) >>> 0;
+ }
+ return hash.toString(16).padStart(8, "0");
+}
+
+async function readArtwork(project: string | null, key: string): Promise<ArrayBuffer | null> {
+ if (!project) return null;
+ try {
+ return await invoke<ArrayBuffer>("read_artwork", { project, key });
+ } catch {
+ return null;
+ }
+}
+
+function writeArtwork(project: string | null, key: string, bytes: Uint8Array): void {
+ if (!project) return;
+ // Serialising the payload for IPC is main-thread work, and nothing about a
+ // cache write is urgent - it waits for an idle moment so it never competes
+ // with playback.
+ const send = () =>
+ void invoke("write_artwork", { project, key, bytes: Array.from(bytes) }).catch(
+ () => undefined,
+ );
+ if (typeof window.requestIdleCallback === "function") window.requestIdleCallback(() => send());
+ else window.setTimeout(send, 500);
+}
+
+/** [bucketsPerSecond f32][count u32][min f32...][max f32...], little-endian. */
+function encodePeaks(peaks: Peaks): Uint8Array {
+ const count = peaks.min.length;
+ const buffer = new ArrayBuffer(8 + count * 8);
+ const view = new DataView(buffer);
+ view.setFloat32(0, peaks.bucketsPerSecond, true);
+ view.setUint32(4, count, true);
+ new Float32Array(buffer, 8, count).set(peaks.min);
+ new Float32Array(buffer, 8 + count * 4, count).set(peaks.max);
+ return new Uint8Array(buffer);
+}
+
+function decodePeaks(bytes: ArrayBuffer): Peaks | null {
+ if (bytes.byteLength < 8) return null;
+ const view = new DataView(bytes);
+ const bucketsPerSecond = view.getFloat32(0, true);
+ const count = view.getUint32(4, true);
+ if (!Number.isFinite(bucketsPerSecond) || bucketsPerSecond <= 0) return null;
+ if (bytes.byteLength !== 8 + count * 8) return null;
+ return {
+ min: new Float32Array(bytes, 8, count),
+ max: new Float32Array(bytes, 8 + count * 4, count),
+ bucketsPerSecond,
+ };
+}
+
/** Decodes a file and reduces it to min/max pairs. */
-async function loadPeaks(path: string): Promise<Peaks | null> {
+async function loadPeaks(path: string, project: string | null): Promise<Peaks | null> {
+ const key = `${artworkKey(path)}-b${BUCKETS_PER_SECOND}.peaks`;
+ const cached = await readArtwork(project, key);
+ if (cached) {
+ const peaks = decodePeaks(cached);
+ if (peaks) return peaks;
+ }
+
try {
const bytes = await readMediaBytes(path);
@@ -153,7 +280,9 @@ async function loadPeaks(path: string): Promise<Peaks | null> {
max[bucket] = high;
}
- return { min, max, bucketsPerSecond: buffer.sampleRate / bucketSize };
+ const peaks = { min, max, bucketsPerSecond: buffer.sampleRate / bucketSize };
+ writeArtwork(project, key, encodePeaks(peaks));
+ return peaks;
} catch (cause) {
console.warn(`WolfCut: no waveform for ${path}`, cause);
return null;
@@ -172,15 +301,32 @@ async function loadImage(path: string): Promise<ImageBitmap | null> {
}
/** Asks the engine for a strip of frames and decodes it into an ImageBitmap. */
-async function loadStrip(path: string, duration: number | null): Promise<ImageBitmap | null> {
+async function loadStrip(
+ path: string,
+ duration: number | null,
+ project: string | null,
+): Promise<ImageBitmap | null> {
if (!duration || duration <= 0) return null;
+ // The key carries the strip's shape, so a density or frame-count change
+ // regenerates rather than serving yesterday's resolution.
+ const key = `${artworkKey(path)}-f${STRIP_FRAMES}-h${STRIP_HEIGHT}.strip.jpg`;
+ const cached = await readArtwork(project, key);
+ if (cached) {
+ try {
+ return await createImageBitmap(new Blob([cached], { type: "image/jpeg" }));
+ } catch {
+ // A corrupt cache entry falls through to regeneration.
+ }
+ }
+
try {
const bytes = await invoke<ArrayBuffer>("extract_filmstrip", {
path,
count: STRIP_FRAMES,
height: STRIP_HEIGHT,
});
+ writeArtwork(project, key, new Uint8Array(bytes));
// createImageBitmap decodes off the main thread, so a slow JPEG does not
// stall the timeline.
return await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
diff --git a/desktop/src/lib/audio.ts b/desktop/src/lib/audio.ts
deleted file mode 100644
index b395b8c5..00000000
--- a/desktop/src/lib/audio.ts
+++ /dev/null
@@ -1,337 +0,0 @@
-/**
- * Audio preview.
- *
- * Each audible clip gets a media element for decoding, routed through a Web
- * Audio `GainNode` for level. The gain node is the whole reason this is not
- * just `element.volume`: that property is hard-capped at 1.0 by every browser,
- * so a boost could not be previewed at all - it could only be promised and
- * applied later at export, which is a useless thing to tell someone mixing.
- * A `GainNode` has no ceiling, so what you hear is what gets rendered.
- *
- * Audio is loaded as a blob rather than through the asset protocol, and that
- * is deliberate too. `createMediaElementSource` on a cross-origin element
- * yields *silence* with no error - the element plays, the graph outputs
- * nothing. A blob URL is same-origin, so the graph is guaranteed to work. The
- * cost is holding the file in memory, which we already do for the waveform.
- *
- * The known limits, so nobody mistakes this for a finished mixer:
- * - Sync is corrective, not sample-accurate. Drift past a tolerance is
- * fixed by reseeking, which is audible when it happens.
- * - No effects and no automation beyond the clip's constant gain and fades.
- * - Scrubbing repositions audio, it does not scrub it.
- *
- * See docs/decisions/0004-audio-preview-in-the-webview.md.
- */
-
-import { readMediaBytes, renderFilteredAudio } from "./engine";
-
-/** One audio clip that should be sounding right now. */
-export interface Voice {
- clipId: string;
- path: string;
- /** Where in the source file the playhead currently sits, in seconds. */
- time: number;
- /** Linear gain, fades already applied. May exceed 1. */
- volume: number;
- /** Playback rate. */
- speed: number;
- /** False lets pitch rise with speed, like tape. */
- preservePitch: boolean;
- /**
- * A filtered render of this clip, when it has filters.
- *
- * The rendered audio begins at the clip's in-point rather than the file's,
- * so a position inside it is offset by `sourceStart`.
- */
- filter?: {
- /** Identity of this exact render. A change means re-render. */
- key: string;
- chain: string;
- sourceStart: number;
- duration: number;
- };
-}
-
-/**
- * How far out of step a voice may drift before it is reseeked.
- *
- * Generous while playing: a reseek is an audible click, and browsers do not
- * keep a media element perfectly locked to an external clock anyway. Tight
- * while paused, where "seek" is the entire operation.
- */
-const TOLERANCE_PLAYING = 0.3;
-const TOLERANCE_PAUSED = 0.02;
-
-/**
- * How long the filter settings must hold still before re-rendering.
- *
- * Without this, every step of a slider drag spawned an FFmpeg process and tore
- * down the audio element mid-playback - dozens of renders for one gesture, and
- * audio that cut out while you were trying to hear what you were adjusting.
- */
-const RENDER_DEBOUNCE_MS = 400;
-
-let sharedContext: AudioContext | null = null;
-
-/** The one AudioContext. Browsers cap how many a page may create. */
-export function audioContext(): AudioContext {
- sharedContext ??= new AudioContext();
- return sharedContext;
-}
-
-/**
- * A context created before any user interaction starts suspended. Resuming is
- * safe to call repeatedly and only does anything the first time.
- */
-function wake(): void {
- const context = audioContext();
- if (context.state === "suspended") void context.resume().catch(() => undefined);
-}
-
-// One source node per element, forever: calling createMediaElementSource twice
-// on the same element throws, and there is no way to undo the first call.
-const routed = new WeakMap<HTMLMediaElement, GainNode>();
-
-/**
- * Routes a media element through a gain node and returns it.
- *
- * Shared with the video preview, so picture and sound obey the same clip gain
- * through the same graph.
- */
-export function connectElement(element: HTMLMediaElement): GainNode {
- const existing = routed.get(element);
- if (existing) return existing;
-
- const context = audioContext();
- const gain = context.createGain();
- context.createMediaElementSource(element).connect(gain);
- gain.connect(context.destination);
- routed.set(element, gain);
- return gain;
-}
-
-interface Loaded {
- element: HTMLAudioElement;
- gain: GainNode;
- objectUrl?: string;
- loading: boolean;
- /** Which render this element holds; undefined means the unfiltered file. */
- key?: string;
- /** How far into the file this element's audio begins. */
- offset: number;
-}
-
-export class AudioPreview {
- private voices = new Map<string, Loaded>();
- private rendering = new Set<string>();
- /** Pending re-renders, keyed by clip. */
- private timers = new Map<string, number>();
- /** Which key each pending timer is waiting to render. */
- private scheduled = new Map<string, string | undefined>();
- /** Notified when a render finishes, so the UI can drop its spinner. */
- onRenderChange: (() => void) | null = null;
-
- /** Clips whose filtered audio is still rendering. */
- pending(): ReadonlySet<string> {
- return this.rendering;
- }
-
- /**
- * Brings the audio output in line with the transport.
- *
- * Idempotent - calling it every animation frame is fine, and is in fact how
- * drift gets corrected.
- */
- sync(active: Voice[], playing: boolean): void {
- if (playing) wake();
-
- const wanted = new Set(active.map((voice) => voice.clipId));
- for (const [clipId, loaded] of this.voices) {
- if (!wanted.has(clipId) && !loaded.element.paused) loaded.element.pause();
- }
-
- const tolerance = playing ? TOLERANCE_PLAYING : TOLERANCE_PAUSED;
-
- for (const voice of active) {
- const loaded = this.load(voice);
- const element = loaded.element;
-
- // No clamp. A boost is meant to be audible.
- loaded.gain.gain.value = Math.max(0, voice.volume);
-
- // `preservesPitch` is exactly the toggle: on, the browser time-stretches
- // and the voice stays put; off, it resamples and pitch rides the rate.
- element.playbackRate = Math.max(0.0625, Math.min(16, voice.speed));
- element.preservesPitch = voice.preservePitch;
-
- // A filtered render starts at the clip's in-point, so the same instant
- // sits at a different offset inside it than in the original file.
- const target = Math.max(0, voice.time - loaded.offset);
-
- // readyState 0 means metadata has not arrived; setting currentTime then
- // throws in some engines and is discarded in others.
-
- if (element.readyState > 0 && Math.abs(element.currentTime - target) > tolerance) {
- try {
- element.currentTime = target;
- } catch {
- // Still opening; the next sync lands it.
- }
- }
-
- if (playing) {
- if (element.paused) void element.play().catch(() => undefined);
- } else if (!element.paused) {
- element.pause();
- }
- }
- }
-
- stopAll(): void {
- for (const loaded of this.voices.values()) loaded.element.pause();
- }
-
- /** Drops a voice, for instance when its clip is deleted. */
- release(clipId: string): void {
- const timer = this.timers.get(clipId);
- if (timer !== undefined) {
- clearTimeout(timer);
- this.timers.delete(clipId);
- this.scheduled.delete(clipId);
- }
-
- const loaded = this.voices.get(clipId);
- if (!loaded) return;
- this.voices.delete(clipId);
- this.destroy(loaded);
-
- if (this.rendering.delete(clipId)) this.onRenderChange?.();
- }
-
- /** Tears down one element and its graph node. */
- private destroy(loaded: Loaded): void {
- loaded.element.pause();
- loaded.element.removeAttribute("src");
- loaded.element.load();
- loaded.gain.disconnect();
- if (loaded.objectUrl) URL.revokeObjectURL(loaded.objectUrl);
- }
-
- dispose(): void {
- for (const timer of this.timers.values()) clearTimeout(timer);
- this.timers.clear();
- this.scheduled.clear();
- for (const clipId of [...this.voices.keys()]) this.release(clipId);
- }
-
- private load(voice: Voice): Loaded {
- const key = voice.filter?.key;
- const existing = this.voices.get(voice.clipId);
-
- if (existing && existing.key === key) return existing;
-
- // The settings changed. Do *not* tear down what is currently audible:
- // schedule the swap and keep playing the old render until the new one is
- // ready, so adjusting a filter does not silence the thing you are judging.
- if (existing) {
- this.schedule(voice);
- return existing;
- }
-
- return this.create(voice);
- }
-
- /** Queues a re-render, restarting the clock on every further change. */
- private schedule(voice: Voice): void {
- const key = voice.filter?.key;
-
- // `has` rather than a value comparison: removing the last filter makes the
- // wanted key `undefined`, and a missing entry also reads as `undefined`.
- // Comparing values there comes out equal, so the swap was never scheduled
- // and the clip kept playing its filtered render for good.
- if (this.timers.has(voice.clipId) && this.scheduled.get(voice.clipId) === key) return;
-
- const timer = this.timers.get(voice.clipId);
- if (timer !== undefined) clearTimeout(timer);
-
- this.scheduled.set(voice.clipId, key);
- this.timers.set(
- voice.clipId,
- window.setTimeout(() => {
- this.timers.delete(voice.clipId);
- this.scheduled.delete(voice.clipId);
-
- const previous = this.voices.get(voice.clipId);
- this.voices.delete(voice.clipId);
- this.create(voice);
- if (previous) this.destroy(previous);
- }, RENDER_DEBOUNCE_MS),
- );
- }
-
- private create(voice: Voice): Loaded {
- const key = voice.filter?.key;
- const element = new Audio();
- element.preload = "auto";
-
- const loaded: Loaded = {
- element,
- gain: connectElement(element),
- loading: true,
- key,
- offset: voice.filter ? voice.filter.sourceStart : 0,
- };
- this.voices.set(voice.clipId, loaded);
-
- const source = voice.filter
- ? renderFilteredAudio({
- path: voice.path,
- sourceStart: voice.filter.sourceStart,
- duration: voice.filter.duration,
- chain: voice.filter.chain,
- })
- : readMediaBytes(voice.path);
-
- if (voice.filter) {
- this.rendering.add(voice.clipId);
- this.onRenderChange?.();
- }
-
- // Source arrives asynchronously; until then the element is silent and the
- // sync pass simply has nothing to play, which is correct.
- void source
- // A filter chain that FFmpeg rejects would otherwise leave this element
- // with no source at all - permanently silent, and never retried because
- // its key already matches. Falling back to the unfiltered file keeps the
- // clip audible; the filters are simply not heard, which is a far better
- // failure than a clip that has gone quiet for no visible reason.
- .catch((cause) => {
- if (!voice.filter) throw cause;
- console.error(`WolfCut: filter render failed for ${voice.path}; playing it dry`, cause);
- loaded.offset = 0;
- return readMediaBytes(voice.path);
- })
- .then((bytes) => {
- const url = URL.createObjectURL(new Blob([bytes]));
-
- // A render that finishes after its clip was replaced or removed has
- // nothing to play into; hand the URL straight back rather than
- // leaking it into a detached element.
- if (this.voices.get(voice.clipId) !== loaded) {
- URL.revokeObjectURL(url);
- return;
- }
-
- loaded.objectUrl = url;
- loaded.element.src = url;
- loaded.element.load();
- })
- .catch((cause) => console.error(`WolfCut: could not load audio for ${voice.path}`, cause))
- .finally(() => {
- loaded.loading = false;
- if (this.rendering.delete(voice.clipId)) this.onRenderChange?.();
- });
-
- return loaded;
- }
-}
diff --git a/desktop/src/lib/engine.ts b/desktop/src/lib/engine.ts
index 631e7e20..fdb25677 100644
--- a/desktop/src/lib/engine.ts
+++ b/desktop/src/lib/engine.ts
@@ -60,20 +60,6 @@ export async function readMediaBytes(path: string): Promise<ArrayBuffer> {
return invoke<ArrayBuffer>("read_media_bytes", { path });
}
-/**
- * Renders a clip's audio through an FFmpeg filter chain, returning WAV bytes.
- *
- * The same chain string the exporter uses, so the preview cannot drift from
- * the rendered file.
- */
-export async function renderFilteredAudio(request: {
- path: string;
- sourceStart: number;
- duration: number;
- chain: string;
-}): Promise<ArrayBuffer> {
- return invoke<ArrayBuffer>("render_filtered_audio", request);
-}
/** A project on disk, as the launch screen sees it. */
export interface ProjectInfo {
diff --git a/desktop/src/lib/persist.ts b/desktop/src/lib/persist.ts
index 68b5bfba..45ae5bf6 100644
--- a/desktop/src/lib/persist.ts
+++ b/desktop/src/lib/persist.ts
@@ -157,6 +157,8 @@ export function fromDocument(raw: unknown): Project | null {
offsetY: number(clip.offsetY, 0),
speed: Math.max(0.1, number(clip.speed, 1)),
preservePitch: flag(clip.preservePitch, true),
+ muted: flag(clip.muted, false) || undefined,
+ detachedFrom: typeof clip.detachedFrom === "string" ? clip.detachedFrom : undefined,
filters: Array.isArray(clip.filters)
? clip.filters.flatMap((filter) => {
if (typeof filter !== "object" || filter === null) return [];
diff --git a/desktop/src/lib/project.ts b/desktop/src/lib/project.ts
index 5ffb8abf..b548ed17 100644
--- a/desktop/src/lib/project.ts
+++ b/desktop/src/lib/project.ts
@@ -147,6 +147,18 @@ export interface Clip {
*/
filters: ClipFilter[];
+ /**
+ * True when this video clip's embedded audio is left out of preview and
+ * export - the state "Detach audio" puts it in. Meaningless on other kinds.
+ */
+ muted?: boolean;
+
+ /**
+ * On a detached audio clip: the id of the video clip the sound came from.
+ * The link is what makes "Reattach audio" possible from either side.
+ */
+ detachedFrom?: string;
+
/**
* The overlay, when this is a text clip. Absent on every other kind.
*
@@ -475,6 +487,95 @@ export function removeClips(project: Project, clipIds: readonly string[]): Proje
return { ...project, clips: project.clips.filter((clip) => !doomed.has(clip.id)) };
}
+/** The audio clips holding sound detached from this video clip, if any. */
+export function detachedAudioOf(project: Project, videoClipId: string): Clip[] {
+ return project.clips.filter((clip) => clip.detachedFrom === videoClipId);
+}
+
+/**
+ * Splits a video clip's sound onto its own audio clip - "Detach audio".
+ *
+ * The video clip goes quiet (`muted`) and a linked audio clip appears with the
+ * same span of the same file, taking the audio filters with it - they were
+ * always about the sound. The link (`detachedFrom`) is what reattachment
+ * follows later. Does nothing for a clip that has no audio to give.
+ */
+export function detachAudio(project: Project, clipId: string): Project {
+ const clip = findClip(project, clipId);
+ const media = clip ? findMedia(project, clip.mediaId) : null;
+ if (!clip || clip.kind !== "video" || clip.muted || !media?.hasAudio) return project;
+ if (detachedAudioOf(project, clipId).length > 0) return project;
+
+ // A lane free for the whole span, or a fresh one - never on top of a clip.
+ const end = clip.start + clip.duration;
+ const free = project.tracks.find(
+ (track) =>
+ !project.clips.some(
+ (other) =>
+ other.trackId === track.id &&
+ other.start < end &&
+ clip.start < other.start + other.duration,
+ ),
+ );
+ let base = project;
+ let trackId = free?.id ?? "";
+ if (!trackId) {
+ const grown = addTrack(project);
+ base = grown.project;
+ trackId = grown.trackId;
+ }
+
+ const sound: Clip = {
+ ...clip,
+ id: nextId("c"),
+ trackId,
+ kind: "audio",
+ filters: [...clip.filters],
+ detachedFrom: clip.id,
+ muted: undefined,
+ };
+
+ return {
+ ...base,
+ clips: [
+ ...base.clips.map((other) =>
+ other.id === clipId ? { ...other, muted: true, filters: [] } : other,
+ ),
+ sound,
+ ],
+ };
+}
+
+/**
+ * Puts detached audio back into its video clip - "Reattach audio".
+ *
+ * Accepts either side of the link: the muted video clip or any of its
+ * detached audio clips. Every linked audio clip is absorbed (a detached clip
+ * that was split leaves several), the video clip sounds again, and the audio
+ * filters ride back with it. Does nothing when the other side is gone.
+ */
+export function reattachAudio(project: Project, clipId: string): Project {
+ const clip = findClip(project, clipId);
+ if (!clip) return project;
+
+ const videoId = clip.kind === "audio" && clip.detachedFrom ? clip.detachedFrom : clip.id;
+ const video = findClip(project, videoId);
+ const sounds = detachedAudioOf(project, videoId);
+ if (!video || sounds.length === 0) return project;
+
+ const doomed = new Set(sounds.map((sound) => sound.id));
+ return {
+ ...project,
+ clips: project.clips
+ .filter((other) => !doomed.has(other.id))
+ .map((other) =>
+ other.id === videoId
+ ? { ...other, muted: undefined, filters: [...sounds[0].filters] }
+ : other,
+ ),
+ };
+}
+
/** Cuts a clip in two at `time`. Does nothing if the cut misses the clip. */
export function splitClip(project: Project, clipId: string, time: number): Project {
const clip = findClip(project, clipId);
diff --git a/desktop/src/lib/transport.ts b/desktop/src/lib/transport.ts
index 3ca0e20d..6f04d8a4 100644
--- a/desktop/src/lib/transport.ts
+++ b/desktop/src/lib/transport.ts
@@ -1,17 +1,19 @@
import { useCallback, useEffect, useRef, useState } from "react";
+import { invoke, isTauri } from "@tauri-apps/api/core";
+import { listen } from "@tauri-apps/api/event";
/**
* The playback clock.
*
- * The playhead is computed as `origin + (now - startedAt)`, never accumulated
- * frame by frame. Accumulation drifts - it is the same mistake as summing
- * `1/29.97` in a loop, and it is why `relay-core` has a rational time type at
- * all. Here it also means a dropped frame costs nothing: the next tick reads
- * the wall clock and lands in the right place.
+ * The engine owns it: the position that matters is the audio device's sample
+ * counter, reported through `transport` events at ~30Hz. Between events the
+ * UI interpolates on the wall clock from the last reported anchor, so the
+ * playhead moves at animation rate while never drifting further than one
+ * event interval from the truth.
*
- * Playback is still paced by the browser here. That is a prototype
- * compromise: the engine owns the clock once it can decode and present
- * frames itself.
+ * In a plain browser (`npm run dev` with no Rust host) there is no engine and
+ * the anchor simply never gets corrected - the wall clock alone is the same
+ * prototype behaviour this file always had there.
*/
export interface Transport {
playhead: number;
@@ -26,64 +28,87 @@ export interface Transport {
}
export function useTransport({ duration }: { duration: number }): Transport {
+ const native = isTauri();
const [playhead, setPlayhead] = useState(0);
const [playing, setPlaying] = useState(false);
- // Where the clock was when playback started, and when that was.
- const origin = useRef(0);
- const startedAt = useRef(0);
- // Read inside the rAF loop so the effect does not restart on every change.
+ // The last known-true position and when it was known. While playing, the
+ // rendered playhead is `position + (now - at)`.
+ const anchor = useRef({ position: 0, at: 0 });
+ // Read inside loops and callbacks without restarting them.
+ const playheadRef = useRef(0);
+ playheadRef.current = playhead;
+ const playingRef = useRef(false);
+ playingRef.current = playing;
const durationRef = useRef(duration);
durationRef.current = duration;
- const pause = useCallback(() => setPlaying(false), []);
+ const pause = useCallback(() => {
+ setPlaying(false);
+ if (native) void invoke("transport_pause").catch(() => undefined);
+ }, [native]);
const play = useCallback(() => {
- setPlayhead((current) => {
- origin.current = current;
- return current;
- });
- startedAt.current = performance.now();
+ const from = playheadRef.current;
+ anchor.current = { position: from, at: performance.now() };
+ if (native) void invoke("transport_play", { position: from }).catch(() => undefined);
setPlaying(true);
- }, []);
+ }, [native]);
const toggle = useCallback(() => {
- if (playing) pause();
+ if (playingRef.current) pause();
else play();
- }, [pause, play, playing]);
+ }, [pause, play]);
- const seek = useCallback((seconds: number) => {
- const next = Math.max(0, seconds);
- origin.current = next;
- startedAt.current = performance.now();
- setPlayhead(next);
- }, []);
+ const seek = useCallback(
+ (seconds: number) => {
+ const next = Math.max(0, seconds);
+ anchor.current = { position: next, at: performance.now() };
+ setPlayhead(next);
+ if (native) void invoke("transport_seek", { position: next }).catch(() => undefined);
+ },
+ [native],
+ );
const step = useCallback(
(frames: number, frameRate: number) => {
// Land on an exact frame boundary rather than adding an offset to
// wherever the playhead happened to stop.
- setPlayhead((current) => {
- const next = Math.max(0, (Math.round(current * frameRate) + frames) / frameRate);
- origin.current = next;
- startedAt.current = performance.now();
- return next;
- });
+ const next = Math.max(0, (Math.round(playheadRef.current * frameRate) + frames) / frameRate);
+ seek(next);
},
- [],
+ [seek],
);
+ // Engine position reports re-anchor the interpolation. Ignored while
+ // paused: a report can arrive just after a local pause, and rewinding the
+ // playhead to it would visibly snap.
+ useEffect(() => {
+ if (!native) return;
+ let stopped = false;
+ const stopping = listen<number>("transport", (event) => {
+ if (stopped || !playingRef.current) return;
+ anchor.current = { position: event.payload, at: performance.now() };
+ });
+ return () => {
+ stopped = true;
+ void stopping.then((stop) => stop());
+ };
+ }, [native]);
+
+ // The render loop: interpolate from the anchor while playing.
useEffect(() => {
if (!playing) return;
let frame = 0;
const tick = (now: number) => {
- const next = origin.current + (now - startedAt.current) / 1000;
+ const next = anchor.current.position + (now - anchor.current.at) / 1000;
const end = durationRef.current;
if (end > 0 && next >= end) {
setPlayhead(end);
setPlaying(false);
+ if (native) void invoke("transport_pause").catch(() => undefined);
return;
}
@@ -93,7 +118,7 @@ export function useTransport({ duration }: { duration: number }): Transport {
frame = requestAnimationFrame(tick);
return () => cancelAnimationFrame(frame);
- }, [playing]);
+ }, [playing, native]);
return { playhead, playing, play, pause, toggle, seek, step };
}
diff --git a/engine/crates/relay-core/src/timeline.rs b/engine/crates/relay-core/src/timeline.rs
index c002783b..c97cdad8 100644
--- a/engine/crates/relay-core/src/timeline.rs
+++ b/engine/crates/relay-core/src/timeline.rs
@@ -61,12 +61,56 @@ pub struct Clip {
pub duration: Rational,
/// Blend factor in `0.0..=1.0`, applied over whatever is beneath.
pub opacity: f32,
+ /// How the picture sits in the frame. Identity is fitted and centred.
+ pub transform: Transform,
+}
+
+/// A clip's placement in the output frame.
+///
+/// Deliberately resolution-independent: `scale` is relative to the fitted
+/// size and the offsets are fractions of the frame, so the same transform
+/// means the same picture on a 720p export and a 4K one.
+#[derive(Clone, Copy, PartialEq, Debug)]
+pub struct Transform {
+ /// Multiplier over the fitted size. 1 fills the frame, preserving aspect.
+ pub scale: f64,
+ /// Offset of the picture's centre from frame centre, as a fraction of
+ /// frame width.
+ pub offset_x: f64,
+ /// Offset as a fraction of frame height.
+ pub offset_y: f64,
+ /// Clockwise rotation about the picture's centre, in degrees.
+ pub rotation: f64,
+}
+
+impl Transform {
+ /// Fitted, centred, unrotated.
+ pub const IDENTITY: Transform =
+ Transform { scale: 1.0, offset_x: 0.0, offset_y: 0.0, rotation: 0.0 };
+
+ /// True when applying this transform would change nothing.
+ pub fn is_identity(&self) -> bool {
+ *self == Self::IDENTITY
+ }
+}
+
+impl Default for Transform {
+ fn default() -> Self {
+ Self::IDENTITY
+ }
}
impl Clip {
/// A clip that starts at the beginning of its media and is fully opaque.
pub fn new(media: MediaRef, start: Rational, duration: Rational) -> Self {
- Self { media, source_start: Rational::ZERO, start, duration, opacity: 1.0 }
+ Self {
+ media,
+ source_start: Rational::ZERO,
+ start,
+ duration,
+ opacity: 1.0,
+ transform: Transform::IDENTITY,
+ }
}
/// The span of timeline this clip occupies.
diff --git a/engine/crates/relay-render/src/compositor.rs b/engine/crates/relay-render/src/compositor.rs
index cebc627f..6bf2a420 100644
--- a/engine/crates/relay-render/src/compositor.rs
+++ b/engine/crates/relay-render/src/compositor.rs
@@ -7,6 +7,40 @@
use relay_core::frame::{BYTES_PER_PIXEL, Frame};
+/// A layer's placement beyond its base position, in output pixels.
+///
+/// Applied about the layer's centre: scale first, then rotation, then the
+/// translation. Pixel-space on purpose - the resolution-independent form
+/// lives in `relay-core::timeline::Transform`, and whoever builds layers
+/// converts exactly once.
+#[derive(Clone, Copy, PartialEq, Debug)]
+pub struct Placement {
+ /// Multiplier over the layer's own size.
+ pub scale: f32,
+ /// Clockwise rotation about the layer's centre, in radians.
+ pub rotation: f32,
+ /// Offset of the layer's centre from its base position, in pixels.
+ pub translate_x: f32,
+ pub translate_y: f32,
+}
+
+impl Placement {
+ /// Unscaled, unrotated, unmoved.
+ pub const IDENTITY: Placement =
+ Placement { scale: 1.0, rotation: 0.0, translate_x: 0.0, translate_y: 0.0 };
+
+ /// True when applying this placement would change nothing.
+ pub fn is_identity(&self) -> bool {
+ *self == Self::IDENTITY
+ }
+}
+
+impl Default for Placement {
+ fn default() -> Self {
+ Self::IDENTITY
+ }
+}
+
/// One source image, positioned and weighted.
#[derive(Clone, Copy, Debug)]
pub struct Layer<'a> {
@@ -18,12 +52,14 @@ pub struct Layer<'a> {
pub x: i32,
/// Vertical offset of the layer's top-left corner. May be negative.
pub y: i32,
+ /// Scale, rotation and translation about the layer's centre.
+ pub placement: Placement,
}
impl<'a> Layer<'a> {
/// A layer drawn at the origin, fully opaque.
pub fn new(frame: &'a Frame) -> Self {
- Self { frame, opacity: 1.0, x: 0, y: 0 }
+ Self { frame, opacity: 1.0, x: 0, y: 0, placement: Placement::IDENTITY }
}
/// Sets the blend strength.
@@ -38,6 +74,12 @@ impl<'a> Layer<'a> {
self.y = y;
self
}
+
+ /// Sets the scale/rotation/translation.
+ pub fn with_placement(mut self, placement: Placement) -> Self {
+ self.placement = placement;
+ self
+ }
}
/// Blends layers into a single output frame.
@@ -65,6 +107,12 @@ impl Compositor for CpuCompositor {
}
let source = layer.frame;
+
+ if !layer.placement.is_identity() {
+ blend_transformed(&mut output, layer, opacity);
+ continue;
+ }
+
let Some((dst_x, src_x, columns)) = overlap(layer.x, source.width(), width) else {
continue;
};
@@ -86,6 +134,93 @@ impl Compositor for CpuCompositor {
}
}
+/// Draws one layer through its placement: inverse-mapped, bilinearly sampled.
+///
+/// Each covered output pixel is carried backwards through the placement into
+/// source coordinates and sampled there. Inverse mapping is what makes the
+/// result hole-free at any scale or angle; bilinear is the cheapest filter
+/// that does not shimmer on motion. Only the transformed bounding box is
+/// visited, so a small layer stays cheap on a large frame.
+fn blend_transformed(output: &mut Frame, layer: &Layer<'_>, opacity: f32) {
+ let source = layer.frame;
+ let placement = layer.placement;
+ let scale = placement.scale.max(1e-6);
+ let (sin, cos) = placement.rotation.sin_cos();
+
+ let src_w = source.width() as f32;
+ let src_h = source.height() as f32;
+ // The layer's centre in output space, after translation.
+ let centre_x = layer.x as f32 + src_w / 2.0 + placement.translate_x;
+ let centre_y = layer.y as f32 + src_h / 2.0 + placement.translate_y;
+
+ // Bounding box of the transformed rectangle, clamped to the output.
+ let half_w = src_w * scale / 2.0;
+ let half_h = src_h * scale / 2.0;
+ let reach_x = (half_w * cos.abs()) + (half_h * sin.abs());
+ let reach_y = (half_w * sin.abs()) + (half_h * cos.abs());
+
+ let x_from = ((centre_x - reach_x).floor().max(0.0)) as u32;
+ let y_from = ((centre_y - reach_y).floor().max(0.0)) as u32;
+ let x_to = ((centre_x + reach_x).ceil().min(output.width() as f32)) as u32;
+ let y_to = ((centre_y + reach_y).ceil().min(output.height() as f32)) as u32;
+ if x_from >= x_to || y_from >= y_to {
+ return;
+ }
+
+ let dst_stride = output.width() as usize * BYTES_PER_PIXEL;
+ let src_stride = source.width() as usize * BYTES_PER_PIXEL;
+ let src_pixels = source.pixels();
+ let dst_pixels = output.pixels_mut();
+
+ for y in y_from..y_to {
+ for x in x_from..x_to {
+ // Sample at the pixel centre, mapped back into source space:
+ // untranslate, unrotate, unscale, then re-origin at the corner.
+ let dx = (x as f32 + 0.5) - centre_x;
+ let dy = (y as f32 + 0.5) - centre_y;
+ let sx = (dx * cos + dy * sin) / scale + src_w / 2.0 - 0.5;
+ let sy = (-dx * sin + dy * cos) / scale + src_h / 2.0 - 0.5;
+
+ if sx < -0.5 || sy < -0.5 || sx > src_w - 0.5 || sy > src_h - 0.5 {
+ continue;
+ }
+
+ let x0 = sx.floor().max(0.0) as usize;
+ let y0 = sy.floor().max(0.0) as usize;
+ let x1 = (x0 + 1).min(source.width() as usize - 1);
+ let y1 = (y0 + 1).min(source.height() as usize - 1);
+ let fx = (sx - x0 as f32).clamp(0.0, 1.0);
+ let fy = (sy - y0 as f32).clamp(0.0, 1.0);
+
+ let mut sample = [0.0f32; 4];
+ for (corner_x, corner_y, weight) in [
+ (x0, y0, (1.0 - fx) * (1.0 - fy)),
+ (x1, y0, fx * (1.0 - fy)),
+ (x0, y1, (1.0 - fx) * fy),
+ (x1, y1, fx * fy),
+ ] {
+ let at = corner_y * src_stride + corner_x * BYTES_PER_PIXEL;
+ for channel in 0..4 {
+ sample[channel] += f32::from(src_pixels[at + channel]) * weight;
+ }
+ }
+
+ let alpha = (sample[3] / 255.0) * opacity;
+ if alpha <= 0.0 {
+ continue;
+ }
+
+ let at = y as usize * dst_stride + x as usize * BYTES_PER_PIXEL;
+ for channel in 0..3 {
+ let over = sample[channel] * alpha;
+ let under = f32::from(dst_pixels[at + channel]) * (1.0 - alpha);
+ dst_pixels[at + channel] = (over + under).round().clamp(0.0, 255.0) as u8;
+ }
+ dst_pixels[at + 3] = 255;
+ }
+ }
+}
+
/// Works out the visible span along one axis when a layer of `source` pixels is
/// placed at `offset` inside a destination of `destination` pixels.
///
@@ -220,6 +355,47 @@ mod tests {
assert_eq!(frame.pixel(1, 1), Some([255, 0, 0, 255]));
}
+ #[test]
+ fn scaling_doubles_coverage() {
+ let red = solid(2, 2, [255, 0, 0, 255]);
+ let placement = Placement { scale: 2.0, ..Placement::IDENTITY };
+ let frame =
+ CpuCompositor.composite(4, 4, &[Layer::new(&red).at(1, 1).with_placement(placement)]);
+ // A 2x2 layer based at (1,1) scaled x2 about its centre covers the
+ // whole 4x4 output.
+ assert_eq!(frame.pixel(0, 0), Some([255, 0, 0, 255]));
+ assert_eq!(frame.pixel(3, 3), Some([255, 0, 0, 255]));
+ }
+
+ #[test]
+ fn translation_moves_the_layer() {
+ let red = solid(1, 1, [255, 0, 0, 255]);
+ let placement = Placement { translate_x: 2.0, ..Placement::IDENTITY };
+ let frame = CpuCompositor.composite(3, 1, &[Layer::new(&red).with_placement(placement)]);
+ assert_eq!(frame.pixel(0, 0), Some([0, 0, 0, 255]));
+ assert_eq!(frame.pixel(2, 0), Some([255, 0, 0, 255]));
+ }
+
+ #[test]
+ fn a_half_turn_swaps_the_ends() {
+ let mut strip = Frame::transparent(2, 1);
+ strip.set_pixel(0, 0, [255, 0, 0, 255]);
+ strip.set_pixel(1, 0, [0, 0, 255, 255]);
+ let placement = Placement { rotation: std::f32::consts::PI, ..Placement::IDENTITY };
+ let frame = CpuCompositor.composite(2, 1, &[Layer::new(&strip).with_placement(placement)]);
+ assert_eq!(frame.pixel(0, 0), Some([0, 0, 255, 255]));
+ assert_eq!(frame.pixel(1, 0), Some([255, 0, 0, 255]));
+ }
+
+ #[test]
+ fn a_transformed_layer_is_clipped_at_the_frame_edge() {
+ let red = solid(2, 2, [255, 0, 0, 255]);
+ let placement = Placement { scale: 100.0, ..Placement::IDENTITY };
+ let frame = CpuCompositor.composite(2, 2, &[Layer::new(&red).with_placement(placement)]);
+ assert_eq!(frame.width(), 2);
+ assert_eq!(frame.pixel(1, 1), Some([255, 0, 0, 255]));
+ }
+
#[test]
fn overlap_spans_are_correct() {
assert_eq!(overlap(0, 4, 4), Some((0, 0, 4)));
diff --git a/engine/crates/relay-render/src/lib.rs b/engine/crates/relay-render/src/lib.rs
index 3b249267..4a7408b1 100644
--- a/engine/crates/relay-render/src/lib.rs
+++ b/engine/crates/relay-render/src/lib.rs
@@ -13,5 +13,5 @@
pub mod compositor;
pub mod plan;
-pub use compositor::{Compositor, CpuCompositor, Layer};
+pub use compositor::{Compositor, CpuCompositor, Layer, Placement};
pub use plan::{FramePlan, PlannedLayer, plan_frame};
diff --git a/engine/crates/relay-render/src/plan.rs b/engine/crates/relay-render/src/plan.rs
index 56ec67a4..da5c92f3 100644
--- a/engine/crates/relay-render/src/plan.rs
+++ b/engine/crates/relay-render/src/plan.rs
@@ -11,7 +11,7 @@
use std::path::PathBuf;
use relay_core::time::Rational;
-use relay_core::timeline::{ClipId, Timeline, TrackKind};
+use relay_core::timeline::{ClipId, Timeline, TrackKind, Transform};
/// One visible layer at one instant.
#[derive(Clone, PartialEq, Debug)]
@@ -24,6 +24,8 @@ pub struct PlannedLayer {
pub source_time: Rational,
/// Blend strength over everything beneath, in `0.0..=1.0`.
pub opacity: f32,
+ /// The clip's placement in the frame, resolution-independent.
+ pub transform: Transform,
}
/// Everything needed to draw one output frame.
@@ -73,6 +75,7 @@ pub fn plan_frame(timeline: &Timeline, time: Rational) -> FramePlan {
media: clip.media.path.clone(),
source_time,
opacity: clip.opacity.clamp(0.0, 1.0),
+ transform: clip.transform,
});
}