Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.
TypeScript cross-platformelectronffmpegopen-sourceproduct-demorustscreen-capturescreen-recorderscreen-recordingscreen-studioscreencastspeech-to-texttypescriptvideo-editingvideo-editorwaylandwgpuwhisper
//! H.264 encoding and MP4 muxing for the captured screen.
//!
//! THE LADDER. Three backends are tried in order and the first one that actually
//! opens wins. "Actually opens" is the whole point: `avcodec_find_encoder_by_name`
//! succeeds for every encoder compiled into the library, including ones this
//! machine's GPU cannot run, so the only honest capability probe is to open the
//! encoder with the real parameters and see what happens.
//!
//! 1. VAAPI — the broadest hardware path on Linux (AMD, Intel, and NVIDIA
//! through nvidia-vaapi-driver).
//! 2. Vulkan — VK_KHR_video_encode_queue. Newer, and on RADV it needs
//! `RADV_PERFTEST=video_encode` / `RADV_EXPERIMENTAL=video_encode`;
//! see [`prepare_environment`].
//! 3. libopenh264 — software. Always available, always last.
//!
//! WHY VAAPI IS GUARDED BY A dlsym. The vendored ffmpeg 8.1 calls `vaMapBuffer2`,
//! which libva only grew in 2.22. On a system with an older libva (Ubuntu 24.04
//! ships 2.20) the call goes through an implib-generated trampoline that does not
//! return an error — it calls `assert(0)` and the process dumps core. A probe
//! that crashes the helper is worse than no probe, so the symbol is checked
//! before libavutil is ever allowed to touch VA-API.
use std::ffi::{CStr, CString};
use std::path::Path;
use std::ptr;
use crate::capture::BYTES_PER_SOURCE_PIXEL;
use crate::ffmpeg as ff;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Backend {
Vaapi,
Vulkan,
Software,
}
/// Whether the dmabuf → VAAPI zero-copy import may be used for a `forced` encoder
/// choice. The import can only feed VAAPI, so `software`/`vulkan` must skip it —
/// otherwise the documented `OPENSCREEN_LINUX_ENCODER` override is silently
/// ignored on the dmabuf path.
///
/// The SINGLE source of truth for this: the negotiation offer (`prefer_dmabuf` in
/// main) and the importer build (`Capture::start`) both call it, so they cannot
/// drift into offering dmabuf that no importer will consume.
pub fn forced_allows_dmabuf(forced: Option<Backend>) -> bool {
matches!(forced, None | Some(Backend::Vaapi))
}
impl Backend {
/// The name the app reports and the tests match on. Kept in the same
/// vocabulary as the Windows helper's `encoder-selection` event.
pub fn as_str(self) -> &'static str {
match self {
Self::Vaapi => "vaapi",
Self::Vulkan => "vulkan",
Self::Software => "software",
}
}
fn codec_name(self) -> &'static CStr {
match self {
Self::Vaapi => c"h264_vaapi",
Self::Vulkan => c"h264_vulkan",
Self::Software => c"libopenh264",
}
}
fn hw_device_type(self) -> Option<ff::AVHWDeviceType> {
match self {
Self::Vaapi => Some(ff::AV_HWDEVICE_TYPE_VAAPI),
Self::Vulkan => Some(ff::AV_HWDEVICE_TYPE_VULKAN),
Self::Software => None,
}
}
/// The pixel format the codec context takes. Hardware encoders take an
/// opaque handle; the real pixels live in [`Self::upload_format`].
fn codec_pixel_format(self) -> ff::AVPixelFormat {
match self {
Self::Vaapi => ff::AV_PIX_FMT_VAAPI,
Self::Vulkan => ff::AV_PIX_FMT_VULKAN,
Self::Software => ff::AV_PIX_FMT_YUV420P,
}
}
/// What swscale converts the captured RGB into.
fn upload_format(self) -> ff::AVPixelFormat {
match self {
// Every VCN/QuickSync/NVENC block wants NV12; YUV420P would force
// ffmpeg into an extra internal conversion on upload.
Self::Vaapi | Self::Vulkan => ff::AV_PIX_FMT_NV12,
Self::Software => ff::AV_PIX_FMT_YUV420P,
}
}
/// The highest QP the encoder's rate control may pick, or `None` to leave
/// the codec's own ceiling (51) in place.
///
/// THE FIRST KEYFRAME, NOT THE BITRATE. On AMD (VCN through Mesa's VA-API
/// driver) the rate control gives the first IDR of a session about eight
/// times the average frame budget, whatever the VBV size, its initial
/// fullness or the peak rate say — all three were varied and none moved it.
/// When the picture needs more than that, the macroblock-level control
/// raises QP down the frame to stay inside it: a 866x628 terminal window at
/// the default 3.3 Mbit/s came out QP 13 at the top and 51 for the whole
/// lower half, unreadable, in a 54 KB frame. The next frames repair it, but
/// the editor opens on frame 0, so that is the picture the user sees.
/// Later keyframes are fine (95 KB at QP 20-25 in the same file): it is only
/// the session's first one that has no history to be budgeted from.
///
/// ffmpeg passes `qmax` as `VAEncMiscParameterRateControl::max_qp`, which
/// the firmware honours: the same frame comes out QP 13-30, 70 KB, and
/// 41 dB PSNR against the source instead of 29, while the first second
/// costs the same within 2% because the following P-frames have less to
/// repair. QP 30 rarely binds otherwise — about 3% of the macroblocks of a
/// real 1080p60 take sat above it, in bursts of change — and where it does
/// it buys quality with bitrate (a 1080p segment starved at 3 Mbit/s came
/// out at 5.6 instead of 4.2), the right trade for a file the editor
/// re-encodes.
///
/// VAAPI only, because VAAPI is where it was measured. Vulkan (rung 2) and
/// libopenh264 keep their defaults.
fn max_qp(self) -> Option<i32> {
match self {
Self::Vaapi => Some(30),
Self::Vulkan | Self::Software => None,
}
}
}
/// Order of preference. Public so the probe can be driven from a test or from
/// `OPENSCREEN_LINUX_ENCODER` without duplicating the list.
pub const LADDER: [Backend; 3] = [Backend::Vaapi, Backend::Vulkan, Backend::Software];
/// Must run before anything creates a Vulkan instance — which, in this process,
/// means before the first `av_hwdevice_ctx_create`.
///
/// Mesa gates RADV's video encode queues behind a flag; without it
/// `h264_vulkan` fails with "Device does not support the VK_KHR_video_encode_queue
/// extension" on hardware that supports it perfectly well.
///
/// Mesa 24.1+ renamed `RADV_PERFTEST=video_encode` to `RADV_EXPERIMENTAL=video_encode`,
/// deprecating the older variable. We populate both so older and newer Mesa versions
/// both enable the queue. Setting the variables only affects this process, and
/// drivers that do not know them ignore them. Any existing value is preserved.
pub fn prepare_environment() {
fn add_flag(key: &str, flag: &str) {
match std::env::var(key) {
Ok(existing) if existing.split(',').any(|f| f.trim() == flag) => {}
Ok(existing) if !existing.is_empty() => {
std::env::set_var(key, format!("{existing},{flag}"));
}
_ => std::env::set_var(key, flag),
}
}
add_flag("RADV_PERFTEST", "video_encode");
add_flag("RADV_EXPERIMENTAL", "video_encode");
}
/// True when libva is new enough that libavutil's VA-API code will not abort.
///
/// See the module header. This deliberately checks the SYMBOL rather than a
/// version string: the version macro reflects the headers ffmpeg was built
/// against, and what matters is what is loadable here and now.
fn vaapi_is_safe_to_probe() -> bool {
// SAFETY: dlopen/dlsym with a literal name; every pointer is checked before
// use and the handle is closed on both paths.
unsafe {
let handle = dlopen(c"libva.so.2".as_ptr(), RTLD_LAZY | RTLD_LOCAL);
if handle.is_null() {
return false;
}
let symbol = dlsym(handle, c"vaMapBuffer2".as_ptr());
dlclose(handle);
!symbol.is_null()
}
}
const RTLD_LAZY: i32 = 1;
const RTLD_LOCAL: i32 = 0;
extern "C" {
fn dlopen(filename: *const std::ffi::c_char, flags: i32) -> *mut std::ffi::c_void;
fn dlsym(
handle: *mut std::ffi::c_void,
symbol: *const std::ffi::c_char,
) -> *mut std::ffi::c_void;
fn dlclose(handle: *mut std::ffi::c_void) -> i32;
}
pub struct VideoParams {
pub width: i32,
pub height: i32,
pub fps: i32,
pub bitrate: i64,
}
/// Where the per-frame time actually goes.
///
/// Not debug scaffolding: the capture loop has a hard real-time budget (16.7 ms
/// at 60 fps) and when it is missed the useful question is always *which* of the
/// three stages missed it. Reported on `capture-stopped` so a slow recording on
/// a user's machine is diagnosable without a rebuild. Three `Instant::now()`
/// calls per frame is about 60 ns against a budget four orders of magnitude
/// larger.
#[derive(Debug, Default, Clone, Copy)]
pub struct EncodeStats {
pub frames: u64,
/// swscale: captured RGB → NV12/YUV420P in system memory.
pub convert_ns: u128,
/// `av_hwframe_transfer_data`: system memory → GPU. Zero for software.
pub upload_ns: u128,
/// `avcodec_send_frame` plus draining whatever packets came back.
pub encode_ns: u128,
}
impl EncodeStats {
fn mean_ms(total_ns: u128, frames: u64) -> f64 {
if frames == 0 {
return 0.0;
}
total_ns as f64 / frames as f64 / 1_000_000.0
}
pub fn convert_ms(&self) -> f64 {
Self::mean_ms(self.convert_ns, self.frames)
}
pub fn upload_ms(&self) -> f64 {
Self::mean_ms(self.upload_ns, self.frames)
}
pub fn encode_ms(&self) -> f64 {
Self::mean_ms(self.encode_ns, self.frames)
}
}
/// Everything the encoder owns, freed in reverse order of acquisition by `Drop`.
pub struct VideoEncoder {
backend: Backend,
codec_ctx: *mut ff::AVCodecContext,
hw_device: *mut ff::AVBufferRef,
hw_frames: *mut ff::AVBufferRef,
/// NV12/YUV420P staging in system memory; the target of swscale and the
/// source of the hardware upload.
sw_frame: *mut ff::AVFrame,
/// The GPU-side frame handed to a hardware encoder. Null for software.
hw_frame: *mut ff::AVFrame,
/// A ready-to-encode VAAPI NV12 surface produced by the dmabuf importer
/// (issue #507). When non-null it is sent directly — no sws_scale, no upload —
/// and held across the clock-driven re-encodes until the next frame replaces
/// it. Null on the shm/software path.
hw_staged: *mut ff::AVFrame,
/// False only for an encoder from [`Self::open_importing`] on a libva
/// without `vaMapBuffer2`, where [`Self::stage`] refuses CPU frames because
/// their upload would abort the process (issue #534).
upload_is_safe: bool,
sws: *mut ff::SwsContext,
sws_src_format: ff::AVPixelFormat,
packet: *mut ff::AVPacket,
params: VideoParams,
stats: EncodeStats,
/// Whether `sw_frame` holds a real picture yet.
staged: bool,
/// What libswscale reported back after `sws_init_context`, not what we asked
/// for. Requesting threads and silently getting one is indistinguishable
/// from threading not helping.
sws_threads: i64,
}
impl VideoEncoder {
/// Walks [`LADDER`] and returns the first backend that opens, or every
/// backend's error if none does.
///
/// `forced` short-circuits the walk — it is how the verification step
/// compares backends on the same machine, and how a user works around a
/// driver that opens successfully and then produces garbage.
pub fn open(
params: VideoParams,
forced: Option<Backend>,
mut on_attempt: impl FnMut(Backend, &str),
) -> Result<Self, String> {
prepare_environment();
let candidates: Vec<Backend> = match forced {
Some(backend) => vec![backend],
None => LADDER.to_vec(),
};
let mut failures = Vec::new();
for backend in candidates {
// Forced too: every CPU frame on this path is uploaded, so a forced VAAPI
// encoder on such a libva would abort on its first frame instead of
// failing here with a readable reason.
if backend == Backend::Vaapi && !vaapi_is_safe_to_probe() {
let reason = "libva.so.2 does not export vaMapBuffer2, so this ffmpeg build \
would abort inside VA-API rather than fail cleanly"
.to_owned();
on_attempt(backend, &reason);
failures.push(format!("{}: {reason}", backend.as_str()));
continue;
}
match Self::open_backend(backend, ¶ms, None) {
Ok(encoder) => return Ok(encoder),
Err(error) => {
on_attempt(backend, &error);
failures.push(format!("{}: {error}", backend.as_str()));
}
}
}
Err(format!(
"no H.264 encoder could be opened at {}x{}. Attempts: {}",
params.width,
params.height,
failures.join(" | ")
))
}
/// Opens the VAAPI encoder to consume surfaces from an EXISTING device and
/// NV12 frames pool — the ones the dmabuf importer built. Sharing the pool is
/// what lets `encode_staged` send an imported surface straight to
/// `avcodec_send_frame` without a copy (issue #507).
///
/// SAFETY: `device` and `frames_ctx` must be a live VAAPI device and an NV12
/// VAAPI frames context on it; the encoder takes its own references.
///
/// BYPASSES THE LADDER'S `vaMapBuffer2` GUARD. The importer only maps, so
/// nothing it does reaches that symbol. A CPU frame staged later (after a
/// renegotiation to a modifier-less format) would be uploaded into this pool
/// through `av_hwframe_transfer_data`, which does. So the guard is taken here
/// and enforced in [`Self::stage`], the only way to reach that upload.
pub unsafe fn open_importing(
params: VideoParams,
device: *mut ff::AVBufferRef,
frames_ctx: *mut ff::AVBufferRef,
) -> Result<Self, String> {
let mut encoder = Self::open_backend(Backend::Vaapi, ¶ms, Some((device, frames_ctx)))?;
encoder.upload_is_safe = vaapi_is_safe_to_probe();
Ok(encoder)
}
fn open_backend(
backend: Backend,
params: &VideoParams,
external: Option<(*mut ff::AVBufferRef, *mut ff::AVBufferRef)>,
) -> Result<Self, String> {
// SAFETY: this whole function is a single ffmpeg setup sequence. Every
// allocation is stored in `encoder` as soon as it succeeds, so the Drop
// impl frees whatever was reached if a later step fails.
unsafe {
let codec = ff::avcodec_find_encoder_by_name(backend.codec_name().as_ptr());
if codec.is_null() {
return Err(format!(
"{} is not compiled into this ffmpeg build",
backend.codec_name().to_string_lossy()
));
}
let codec_ctx = ff::avcodec_alloc_context3(codec);
if codec_ctx.is_null() {
return Err("avcodec_alloc_context3 returned null".to_owned());
}
let mut encoder = Self {
backend,
codec_ctx,
hw_device: ptr::null_mut(),
hw_frames: ptr::null_mut(),
sw_frame: ptr::null_mut(),
hw_frame: ptr::null_mut(),
hw_staged: ptr::null_mut(),
upload_is_safe: true,
sws: ptr::null_mut(),
sws_src_format: ff::AV_PIX_FMT_NONE,
packet: ptr::null_mut(),
params: VideoParams {
width: params.width,
height: params.height,
fps: params.fps,
bitrate: params.bitrate,
},
stats: EncodeStats::default(),
staged: false,
sws_threads: 0,
};
(*codec_ctx).width = params.width;
(*codec_ctx).height = params.height;
// Constant frame rate: `time_base` is one frame, so a PTS is simply
// a frame index. Everything downstream — the muxer, the editor's
// timeline, the audio interleave — is easier to reason about than it
// would be with wall-clock timestamps, and duplicated frames of a
// static screen cost a few hundred bytes each.
(*codec_ctx).time_base = ff::AVRational { num: 1, den: params.fps };
(*codec_ctx).framerate = ff::AVRational { num: params.fps, den: 1 };
(*codec_ctx).pix_fmt = backend.codec_pixel_format();
(*codec_ctx).bit_rate = params.bitrate;
if let Some(max_qp) = backend.max_qp() {
(*codec_ctx).qmax = max_qp;
}
// What the samples are (see `ensure_sws` and the dmabuf VPP), said in
// the stream so no player has to guess from the frame size (#926).
(*codec_ctx).color_range = ff::AVCOL_RANGE_MPEG;
(*codec_ctx).colorspace = ff::AVCOL_SPC_BT709;
(*codec_ctx).color_primaries = ff::AVCOL_PRI_BT709;
(*codec_ctx).color_trc = ff::AVCOL_TRC_BT709;
// Half a second, not the two seconds a streaming preset would use.
//
// This file is an EDITING SOURCE, and the editor scrubs it. Seeking
// to an arbitrary point means decoding forward from the previous
// keyframe, so the GOP length is exactly the worst-case scrub cost:
// at 60 fps a two-second GOP makes the editor decode up to 120
// frames to show one. Halving it to 30 quarters that, and the extra
// keyframes cost a few percent of bitrate on screen content, which
// is mostly static and compresses well anyway.
(*codec_ctx).gop_size = (params.fps / 2).max(1);
// No B-frames. A screen recording has no use for them, they add
// reorder delay, and they force DTS bookkeeping the muxer would
// otherwise not need.
(*codec_ctx).max_b_frames = 0;
// The muxer needs an extradata-carrying global header for MP4.
(*codec_ctx).flags |= ff::AV_CODEC_FLAG_GLOBAL_HEADER as i32;
if let Some(device_type) = backend.hw_device_type() {
encoder.attach_hardware(device_type, params, external)?;
}
let opened = ff::avcodec_open2(codec_ctx, codec, ptr::null_mut());
if opened < 0 {
return Err(format!("avcodec_open2: {}", ff::err_to_string(opened)));
}
encoder.packet = ff::av_packet_alloc();
if encoder.packet.is_null() {
return Err("av_packet_alloc returned null".to_owned());
}
encoder.alloc_software_frame(backend, params)?;
if backend.hw_device_type().is_some() {
encoder.hw_frame = ff::av_frame_alloc();
if encoder.hw_frame.is_null() {
return Err("av_frame_alloc returned null".to_owned());
}
}
Ok(encoder)
}
}
/// SAFETY: called only from `open_backend`, with `self.codec_ctx` valid.
unsafe fn attach_hardware(
&mut self,
device_type: ff::AVHWDeviceType,
params: &VideoParams,
external: Option<(*mut ff::AVBufferRef, *mut ff::AVBufferRef)>,
) -> Result<(), String> {
// The dmabuf importer already built a VAAPI device and an NV12 pool; the
// encoder must consume from THAT pool, so take references to it instead of
// creating a second, incompatible one. See `open_importing`.
if let Some((device, frames_ctx)) = external {
// The guard belongs on the INPUTS, before the refs are taken:
// av_buffer_ref dereferences its argument (`*ret = *buf`), so a null
// one faults inside ffmpeg instead of coming back as a null return
// value the old check could inspect. And null IS reachable here --
// `DmabufImporter::output_frames_ctx` is av_buffersink_get_hw_frames_ctx,
// which returns NULL whenever the sink's input link carries no hw
// frames context.
if device.is_null() || frames_ctx.is_null() {
return Err("av_buffer_ref on the shared VAAPI context returned null".to_owned());
}
self.hw_device = ff::av_buffer_ref(device);
self.hw_frames = ff::av_buffer_ref(frames_ctx);
// The results still get checked: av_buffer_ref allocates, and the
// hw_frames_ctx ref below would dereference whatever it handed back.
if self.hw_device.is_null() || self.hw_frames.is_null() {
return Err("av_buffer_ref on the shared VAAPI context returned null".to_owned());
}
(*self.codec_ctx).hw_frames_ctx = ff::av_buffer_ref(self.hw_frames);
if (*self.codec_ctx).hw_frames_ctx.is_null() {
return Err("av_buffer_ref on the shared frames context returned null".to_owned());
}
return Ok(());
}
let created = ff::av_hwdevice_ctx_create(
&mut self.hw_device,
device_type,
ptr::null(),
ptr::null_mut(),
0,
);
if created < 0 {
return Err(format!(
"av_hwdevice_ctx_create: {}",
ff::err_to_string(created)
));
}
self.hw_frames = ff::av_hwframe_ctx_alloc(self.hw_device);
if self.hw_frames.is_null() {
return Err("av_hwframe_ctx_alloc returned null".to_owned());
}
let frames_ctx = (*self.hw_frames).data as *mut ff::AVHWFramesContext;
(*frames_ctx).format = self.backend.codec_pixel_format();
(*frames_ctx).sw_format = self.backend.upload_format();
(*frames_ctx).width = params.width;
(*frames_ctx).height = params.height;
// Enough surfaces that the encoder can hold a few references while we
// are filling the next one. Too small a pool stalls; too large just
// wastes VRAM.
(*frames_ctx).initial_pool_size = 8;
let initialised = ff::av_hwframe_ctx_init(self.hw_frames);
if initialised < 0 {
return Err(format!(
"av_hwframe_ctx_init: {}",
ff::err_to_string(initialised)
));
}
(*self.codec_ctx).hw_frames_ctx = ff::av_buffer_ref(self.hw_frames);
if (*self.codec_ctx).hw_frames_ctx.is_null() {
return Err("av_buffer_ref on the frames context returned null".to_owned());
}
Ok(())
}
/// SAFETY: called only from `open_backend`.
unsafe fn alloc_software_frame(
&mut self,
backend: Backend,
params: &VideoParams,
) -> Result<(), String> {
self.sw_frame = ff::av_frame_alloc();
if self.sw_frame.is_null() {
return Err("av_frame_alloc returned null".to_owned());
}
(*self.sw_frame).format = backend.upload_format();
(*self.sw_frame).width = params.width;
(*self.sw_frame).height = params.height;
let allocated = ff::av_frame_get_buffer(self.sw_frame, 0);
if allocated < 0 {
return Err(format!(
"av_frame_get_buffer: {}",
ff::err_to_string(allocated)
));
}
Ok(())
}
pub fn backend(&self) -> Backend {
self.backend
}
/// Borrows the opened codec context so the muxer can copy its parameters.
pub fn codec_context(&self) -> *mut ff::AVCodecContext {
self.codec_ctx
}
/// Converts one captured RGB frame into the staging buffer, without encoding
/// it.
///
/// Split from [`Self::encode_staged`] so that a screen which stops changing
/// still produces frames: the capture writes at a constant rate, and holding
/// the last picture costs an upload and an encode but no conversion, which
/// is where three quarters of the per-frame time goes.
///
/// `src_format` is whatever PipeWire negotiated; the swscale context is
/// rebuilt if it ever changes, which it should not mid-stream.
pub fn stage(
&mut self,
pixels: &[u8],
stride: usize,
src_format: ff::AVPixelFormat,
) -> Result<(), String> {
// Before anything is touched, `hw_staged` above all: `Capture::finish`
// re-sends that last imported surface as its tail write, so refusing here
// ends the recording with a playable file instead of a core dump.
if !self.upload_is_safe {
return Err(concat!(
"a CPU frame reached the dmabuf encoder, and libva.so.2 does not export ",
"vaMapBuffer2: uploading it would abort the process"
)
.to_owned());
}
// The LAST row needs only its own pixels, not a further stride's worth of
// padding. Demanding `stride * height` rejected exactly the frames a
// window crop produces: `pixels` there starts partway into the buffer, so
// the tail is short by the offset even though every row is complete.
let needed = stride
.checked_mul(self.params.height.saturating_sub(1) as usize)
.and_then(|rows| {
rows.checked_add(self.params.width as usize * BYTES_PER_SOURCE_PIXEL)
})
.ok_or_else(|| "frame size overflows".to_owned())?;
if pixels.len() < needed {
return Err(format!(
"captured frame is truncated: {} bytes for {} rows of {} px at stride {stride}",
pixels.len(),
self.params.height,
self.params.width
));
}
// SAFETY: the slice was just bounds-checked against the geometry we hand
// to swscale, and every ffmpeg pointer below is owned by `self`.
unsafe {
self.ensure_sws(src_format)?;
// The staging frame is reused every frame; ffmpeg requires it be
// writable before we scribble into it, which matters once the
// encoder starts holding references to earlier frames.
let writable = ff::av_frame_make_writable(self.sw_frame);
if writable < 0 {
return Err(format!(
"av_frame_make_writable: {}",
ff::err_to_string(writable)
));
}
let src_data = [pixels.as_ptr(), ptr::null(), ptr::null(), ptr::null()];
let src_stride = [stride as i32, 0, 0, 0];
let convert_started = std::time::Instant::now();
let scaled = ff::sws_scale(
self.sws,
src_data.as_ptr(),
src_stride.as_ptr(),
0,
self.params.height,
(*self.sw_frame).data.as_mut_ptr(),
(*self.sw_frame).linesize.as_mut_ptr(),
);
self.stats.convert_ns += convert_started.elapsed().as_nanos();
if scaled < 0 {
return Err(format!("sws_scale: {}", ff::err_to_string(scaled)));
}
// This CPU frame is now what `encode_staged` must send, so drop any hw
// surface a previous dmabuf frame left staged — e.g. after the stream
// renegotiates to a modifier-less format and frames start arriving on
// the sws path. `encode_staged` prefers `hw_staged` whenever it is
// non-null, so without this it would keep re-sending that stale surface
// and freeze the video. `av_frame_free` nulls the pointer.
if !self.hw_staged.is_null() {
ff::av_frame_free(&mut self.hw_staged);
}
self.staged = true;
}
Ok(())
}
/// Stages a ready VAAPI NV12 surface produced by the dmabuf importer. Takes
/// ownership of `frame`; the previous one is released. Unlike [`Self::stage`]
/// there is no conversion or upload — the surface is encoded as-is and held
/// across the clock-driven re-encodes until the next frame replaces it.
///
/// SAFETY: `frame` must be a valid VAAPI NV12 `AVFrame` from the shared pool
/// the encoder was opened against (see `open_importing`).
pub unsafe fn stage_hw(&mut self, frame: *mut ff::AVFrame) {
if !self.hw_staged.is_null() {
let mut old = self.hw_staged;
ff::av_frame_free(&mut old);
}
self.hw_staged = frame;
self.staged = true;
}
/// True once a picture has been staged — via [`Self::stage`] (shm/software)
/// or [`Self::stage_hw`] (dmabuf). Before that there is nothing to encode.
pub fn has_staged_frame(&self) -> bool {
self.staged
}
/// Encodes whatever is currently staged at `pts` (a frame index).
///
/// Called once per staged frame and then once more per frame of a static
/// screen, which is what holds the constant frame rate.
pub fn encode_staged(
&mut self,
pts: i64,
mut on_packet: impl FnMut(*mut ff::AVPacket) -> Result<(), String>,
) -> Result<(), String> {
if !self.staged {
return Err("encode_staged called before any frame was staged".to_owned());
}
// SAFETY: `staged` is only set after a successful sws_scale into
// `sw_frame`, and every pointer below is owned by `self`.
unsafe {
let upload_started = std::time::Instant::now();
let mut used_upload = false;
let frame = if !self.hw_staged.is_null() {
// Imported dmabuf surface: already NV12 on the GPU. No upload, no
// conversion — just timestamp it. Held for the next re-encode.
(*self.hw_staged).pts = pts;
self.hw_staged
} else if self.hw_frames.is_null() {
(*self.sw_frame).pts = pts;
self.sw_frame
} else {
let got = ff::av_hwframe_get_buffer(self.hw_frames, self.hw_frame, 0);
if got < 0 {
return Err(format!(
"av_hwframe_get_buffer: {}",
ff::err_to_string(got)
));
}
let transferred = ff::av_hwframe_transfer_data(self.hw_frame, self.sw_frame, 0);
if transferred < 0 {
return Err(format!(
"av_hwframe_transfer_data: {}",
ff::err_to_string(transferred)
));
}
(*self.hw_frame).pts = pts;
used_upload = true;
self.hw_frame
};
self.stats.upload_ns += upload_started.elapsed().as_nanos();
let encode_started = std::time::Instant::now();
self.send_and_drain(frame, &mut on_packet)?;
self.stats.encode_ns += encode_started.elapsed().as_nanos();
self.stats.frames += 1;
if used_upload {
// Release our reference to the per-encode upload surface; the
// encoder keeps its own for as long as it needs one. Without this
// the pool drains after `initial_pool_size` frames and every
// subsequent av_hwframe_get_buffer blocks. The imported
// `hw_staged` surface is NOT released here — it is held for the
// next clock-driven re-encode and freed in `stage_hw`/`Drop`.
ff::av_frame_unref(self.hw_frame);
}
}
Ok(())
}
/// [`Self::stage`] then [`Self::encode_staged`], for callers with nothing to
/// gain from the split.
pub fn submit(
&mut self,
pixels: &[u8],
stride: usize,
src_format: ff::AVPixelFormat,
pts: i64,
on_packet: impl FnMut(*mut ff::AVPacket) -> Result<(), String>,
) -> Result<(), String> {
self.stage(pixels, stride, src_format)?;
self.encode_staged(pts, on_packet)
}
/// Flushes the encoder. Call once, at stop.
pub fn finish(
&mut self,
mut on_packet: impl FnMut(*mut ff::AVPacket) -> Result<(), String>,
) -> Result<(), String> {
// SAFETY: a null frame is how ffmpeg is told to drain.
unsafe { self.send_and_drain(ptr::null_mut(), &mut on_packet) }
}
/// SAFETY: `frame` is either null (drain) or a frame this encoder owns.
unsafe fn send_and_drain(
&mut self,
frame: *mut ff::AVFrame,
on_packet: &mut impl FnMut(*mut ff::AVPacket) -> Result<(), String>,
) -> Result<(), String> {
let sent = ff::avcodec_send_frame(self.codec_ctx, frame);
if sent < 0 && sent != ff::AVERROR_EOF {
return Err(format!("avcodec_send_frame: {}", ff::err_to_string(sent)));
}
loop {
let received = ff::avcodec_receive_packet(self.codec_ctx, self.packet);
if received == ff::AVERROR_EAGAIN || received == ff::AVERROR_EOF {
return Ok(());
}
if received < 0 {
return Err(format!(
"avcodec_receive_packet: {}",
ff::err_to_string(received)
));
}
let result = on_packet(self.packet);
ff::av_packet_unref(self.packet);
result?;
}
}
/// SAFETY: called with `self` valid; rebuilds `self.sws` when the source
/// format changes.
unsafe fn ensure_sws(&mut self, src_format: ff::AVPixelFormat) -> Result<(), String> {
if !self.sws.is_null() && self.sws_src_format == src_format {
return Ok(());
}
if !self.sws.is_null() {
ff::sws_freeContext(self.sws);
self.sws = ptr::null_mut();
}
// Built through the option API rather than `sws_getContext` so that
// `threads` can be set at all — `sws_getContext` has no way to express
// it. It is set to 1 by default; see [`sws_thread_count`] for the
// measurement behind that.
//
// Source and destination are the same size, so no resampling filter is
// involved and the cheapest kernel is the right one: this is a pure
// colour-space conversion, and a bilinear kernel would cost more for
// pixels it never moves.
self.sws = ff::sws_alloc_context();
if self.sws.is_null() {
return Err("sws_alloc_context returned null".to_owned());
}
let dst_format = self.backend.upload_format();
let options: [(&CStr, i64); 8] = [
(c"srcw", self.params.width as i64),
(c"srch", self.params.height as i64),
(c"src_format", src_format as i64),
(c"dstw", self.params.width as i64),
(c"dsth", self.params.height as i64),
(c"dst_format", dst_format as i64),
(c"sws_flags", ff::SWS_POINT as i64),
(c"threads", sws_thread_count() as i64),
];
for (name, value) in options {
let set = ff::av_opt_set_int(self.sws.cast(), name.as_ptr(), value, 0);
if set < 0 {
return Err(format!(
"av_opt_set_int({}, {value}): {}",
name.to_string_lossy(),
ff::err_to_string(set)
));
}
}
let initialised = ff::sws_init_context(self.sws, ptr::null_mut(), ptr::null_mut());
if initialised < 0 {
return Err(format!(
"sws_init_context could not convert pixel format {src_format} to {dst_format}: {}",
ff::err_to_string(initialised)
));
}
// BT.709 studio range, which is what the compositor decodes every
// recording as. swscale's default matrix is BT.601, so left alone every
// Linux take played back with shifted hue and saturation
// (getopenscreen/openscreen#926). Full range in: the source is RGB.
let bt709 = ff::sws_getCoefficients(ff::SWS_CS_ITU709 as i32);
let unity = 1 << 16;
if ff::sws_setColorspaceDetails(self.sws, bt709, 1, bt709, 0, 0, unity, unity) < 0 {
return Err(format!(
"swscale refused a BT.709 conversion from pixel format {src_format} to {dst_format}"
));
}
// Read back rather than assume: libswscale is free to ignore a thread
// count it cannot use for a given conversion, and a request that was
// silently dropped looks exactly like threading that did not help.
let mut granted = 0i64;
if ff::av_opt_get_int(self.sws.cast(), c"threads".as_ptr(), 0, &mut granted) >= 0 {
self.sws_threads = granted;
}
self.sws_src_format = src_format;
Ok(())
}
pub fn stats(&self) -> EncodeStats {
self.stats
}
/// Threads libswscale actually granted, once a frame has been submitted.
pub fn sws_threads(&self) -> i64 {
self.sws_threads
}
}
impl Drop for VideoEncoder {
fn drop(&mut self) {
// SAFETY: every pointer is either null or owned by this struct, and each
// free function tolerates a pointer-to-null.
unsafe {
if !self.sws.is_null() {
ff::sws_freeContext(self.sws);
}
if !self.packet.is_null() {
ff::av_packet_free(&mut self.packet);
}
if !self.hw_frame.is_null() {
ff::av_frame_free(&mut self.hw_frame);
}
if !self.hw_staged.is_null() {
ff::av_frame_free(&mut self.hw_staged);
}
if !self.sw_frame.is_null() {
ff::av_frame_free(&mut self.sw_frame);
}
if !self.codec_ctx.is_null() {
ff::avcodec_free_context(&mut self.codec_ctx);
}
if !self.hw_frames.is_null() {
ff::av_buffer_unref(&mut self.hw_frames);
}
if !self.hw_device.is_null() {
ff::av_buffer_unref(&mut self.hw_device);
}
}
}
}
/// Sample rate every audio track is captured and encoded at. PipeWire's stream
/// adapter converts whatever the device runs at, so this is a free choice, and
/// 48 kHz is what the rest of the pipeline (and MP4) expects.
pub const AUDIO_SAMPLE_RATE: i32 = 48_000;
pub const AUDIO_CHANNELS: usize = 2;
/// One AAC track, fed interleaved stereo f32.
///
/// AAC encodes in fixed blocks (1024 samples per channel), but PipeWire delivers
/// whatever quantum the graph is running — 512, 1024, 2048, and it changes at
/// runtime. So samples accumulate here until a full block exists rather than the
/// caller having to buffer.
pub struct AudioEncoder {
codec_ctx: *mut ff::AVCodecContext,
frame: *mut ff::AVFrame,
packet: *mut ff::AVPacket,
/// Interleaved samples not yet forming a whole AAC block.
pending: Vec<f32>,
/// Samples per channel in one block.
block: usize,
/// Presentation time of the next block, counted in samples — which is
/// exactly the codec's time base, so no rounding accumulates.
next_pts: i64,
}
impl AudioEncoder {
pub fn open(bitrate: i64) -> Result<Self, String> {
// SAFETY: an ffmpeg setup sequence; every allocation is stored in
// `encoder` as soon as it succeeds so Drop can free it.
unsafe {
let codec = ff::avcodec_find_encoder_by_name(c"aac".as_ptr());
if codec.is_null() {
return Err("the vendored ffmpeg has no AAC encoder".to_owned());
}
let codec_ctx = ff::avcodec_alloc_context3(codec);
if codec_ctx.is_null() {
return Err("avcodec_alloc_context3 returned null".to_owned());
}
let mut encoder = Self {
codec_ctx,
frame: ptr::null_mut(),
packet: ptr::null_mut(),
pending: Vec::new(),
block: 0,
next_pts: 0,
};
(*codec_ctx).sample_rate = AUDIO_SAMPLE_RATE;
// The native AAC encoder takes PLANAR float; interleaved is rejected
// outright by avcodec_open2, which is why `push` deinterleaves.
(*codec_ctx).sample_fmt = ff::AV_SAMPLE_FMT_FLTP;
(*codec_ctx).bit_rate = bitrate;
(*codec_ctx).time_base = ff::AVRational { num: 1, den: AUDIO_SAMPLE_RATE };
ff::av_channel_layout_default(&mut (*codec_ctx).ch_layout, AUDIO_CHANNELS as i32);
(*codec_ctx).flags |= ff::AV_CODEC_FLAG_GLOBAL_HEADER as i32;
let opened = ff::avcodec_open2(codec_ctx, codec, ptr::null_mut());
if opened < 0 {
return Err(format!("aac avcodec_open2: {}", ff::err_to_string(opened)));
}
// Only valid after open: the encoder decides its own block size.
encoder.block = match (*codec_ctx).frame_size {
size if size > 0 => size as usize,
_ => 1024,
};
encoder.packet = ff::av_packet_alloc();
encoder.frame = ff::av_frame_alloc();
if encoder.packet.is_null() || encoder.frame.is_null() {
return Err("av_packet_alloc/av_frame_alloc returned null".to_owned());
}
(*encoder.frame).format = ff::AV_SAMPLE_FMT_FLTP as i32;
(*encoder.frame).nb_samples = encoder.block as i32;
(*encoder.frame).sample_rate = AUDIO_SAMPLE_RATE;
ff::av_channel_layout_default(
&mut (*encoder.frame).ch_layout,
AUDIO_CHANNELS as i32,
);
let allocated = ff::av_frame_get_buffer(encoder.frame, 0);
if allocated < 0 {
return Err(format!(
"audio av_frame_get_buffer: {}",
ff::err_to_string(allocated)
));
}
Ok(encoder)
}
}
pub fn codec_context(&self) -> *mut ff::AVCodecContext {
self.codec_ctx
}
/// Feeds interleaved stereo samples, encoding every whole block they complete.
pub fn push(
&mut self,
interleaved: &[f32],
mut on_packet: impl FnMut(*mut ff::AVPacket) -> Result<(), String>,
) -> Result<(), String> {
self.pending.extend_from_slice(interleaved);
let per_block = self.block * AUDIO_CHANNELS;
while self.pending.len() >= per_block {
// SAFETY: the frame was allocated for exactly `block` samples per
// channel, and `pending` has been checked to hold a whole block.
unsafe { self.encode_block(per_block, &mut on_packet)? };
}
Ok(())
}
/// SAFETY: `self.pending` must hold at least `per_block` samples.
unsafe fn encode_block(
&mut self,
per_block: usize,
on_packet: &mut impl FnMut(*mut ff::AVPacket) -> Result<(), String>,
) -> Result<(), String> {
let writable = ff::av_frame_make_writable(self.frame);
if writable < 0 {
return Err(format!(
"audio av_frame_make_writable: {}",
ff::err_to_string(writable)
));
}
// Interleaved LRLRLR… in, one contiguous plane per channel out.
for channel in 0..AUDIO_CHANNELS {
let plane = (*self.frame).data[channel] as *mut f32;
for sample in 0..self.block {
*plane.add(sample) = self.pending[sample * AUDIO_CHANNELS + channel];
}
}
(*self.frame).pts = self.next_pts;
self.next_pts += self.block as i64;
self.pending.drain(..per_block);
self.send_and_drain(self.frame, on_packet)
}
/// Pads any partial block with silence, encodes it, and flushes.
pub fn finish(
&mut self,
mut on_packet: impl FnMut(*mut ff::AVPacket) -> Result<(), String>,
) -> Result<(), String> {
// SAFETY: padding only grows `pending` to exactly one block, which is
// what encode_block requires; a null frame is how ffmpeg is drained.
unsafe {
if !self.pending.is_empty() {
let per_block = self.block * AUDIO_CHANNELS;
self.pending.resize(per_block, 0.0);
self.encode_block(per_block, &mut on_packet)?;
}
self.send_and_drain(ptr::null_mut(), &mut on_packet)
}
}
/// SAFETY: `frame` is either null (drain) or the frame this encoder owns.
unsafe fn send_and_drain(
&mut self,
frame: *mut ff::AVFrame,
on_packet: &mut impl FnMut(*mut ff::AVPacket) -> Result<(), String>,
) -> Result<(), String> {
let sent = ff::avcodec_send_frame(self.codec_ctx, frame);
if sent < 0 && sent != ff::AVERROR_EOF {
return Err(format!("aac avcodec_send_frame: {}", ff::err_to_string(sent)));
}
loop {
let received = ff::avcodec_receive_packet(self.codec_ctx, self.packet);
if received == ff::AVERROR_EAGAIN || received == ff::AVERROR_EOF {
return Ok(());
}
if received < 0 {
return Err(format!(
"aac avcodec_receive_packet: {}",
ff::err_to_string(received)
));
}
let result = on_packet(self.packet);
ff::av_packet_unref(self.packet);
result?;
}
}
}
impl Drop for AudioEncoder {
fn drop(&mut self) {
// SAFETY: every pointer is either null or owned here.
unsafe {
if !self.packet.is_null() {
ff::av_packet_free(&mut self.packet);
}
if !self.frame.is_null() {
ff::av_frame_free(&mut self.frame);
}
if !self.codec_ctx.is_null() {
ff::avcodec_free_context(&mut self.codec_ctx);
}
}
}
}
/// Identifies a track added with [`Muxer::add_stream`].
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct TrackId(usize);
struct MuxTrack {
/// libavformat's own stream index, which need not equal our `TrackId`.
index: i32,
/// The time base packets arrive in.
source_time_base: ff::AVRational,
/// The time base libavformat CHOSE, which is what they must be rescaled to.
stream_time_base: ff::AVRational,
}
/// The mov muxer options that make the output fragmented.
///
/// A fragment is cut at a video keyframe once it holds at least one second, the
/// same floor the Windows sink uses (`kFragmentDurationHns` in
/// wgc-capture/src/mf_encoder.cpp). The GOP is half a second (see
/// `VideoEncoder::open_backend`), so a fragment is two GOPs and a kill loses at
/// most about a second. `frag_keyframe` alone would cut every half second;
/// `frag_duration` alone would cut mid-GOP, leaving fragments that open on a
/// frame nothing can decode without the previous one. `min_frag_duration` is
/// the floor that waits for the keyframe. `default_base_moof` makes each moof's
/// data offsets relative to that moof rather than to the file, so a fragment
/// parses on its own.
///
/// `flush_packets` hands every closed fragment to the kernel at once. Without
/// it the moov and the fragments wait in avio's 32 KB buffer, and a kill loses
/// that too: measured, a low-bitrate take killed after three seconds left a
/// 28-byte file. It costs one write per fragment, since the muxer holds each
/// fragment's samples until it closes.
///
/// `delay_moov` is for the audio track. AAC's first packet sits 1024 samples
/// before zero (encoder priming), and an empty moov written with the header
/// cannot know that, so libavformat drops the edit list and shifts every track
/// 21 ms later instead: measured, the first video frame then lasts 55 ms and
/// every later one lands 21 ms after its capture time, against a cursor track
/// that does not move. Delaying the moov to the first fragment lets it carry
/// the same edit list the plain MP4 had; timestamps match it packet for packet.
/// The price is that a take killed in its first second keeps nothing, as every
/// killed take did before.
const FRAGMENT_OPTIONS: [(&CStr, &CStr); 3] = [
(c"movflags", c"+frag_keyframe+empty_moov+default_base_moof+delay_moov"),
(c"min_frag_duration", c"1000000"),
(c"flush_packets", c"1"),
];
/// An MP4 writer, taking however many streams [`Self::add_stream`] is called for.
///
/// WHAT A RECORDING ACTUALLY USES IS TWO: one H.264 video stream and one AAC
/// track carrying system audio and the microphone already mixed together
/// (`AudioMix` in crate::capture). This wrote them as two separate audio tracks
/// once, matching what the macOS helper did, on the grounds that the export path
/// mixes every track it finds (crates/compositor/src/audio.rs). The preview does
/// not: it is an HTML5 `<video>`, which plays the default audio track and cannot
/// be told to switch, because Chromium implements no `audioTracks` API — so a
/// take with silent system audio played silence while the microphone sat in a
/// track nothing would ever select. Mixing before the muxer is what fixed it,
/// and it also removes the question of reconciling two capture clocks in the
/// container: there is only one audio timeline now, built in `AudioMix::pump`.
///
/// The file is a FRAGMENTED MP4, like the Windows and macOS helpers write
/// (#338): a moov with no samples in front, then self-describing moof+mdat pairs. A plain
/// MP4 has no index until the trailer, so a killed helper, a crash or a power
/// loss used to leave gigabytes of frames and no way to read them (#944). Now it
/// plays up to the last complete fragment. See [`FRAGMENT_OPTIONS`].
///
/// `+faststart` is not used: it rewrites the whole file on close, which on a
/// long recording means copying gigabytes, and a fragmented file has its moov
/// in front anyway.
pub struct Muxer {
fmt: *mut ff::AVFormatContext,
tracks: Vec<MuxTrack>,
header_written: bool,
}
impl Muxer {
/// Allocates the output and opens the file. No stream exists yet: add them
/// all with [`Self::add_stream`], then call [`Self::write_header`].
pub fn create(path: &Path) -> Result<Self, String> {
let path_c = CString::new(path.as_os_str().as_encoded_bytes())
.map_err(|_| "output path contains a NUL byte".to_owned())?;
// SAFETY: standard libavformat output setup; `fmt` is stored before any
// fallible step so Drop can clean up.
unsafe {
let mut fmt: *mut ff::AVFormatContext = ptr::null_mut();
let allocated = ff::avformat_alloc_output_context2(
&mut fmt,
ptr::null_mut(),
c"mp4".as_ptr(),
path_c.as_ptr(),
);
if allocated < 0 || fmt.is_null() {
return Err(format!(
"avformat_alloc_output_context2: {}",
ff::err_to_string(allocated)
));
}
let muxer = Self { fmt, tracks: Vec::new(), header_written: false };
let pb = ff::osc_avformat_pb(fmt);
if pb.is_null() {
return Err("FFmpeg did not expose the output IO context".to_owned());
}
let opened = ff::avio_open(pb, path_c.as_ptr(), ff::AVIO_FLAG_WRITE as i32);
if opened < 0 {
return Err(format!(
"avio_open({}): {}",
path.display(),
ff::err_to_string(opened)
));
}
Ok(muxer)
}
}
/// Adds a track fed by `codec_ctx`. Every track must be added before
/// [`Self::write_header`]: MP4 fixes its track list in the header.
pub fn add_stream(&mut self, codec_ctx: *mut ff::AVCodecContext) -> Result<TrackId, String> {
if self.header_written {
return Err("a stream cannot be added after the header is written".to_owned());
}
// SAFETY: `codec_ctx` is an opened context owned by the caller, and
// `fmt` is ours.
unsafe {
let stream = ff::avformat_new_stream(self.fmt, ptr::null());
if stream.is_null() {
return Err("avformat_new_stream returned null".to_owned());
}
let source_time_base = (*codec_ctx).time_base;
(*stream).time_base = source_time_base;
let copied = ff::avcodec_parameters_from_context((*stream).codecpar, codec_ctx);
if copied < 0 {
return Err(format!(
"avcodec_parameters_from_context: {}",
ff::err_to_string(copied)
));
}
self.tracks.push(MuxTrack {
index: (*stream).index,
source_time_base,
// Filled in by write_header, which is free to change it.
stream_time_base: source_time_base,
});
Ok(TrackId(self.tracks.len() - 1))
}
}
pub fn write_header(&mut self) -> Result<(), String> {
if self.tracks.is_empty() {
return Err("the output has no streams".to_owned());
}
// SAFETY: `fmt` is ours and every stream was added through add_stream;
// the dictionary is ours and freed before leaving the block.
unsafe {
let mut options: *mut ff::AVDictionary = ptr::null_mut();
for (key, value) in FRAGMENT_OPTIONS {
ff::av_dict_set(&mut options, key.as_ptr(), value.as_ptr(), 0);
}
let header = ff::avformat_write_header(self.fmt, &mut options);
// The muxer takes out every option it knows; one left over is a typo
// that would silently bring back the unfragmented file.
let unknown = ff::av_dict_count(options);
ff::av_dict_free(&mut options);
if header >= 0 && unknown > 0 {
return Err(format!("the mp4 muxer ignored {unknown} fragment option(s)"));
}
if header < 0 {
return Err(format!(
"avformat_write_header: {}",
ff::err_to_string(header)
));
}
self.header_written = true;
// libavformat may have rewritten a stream's time base while writing
// the header (MP4 prefers 1/90000-style bases for video and the
// sample rate for audio). Everything after this must rescale into
// the value it CHOSE, not the value we asked for — using ours
// produces a file whose duration is wrong by the ratio between them.
for track in &mut self.tracks {
let stream = ff::osc_avformat_stream(self.fmt, track.index as u32);
if stream.is_null() {
return Err(format!("FFmpeg output stream {} disappeared", track.index));
}
track.stream_time_base = (*stream).time_base;
}
}
Ok(())
}
/// Writes one encoded packet, rescaling it out of its encoder's time base.
pub fn write(&mut self, track: TrackId, packet: *mut ff::AVPacket) -> Result<(), String> {
let Some(track) = self.tracks.get(track.0) else {
return Err("packet written to a track that was never added".to_owned());
};
// SAFETY: `packet` comes from the encoder this track was added for, and
// is valid until the caller unrefs it.
unsafe {
(*packet).stream_index = track.index;
ff::av_packet_rescale_ts(packet, track.source_time_base, track.stream_time_base);
let written = ff::av_interleaved_write_frame(self.fmt, packet);
if written < 0 {
return Err(format!(
"av_interleaved_write_frame: {}",
ff::err_to_string(written)
));
}
}
Ok(())
}
/// Writes the trailer and closes the file. Consuming `self` makes it a
/// compile error to write another packet afterwards.
pub fn finish(mut self) -> Result<(), String> {
// SAFETY: `fmt` is valid and the header was written, which is what
// av_write_trailer requires.
unsafe {
let written = ff::av_write_trailer(self.fmt);
self.header_written = false;
if written < 0 {
return Err(format!("av_write_trailer: {}", ff::err_to_string(written)));
}
}
Ok(())
}
}
impl Drop for Muxer {
fn drop(&mut self) {
// SAFETY: `fmt` is either null or ours. `header_written` is still true
// only on the error path — `finish` clears it — and a file left without
// its trailer would lose its last fragment, so write one on the way out.
unsafe {
if self.fmt.is_null() {
return;
}
if self.header_written {
ff::av_write_trailer(self.fmt);
}
let pb = ff::osc_avformat_pb(self.fmt);
if !pb.is_null() && !(*pb).is_null() {
ff::avio_closep(pb);
}
ff::avformat_free_context(self.fmt);
self.fmt = ptr::null_mut();
}
}
}
/// Threads to give libswscale. One, by measurement.
///
/// BGRA→NV12 at 1920×1080 was timed at 3.80 ms/frame with one thread and
/// 3.84 ms with eight, on an 8-thread Radeon 610M APU. The request is not being
/// ignored — [`VideoEncoder::sws_threads`] reads the granted count back out of
/// libswscale and it matches — the conversion is simply memory-bandwidth bound,
/// and extra threads contend for the same bandwidth instead of adding any.
/// Spawning eight threads per recording to gain 1% of nothing is not worth the
/// scheduler churn on a machine that is simultaneously compositing the screen
/// being captured.
///
/// The override exists because that measurement is one machine at one
/// resolution; a 4K capture on a many-core desktop with more memory channels may
/// well behave differently, and this makes checking it a one-liner.
fn sws_thread_count() -> usize {
if let Ok(raw) = std::env::var("OPENSCREEN_LINUX_SWS_THREADS") {
if let Ok(count) = raw.trim().parse::<usize>() {
return count.clamp(1, 32);
}
}
1
}
/// Parses `OPENSCREEN_LINUX_ENCODER`. Returns `None` when unset, which means
/// "walk the ladder".
pub fn forced_backend_from_env() -> Result<Option<Backend>, String> {
let Ok(raw) = std::env::var("OPENSCREEN_LINUX_ENCODER") else {
return Ok(None);
};
match raw.trim().to_ascii_lowercase().as_str() {
"" | "auto" => Ok(None),
"vaapi" => Ok(Some(Backend::Vaapi)),
"vulkan" => Ok(Some(Backend::Vulkan)),
"software" | "openh264" | "libopenh264" => Ok(Some(Backend::Software)),
other => Err(format!(
"OPENSCREEN_LINUX_ENCODER={other} is not one of auto, vaapi, vulkan, software"
)),
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn every_ladder_entry_names_a_codec_this_build_has() {
// Catches a vendored-ffmpeg swap that drops an encoder we depend on:
// without this the loss shows up only as a silent fall through to
// software on a user's machine.
for backend in LADDER {
// SAFETY: a lookup by name against a static string.
let codec = unsafe { ff::avcodec_find_encoder_by_name(backend.codec_name().as_ptr()) };
assert!(
!codec.is_null(),
"{} is missing from the vendored ffmpeg",
backend.codec_name().to_string_lossy()
);
}
}
#[test]
fn software_always_opens() {
// The bottom of the ladder is the one rung that must never fail, on any
// machine, with no GPU at all.
let encoder = VideoEncoder::open(
VideoParams { width: 320, height: 240, fps: 30, bitrate: 1_000_000 },
Some(Backend::Software),
|_, _| {},
)
.expect("libopenh264 must open");
assert_eq!(encoder.backend(), Backend::Software);
}
#[test]
fn software_encodes_a_frame_into_packets() {
let mut encoder = VideoEncoder::open(
VideoParams { width: 320, height: 240, fps: 30, bitrate: 1_000_000 },
Some(Backend::Software),
|_, _| {},
)
.expect("open");
let stride = 320 * 4;
let pixels = vec![0x40u8; stride * 240];
let mut packets = 0;
for pts in 0..10 {
encoder
.submit(&pixels, stride, ff::AV_PIX_FMT_BGRA, pts, |_| {
packets += 1;
Ok(())
})
.expect("submit");
}
encoder
.finish(|_| {
packets += 1;
Ok(())
})
.expect("finish");
assert!(packets > 0, "encoder produced no packets at all");
}
#[test]
fn only_vaapi_caps_the_qp_and_the_cap_reaches_the_codec() {
// The first VAAPI keyframe of a window take came out QP 51 across its
// lower half; `max_qp` is the fix (see its comment). The other rungs
// were not measured, so they must keep the codec's own default.
assert_eq!(Backend::Vaapi.max_qp(), Some(30));
assert_eq!(Backend::Vulkan.max_qp(), None);
assert_eq!(Backend::Software.max_qp(), None);
let params = || VideoParams { width: 320, height: 240, fps: 60, bitrate: 2_000_000 };
let software = VideoEncoder::open(params(), Some(Backend::Software), |_, _| {})
.expect("libopenh264 must open");
// SAFETY: the context is owned by `software`, alive for this read.
let software_qmax = unsafe { (*software.codec_context()).qmax };
assert_eq!(software_qmax, -1, "libopenh264 must keep its default ceiling");
// Opportunistic: only a machine with a usable VA-API driver can open it,
// which CI cannot. Where it opens, the cap must have survived avcodec_open2.
match VideoEncoder::open(params(), Some(Backend::Vaapi), |_, _| {}) {
Ok(vaapi) => {
// SAFETY: as above.
let vaapi_qmax = unsafe { (*vaapi.codec_context()).qmax };
assert_eq!(vaapi_qmax, 30, "the VAAPI rate control must be capped");
}
Err(error) => eprintln!("VAAPI part skipped: {error}"),
}
}
#[test]
fn a_truncated_frame_is_refused_rather_than_read_past() {
let mut encoder = VideoEncoder::open(
VideoParams { width: 320, height: 240, fps: 30, bitrate: 1_000_000 },
Some(Backend::Software),
|_, _| {},
)
.expect("open");
let error = encoder
.submit(&[0u8; 16], 320 * 4, ff::AV_PIX_FMT_BGRA, 0, |_| Ok(()))
.expect_err("must refuse");
assert!(error.contains("truncated"), "{error}");
}
#[test]
fn forced_backend_parsing_rejects_typos_instead_of_falling_back() {
std::env::set_var("OPENSCREEN_LINUX_ENCODER", "vulcan");
let error = forced_backend_from_env().expect_err("must reject");
assert!(error.contains("vulcan"), "{error}");
std::env::set_var("OPENSCREEN_LINUX_ENCODER", "vaapi");
assert_eq!(forced_backend_from_env().unwrap(), Some(Backend::Vaapi));
std::env::remove_var("OPENSCREEN_LINUX_ENCODER");
assert_eq!(forced_backend_from_env().unwrap(), None);
}
/// The whole pipeline — ladder, encode, mux — against a real file, with no
/// portal and no compositor involved.
///
/// Opt-in via `OPENSCREEN_LINUX_ENCODE_TEST=1` because it depends on the
/// GPU: on this hardware it selects Vulkan, on a headless CI box it falls
/// through to software, and both are correct. Run it after any change to the
/// ladder, and read the printed backend — that line is the measurement.
#[test]
fn ladder_encodes_a_playable_mp4() {
if std::env::var("OPENSCREEN_LINUX_ENCODE_TEST").is_err() {
eprintln!("skipped: set OPENSCREEN_LINUX_ENCODE_TEST=1 to run");
return;
}
let (width, height, fps) = (1920, 1080, 60);
let output = std::env::temp_dir().join("openscreen-encoder-smoke.mp4");
let _ = std::fs::remove_file(&output);
let mut encoder = VideoEncoder::open(
VideoParams { width, height, fps, bitrate: 8_000_000 },
forced_backend_from_env().expect("valid encoder override"),
|backend, error| eprintln!(" {} unavailable: {error}", backend.as_str()),
)
.expect("some backend must open");
eprintln!("selected backend: {}", encoder.backend().as_str());
let mut muxer = Muxer::create(&output).expect("muxer");
let track = muxer.add_stream(encoder.codec_context()).expect("video track");
muxer.write_header().expect("header");
let stride = width as usize * 4;
let mut frame = vec![0u8; stride * height as usize];
let started = std::time::Instant::now();
const FRAMES: i64 = 120;
for pts in 0..FRAMES {
// Vary the content so the encoder cannot collapse the whole clip
// into one keyframe plus 119 skip frames, which would make the
// timing meaningless.
for (index, byte) in frame.iter_mut().enumerate() {
*byte = ((index as i64 + pts * 7919) % 251) as u8;
}
encoder
.submit(&frame, stride, ff::AV_PIX_FMT_BGRA, pts, |packet| muxer.write(track, packet))
.expect("submit");
}
encoder.finish(|packet| muxer.write(track, packet)).expect("finish");
let elapsed = started.elapsed();
muxer.finish().expect("trailer");
let stats = encoder.stats();
eprintln!(
"{FRAMES} frames of {width}x{height} in {:.2}s = {:.0} fps\n \
convert {:.2} ms/frame ({} swscale threads) | upload {:.2} ms | encode {:.2} ms",
elapsed.as_secs_f64(),
FRAMES as f64 / elapsed.as_secs_f64(),
stats.convert_ms(),
encoder.sws_threads(),
stats.upload_ms(),
stats.encode_ms(),
);
let size = std::fs::metadata(&output).expect("output exists").len();
assert!(size > 1024, "the muxed file is {size} bytes, which cannot be 120 frames");
}
#[test]
fn software_path_converts_and_tags_bt709() {
// The compositor decodes every recording as BT.709 studio range; swscale's
// default is BT.601, which puts pure red at Y 81, Cb 90 (#926).
let (width, height) = (64, 32);
let mut encoder = VideoEncoder::open(
VideoParams { width, height, fps: 30, bitrate: 2_000_000 },
Some(Backend::Software),
|_, _| {},
)
.expect("the software encoder always opens");
let stride = width as usize * 4;
let mut frame = vec![0u8; stride * height as usize];
for pixel in frame.chunks_exact_mut(4) {
pixel.copy_from_slice(&[0, 0, 255, 255]); // BGRA pure red
}
encoder.stage(&frame, stride, ff::AV_PIX_FMT_BGRA).expect("stage");
// SAFETY: `stage` just filled `sw_frame`, a YUV420P frame of this size.
let (y, cb, cr, ctx) = unsafe {
let staged = encoder.sw_frame;
(*(*staged).data[0], *(*staged).data[1], *(*staged).data[2], &*encoder.codec_ctx)
};
let close = |got: u8, want: i32| (i32::from(got) - want).abs() <= 1;
assert!(close(y, 63) && close(cb, 102) && close(cr, 240), "red came out Y {y} Cb {cb} Cr {cr}");
assert_eq!(ctx.color_range, ff::AVCOL_RANGE_MPEG);
assert_eq!(ctx.colorspace, ff::AVCOL_SPC_BT709);
assert_eq!(ctx.color_primaries, ff::AVCOL_PRI_BT709);
assert_eq!(ctx.color_trc, ff::AVCOL_TRC_BT709);
}
#[test]
fn a_killed_muxer_leaves_a_file_that_opens_and_holds_its_frames() {
// Three seconds at 30 fps with audio, then a simulated kill: no trailer,
// no flush of the write buffer, no interleave queue drained (#944).
let (width, height, fps) = (64, 32, 30);
// Per process, so two test runs on one machine never share the file.
let output = std::env::temp_dir()
.join(format!("openscreen-killed-muxer-{}.mp4", std::process::id()));
let _ = std::fs::remove_file(&output);
let mut video = VideoEncoder::open(
VideoParams { width, height, fps, bitrate: 1_000_000 },
Some(Backend::Software),
|_, _| {},
)
.expect("the software encoder always opens");
let mut audio = AudioEncoder::open(128_000).expect("aac");
let mut muxer = Muxer::create(&output).expect("muxer");
let video_track = muxer.add_stream(video.codec_context()).expect("video track");
let audio_track = muxer.add_stream(audio.codec_context()).expect("audio track");
muxer.write_header().expect("header");
let stride = width as usize * 4;
let mut frame = vec![0u8; stride * height as usize];
let samples = vec![0.0f32; (AUDIO_SAMPLE_RATE / fps) as usize * AUDIO_CHANNELS];
for pts in 0..3 * i64::from(fps) {
frame.fill(pts as u8);
video
.submit(&frame, stride, ff::AV_PIX_FMT_BGRA, pts, |p| muxer.write(video_track, p))
.expect("video");
audio.push(&samples, |p| muxer.write(audio_track, p)).expect("audio");
}
std::mem::forget(muxer);
// SAFETY: ffmpeg's own demuxer on the file just written; the context is
// closed before the block ends.
let (streams, video_frames, second_frame_at) = unsafe {
let path = CString::new(output.as_os_str().as_encoded_bytes()).unwrap();
let mut input: *mut ff::AVFormatContext = ptr::null_mut();
let opened =
ff::avformat_open_input(&mut input, path.as_ptr(), ptr::null(), ptr::null_mut());
assert!(opened >= 0, "a killed take must open: {}", ff::err_to_string(opened));
let probed = ff::avformat_find_stream_info(input, ptr::null_mut());
assert!(probed >= 0, "stream info: {}", ff::err_to_string(probed));
let packet = ff::av_packet_alloc();
let mut video_frames = 0;
let mut second_frame_at = 0.0;
while ff::av_read_frame(input, packet) >= 0 {
let stream = ff::osc_avformat_stream(input, (*packet).stream_index as u32);
assert!(!stream.is_null(), "packet stream index must be valid");
if (*(*stream).codecpar).codec_type == ff::AVMEDIA_TYPE_VIDEO {
if video_frames == 1 {
let base = (*stream).time_base;
second_frame_at = (*packet).pts as f64 * f64::from(base.num)
/ f64::from(base.den);
}
video_frames += 1;
}
ff::av_packet_unref(packet);
}
let mut packet = packet;
ff::av_packet_free(&mut packet);
let streams = ff::osc_avformat_nb_streams(input);
ff::avformat_close_input(&mut input);
(streams, video_frames, second_frame_at)
};
assert_eq!(streams, 2, "both tracks are declared up front");
// Fragments close at the keyframes at 1 s and 2 s; only the one in
// flight, the last second, is lost.
assert!(video_frames >= 2 * fps, "only {video_frames} frames survived the kill");
// Without an edit list for AAC's priming, every frame after the first
// slides 21 ms off its capture time (see FRAGMENT_OPTIONS).
let one_frame = 1.0 / f64::from(fps);
assert!((second_frame_at - one_frame).abs() < 1e-3, "frame 1 at {second_frame_at} s");
let _ = std::fs::remove_file(&output);
}
#[test]
fn radv_flags_are_added_without_clobbering_an_existing_value() {
std::env::set_var("RADV_PERFTEST", "gpl");
std::env::set_var("RADV_EXPERIMENTAL", "other");
prepare_environment();
let perftest = std::env::var("RADV_PERFTEST").unwrap();
assert!(perftest.contains("gpl"), "existing flags must survive: {perftest}");
assert!(perftest.contains("video_encode"), "{perftest}");
let experimental = std::env::var("RADV_EXPERIMENTAL").unwrap();
assert!(
experimental.contains("other"),
"existing flags must survive: {experimental}"
);
assert!(experimental.contains("video_encode"), "{experimental}");
// Idempotent: a second call must not append a duplicate.
prepare_environment();
assert_eq!(std::env::var("RADV_PERFTEST").unwrap(), perftest);
assert_eq!(std::env::var("RADV_EXPERIMENTAL").unwrap(), experimental);
std::env::remove_var("RADV_PERFTEST");
std::env::remove_var("RADV_EXPERIMENTAL");
}
}