The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.
Rust ffmpegfree-video-editormodel-context-protocolnon-linear-editoroffline-firstopen-source-video-editorself-hostedvideo-editingvideo-editing-automationvideo-editing-softwarevideo-editing-toolsvideo-editorvoice-cloningyoutube-automationyoutube-shorts
feat(hdr): HDR clips are decoded deep and converted on the GPU - iPhone HLG previews five times faster
Jareerd854df3d5255
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.mdindex 4f1a6a76..a63a5961 100644--- a/ARCHITECTURE.md+++ b/ARCHITECTURE.md@@ -107,7 +107,14 @@ flowchart LRWindows calls scRGB and Apple extended linear sRGB. A frame is convertedinto it on the GPU as it is uploaded and out of it once, at the resolve(`concat-render/src/gpu.rs`, `COPY_SHADER`); blending, opacity and fades- happen in light, as in Resolve and Final Cut. Effect packages written for+ happen in light, as in Resolve and Final Cut. A Rec. 2020 source - an+ iPhone's HLG, HDR10's PQ - is decoded deep (`DecodeOptions::deep`: sixteen+ bits a channel, its own signal, no CPU tone map) wherever its frame goes+ straight to the GPU, and converted as it uploads (`DEEP_SHADER`): the+ transfer undone, the primaries brought to Rec. 709's, and, while every+ timeline is SDR, conformed to it by BT.2390's roll-off. A clip with an+ FFmpeg chain or a cutout keeps the eight-bit tone map in the decoder.+ Effect packages written forgamma-encoded `0..1` see that through the prelude's `legacy_in` and`legacy_out` (`concat-effects/src/shader.rs`) until the GPU-only effectsreplace them.diff --git a/src/crates/concat-cli/src/main.rs b/src/crates/concat-cli/src/main.rsindex f4a90cf6..f4457e3d 100644--- a/src/crates/concat-cli/src/main.rs+++ b/src/crates/concat-cli/src/main.rs@@ -404,6 +404,8 @@ fn render(input: &PathBuf, output: &PathBuf, frames: u64, fade: u64) -> Result<(None,None,None,+ // Straight to the GPU compositor: deep for an HDR source.+ true,)?;layer.source = Some(frame);layer.opacity *= fade_in(index, fade);diff --git a/src/crates/concat-core/src/frame.rs b/src/crates/concat-core/src/frame.rsindex 77083f3a..37fcb136 100644--- a/src/crates/concat-core/src/frame.rs+++ b/src/crates/concat-core/src/frame.rs@@ -3,21 +3,55 @@//! Decoded image buffers.//!-//! One pixel format, everywhere: 8-bit RGBA, straight (non-premultiplied)-//! alpha, sRGB, rows packed tightly with no padding. Every decoder converts to-//! it and every encoder converts from it.+//! One pixel format almost everywhere: 8-bit RGBA, straight (non-premultiplied)+//! alpha, gamma-encoded Rec. 709, rows packed tightly with no padding. Every+//! encoder converts from it, and every consumer but the compositor's upload+//! reads it.//!-//! That is a deliberate simplification, not an oversight. A real grade needs-//! higher precision and a linear working space, and when that day comes it-//! arrives as a second frame type next to this one - not as a `format` field-//! that every call site has to branch on.+//! The exception is a deep frame ([`Depth::Sixteen`]): sixteen bits a+//! channel, still RGBA, in the source's own signal ([`Signal`]) - an HDR clip+//! as it was shot, not tone-mapped to eight bits on the CPU. The decoder+//! makes one only when a caller asks (`DecodeOptions::deep`) and the source+//! is wide, and the only caller that asks is the one whose frames go+//! straight to the GPU, which brings them into its working space as it+//! uploads them. Nothing else ever sees one, so no call site branches on+//! the depth: the note that once said a deeper frame would arrive as a+//! type of its own gave way to that, since a second type would have run+//! through the frame pool, the plan and every cache for one reader.use std::fmt;use std::sync::atomic::{AtomicU64, Ordering};-/// Bytes per pixel in the one format Concat uses.+/// Bytes per pixel of an eight-bit frame, the format nearly everything uses.pub const BYTES_PER_PIXEL: usize = 4;+/// How many bits a channel a frame's pixels hold.+#[derive(Clone, Copy, PartialEq, Eq, Debug, Default)]+pub enum Depth {+ /// Eight: `u8` channels, four bytes a pixel.+ #[default]+ Eight,+ /// Sixteen: little-endian `u16` channels, eight bytes a pixel.+ Sixteen,+}++/// What a frame's RGB values mean: the transfer they are encoded with and+/// the primaries they are on. An eight-bit frame is always [`Signal::Sdr`];+/// a deep one carries its source's.+#[derive(Clone, Copy, PartialEq, Eq, Debug, Default)]+pub enum Signal {+ /// Gamma-encoded (BT.1886, 2.4) on Rec. 709 primaries.+ #[default]+ Sdr,+ /// Gamma-encoded (BT.1886, 2.4) on Rec. 2020 primaries: a wide-gamut+ /// SDR source.+ SdrWide,+ /// ARIB STD-B67 hybrid log-gamma on Rec. 2020: an iPhone's HDR.+ Hlg,+ /// SMPTE ST 2084 perceptual quantiser on Rec. 2020: HDR10.+ Pq,+}+/// The next frame identity; see [`Frame::id`].static NEXT_ID: AtomicU64 = AtomicU64::new(1);@@ -36,6 +70,10 @@ pub struct Frame {pixels: Vec<u8>,/// Which picture this is; see [`Frame::id`].id: u64,+ /// Eight bits a channel, or sixteen for a deep frame.+ depth: Depth,+ /// What the values mean.+ signal: Signal,}impl Clone for Frame {@@ -47,6 +85,8 @@ impl Clone for Frame {height: self.height,pixels: self.pixels.clone(),id: mint(),+ depth: self.depth,+ signal: self.signal,}}}@@ -54,7 +94,11 @@ impl Clone for Frame {impl PartialEq for Frame {/// The same picture, whatever its identity.fn eq(&self, other: &Self) -> bool {- self.width == other.width && self.height == other.height && self.pixels == other.pixels+ self.width == other.width+ && self.height == other.height+ && self.depth == other.depth+ && self.signal == other.signal+ && self.pixels == other.pixels}}@@ -85,6 +129,8 @@ impl Frame {height,pixels,id: mint(),+ depth: Depth::Eight,+ signal: Signal::Sdr,}}@@ -95,6 +141,8 @@ impl Frame {height,pixels: vec![0u8; Self::byte_len(width, height)],id: mint(),+ depth: Depth::Eight,+ signal: Signal::Sdr,}}@@ -108,9 +156,42 @@ impl Frame {height,pixels,id: mint(),+ depth: Depth::Eight,+ signal: Signal::Sdr,})}+ /// Wraps a deep buffer: little-endian `u16` RGBA, exactly `width *+ /// height * 8` bytes, in `signal`. `None` if the size is wrong.+ pub fn from_rgba64(width: u32, height: u32, pixels: Vec<u8>, signal: Signal) -> Option<Self> {+ (pixels.len() == Self::byte_len(width, height) * 2).then_some(Self {+ width,+ height,+ pixels,+ id: mint(),+ depth: Depth::Sixteen,+ signal,+ })+ }++ /// Eight bits a channel, or sixteen.+ pub const fn depth(&self) -> Depth {+ self.depth+ }++ /// What the values mean: see [`Signal`].+ pub const fn signal(&self) -> Signal {+ self.signal+ }++ /// Bytes a pixel of this frame takes.+ pub const fn bytes_per_pixel(&self) -> usize {+ match self.depth {+ Depth::Eight => BYTES_PER_PIXEL,+ Depth::Sixteen => BYTES_PER_PIXEL * 2,+ }+ }+/// Width in pixels.pub const fn width(&self) -> u32 {self.width@@ -152,15 +233,27 @@ impl Frame {Some((y as usize * self.width as usize + x as usize) * BYTES_PER_PIXEL)}- /// Reads one pixel as `[r, g, b, a]`.+ /// Reads one pixel as `[r, g, b, a]`: a deep frame's to eight bits, its+ /// channels' high bytes, for a test or a log to look at.pub fn pixel(&self, x: u32, y: u32) -> Option<[u8; 4]> {let offset = self.offset_of(x, y)?;- Some([- self.pixels[offset],- self.pixels[offset + 1],- self.pixels[offset + 2],- self.pixels[offset + 3],- ])+ Some(match self.depth {+ Depth::Eight => [+ self.pixels[offset],+ self.pixels[offset + 1],+ self.pixels[offset + 2],+ self.pixels[offset + 3],+ ],+ Depth::Sixteen => {+ let at = offset * 2;+ [+ self.pixels[at + 1],+ self.pixels[at + 3],+ self.pixels[at + 5],+ self.pixels[at + 7],+ ]+ }+ })}/// Writes one pixel. Out-of-bounds writes are ignored.@@ -215,6 +308,8 @@ impl fmt::Debug for Frame {.field("width", &self.width).field("height", &self.height).field("bytes", &self.pixels.len())+ .field("depth", &self.depth)+ .field("signal", &self.signal).finish()}}diff --git a/src/crates/concat-export/src/lib.rs b/src/crates/concat-export/src/lib.rsindex 74c7d7b5..d5cff700 100644--- a/src/crates/concat-export/src/lib.rs+++ b/src/crates/concat-export/src/lib.rs@@ -1174,6 +1174,9 @@ fn render_picture(chain,pre,ranges.get(&layer.clip).copied(),+ // Deep where the frame goes to the GPU as it is: a+ // cutout cuts eight bits.+ !cutouts.contains_key(&layer.clip),) {let frame = cutouts.get(&layer.clip)@@ -1212,11 +1215,15 @@ fn render_picture(FrameRate::new(rate.fps() / layer.speed)};+ // Deep - an HDR clip in its own signal, converted on+ // the GPU - unless a cutout has to cut it in eight bits;+ // the decoder keeps eight bits for a chain itself.let mut options = DecodeOptions::default().starting_at(layer.source_time).scaled_to(decode_width, decode_height).at_rate(decode_rate)- .in_range(ranges.get(&layer.clip).copied());+ .in_range(ranges.get(&layer.clip).copied())+ .deep(!cutouts.contains_key(&layer.clip));// Effects and transition fades, as one FFmpeg chain. The// decoder guards the frame size after it, so an effect@@ -1664,8 +1671,10 @@ fn frame_request(.from_proxy(proxy).in_range(plan.built.ranges.get(&layer.clip).copied())// The compositor fits the picture into its place, so an untreated- // frame is drawn at the level it was decoded at.+ // frame is drawn at the level it was decoded at - deep, for an HDR+ // clip, unless a cutout has to cut it in eight bits..at_any_size(true)+ .deep_when_untreated(!plan.built.cutouts.contains_key(&layer.clip))}/// [`preview_sources`] for one instant of a plan already built. Withdiff --git a/src/crates/concat-host/tests/export.rs b/src/crates/concat-host/tests/export.rsindex d0ad6388..aa4d105f 100644--- a/src/crates/concat-host/tests/export.rs+++ b/src/crates/concat-host/tests/export.rs@@ -916,6 +916,63 @@ fn fades_and_wipes_reach_the_picture() {}}+/// An iPhone's HDR - ten-bit HEVC tagged HLG - exports through the new road:+/// decoded deep in its own signal, converted and conformed to the SDR+/// timeline on the GPU as it uploads. The picture is the grey it was+/// written as, conformed: neither black nor blown out, and not tinted.+#[test]+fn an_hlg_clip_exports_through_the_gpu_conversion() {+ if !VideoCodec::Hevc.available() {+ eprintln!("no HEVC encoder in the linked FFmpeg; skipped");+ return;+ }+ let scratch = Scratch::new("hlg");+ let path = scratch.path().join("hlg.mp4");+ {+ let options = EncodeOptions {+ codec: VideoCodec::Hevc,+ preset: "ultrafast".to_owned(),+ crf: 12,+ rate_mode: RateMode::Vbr,+ bitrate_kbps: 0,+ ten_bit: true,+ color_range: concat_media::ColorRange::Limited,+ hardware: false,+ threads: 0,+ };+ let mut encoder = Encoder::create_mislabelled_hdr(+ &path,+ WIDTH,+ HEIGHT,+ FrameRate::THIRTY,+ &options,+ false,+ )+ .expect("an HEVC encoder");+ let mut frame = Frame::black(WIDTH, HEIGHT);+ frame.fill([160, 160, 160, 255]);+ for _ in 0..30 {+ encoder.write_frame(&frame).expect("writes");+ }+ encoder.finish().expect("finishes");+ }+ let mut studio = Studio::new(scratch.path(), "HLG", video(WIDTH, HEIGHT, 30, 1));+ let media = studio.import(&path);+ studio.apply(Command::AddClipAtFirstFree {+ media_id: media,+ start: 0.0,+ });+ let exported = studio.export("an HLG clip");+ let [r, g, b] = exported.colour_at(0.5);+ assert!(+ [r, g, b].iter().all(|channel| (40..=240).contains(channel))+ && r.abs_diff(g) <= 6+ && g.abs_diff(b) <= 6,+ "the HLG grey exported as {:?}",+ [r, g, b]+ );+}+/// Issue #103: the levels a file is read as reach the export. A source/// written video range reads as it is by default; told it is full range,/// its levels expand and the colour of a second is no longer the colourdiff --git a/src/crates/concat-media/src/decode.rs b/src/crates/concat-media/src/decode.rsindex eda0593c..3936417c 100644--- a/src/crates/concat-media/src/decode.rs+++ b/src/crates/concat-media/src/decode.rs@@ -26,7 +26,7 @@use std::path::{Path, PathBuf};-use concat_core::frame::Frame;+use concat_core::frame::{Frame, Signal};use concat_core::time::{FrameRate, Rational};use ffmpeg_the_third as ffmpeg;use ffmpeg_the_third::codec::decoder;@@ -132,6 +132,13 @@ pub struct DecodeOptions {/// is none, takes video range the way a player does. See/// [`ColorRange`].pub color_range: Option<ColorRange>,+ /// Keep a wide or HDR source's own signal, at sixteen bits a channel,+ /// rather than tone-map it to eight-bit Rec. 709 on the CPU: for a+ /// caller whose frames go straight to the GPU, which converts them as it+ /// uploads. Ignored for a narrow source, and where a chain or a+ /// pre-chain runs, since those are FFmpeg filters on eight bits. See+ /// [`Frame::signal`].+ pub deep: bool,}impl DecodeOptions {@@ -204,6 +211,12 @@ impl DecodeOptions {self}+ /// Keeps a wide or HDR source deep; see [`DecodeOptions::deep`].+ pub fn deep(mut self, deep: bool) -> Self {+ self.deep = deep;+ self+ }+/// Reads the picture as `range`, whatever the file says; `None` goes/// back to the file's tag. See [`DecodeOptions::color_range`].pub fn in_range(mut self, range: Option<ColorRange>) -> Self {@@ -431,6 +444,47 @@ impl ColorSignal {:out_color_matrix=bt709:out_range=tv:intent=perceptual")}++ /// What a deep frame of this source says it holds, when the source is+ /// one the compositor converts on the GPU: Rec. 2020 primaries, with+ /// PQ, HLG or a gamma. `None` for anything else - a P3 source, say -+ /// which keeps the eight-bit conversion here.+ pub fn deep_signal(&self) -> Option<Signal> {+ use ffmpeg::color::{Primaries, TransferCharacteristic as Transfer};+ match self.transfer {+ Transfer::SMPTE2084 => Some(Signal::Pq),+ Transfer::ARIB_STD_B67 => Some(Signal::Hlg),+ _ if self.primaries == Primaries::BT2020 => Some(Signal::SdrWide),+ _ => None,+ }+ }++ /// The `scale` filter's arguments for a deep frame: the source's own+ /// matrix and range, to RGB and nothing more - its primaries and+ /// transfer are left as they are, for the GPU to convert.+ fn native_args(self) -> String {+ use ffmpeg::color::{Range, Space};+ let matrix = match self.matrix {+ Space::BT2020CL => "bt2020c",+ Space::BT709 => "bt709",+ _ => "bt2020nc",+ };+ let range = match self.range {+ Range::JPEG => "pc",+ _ => "tv",+ };+ format!(":in_color_matrix={matrix}:in_range={range}")+ }+}++/// The signal a deep frame is decoded in, when `options` asks for one and+/// the source is one the compositor converts: nothing in the decoder's+/// graph may need eight bits, so no chain and no pre-chain.+fn deep_output(options: &DecodeOptions, color: Option<&ColorSignal>) -> Option<Signal> {+ if !options.deep || options.filter_chain.is_some() || options.pre_chain.is_some() {+ return None;+ }+ color?.deep_signal()}/// The filtergraph between the decoder and the caller, as one string.@@ -462,7 +516,9 @@ fn video_filter(// for a file that lies about it. A range read off the codec in place// of a tag is named only when it is full: the frames carry no tag for// swscale to read, and untagged already means limited to it.+ let deep = deep_output(options, color);let convert = match (color, options.color_range) {+ (Some(signal), _) if deep.is_some() => signal.native_args(),(Some(signal), _) if signal.is_wide() => signal.bt709_args(),(_, Some(range)) => format!(":in_range={}", range.scale_name()),(Some(signal), None) if signal.implied && signal.range == ffmpeg::color::Range::JPEG => {@@ -475,7 +531,14 @@ fn video_filter(parts.push(chain.clone());parts.push(format!("scale={width}:{height}:flags=bilinear"));}- parts.push("format=rgba".to_owned());+ parts.push(+ if deep.is_some() {+ "format=rgba64le"+ } else {+ "format=rgba"+ }+ .to_owned(),+ );parts.join(",")}@@ -884,18 +947,24 @@ impl Decoder {.map_err(|error| ffi::fail("filter output", &self.path, error))?;}- // The sink's picture is RGBA already; only its row padding differs- // from the packed buffer a Frame is.+ // The sink's picture is RGBA already - sixteen bits a channel for a+ // deep frame - and only its row padding differs from the packed+ // buffer a Frame is.+ let deep = deep_output(&self.options, Some(&self.color));let width = filtered.width();let height = filtered.height();- let row = width as usize * 4;+ let row = width as usize * if deep.is_some() { 8 } else { 4 };let stride = filtered.stride(0);let data = filtered.data(0);let mut pixels = Vec::with_capacity(row * height as usize);for y in 0..height as usize {pixels.extend_from_slice(&data[y * stride..y * stride + row]);}- Frame::from_rgba(width, height, pixels).ok_or_else(|| Error::Probe {+ let frame = match deep {+ Some(signal) => Frame::from_rgba64(width, height, pixels, signal),+ None => Frame::from_rgba(width, height, pixels),+ };+ frame.ok_or_else(|| Error::Probe {path: self.path.clone(),detail: "the filtergraph produced a frame of the wrong size".to_owned(),})@@ -1353,9 +1422,36 @@ mod tests {let mut decoder = Decoder::open(&path, &DecodeOptions::default()).expect("opens");assert!(decoder.color().is_hdr(), "{:?}", decoder.color());let decoded = decoder.next_frame().expect("decodes").expect("a frame");- let _ = std::fs::remove_file(&path);let luma = decoded.pixels()[0];assert!(luma > 8 && luma < 250, "tone-mapped grey came out {luma}");++ // Asked deep, the same file comes out in its own signal at sixteen+ // bits a channel, its grey where it was written - about 180 of 255+ // of full scale - rather than tone-mapped.+ let mut deep = Decoder::open(&path, &DecodeOptions::default().deep(true)).expect("opens");+ let frame = deep.next_frame().expect("decodes").expect("a frame");+ assert_eq!(frame.depth(), concat_core::frame::Depth::Sixteen);+ assert_eq!(frame.signal(), concat_core::frame::Signal::Pq);+ assert_eq!(frame.pixels().len(), 64 * 64 * 8);+ let channel = |at: usize| u16::from_le_bytes([frame.pixels()[at], frame.pixels()[at + 1]]);+ let (r, g, b) = (channel(0), channel(2), channel(4));+ let written = 180.0 / 255.0 * 65535.0;+ for value in [r, g, b] {+ assert!(+ (f64::from(value) - written).abs() < 65535.0 * 0.03,+ "the deep grey came out {r}, {g}, {b}"+ );+ }++ // A chain runs on eight bits, so it keeps the tone-mapped frame.+ let mut chained = Decoder::open(+ &path,+ &DecodeOptions::default().deep(true).filtered("negate"),+ )+ .expect("opens");+ let frame = chained.next_frame().expect("decodes").expect("a frame");+ assert_eq!(frame.depth(), concat_core::frame::Depth::Eight);+ let _ = std::fs::remove_file(&path);}#[test]diff --git a/src/crates/concat-media/src/pool.rs b/src/crates/concat-media/src/pool.rsindex 9081a2b4..4d3881d4 100644--- a/src/crates/concat-media/src/pool.rs+++ b/src/crates/concat-media/src/pool.rs@@ -89,6 +89,11 @@ pub struct FrameRequest {/// and a second cache entry per size; the level is never smaller than/// `cover`, so nothing is lost. See [`FrameRequest::at_any_size`].pub any_size: bool,+ /// Served deep where the source is wide and nothing treats it here:+ /// its own signal at sixteen bits a channel, for a caller whose frames+ /// go straight to the GPU. Only with `any_size`, since a fit here is+ /// FFmpeg on eight bits. See [`DecodeOptions::deep`].+ pub deep: bool,}impl FrameRequest {@@ -106,6 +111,7 @@ impl FrameRequest {proxy: false,range: None,any_size: false,+ deep: false,}}@@ -152,6 +158,12 @@ impl FrameRequest {self}+ /// Served deep where it can be; see [`FrameRequest::deep`].+ pub fn deep_when_untreated(mut self, deep: bool) -> Self {+ self.deep = deep;+ self+ }+/// Reads the proxy where there is one. See [`FrameRequest::proxy`].pub fn from_proxy(mut self, proxy: bool) -> Self {self.proxy = proxy;@@ -171,6 +183,10 @@ struct SourceKey {/// The range the file was read as: the same frame read as full and/// as video range is two different pictures.range: Option<ColorRange>,+ /// Read deep: a wide source in its own signal, sixteen bits a channel+ /// (see [`DecodeOptions::deep`]) - a different picture from the+ /// tone-mapped eight bits.+ deep: bool,}/// A treated frame's identity: the source frame and what was done to it.@@ -339,6 +355,7 @@ impl Reader {chain: Option<&str>,pre: Option<&str>,range: Option<ColorRange>,+ deep: bool,rate: FrameRate,still: bool,index: i64,@@ -349,7 +366,8 @@ impl Reader {let mut options = DecodeOptions::default().starting_at(rate.time_of_frame(index)).scaled_to(width, height)- .in_range(range);+ .in_range(range)+ .deep(deep);if still {// One frame, served for every time, and never sought: a seek// on a single-image JPEG leaves FFmpeg's image demuxer with@@ -432,6 +450,7 @@ type ReaderKey = (Option<String>,Option<String>,Option<ColorRange>,+ bool,);/// The warm readers and their recency, behind one short lock. A reader is@@ -696,6 +715,10 @@ impl ReaderPool {// A proxy is read as it is tagged: it was written from the// corrected picture, so the correction applied twice would undo it.let range = (path == request.path).then_some(request.range).flatten();+ // Deep only where the frame is served as it is: a fit or a chain+ // here is FFmpeg, which takes eight bits.+ let deep =+ request.deep && request.any_size && request.pre.is_none() && request.chain.is_none();Ok((facts,SourceKey {@@ -704,6 +727,7 @@ impl ReaderPool {height,index,range,+ deep,},))}@@ -767,6 +791,9 @@ impl ReaderPool {/// the same options makes. The monitor asks through/// [`ReaderPool::frame`] instead, whose cache the chain does not/// invalidate. Without a crop or a chain the two are the same request.+ /// `deep` reads a wide source in its own signal where there is no crop+ /// and no chain; see [`DecodeOptions::deep`].+ #[allow(clippy::too_many_arguments)]pub fn frame_at(&self,path: &Path,@@ -777,6 +804,7 @@ impl ReaderPool {chain: Option<&str>,pre: Option<&str>,range: Option<ColorRange>,+ deep: bool,) -> Result<Arc<Frame>> {let chain = chain.filter(|chain| !chain.trim().is_empty());let pre = pre.filter(|pre| !pre.trim().is_empty());@@ -788,6 +816,7 @@ impl ReaderPool {height,index,range,+ deep: deep && chain.is_none() && pre.is_none(),};if chain.is_none() && pre.is_none() {return self.source_frame(&facts, &source_key);@@ -817,7 +846,7 @@ impl ReaderPool {self.remember_treated(key, Arc::clone(&frame));return Ok(frame);}- let shared = self.reader(path, width, height, chain, pre, range, &facts, index)?;+ let shared = self.reader(path, width, height, chain, pre, range, false, &facts, index)?;let mut reader = shared.lock().map_err(|_| crate::error::Error::NoFrame {path: path.to_path_buf(),})?;@@ -853,7 +882,7 @@ impl ReaderPool {return Ok(frame);}let shared = self.reader(- path, key.width, key.height, None, None, key.range, facts, key.index,+ path, key.width, key.height, None, None, key.range, key.deep, facts, key.index,)?;let mut reader = shared.lock().map_err(|_| crate::error::Error::NoFrame {path: path.to_path_buf(),@@ -1016,6 +1045,7 @@ impl ReaderPool {chain: Option<&str>,pre: Option<&str>,range: Option<ColorRange>,+ deep: bool,facts: &MediaFacts,target: i64,) -> Result<Arc<Mutex<Reader>>> {@@ -1026,6 +1056,7 @@ impl ReaderPool {chain.map(str::to_owned),pre.map(str::to_owned),range,+ deep,);{let mut readers = self@@ -1051,6 +1082,7 @@ impl ReaderPool {chain,pre,range,+ deep,facts.rate,facts.still,target,@@ -1158,6 +1190,7 @@ pub(crate) mod tests {height: 2,index,range: None,+ deep: false,};cache.insert(key(0), Arc::new(Frame::black(2, 2)));cache.insert(key(1), Arc::new(Frame::black(2, 2)));@@ -1185,7 +1218,7 @@ pub(crate) mod tests {first.fill([255, 0, 0, 255]);pool.hold_still(path, Arc::new(first));let same = pool- .frame_at(path, Rational::ZERO, 64, 32, true, None, None, None)+ .frame_at(path, Rational::ZERO, 64, 32, true, None, None, None, false).expect("its own size");assert_eq!((same.width(), same.height()), (64, 32));assert_eq!(&same.pixels()[..4], &[255, 0, 0, 255]);@@ -1217,6 +1250,7 @@ pub(crate) mod tests {height: 2,index: 0,range: None,+ deep: false,},Arc::new(Frame::black(2, 2)),);@@ -1266,6 +1300,7 @@ pub(crate) mod tests {None,None,None,+ false,).expect("frame decodes");i64::from(frame.pixel(32, 32).expect("in bounds")[0])diff --git a/src/crates/concat-perf/src/main.rs b/src/crates/concat-perf/src/main.rsindex ec487f65..b0089fb9 100644--- a/src/crates/concat-perf/src/main.rs+++ b/src/crates/concat-perf/src/main.rs@@ -716,6 +716,13 @@ fn decode_4k() -> Vec<Measure> {&DecodeOptions::default().scaled_to(960, 540),hlg.frames,);+ // The same stream the way the monitor now reads it: deep, in its own+ // signal, the conversion left to the GPU's upload.+ let (deep_fps, deep_on) = decode_with(+ &hlg.path,+ &DecodeOptions::default().scaled_to(960, 540).deep(true),+ hlg.frames,+ );concat_media::set_hardware_decode(false);vec![Measure {@@ -750,11 +757,24 @@ fn decode_4k() -> Vec<Measure> {note: format!("the slowest stream; {}", used(three_on, &hevc)),},Measure {- name: "decode 4K HLG to the SDR monitor, 960x540",+ name: "decode 4K HLG tone-mapped on the CPU, 960x540",value: hlg_fps,unit: "fps",+ budget: Budget::AtLeast(15.0),+ note: format!(+ "the eight-bit fallback, for a clip with a chain or a cutout; {}",+ used(hlg_on, &hlg)+ ),+ },+ Measure {+ name: "decode 4K HLG deep for the GPU, 960x540",+ value: deep_fps,+ unit: "fps",budget: Budget::AtLeast(30.0),- note: format!("tone-mapped on the CPU; {}", used(hlg_on, &hlg)),+ note: format!(+ "converted on the GPU as it uploads; {}",+ used(deep_on, &hlg)+ ),},]}diff --git a/src/crates/concat-render/src/gpu.rs b/src/crates/concat-render/src/gpu.rsindex 65c9a721..85875178 100644--- a/src/crates/concat-render/src/gpu.rs+++ b/src/crates/concat-render/src/gpu.rs@@ -213,6 +213,116 @@ fn fs_resolve(@builtin(position) at: vec4<f32>) -> @location(0) vec4<f32> {}"#;+/// A deep frame (sixteen bits a channel, Rec. 2020, its source's own signal)+/// copied into the working space: the upload of an HDR or wide-gamut clip.+/// One entry a signal (`concat_core::frame::Signal`): the transfer undone+/// into light, 1.0 the white of an SDR picture (203 nits, ITU-R BT.2408),+/// and the primaries brought to Rec. 709's, where a colour outside them is+/// negative rather than clipped.+///+/// Every timeline is SDR for now, so an HDR clip is conformed to SDR as it+/// arrives, as Final Cut does a clip at a time: BT.2390's roll-off (ITU-R+/// BT.2390, the EETF) takes its highlights from a 1000-nit master down to+/// SDR white, on the brightest channel so a hue keeps its place. An HDR+/// timeline will skip it and keep the highlights.+const DEEP_SHADER: &str = r#"+@group(0) @binding(0) var deep: texture_2d<u32>;++@vertex+fn vs_full(@builtin(vertex_index) index: u32) -> @builtin(position) vec4<f32> {+ let corner = vec2<f32>(f32((index << 1u) & 2u), f32(index & 2u));+ return vec4<f32>(corner * 2.0 - 1.0, 0.0, 1.0);+}++fn deep_at(at: vec4<f32>) -> vec4<f32> {+ return vec4<f32>(textureLoad(deep, vec2<i32>(at.xy), 0)) / 65535.0;+}++// Rec. 2020 primaries to Rec. 709's, linear light (ITU-R BT.2087's matrix+// inverted, to full precision).+fn from_2020(rgb: vec3<f32>) -> vec3<f32> {+ return vec3<f32>(+ dot(vec3<f32>(1.660491002, -0.587641139, -0.072849863), rgb),+ dot(vec3<f32>(-0.124550475, 1.132899897, -0.008349423), rgb),+ dot(vec3<f32>(-0.018150763, -0.100578898, 1.118729661), rgb),+ );+}++// SMPTE ST 2084: a signal to nits, and back.+const PQ_M1: f32 = 0.1593017578125;+const PQ_M2: f32 = 78.84375;+const PQ_C1: f32 = 0.8359375;+const PQ_C2: f32 = 18.8515625;+const PQ_C3: f32 = 18.6875;++fn pq_nits(signal: vec3<f32>) -> vec3<f32> {+ let e = pow(clamp(signal, vec3<f32>(0.0), vec3<f32>(1.0)), vec3<f32>(1.0 / PQ_M2));+ let y = pow(max(e - PQ_C1, vec3<f32>(0.0)) / (PQ_C2 - PQ_C3 * e), vec3<f32>(1.0 / PQ_M1));+ return y * 10000.0;+}++fn pq_signal(nits: f32) -> f32 {+ let y = pow(clamp(nits / 10000.0, 0.0, 1.0), PQ_M1);+ return pow((PQ_C1 + PQ_C2 * y) / (1.0 + PQ_C3 * y), PQ_M2);+}++// BT.2390's EETF from a 1000-nit master to SDR white (203 nits), on the+// brightest channel, the colour scaled with it.+fn to_sdr(nits: vec3<f32>) -> vec3<f32> {+ let peak = max(nits.r, max(nits.g, nits.b));+ if (peak <= 0.0) {+ return nits;+ }+ let master = pq_signal(1000.0);+ let e1 = pq_signal(peak) / master;+ let top = pq_signal(203.0) / master;+ let knee = 1.5 * top - 0.5;+ var e2 = e1;+ if (e1 > knee) {+ let t = (min(e1, 1.0) - knee) / (1.0 - knee);+ let t2 = t * t;+ let t3 = t2 * t;+ e2 = (2.0 * t3 - 3.0 * t2 + 1.0) * knee+ + (t3 - 2.0 * t2 + t) * (1.0 - knee)+ + (-2.0 * t3 + 3.0 * t2) * top;+ }+ let mapped = pq_nits(vec3<f32>(e2 * master)).r;+ return nits * (mapped / peak);+}++@fragment+fn fs_upload_sdr_wide(@builtin(position) at: vec4<f32>) -> @location(0) vec4<f32> {+ let colour = deep_at(at);+ let lin = pow(max(colour.rgb, vec3<f32>(0.0)), vec3<f32>(2.4));+ return vec4<f32>(from_2020(lin), colour.a);+}++@fragment+fn fs_upload_pq(@builtin(position) at: vec4<f32>) -> @location(0) vec4<f32> {+ let colour = deep_at(at);+ let nits = to_sdr(pq_nits(colour.rgb));+ return vec4<f32>(from_2020(nits / 203.0), colour.a);+}++// ARIB STD-B67's inverse OETF to scene light, then BT.2100's OOTF for a+// 1000-nit display (a system gamma of 1.2), which puts HLG's reference+// white - 75 % of the signal - at 203 nits.+@fragment+fn fs_upload_hlg(@builtin(position) at: vec4<f32>) -> @location(0) vec4<f32> {+ let colour = deep_at(at);+ let a = 0.17883277;+ let b = 0.28466892;+ let c = 0.55991073;+ let e = clamp(colour.rgb, vec3<f32>(0.0), vec3<f32>(1.0));+ let low = e * e / 3.0;+ let high = (exp((e - c) / a) + b) / 12.0;+ let scene = select(high, low, e <= vec3<f32>(0.5));+ let ys = dot(scene, vec3<f32>(0.2627, 0.6780, 0.0593));+ let nits = to_sdr(1000.0 * pow(max(ys, 1e-6), 0.2) * scene);+ return vec4<f32>(from_2020(nits / 203.0), colour.a);+}+"#;+/// One package's shader, compiled once and kept: its pipeline, and the two/// uniform buffers every pass through it rewrites.struct CompiledShader {@@ -308,6 +418,14 @@ pub struct WgpuCompositor {/// The eight-bit textures frames are written into on their way to the/// pool, one a size, each with its bind group.staging: HashMap<(u32, u32), (wgpu::Texture, wgpu::BindGroup)>,+ /// A deep frame into the working space, by signal: Rec. 2020 gamma,+ /// HLG, PQ (see `DEEP_SHADER`).+ deep_pipelines: [wgpu::RenderPipeline; 3],+ /// The layout a deep staging texture is bound with: sixteen-bit+ /// integers, read texel by texel.+ deep_layout: wgpu::BindGroupLayout,+ /// The sixteen-bit staging textures, one a size.+ deep_staging: HashMap<(u32, u32), (wgpu::Texture, wgpu::BindGroup)>,bind_layout: wgpu::BindGroupLayout,/// Group 1 of a shader pass: the frame block and the package's params.uniform_layout: wgpu::BindGroupLayout,@@ -691,6 +809,56 @@ impl WgpuCompositor {let upload_pipeline = copy_into(WORK, "fs_upload");let resolve_pipeline = copy_into(wgpu::TextureFormat::Rgba8Unorm, "fs_resolve");+ let deep_layout = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {+ label: Some("concat deep"),+ entries: &[wgpu::BindGroupLayoutEntry {+ binding: 0,+ visibility: wgpu::ShaderStages::FRAGMENT,+ ty: wgpu::BindingType::Texture {+ sample_type: wgpu::TextureSampleType::Uint,+ view_dimension: wgpu::TextureViewDimension::D2,+ multisampled: false,+ },+ count: None,+ }],+ });+ let deep_shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {+ label: Some("concat deep"),+ source: wgpu::ShaderSource::Wgsl(DEEP_SHADER.into()),+ });+ let deep_pipeline_layout = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {+ label: Some("concat deep"),+ bind_group_layouts: &[Some(&deep_layout)],+ immediate_size: 0,+ });+ let deep_pipelines = ["fs_upload_sdr_wide", "fs_upload_hlg", "fs_upload_pq"].map(|entry| {+ device.create_render_pipeline(&wgpu::RenderPipelineDescriptor {+ label: Some(entry),+ layout: Some(&deep_pipeline_layout),+ vertex: wgpu::VertexState {+ module: &deep_shader,+ entry_point: Some("vs_full"),+ compilation_options: Default::default(),+ buffers: &[],+ },+ fragment: Some(wgpu::FragmentState {+ module: &deep_shader,+ entry_point: Some(entry),+ compilation_options: Default::default(),+ targets: &[Some(wgpu::ColorTargetState {+ format: WORK,+ blend: None,+ write_mask: wgpu::ColorWrites::ALL,+ })],+ }),+ primitive: wgpu::PrimitiveState::default(),+ depth_stencil: None,+ multisample: wgpu::MultisampleState::default(),+ multiview_mask: None,+ cache: None,+ })+ });+let sampler = device.create_sampler(&wgpu::SamplerDescriptor {label: Some("concat layer"),address_mode_u: wgpu::AddressMode::ClampToEdge,@@ -715,6 +883,9 @@ impl WgpuCompositor {upload_pipeline,resolve_pipeline,staging: HashMap::new(),+ deep_pipelines,+ deep_layout,+ deep_staging: HashMap::new(),bind_layout,uniform_layout,lut_layout,@@ -1281,6 +1452,7 @@ impl WgpuCompositor {self.used.remove(&key);self.idle.remove(&key);self.staging.remove(&key);+ self.deep_staging.remove(&key);}Self::trim(&mut self.luts,@@ -1404,7 +1576,12 @@ impl WgpuCompositor {// a clip's own colour will be brought into the working space. The// write lands before the submit that copies it, and the next// upload's write after it, so one staging texture a size serves.- let (staging, staging_group) = self.staging_for(frame.width(), frame.height());+ let deep = frame.depth() == concat_core::frame::Depth::Sixteen;+ let (staging, staging_group) = if deep {+ self.deep_staging_for(frame.width(), frame.height())+ } else {+ self.staging_for(frame.width(), frame.height())+ };self.queue.write_texture(wgpu::TexelCopyTextureInfo {texture: &staging,@@ -1415,7 +1592,7 @@ impl WgpuCompositor {frame.pixels(),wgpu::TexelCopyBufferLayout {offset: 0,- bytes_per_row: Some(frame.width() * 4),+ bytes_per_row: Some(frame.width() * frame.bytes_per_pixel() as u32),rows_per_image: Some(frame.height()),},wgpu::Extent3d {@@ -1432,7 +1609,17 @@ impl WgpuCompositor {.create_command_encoder(&wgpu::CommandEncoderDescriptor {label: Some("concat upload"),});- Self::copy(&mut encoder, &self.upload_pipeline, &staging_group, &view);+ let pipeline = if deep {+ use concat_core::frame::Signal;+ &self.deep_pipelines[match frame.signal() {+ Signal::Hlg => 1,+ Signal::Pq => 2,+ Signal::Sdr | Signal::SdrWide => 0,+ }]+ } else {+ &self.upload_pipeline+ };+ Self::copy(&mut encoder, pipeline, &staging_group, &view);self.queue.submit([encoder.finish()]);index}@@ -1465,6 +1652,40 @@ impl WgpuCompositor {);}+ /// This size's sixteen-bit staging texture and its bind group, made on+ /// first use.+ fn deep_staging_for(&mut self, width: u32, height: u32) -> (wgpu::Texture, wgpu::BindGroup) {+ if let Some((texture, group)) = self.deep_staging.get(&(width, height)) {+ return (texture.clone(), group.clone());+ }+ let texture = self.device.create_texture(&wgpu::TextureDescriptor {+ label: Some("concat deep staging"),+ size: wgpu::Extent3d {+ width,+ height,+ depth_or_array_layers: 1,+ },+ mip_level_count: 1,+ sample_count: 1,+ dimension: wgpu::TextureDimension::D2,+ format: wgpu::TextureFormat::Rgba16Uint,+ usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST,+ view_formats: &[],+ });+ let view = texture.create_view(&wgpu::TextureViewDescriptor::default());+ let group = self.device.create_bind_group(&wgpu::BindGroupDescriptor {+ label: Some("concat deep staging"),+ layout: &self.deep_layout,+ entries: &[wgpu::BindGroupEntry {+ binding: 0,+ resource: wgpu::BindingResource::TextureView(&view),+ }],+ });+ self.deep_staging+ .insert((width, height), (texture.clone(), group.clone()));+ (texture, group)+ }+/// This size's staging texture and its bind group, made on first use.fn staging_for(&mut self, width: u32, height: u32) -> (wgpu::Texture, wgpu::BindGroup) {if let Some((texture, group)) = self.staging.get(&(width, height)) {diff --git a/src/crates/concat-render/src/gpu/tests.rs b/src/crates/concat-render/src/gpu/tests.rsindex c90ef238..f0061620 100644--- a/src/crates/concat-render/src/gpu/tests.rs+++ b/src/crates/concat-render/src/gpu/tests.rs@@ -842,3 +842,59 @@ fn a_stack_keeps_its_precision_between_passes() {"down and up again gave {got:?}");}++/// A deep frame of `level` a channel, sixteen bits, in `signal`.+fn deep(width: u32, height: u32, level: f32, signal: concat_core::frame::Signal) -> Frame {+ let value = (level.clamp(0.0, 1.0) * 65535.0).round() as u16;+ let pixel: Vec<u8> = [value, value, value, u16::MAX]+ .iter()+ .flat_map(|channel| channel.to_le_bytes())+ .collect();+ Frame::from_rgba64(+ width,+ height,+ pixel.repeat((width * height) as usize),+ signal,+ )+ .expect("a deep frame")+}++/// An HDR clip is converted into the working space on the GPU as it+/// uploads, and conformed to the SDR timeline there: HLG's reference white+/// (75 % of the signal, 203 nits) lands a little under SDR white, where+/// BT.2390's roll-off leaves room for the highlights above it, grey still;+/// the master's peak - all of HLG's signal, or 1000 nits of PQ - is SDR+/// white; black is black.+#[test]+fn an_hdr_frame_is_conformed_to_sdr_on_the_way_in() {+ use concat_core::frame::Signal;+ let Some(mut gpu) = gpu() else { return };+ let mut drawn = |frame: Frame| {+ let out = gpu.render(&plan(4, 4, vec![layer(frame)]));+ let p = out.pixel(1, 1).expect("in the frame");+ [p[0], p[1], p[2]]+ };+ let white = drawn(deep(4, 4, 0.75, Signal::Hlg));+ assert!(+ white.iter().all(|channel| (205..=245).contains(channel))+ && white[0].abs_diff(white[1]) <= 2+ && white[1].abs_diff(white[2]) <= 2,+ "HLG reference white came out {white:?}"+ );+ let peak = drawn(deep(4, 4, 1.0, Signal::Hlg));+ assert!(+ peak.iter().all(|channel| *channel >= 252),+ "HLG peak came out {peak:?}"+ );+ let black = drawn(deep(4, 4, 0.0, Signal::Hlg));+ assert!(+ black.iter().all(|channel| *channel <= 2),+ "HLG black came out {black:?}"+ );+ // 1000 nits in PQ is 0.7518 of the signal.+ let pq_peak = drawn(deep(4, 4, 0.7518, Signal::Pq));+ assert!(+ pq_peak.iter().all(|channel| *channel >= 250),+ "PQ 1000 nits came out {pq_peak:?}"+ );+}