Skip to content

The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.

feat(hdr): HDR clips are decoded deep and converted on the GPU - iPhone HLG previews five times faster

Jareerd854df3d5255
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
index 4f1a6a76..a63a5961 100644
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -107,7 +107,14 @@ flowchart LR
Windows calls scRGB and Apple extended linear sRGB. A frame is converted
into it on the GPU as it is uploaded and out of it once, at the resolve
(`concat-render/src/gpu.rs`, `COPY_SHADER`); blending, opacity and fades
- happen in light, as in Resolve and Final Cut. Effect packages written for
+ happen in light, as in Resolve and Final Cut. A Rec. 2020 source - an
+ iPhone's HLG, HDR10's PQ - is decoded deep (`DecodeOptions::deep`: sixteen
+ bits a channel, its own signal, no CPU tone map) wherever its frame goes
+ straight to the GPU, and converted as it uploads (`DEEP_SHADER`): the
+ transfer undone, the primaries brought to Rec. 709's, and, while every
+ timeline is SDR, conformed to it by BT.2390's roll-off. A clip with an
+ FFmpeg chain or a cutout keeps the eight-bit tone map in the decoder.
+ Effect packages written for
gamma-encoded `0..1` see that through the prelude's `legacy_in` and
`legacy_out` (`concat-effects/src/shader.rs`) until the GPU-only effects
replace them.
diff --git a/src/crates/concat-cli/src/main.rs b/src/crates/concat-cli/src/main.rs
index f4a90cf6..f4457e3d 100644
--- a/src/crates/concat-cli/src/main.rs
+++ b/src/crates/concat-cli/src/main.rs
@@ -404,6 +404,8 @@ fn render(input: &PathBuf, output: &PathBuf, frames: u64, fade: u64) -> Result<(
None,
None,
None,
+ // Straight to the GPU compositor: deep for an HDR source.
+ true,
)?;
layer.source = Some(frame);
layer.opacity *= fade_in(index, fade);
diff --git a/src/crates/concat-core/src/frame.rs b/src/crates/concat-core/src/frame.rs
index 77083f3a..37fcb136 100644
--- a/src/crates/concat-core/src/frame.rs
+++ b/src/crates/concat-core/src/frame.rs
@@ -3,21 +3,55 @@
//! Decoded image buffers.
//!
-//! One pixel format, everywhere: 8-bit RGBA, straight (non-premultiplied)
-//! alpha, sRGB, rows packed tightly with no padding. Every decoder converts to
-//! it and every encoder converts from it.
+//! One pixel format almost everywhere: 8-bit RGBA, straight (non-premultiplied)
+//! alpha, gamma-encoded Rec. 709, rows packed tightly with no padding. Every
+//! encoder converts from it, and every consumer but the compositor's upload
+//! reads it.
//!
-//! That is a deliberate simplification, not an oversight. A real grade needs
-//! higher precision and a linear working space, and when that day comes it
-//! arrives as a second frame type next to this one - not as a `format` field
-//! that every call site has to branch on.
+//! The exception is a deep frame ([`Depth::Sixteen`]): sixteen bits a
+//! channel, still RGBA, in the source's own signal ([`Signal`]) - an HDR clip
+//! as it was shot, not tone-mapped to eight bits on the CPU. The decoder
+//! makes one only when a caller asks (`DecodeOptions::deep`) and the source
+//! is wide, and the only caller that asks is the one whose frames go
+//! straight to the GPU, which brings them into its working space as it
+//! uploads them. Nothing else ever sees one, so no call site branches on
+//! the depth: the note that once said a deeper frame would arrive as a
+//! type of its own gave way to that, since a second type would have run
+//! through the frame pool, the plan and every cache for one reader.
use std::fmt;
use std::sync::atomic::{AtomicU64, Ordering};
-/// Bytes per pixel in the one format Concat uses.
+/// Bytes per pixel of an eight-bit frame, the format nearly everything uses.
pub const BYTES_PER_PIXEL: usize = 4;
+/// How many bits a channel a frame's pixels hold.
+#[derive(Clone, Copy, PartialEq, Eq, Debug, Default)]
+pub enum Depth {
+ /// Eight: `u8` channels, four bytes a pixel.
+ #[default]
+ Eight,
+ /// Sixteen: little-endian `u16` channels, eight bytes a pixel.
+ Sixteen,
+}
+
+/// What a frame's RGB values mean: the transfer they are encoded with and
+/// the primaries they are on. An eight-bit frame is always [`Signal::Sdr`];
+/// a deep one carries its source's.
+#[derive(Clone, Copy, PartialEq, Eq, Debug, Default)]
+pub enum Signal {
+ /// Gamma-encoded (BT.1886, 2.4) on Rec. 709 primaries.
+ #[default]
+ Sdr,
+ /// Gamma-encoded (BT.1886, 2.4) on Rec. 2020 primaries: a wide-gamut
+ /// SDR source.
+ SdrWide,
+ /// ARIB STD-B67 hybrid log-gamma on Rec. 2020: an iPhone's HDR.
+ Hlg,
+ /// SMPTE ST 2084 perceptual quantiser on Rec. 2020: HDR10.
+ Pq,
+}
+
/// The next frame identity; see [`Frame::id`].
static NEXT_ID: AtomicU64 = AtomicU64::new(1);
@@ -36,6 +70,10 @@ pub struct Frame {
pixels: Vec<u8>,
/// Which picture this is; see [`Frame::id`].
id: u64,
+ /// Eight bits a channel, or sixteen for a deep frame.
+ depth: Depth,
+ /// What the values mean.
+ signal: Signal,
}
impl Clone for Frame {
@@ -47,6 +85,8 @@ impl Clone for Frame {
height: self.height,
pixels: self.pixels.clone(),
id: mint(),
+ depth: self.depth,
+ signal: self.signal,
}
}
}
@@ -54,7 +94,11 @@ impl Clone for Frame {
impl PartialEq for Frame {
/// The same picture, whatever its identity.
fn eq(&self, other: &Self) -> bool {
- self.width == other.width && self.height == other.height && self.pixels == other.pixels
+ self.width == other.width
+ && self.height == other.height
+ && self.depth == other.depth
+ && self.signal == other.signal
+ && self.pixels == other.pixels
}
}
@@ -85,6 +129,8 @@ impl Frame {
height,
pixels,
id: mint(),
+ depth: Depth::Eight,
+ signal: Signal::Sdr,
}
}
@@ -95,6 +141,8 @@ impl Frame {
height,
pixels: vec![0u8; Self::byte_len(width, height)],
id: mint(),
+ depth: Depth::Eight,
+ signal: Signal::Sdr,
}
}
@@ -108,9 +156,42 @@ impl Frame {
height,
pixels,
id: mint(),
+ depth: Depth::Eight,
+ signal: Signal::Sdr,
})
}
+ /// Wraps a deep buffer: little-endian `u16` RGBA, exactly `width *
+ /// height * 8` bytes, in `signal`. `None` if the size is wrong.
+ pub fn from_rgba64(width: u32, height: u32, pixels: Vec<u8>, signal: Signal) -> Option<Self> {
+ (pixels.len() == Self::byte_len(width, height) * 2).then_some(Self {
+ width,
+ height,
+ pixels,
+ id: mint(),
+ depth: Depth::Sixteen,
+ signal,
+ })
+ }
+
+ /// Eight bits a channel, or sixteen.
+ pub const fn depth(&self) -> Depth {
+ self.depth
+ }
+
+ /// What the values mean: see [`Signal`].
+ pub const fn signal(&self) -> Signal {
+ self.signal
+ }
+
+ /// Bytes a pixel of this frame takes.
+ pub const fn bytes_per_pixel(&self) -> usize {
+ match self.depth {
+ Depth::Eight => BYTES_PER_PIXEL,
+ Depth::Sixteen => BYTES_PER_PIXEL * 2,
+ }
+ }
+
/// Width in pixels.
pub const fn width(&self) -> u32 {
self.width
@@ -152,15 +233,27 @@ impl Frame {
Some((y as usize * self.width as usize + x as usize) * BYTES_PER_PIXEL)
}
- /// Reads one pixel as `[r, g, b, a]`.
+ /// Reads one pixel as `[r, g, b, a]`: a deep frame's to eight bits, its
+ /// channels' high bytes, for a test or a log to look at.
pub fn pixel(&self, x: u32, y: u32) -> Option<[u8; 4]> {
let offset = self.offset_of(x, y)?;
- Some([
- self.pixels[offset],
- self.pixels[offset + 1],
- self.pixels[offset + 2],
- self.pixels[offset + 3],
- ])
+ Some(match self.depth {
+ Depth::Eight => [
+ self.pixels[offset],
+ self.pixels[offset + 1],
+ self.pixels[offset + 2],
+ self.pixels[offset + 3],
+ ],
+ Depth::Sixteen => {
+ let at = offset * 2;
+ [
+ self.pixels[at + 1],
+ self.pixels[at + 3],
+ self.pixels[at + 5],
+ self.pixels[at + 7],
+ ]
+ }
+ })
}
/// Writes one pixel. Out-of-bounds writes are ignored.
@@ -215,6 +308,8 @@ impl fmt::Debug for Frame {
.field("width", &self.width)
.field("height", &self.height)
.field("bytes", &self.pixels.len())
+ .field("depth", &self.depth)
+ .field("signal", &self.signal)
.finish()
}
}
diff --git a/src/crates/concat-export/src/lib.rs b/src/crates/concat-export/src/lib.rs
index 74c7d7b5..d5cff700 100644
--- a/src/crates/concat-export/src/lib.rs
+++ b/src/crates/concat-export/src/lib.rs
@@ -1174,6 +1174,9 @@ fn render_picture(
chain,
pre,
ranges.get(&layer.clip).copied(),
+ // Deep where the frame goes to the GPU as it is: a
+ // cutout cuts eight bits.
+ !cutouts.contains_key(&layer.clip),
) {
let frame = cutouts
.get(&layer.clip)
@@ -1212,11 +1215,15 @@ fn render_picture(
FrameRate::new(rate.fps() / layer.speed)
};
+ // Deep - an HDR clip in its own signal, converted on
+ // the GPU - unless a cutout has to cut it in eight bits;
+ // the decoder keeps eight bits for a chain itself.
let mut options = DecodeOptions::default()
.starting_at(layer.source_time)
.scaled_to(decode_width, decode_height)
.at_rate(decode_rate)
- .in_range(ranges.get(&layer.clip).copied());
+ .in_range(ranges.get(&layer.clip).copied())
+ .deep(!cutouts.contains_key(&layer.clip));
// Effects and transition fades, as one FFmpeg chain. The
// decoder guards the frame size after it, so an effect
@@ -1664,8 +1671,10 @@ fn frame_request(
.from_proxy(proxy)
.in_range(plan.built.ranges.get(&layer.clip).copied())
// The compositor fits the picture into its place, so an untreated
- // frame is drawn at the level it was decoded at.
+ // frame is drawn at the level it was decoded at - deep, for an HDR
+ // clip, unless a cutout has to cut it in eight bits.
.at_any_size(true)
+ .deep_when_untreated(!plan.built.cutouts.contains_key(&layer.clip))
}
/// [`preview_sources`] for one instant of a plan already built. With
diff --git a/src/crates/concat-host/tests/export.rs b/src/crates/concat-host/tests/export.rs
index d0ad6388..aa4d105f 100644
--- a/src/crates/concat-host/tests/export.rs
+++ b/src/crates/concat-host/tests/export.rs
@@ -916,6 +916,63 @@ fn fades_and_wipes_reach_the_picture() {
}
}
+/// An iPhone's HDR - ten-bit HEVC tagged HLG - exports through the new road:
+/// decoded deep in its own signal, converted and conformed to the SDR
+/// timeline on the GPU as it uploads. The picture is the grey it was
+/// written as, conformed: neither black nor blown out, and not tinted.
+#[test]
+fn an_hlg_clip_exports_through_the_gpu_conversion() {
+ if !VideoCodec::Hevc.available() {
+ eprintln!("no HEVC encoder in the linked FFmpeg; skipped");
+ return;
+ }
+ let scratch = Scratch::new("hlg");
+ let path = scratch.path().join("hlg.mp4");
+ {
+ let options = EncodeOptions {
+ codec: VideoCodec::Hevc,
+ preset: "ultrafast".to_owned(),
+ crf: 12,
+ rate_mode: RateMode::Vbr,
+ bitrate_kbps: 0,
+ ten_bit: true,
+ color_range: concat_media::ColorRange::Limited,
+ hardware: false,
+ threads: 0,
+ };
+ let mut encoder = Encoder::create_mislabelled_hdr(
+ &path,
+ WIDTH,
+ HEIGHT,
+ FrameRate::THIRTY,
+ &options,
+ false,
+ )
+ .expect("an HEVC encoder");
+ let mut frame = Frame::black(WIDTH, HEIGHT);
+ frame.fill([160, 160, 160, 255]);
+ for _ in 0..30 {
+ encoder.write_frame(&frame).expect("writes");
+ }
+ encoder.finish().expect("finishes");
+ }
+ let mut studio = Studio::new(scratch.path(), "HLG", video(WIDTH, HEIGHT, 30, 1));
+ let media = studio.import(&path);
+ studio.apply(Command::AddClipAtFirstFree {
+ media_id: media,
+ start: 0.0,
+ });
+ let exported = studio.export("an HLG clip");
+ let [r, g, b] = exported.colour_at(0.5);
+ assert!(
+ [r, g, b].iter().all(|channel| (40..=240).contains(channel))
+ && r.abs_diff(g) <= 6
+ && g.abs_diff(b) <= 6,
+ "the HLG grey exported as {:?}",
+ [r, g, b]
+ );
+}
+
/// Issue #103: the levels a file is read as reach the export. A source
/// written video range reads as it is by default; told it is full range,
/// its levels expand and the colour of a second is no longer the colour
diff --git a/src/crates/concat-media/src/decode.rs b/src/crates/concat-media/src/decode.rs
index eda0593c..3936417c 100644
--- a/src/crates/concat-media/src/decode.rs
+++ b/src/crates/concat-media/src/decode.rs
@@ -26,7 +26,7 @@
use std::path::{Path, PathBuf};
-use concat_core::frame::Frame;
+use concat_core::frame::{Frame, Signal};
use concat_core::time::{FrameRate, Rational};
use ffmpeg_the_third as ffmpeg;
use ffmpeg_the_third::codec::decoder;
@@ -132,6 +132,13 @@ pub struct DecodeOptions {
/// is none, takes video range the way a player does. See
/// [`ColorRange`].
pub color_range: Option<ColorRange>,
+ /// Keep a wide or HDR source's own signal, at sixteen bits a channel,
+ /// rather than tone-map it to eight-bit Rec. 709 on the CPU: for a
+ /// caller whose frames go straight to the GPU, which converts them as it
+ /// uploads. Ignored for a narrow source, and where a chain or a
+ /// pre-chain runs, since those are FFmpeg filters on eight bits. See
+ /// [`Frame::signal`].
+ pub deep: bool,
}
impl DecodeOptions {
@@ -204,6 +211,12 @@ impl DecodeOptions {
self
}
+ /// Keeps a wide or HDR source deep; see [`DecodeOptions::deep`].
+ pub fn deep(mut self, deep: bool) -> Self {
+ self.deep = deep;
+ self
+ }
+
/// Reads the picture as `range`, whatever the file says; `None` goes
/// back to the file's tag. See [`DecodeOptions::color_range`].
pub fn in_range(mut self, range: Option<ColorRange>) -> Self {
@@ -431,6 +444,47 @@ impl ColorSignal {
:out_color_matrix=bt709:out_range=tv:intent=perceptual"
)
}
+
+ /// What a deep frame of this source says it holds, when the source is
+ /// one the compositor converts on the GPU: Rec. 2020 primaries, with
+ /// PQ, HLG or a gamma. `None` for anything else - a P3 source, say -
+ /// which keeps the eight-bit conversion here.
+ pub fn deep_signal(&self) -> Option<Signal> {
+ use ffmpeg::color::{Primaries, TransferCharacteristic as Transfer};
+ match self.transfer {
+ Transfer::SMPTE2084 => Some(Signal::Pq),
+ Transfer::ARIB_STD_B67 => Some(Signal::Hlg),
+ _ if self.primaries == Primaries::BT2020 => Some(Signal::SdrWide),
+ _ => None,
+ }
+ }
+
+ /// The `scale` filter's arguments for a deep frame: the source's own
+ /// matrix and range, to RGB and nothing more - its primaries and
+ /// transfer are left as they are, for the GPU to convert.
+ fn native_args(self) -> String {
+ use ffmpeg::color::{Range, Space};
+ let matrix = match self.matrix {
+ Space::BT2020CL => "bt2020c",
+ Space::BT709 => "bt709",
+ _ => "bt2020nc",
+ };
+ let range = match self.range {
+ Range::JPEG => "pc",
+ _ => "tv",
+ };
+ format!(":in_color_matrix={matrix}:in_range={range}")
+ }
+}
+
+/// The signal a deep frame is decoded in, when `options` asks for one and
+/// the source is one the compositor converts: nothing in the decoder's
+/// graph may need eight bits, so no chain and no pre-chain.
+fn deep_output(options: &DecodeOptions, color: Option<&ColorSignal>) -> Option<Signal> {
+ if !options.deep || options.filter_chain.is_some() || options.pre_chain.is_some() {
+ return None;
+ }
+ color?.deep_signal()
}
/// The filtergraph between the decoder and the caller, as one string.
@@ -462,7 +516,9 @@ fn video_filter(
// for a file that lies about it. A range read off the codec in place
// of a tag is named only when it is full: the frames carry no tag for
// swscale to read, and untagged already means limited to it.
+ let deep = deep_output(options, color);
let convert = match (color, options.color_range) {
+ (Some(signal), _) if deep.is_some() => signal.native_args(),
(Some(signal), _) if signal.is_wide() => signal.bt709_args(),
(_, Some(range)) => format!(":in_range={}", range.scale_name()),
(Some(signal), None) if signal.implied && signal.range == ffmpeg::color::Range::JPEG => {
@@ -475,7 +531,14 @@ fn video_filter(
parts.push(chain.clone());
parts.push(format!("scale={width}:{height}:flags=bilinear"));
}
- parts.push("format=rgba".to_owned());
+ parts.push(
+ if deep.is_some() {
+ "format=rgba64le"
+ } else {
+ "format=rgba"
+ }
+ .to_owned(),
+ );
parts.join(",")
}
@@ -884,18 +947,24 @@ impl Decoder {
.map_err(|error| ffi::fail("filter output", &self.path, error))?;
}
- // The sink's picture is RGBA already; only its row padding differs
- // from the packed buffer a Frame is.
+ // The sink's picture is RGBA already - sixteen bits a channel for a
+ // deep frame - and only its row padding differs from the packed
+ // buffer a Frame is.
+ let deep = deep_output(&self.options, Some(&self.color));
let width = filtered.width();
let height = filtered.height();
- let row = width as usize * 4;
+ let row = width as usize * if deep.is_some() { 8 } else { 4 };
let stride = filtered.stride(0);
let data = filtered.data(0);
let mut pixels = Vec::with_capacity(row * height as usize);
for y in 0..height as usize {
pixels.extend_from_slice(&data[y * stride..y * stride + row]);
}
- Frame::from_rgba(width, height, pixels).ok_or_else(|| Error::Probe {
+ let frame = match deep {
+ Some(signal) => Frame::from_rgba64(width, height, pixels, signal),
+ None => Frame::from_rgba(width, height, pixels),
+ };
+ frame.ok_or_else(|| Error::Probe {
path: self.path.clone(),
detail: "the filtergraph produced a frame of the wrong size".to_owned(),
})
@@ -1353,9 +1422,36 @@ mod tests {
let mut decoder = Decoder::open(&path, &DecodeOptions::default()).expect("opens");
assert!(decoder.color().is_hdr(), "{:?}", decoder.color());
let decoded = decoder.next_frame().expect("decodes").expect("a frame");
- let _ = std::fs::remove_file(&path);
let luma = decoded.pixels()[0];
assert!(luma > 8 && luma < 250, "tone-mapped grey came out {luma}");
+
+ // Asked deep, the same file comes out in its own signal at sixteen
+ // bits a channel, its grey where it was written - about 180 of 255
+ // of full scale - rather than tone-mapped.
+ let mut deep = Decoder::open(&path, &DecodeOptions::default().deep(true)).expect("opens");
+ let frame = deep.next_frame().expect("decodes").expect("a frame");
+ assert_eq!(frame.depth(), concat_core::frame::Depth::Sixteen);
+ assert_eq!(frame.signal(), concat_core::frame::Signal::Pq);
+ assert_eq!(frame.pixels().len(), 64 * 64 * 8);
+ let channel = |at: usize| u16::from_le_bytes([frame.pixels()[at], frame.pixels()[at + 1]]);
+ let (r, g, b) = (channel(0), channel(2), channel(4));
+ let written = 180.0 / 255.0 * 65535.0;
+ for value in [r, g, b] {
+ assert!(
+ (f64::from(value) - written).abs() < 65535.0 * 0.03,
+ "the deep grey came out {r}, {g}, {b}"
+ );
+ }
+
+ // A chain runs on eight bits, so it keeps the tone-mapped frame.
+ let mut chained = Decoder::open(
+ &path,
+ &DecodeOptions::default().deep(true).filtered("negate"),
+ )
+ .expect("opens");
+ let frame = chained.next_frame().expect("decodes").expect("a frame");
+ assert_eq!(frame.depth(), concat_core::frame::Depth::Eight);
+ let _ = std::fs::remove_file(&path);
}
#[test]
diff --git a/src/crates/concat-media/src/pool.rs b/src/crates/concat-media/src/pool.rs
index 9081a2b4..4d3881d4 100644
--- a/src/crates/concat-media/src/pool.rs
+++ b/src/crates/concat-media/src/pool.rs
@@ -89,6 +89,11 @@ pub struct FrameRequest {
/// and a second cache entry per size; the level is never smaller than
/// `cover`, so nothing is lost. See [`FrameRequest::at_any_size`].
pub any_size: bool,
+ /// Served deep where the source is wide and nothing treats it here:
+ /// its own signal at sixteen bits a channel, for a caller whose frames
+ /// go straight to the GPU. Only with `any_size`, since a fit here is
+ /// FFmpeg on eight bits. See [`DecodeOptions::deep`].
+ pub deep: bool,
}
impl FrameRequest {
@@ -106,6 +111,7 @@ impl FrameRequest {
proxy: false,
range: None,
any_size: false,
+ deep: false,
}
}
@@ -152,6 +158,12 @@ impl FrameRequest {
self
}
+ /// Served deep where it can be; see [`FrameRequest::deep`].
+ pub fn deep_when_untreated(mut self, deep: bool) -> Self {
+ self.deep = deep;
+ self
+ }
+
/// Reads the proxy where there is one. See [`FrameRequest::proxy`].
pub fn from_proxy(mut self, proxy: bool) -> Self {
self.proxy = proxy;
@@ -171,6 +183,10 @@ struct SourceKey {
/// The range the file was read as: the same frame read as full and
/// as video range is two different pictures.
range: Option<ColorRange>,
+ /// Read deep: a wide source in its own signal, sixteen bits a channel
+ /// (see [`DecodeOptions::deep`]) - a different picture from the
+ /// tone-mapped eight bits.
+ deep: bool,
}
/// A treated frame's identity: the source frame and what was done to it.
@@ -339,6 +355,7 @@ impl Reader {
chain: Option<&str>,
pre: Option<&str>,
range: Option<ColorRange>,
+ deep: bool,
rate: FrameRate,
still: bool,
index: i64,
@@ -349,7 +366,8 @@ impl Reader {
let mut options = DecodeOptions::default()
.starting_at(rate.time_of_frame(index))
.scaled_to(width, height)
- .in_range(range);
+ .in_range(range)
+ .deep(deep);
if still {
// One frame, served for every time, and never sought: a seek
// on a single-image JPEG leaves FFmpeg's image demuxer with
@@ -432,6 +450,7 @@ type ReaderKey = (
Option<String>,
Option<String>,
Option<ColorRange>,
+ bool,
);
/// The warm readers and their recency, behind one short lock. A reader is
@@ -696,6 +715,10 @@ impl ReaderPool {
// A proxy is read as it is tagged: it was written from the
// corrected picture, so the correction applied twice would undo it.
let range = (path == request.path).then_some(request.range).flatten();
+ // Deep only where the frame is served as it is: a fit or a chain
+ // here is FFmpeg, which takes eight bits.
+ let deep =
+ request.deep && request.any_size && request.pre.is_none() && request.chain.is_none();
Ok((
facts,
SourceKey {
@@ -704,6 +727,7 @@ impl ReaderPool {
height,
index,
range,
+ deep,
},
))
}
@@ -767,6 +791,9 @@ impl ReaderPool {
/// the same options makes. The monitor asks through
/// [`ReaderPool::frame`] instead, whose cache the chain does not
/// invalidate. Without a crop or a chain the two are the same request.
+ /// `deep` reads a wide source in its own signal where there is no crop
+ /// and no chain; see [`DecodeOptions::deep`].
+ #[allow(clippy::too_many_arguments)]
pub fn frame_at(
&self,
path: &Path,
@@ -777,6 +804,7 @@ impl ReaderPool {
chain: Option<&str>,
pre: Option<&str>,
range: Option<ColorRange>,
+ deep: bool,
) -> Result<Arc<Frame>> {
let chain = chain.filter(|chain| !chain.trim().is_empty());
let pre = pre.filter(|pre| !pre.trim().is_empty());
@@ -788,6 +816,7 @@ impl ReaderPool {
height,
index,
range,
+ deep: deep && chain.is_none() && pre.is_none(),
};
if chain.is_none() && pre.is_none() {
return self.source_frame(&facts, &source_key);
@@ -817,7 +846,7 @@ impl ReaderPool {
self.remember_treated(key, Arc::clone(&frame));
return Ok(frame);
}
- let shared = self.reader(path, width, height, chain, pre, range, &facts, index)?;
+ let shared = self.reader(path, width, height, chain, pre, range, false, &facts, index)?;
let mut reader = shared.lock().map_err(|_| crate::error::Error::NoFrame {
path: path.to_path_buf(),
})?;
@@ -853,7 +882,7 @@ impl ReaderPool {
return Ok(frame);
}
let shared = self.reader(
- path, key.width, key.height, None, None, key.range, facts, key.index,
+ path, key.width, key.height, None, None, key.range, key.deep, facts, key.index,
)?;
let mut reader = shared.lock().map_err(|_| crate::error::Error::NoFrame {
path: path.to_path_buf(),
@@ -1016,6 +1045,7 @@ impl ReaderPool {
chain: Option<&str>,
pre: Option<&str>,
range: Option<ColorRange>,
+ deep: bool,
facts: &MediaFacts,
target: i64,
) -> Result<Arc<Mutex<Reader>>> {
@@ -1026,6 +1056,7 @@ impl ReaderPool {
chain.map(str::to_owned),
pre.map(str::to_owned),
range,
+ deep,
);
{
let mut readers = self
@@ -1051,6 +1082,7 @@ impl ReaderPool {
chain,
pre,
range,
+ deep,
facts.rate,
facts.still,
target,
@@ -1158,6 +1190,7 @@ pub(crate) mod tests {
height: 2,
index,
range: None,
+ deep: false,
};
cache.insert(key(0), Arc::new(Frame::black(2, 2)));
cache.insert(key(1), Arc::new(Frame::black(2, 2)));
@@ -1185,7 +1218,7 @@ pub(crate) mod tests {
first.fill([255, 0, 0, 255]);
pool.hold_still(path, Arc::new(first));
let same = pool
- .frame_at(path, Rational::ZERO, 64, 32, true, None, None, None)
+ .frame_at(path, Rational::ZERO, 64, 32, true, None, None, None, false)
.expect("its own size");
assert_eq!((same.width(), same.height()), (64, 32));
assert_eq!(&same.pixels()[..4], &[255, 0, 0, 255]);
@@ -1217,6 +1250,7 @@ pub(crate) mod tests {
height: 2,
index: 0,
range: None,
+ deep: false,
},
Arc::new(Frame::black(2, 2)),
);
@@ -1266,6 +1300,7 @@ pub(crate) mod tests {
None,
None,
None,
+ false,
)
.expect("frame decodes");
i64::from(frame.pixel(32, 32).expect("in bounds")[0])
diff --git a/src/crates/concat-perf/src/main.rs b/src/crates/concat-perf/src/main.rs
index ec487f65..b0089fb9 100644
--- a/src/crates/concat-perf/src/main.rs
+++ b/src/crates/concat-perf/src/main.rs
@@ -716,6 +716,13 @@ fn decode_4k() -> Vec<Measure> {
&DecodeOptions::default().scaled_to(960, 540),
hlg.frames,
);
+ // The same stream the way the monitor now reads it: deep, in its own
+ // signal, the conversion left to the GPU's upload.
+ let (deep_fps, deep_on) = decode_with(
+ &hlg.path,
+ &DecodeOptions::default().scaled_to(960, 540).deep(true),
+ hlg.frames,
+ );
concat_media::set_hardware_decode(false);
vec![
Measure {
@@ -750,11 +757,24 @@ fn decode_4k() -> Vec<Measure> {
note: format!("the slowest stream; {}", used(three_on, &hevc)),
},
Measure {
- name: "decode 4K HLG to the SDR monitor, 960x540",
+ name: "decode 4K HLG tone-mapped on the CPU, 960x540",
value: hlg_fps,
unit: "fps",
+ budget: Budget::AtLeast(15.0),
+ note: format!(
+ "the eight-bit fallback, for a clip with a chain or a cutout; {}",
+ used(hlg_on, &hlg)
+ ),
+ },
+ Measure {
+ name: "decode 4K HLG deep for the GPU, 960x540",
+ value: deep_fps,
+ unit: "fps",
budget: Budget::AtLeast(30.0),
- note: format!("tone-mapped on the CPU; {}", used(hlg_on, &hlg)),
+ note: format!(
+ "converted on the GPU as it uploads; {}",
+ used(deep_on, &hlg)
+ ),
},
]
}
diff --git a/src/crates/concat-render/src/gpu.rs b/src/crates/concat-render/src/gpu.rs
index 65c9a721..85875178 100644
--- a/src/crates/concat-render/src/gpu.rs
+++ b/src/crates/concat-render/src/gpu.rs
@@ -213,6 +213,116 @@ fn fs_resolve(@builtin(position) at: vec4<f32>) -> @location(0) vec4<f32> {
}
"#;
+/// A deep frame (sixteen bits a channel, Rec. 2020, its source's own signal)
+/// copied into the working space: the upload of an HDR or wide-gamut clip.
+/// One entry a signal (`concat_core::frame::Signal`): the transfer undone
+/// into light, 1.0 the white of an SDR picture (203 nits, ITU-R BT.2408),
+/// and the primaries brought to Rec. 709's, where a colour outside them is
+/// negative rather than clipped.
+///
+/// Every timeline is SDR for now, so an HDR clip is conformed to SDR as it
+/// arrives, as Final Cut does a clip at a time: BT.2390's roll-off (ITU-R
+/// BT.2390, the EETF) takes its highlights from a 1000-nit master down to
+/// SDR white, on the brightest channel so a hue keeps its place. An HDR
+/// timeline will skip it and keep the highlights.
+const DEEP_SHADER: &str = r#"
+@group(0) @binding(0) var deep: texture_2d<u32>;
+
+@vertex
+fn vs_full(@builtin(vertex_index) index: u32) -> @builtin(position) vec4<f32> {
+ let corner = vec2<f32>(f32((index << 1u) & 2u), f32(index & 2u));
+ return vec4<f32>(corner * 2.0 - 1.0, 0.0, 1.0);
+}
+
+fn deep_at(at: vec4<f32>) -> vec4<f32> {
+ return vec4<f32>(textureLoad(deep, vec2<i32>(at.xy), 0)) / 65535.0;
+}
+
+// Rec. 2020 primaries to Rec. 709's, linear light (ITU-R BT.2087's matrix
+// inverted, to full precision).
+fn from_2020(rgb: vec3<f32>) -> vec3<f32> {
+ return vec3<f32>(
+ dot(vec3<f32>(1.660491002, -0.587641139, -0.072849863), rgb),
+ dot(vec3<f32>(-0.124550475, 1.132899897, -0.008349423), rgb),
+ dot(vec3<f32>(-0.018150763, -0.100578898, 1.118729661), rgb),
+ );
+}
+
+// SMPTE ST 2084: a signal to nits, and back.
+const PQ_M1: f32 = 0.1593017578125;
+const PQ_M2: f32 = 78.84375;
+const PQ_C1: f32 = 0.8359375;
+const PQ_C2: f32 = 18.8515625;
+const PQ_C3: f32 = 18.6875;
+
+fn pq_nits(signal: vec3<f32>) -> vec3<f32> {
+ let e = pow(clamp(signal, vec3<f32>(0.0), vec3<f32>(1.0)), vec3<f32>(1.0 / PQ_M2));
+ let y = pow(max(e - PQ_C1, vec3<f32>(0.0)) / (PQ_C2 - PQ_C3 * e), vec3<f32>(1.0 / PQ_M1));
+ return y * 10000.0;
+}
+
+fn pq_signal(nits: f32) -> f32 {
+ let y = pow(clamp(nits / 10000.0, 0.0, 1.0), PQ_M1);
+ return pow((PQ_C1 + PQ_C2 * y) / (1.0 + PQ_C3 * y), PQ_M2);
+}
+
+// BT.2390's EETF from a 1000-nit master to SDR white (203 nits), on the
+// brightest channel, the colour scaled with it.
+fn to_sdr(nits: vec3<f32>) -> vec3<f32> {
+ let peak = max(nits.r, max(nits.g, nits.b));
+ if (peak <= 0.0) {
+ return nits;
+ }
+ let master = pq_signal(1000.0);
+ let e1 = pq_signal(peak) / master;
+ let top = pq_signal(203.0) / master;
+ let knee = 1.5 * top - 0.5;
+ var e2 = e1;
+ if (e1 > knee) {
+ let t = (min(e1, 1.0) - knee) / (1.0 - knee);
+ let t2 = t * t;
+ let t3 = t2 * t;
+ e2 = (2.0 * t3 - 3.0 * t2 + 1.0) * knee
+ + (t3 - 2.0 * t2 + t) * (1.0 - knee)
+ + (-2.0 * t3 + 3.0 * t2) * top;
+ }
+ let mapped = pq_nits(vec3<f32>(e2 * master)).r;
+ return nits * (mapped / peak);
+}
+
+@fragment
+fn fs_upload_sdr_wide(@builtin(position) at: vec4<f32>) -> @location(0) vec4<f32> {
+ let colour = deep_at(at);
+ let lin = pow(max(colour.rgb, vec3<f32>(0.0)), vec3<f32>(2.4));
+ return vec4<f32>(from_2020(lin), colour.a);
+}
+
+@fragment
+fn fs_upload_pq(@builtin(position) at: vec4<f32>) -> @location(0) vec4<f32> {
+ let colour = deep_at(at);
+ let nits = to_sdr(pq_nits(colour.rgb));
+ return vec4<f32>(from_2020(nits / 203.0), colour.a);
+}
+
+// ARIB STD-B67's inverse OETF to scene light, then BT.2100's OOTF for a
+// 1000-nit display (a system gamma of 1.2), which puts HLG's reference
+// white - 75 % of the signal - at 203 nits.
+@fragment
+fn fs_upload_hlg(@builtin(position) at: vec4<f32>) -> @location(0) vec4<f32> {
+ let colour = deep_at(at);
+ let a = 0.17883277;
+ let b = 0.28466892;
+ let c = 0.55991073;
+ let e = clamp(colour.rgb, vec3<f32>(0.0), vec3<f32>(1.0));
+ let low = e * e / 3.0;
+ let high = (exp((e - c) / a) + b) / 12.0;
+ let scene = select(high, low, e <= vec3<f32>(0.5));
+ let ys = dot(scene, vec3<f32>(0.2627, 0.6780, 0.0593));
+ let nits = to_sdr(1000.0 * pow(max(ys, 1e-6), 0.2) * scene);
+ return vec4<f32>(from_2020(nits / 203.0), colour.a);
+}
+"#;
+
/// One package's shader, compiled once and kept: its pipeline, and the two
/// uniform buffers every pass through it rewrites.
struct CompiledShader {
@@ -308,6 +418,14 @@ pub struct WgpuCompositor {
/// The eight-bit textures frames are written into on their way to the
/// pool, one a size, each with its bind group.
staging: HashMap<(u32, u32), (wgpu::Texture, wgpu::BindGroup)>,
+ /// A deep frame into the working space, by signal: Rec. 2020 gamma,
+ /// HLG, PQ (see `DEEP_SHADER`).
+ deep_pipelines: [wgpu::RenderPipeline; 3],
+ /// The layout a deep staging texture is bound with: sixteen-bit
+ /// integers, read texel by texel.
+ deep_layout: wgpu::BindGroupLayout,
+ /// The sixteen-bit staging textures, one a size.
+ deep_staging: HashMap<(u32, u32), (wgpu::Texture, wgpu::BindGroup)>,
bind_layout: wgpu::BindGroupLayout,
/// Group 1 of a shader pass: the frame block and the package's params.
uniform_layout: wgpu::BindGroupLayout,
@@ -691,6 +809,56 @@ impl WgpuCompositor {
let upload_pipeline = copy_into(WORK, "fs_upload");
let resolve_pipeline = copy_into(wgpu::TextureFormat::Rgba8Unorm, "fs_resolve");
+ let deep_layout = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
+ label: Some("concat deep"),
+ entries: &[wgpu::BindGroupLayoutEntry {
+ binding: 0,
+ visibility: wgpu::ShaderStages::FRAGMENT,
+ ty: wgpu::BindingType::Texture {
+ sample_type: wgpu::TextureSampleType::Uint,
+ view_dimension: wgpu::TextureViewDimension::D2,
+ multisampled: false,
+ },
+ count: None,
+ }],
+ });
+ let deep_shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
+ label: Some("concat deep"),
+ source: wgpu::ShaderSource::Wgsl(DEEP_SHADER.into()),
+ });
+ let deep_pipeline_layout = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {
+ label: Some("concat deep"),
+ bind_group_layouts: &[Some(&deep_layout)],
+ immediate_size: 0,
+ });
+ let deep_pipelines = ["fs_upload_sdr_wide", "fs_upload_hlg", "fs_upload_pq"].map(|entry| {
+ device.create_render_pipeline(&wgpu::RenderPipelineDescriptor {
+ label: Some(entry),
+ layout: Some(&deep_pipeline_layout),
+ vertex: wgpu::VertexState {
+ module: &deep_shader,
+ entry_point: Some("vs_full"),
+ compilation_options: Default::default(),
+ buffers: &[],
+ },
+ fragment: Some(wgpu::FragmentState {
+ module: &deep_shader,
+ entry_point: Some(entry),
+ compilation_options: Default::default(),
+ targets: &[Some(wgpu::ColorTargetState {
+ format: WORK,
+ blend: None,
+ write_mask: wgpu::ColorWrites::ALL,
+ })],
+ }),
+ primitive: wgpu::PrimitiveState::default(),
+ depth_stencil: None,
+ multisample: wgpu::MultisampleState::default(),
+ multiview_mask: None,
+ cache: None,
+ })
+ });
+
let sampler = device.create_sampler(&wgpu::SamplerDescriptor {
label: Some("concat layer"),
address_mode_u: wgpu::AddressMode::ClampToEdge,
@@ -715,6 +883,9 @@ impl WgpuCompositor {
upload_pipeline,
resolve_pipeline,
staging: HashMap::new(),
+ deep_pipelines,
+ deep_layout,
+ deep_staging: HashMap::new(),
bind_layout,
uniform_layout,
lut_layout,
@@ -1281,6 +1452,7 @@ impl WgpuCompositor {
self.used.remove(&key);
self.idle.remove(&key);
self.staging.remove(&key);
+ self.deep_staging.remove(&key);
}
Self::trim(
&mut self.luts,
@@ -1404,7 +1576,12 @@ impl WgpuCompositor {
// a clip's own colour will be brought into the working space. The
// write lands before the submit that copies it, and the next
// upload's write after it, so one staging texture a size serves.
- let (staging, staging_group) = self.staging_for(frame.width(), frame.height());
+ let deep = frame.depth() == concat_core::frame::Depth::Sixteen;
+ let (staging, staging_group) = if deep {
+ self.deep_staging_for(frame.width(), frame.height())
+ } else {
+ self.staging_for(frame.width(), frame.height())
+ };
self.queue.write_texture(
wgpu::TexelCopyTextureInfo {
texture: &staging,
@@ -1415,7 +1592,7 @@ impl WgpuCompositor {
frame.pixels(),
wgpu::TexelCopyBufferLayout {
offset: 0,
- bytes_per_row: Some(frame.width() * 4),
+ bytes_per_row: Some(frame.width() * frame.bytes_per_pixel() as u32),
rows_per_image: Some(frame.height()),
},
wgpu::Extent3d {
@@ -1432,7 +1609,17 @@ impl WgpuCompositor {
.create_command_encoder(&wgpu::CommandEncoderDescriptor {
label: Some("concat upload"),
});
- Self::copy(&mut encoder, &self.upload_pipeline, &staging_group, &view);
+ let pipeline = if deep {
+ use concat_core::frame::Signal;
+ &self.deep_pipelines[match frame.signal() {
+ Signal::Hlg => 1,
+ Signal::Pq => 2,
+ Signal::Sdr | Signal::SdrWide => 0,
+ }]
+ } else {
+ &self.upload_pipeline
+ };
+ Self::copy(&mut encoder, pipeline, &staging_group, &view);
self.queue.submit([encoder.finish()]);
index
}
@@ -1465,6 +1652,40 @@ impl WgpuCompositor {
);
}
+ /// This size's sixteen-bit staging texture and its bind group, made on
+ /// first use.
+ fn deep_staging_for(&mut self, width: u32, height: u32) -> (wgpu::Texture, wgpu::BindGroup) {
+ if let Some((texture, group)) = self.deep_staging.get(&(width, height)) {
+ return (texture.clone(), group.clone());
+ }
+ let texture = self.device.create_texture(&wgpu::TextureDescriptor {
+ label: Some("concat deep staging"),
+ size: wgpu::Extent3d {
+ width,
+ height,
+ depth_or_array_layers: 1,
+ },
+ mip_level_count: 1,
+ sample_count: 1,
+ dimension: wgpu::TextureDimension::D2,
+ format: wgpu::TextureFormat::Rgba16Uint,
+ usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST,
+ view_formats: &[],
+ });
+ let view = texture.create_view(&wgpu::TextureViewDescriptor::default());
+ let group = self.device.create_bind_group(&wgpu::BindGroupDescriptor {
+ label: Some("concat deep staging"),
+ layout: &self.deep_layout,
+ entries: &[wgpu::BindGroupEntry {
+ binding: 0,
+ resource: wgpu::BindingResource::TextureView(&view),
+ }],
+ });
+ self.deep_staging
+ .insert((width, height), (texture.clone(), group.clone()));
+ (texture, group)
+ }
+
/// This size's staging texture and its bind group, made on first use.
fn staging_for(&mut self, width: u32, height: u32) -> (wgpu::Texture, wgpu::BindGroup) {
if let Some((texture, group)) = self.staging.get(&(width, height)) {
diff --git a/src/crates/concat-render/src/gpu/tests.rs b/src/crates/concat-render/src/gpu/tests.rs
index c90ef238..f0061620 100644
--- a/src/crates/concat-render/src/gpu/tests.rs
+++ b/src/crates/concat-render/src/gpu/tests.rs
@@ -842,3 +842,59 @@ fn a_stack_keeps_its_precision_between_passes() {
"down and up again gave {got:?}"
);
}
+
+/// A deep frame of `level` a channel, sixteen bits, in `signal`.
+fn deep(width: u32, height: u32, level: f32, signal: concat_core::frame::Signal) -> Frame {
+ let value = (level.clamp(0.0, 1.0) * 65535.0).round() as u16;
+ let pixel: Vec<u8> = [value, value, value, u16::MAX]
+ .iter()
+ .flat_map(|channel| channel.to_le_bytes())
+ .collect();
+ Frame::from_rgba64(
+ width,
+ height,
+ pixel.repeat((width * height) as usize),
+ signal,
+ )
+ .expect("a deep frame")
+}
+
+/// An HDR clip is converted into the working space on the GPU as it
+/// uploads, and conformed to the SDR timeline there: HLG's reference white
+/// (75 % of the signal, 203 nits) lands a little under SDR white, where
+/// BT.2390's roll-off leaves room for the highlights above it, grey still;
+/// the master's peak - all of HLG's signal, or 1000 nits of PQ - is SDR
+/// white; black is black.
+#[test]
+fn an_hdr_frame_is_conformed_to_sdr_on_the_way_in() {
+ use concat_core::frame::Signal;
+ let Some(mut gpu) = gpu() else { return };
+ let mut drawn = |frame: Frame| {
+ let out = gpu.render(&plan(4, 4, vec![layer(frame)]));
+ let p = out.pixel(1, 1).expect("in the frame");
+ [p[0], p[1], p[2]]
+ };
+ let white = drawn(deep(4, 4, 0.75, Signal::Hlg));
+ assert!(
+ white.iter().all(|channel| (205..=245).contains(channel))
+ && white[0].abs_diff(white[1]) <= 2
+ && white[1].abs_diff(white[2]) <= 2,
+ "HLG reference white came out {white:?}"
+ );
+ let peak = drawn(deep(4, 4, 1.0, Signal::Hlg));
+ assert!(
+ peak.iter().all(|channel| *channel >= 252),
+ "HLG peak came out {peak:?}"
+ );
+ let black = drawn(deep(4, 4, 0.0, Signal::Hlg));
+ assert!(
+ black.iter().all(|channel| *channel <= 2),
+ "HLG black came out {black:?}"
+ );
+ // 1000 nits in PQ is 0.7518 of the signal.
+ let pq_peak = drawn(deep(4, 4, 0.7518, Signal::Pq));
+ assert!(
+ pq_peak.iter().all(|channel| *channel >= 250),
+ "PQ 1000 nits came out {pq_peak:?}"
+ );
+}