Skip to content

LightCraft: An open-source non-destructive photo editor written by scratch in pure Rust

LR-MASK-SKY: AI masks use DNG semantic mattes (iPhone ProRAW sky matte)

Exalted5373ed59d78e837e
diff --git a/ROADMAP.md b/ROADMAP.md
index 26acc1765..2ad5884ea 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -25,7 +25,7 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
| **Feature checklist** | 79.7% | P0 core and P1 nearly complete: import (Add / Copy / Move, templates, devices), library, grid/loupe/compare/survey, every Edit slider, curves, colour grading, masking tools, crop/Upright, heal/clone, presets/profiles, versions/history, sync, export, menus, shortcuts | P0: full-resolution zoom detail; P1: lens-profile database, content-aware fill (patch synthesis), video playback/trim |
| **RAW coverage** (formats people shoot) | ~55% | DNG (all kinds), CR2, CR3 lossless Bayer and version 0x100/0x200 C-RAW (M50/R100/R8 pixel-exact corpus), ARW, NEF (uncompressed + Huffman lossless/lossy), RAF (uncompressed, lossless/lossy compressed Bayer and X-Trans), uncompressed ORF, RW2 / RWL / Panasonic RAW (every raw format, checked on 178 files from 118 Panasonic and Leica bodies), PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed ORF, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. NEFs labelled compressed but stored uncompressed (Z 6 packed 14-bit, D850 12-bit uncompressed). Per-model verification is thin (~55 corpus files vs >1,000 models) |
| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW, NEF, RW2, RAF, CR3, CR2 and PEF have guarded camera-JPEG colour estimates, with bundled ILCE-7M4, X-H2S and X-T4 profiles (docs/camera-preview-colour.md); other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
-| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |
+| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics, or the photo's own segmentation mattes when it ships them (DNG semantic masks, e.g. iPhone ProRAW's sky matte) | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |
| **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering |
| **Classic modules** | ~30% | Geotagging from GPX track logs, soft proofing (partial), slideshow (basic) | **Map view, Book, Print, Slideshow module, Web, publish services**: ~40 tracker rows ⬜ |
| **HDR & video** | 0% | | HDR edit/display/export; video play/trim/edit/export |
diff --git a/crates/engine/src/files.rs b/crates/engine/src/files.rs
index bad30e149..8e14c68f7 100644
--- a/crates/engine/src/files.rs
+++ b/crates/engine/src/files.rs
@@ -245,7 +245,8 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
raw.opcodes.list3.retain(|op| !op.is_lens_correction());
let xy = lightcraft_raw::color::as_shot_white_xy(&raw);
let t = lightcraft_raw::color::camera_transform(&raw, xy);
- let camera_look = crate::camera_preview::fit_preview(&raw, &bytes, &t);
+ // the source's segmentation mattes (DNG semantic masks), read while the preview is fitted
+ let (camera_look, mattes) = rayon::join(|| crate::camera_preview::fit_preview(&raw, &bytes, &t), || dng_mattes(&bytes, &raw));
drop(bytes);
// Previews and thumbnails bin the mosaic straight to (about) the size they need; only
// larger levels (exports, 1:1) demosaic the whole sensor.
@@ -308,7 +309,7 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
let relative = crate::camera_preview::file_local_look(raw.format) && t.matrix_is_fallback;
let camera_tone = camera_look.as_ref().map(|p| p.tone).or_else(|| raw.color.profile.tone_curve.as_ref().and_then(dng_tone_curve));
let (temp, tint) = if relative { (6500.0, 0.0) } else { (temp.round(), tint.round()) };
- return Ok((img, SourceInfo { raw: true, as_shot_temp: temp, as_shot_tint: tint, lens, relative_wb: relative, camera_tone }));
+ return Ok((img, SourceInfo { raw: true, as_shot_temp: temp, as_shot_tint: tint, lens, relative_wb: relative, camera_tone, mattes }));
}
let d = lightcraft_codecs::decode(&bytes, fit_box(max_edge)).map_err(|e| e.to_string())?;
drop(bytes);
@@ -317,6 +318,42 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
Ok((img.into_oriented(Orientation::from_exif(d.orientation)), SourceInfo::default()))
}
+/// The semantic masks of a DNG that AI masks understand, over the developed image (default crop,
+/// oriented like it). Person mattes without a confidently selected pixel are left out (the
+/// Subject mask then falls back to its heuristic); a sky matte counts even when empty (no sky).
+fn dng_mattes(bytes: &[u8], raw: &lightcraft_raw::RawImage) -> Option<Arc<lightcraft_pipeline::masks::Mattes>> {
+ use lightcraft_pipeline::masks::{MatteKind, Mattes};
+ if raw.format != lightcraft_raw::RawFormat::Dng {
+ return None;
+ }
+ let mut mattes = Mattes::default();
+ for m in lightcraft_raw::semantic_masks(bytes) {
+ let Some(kind) = matte_kind(&m.name) else { continue };
+ let Some(img) = m.developed(raw.active_area, raw.crop) else { continue };
+ if kind != MatteKind::Sky && !img.data.iter().any(|&v| v >= 128) {
+ continue;
+ }
+ mattes.push(kind, img.into_oriented(raw.orientation));
+ }
+ (!mattes.is_empty()).then(|| Arc::new(mattes))
+}
+
+/// What a semantic mask selects, by its `SemanticName`: Apple's (iPhone ProRAW) are named after
+/// their auxiliary image types, `urn:com:apple:photo:<year>:aux:<type>`. Other names: `None`.
+fn matte_kind(name: &str) -> Option<lightcraft_pipeline::masks::MatteKind> {
+ use lightcraft_pipeline::masks::MatteKind::*;
+ let kind = name.strip_prefix("urn:com:apple:photo:")?.rsplit(':').next()?;
+ Some(match kind {
+ "semanticskymatte" => Sky,
+ "semanticskinmatte" => Skin,
+ "semantichairmatte" => Hair,
+ "semanticteethmatte" => Teeth,
+ "semanticglassesmatte" => Glasses,
+ "portraiteffectsmatte" => Person,
+ _ => return None,
+ })
+}
+
/// A DNG `ProfileToneCurve` (linear in, linear out, 1.0 = white after exposure compensation) as the
/// finish stage's camera tone curve: 32 knots, log-spaced over 12 stops below white; above white
/// the camera tone's shoulder continues it.
@@ -699,6 +736,70 @@ mod tests {
assert!((mean(&dim) / mean(&before) - 0.5).abs() < 0.02, "{} vs {}", mean(&dim), mean(&before));
}
+ /// iPhone ProRAW-style semantic masks drive the Sky mask: Apple's sky matte (here the right
+ /// half of the sensor, i.e. the bottom of the photo once turned 90° clockwise — where the sky
+ /// heuristic would never look) is used; a matte with an unknown name is ignored.
+ #[test]
+ fn dng_sky_matte_drives_the_sky_mask() {
+ use lightcraft_develop::{DevelopSettings, LocalAdjustments, Mask, MaskComponent, MaskOp, MaskShape};
+ use lightcraft_pipeline::{RenderRequest, render};
+ use lightcraft_tiff::tags::{self as t, photometric};
+ use lightcraft_tiff::{ByteOrder, IfdBuilder, ImageData, TiffWriter, Value};
+ let (w, h) = (16u32, 8u32);
+ let image = |ifd: &mut IfdBuilder, w: u32, h: u32, cpp: u16, bits: u16, data: Vec<u8>| {
+ ifd.set(t::IMAGE_WIDTH, Value::Long(vec![w]));
+ ifd.set(t::IMAGE_LENGTH, Value::Long(vec![h]));
+ ifd.set(t::BITS_PER_SAMPLE, Value::Short(vec![bits; cpp as usize]));
+ ifd.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![cpp]));
+ ifd.set(t::COMPRESSION, Value::Short(vec![1]));
+ ifd.set_image(ImageData::Strips { rows_per_strip: h, strips: vec![data] });
+ };
+ let mut raw = IfdBuilder::new();
+ raw.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![0]));
+ raw.set(t::PHOTOMETRIC, Value::Short(vec![photometric::LINEAR_RAW]));
+ image(&mut raw, w, h, 3, 16, 12000u16.to_le_bytes().repeat((w * h * 3) as usize));
+ let matte = |name: &str, right: bool| {
+ let mut m = IfdBuilder::new();
+ m.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![t::SUBFILE_SEMANTIC_MASK]));
+ m.set(t::PHOTOMETRIC, Value::Short(vec![photometric::MASK]));
+ m.set(t::SEMANTIC_NAME, Value::Ascii(name.into()));
+ image(&mut m, w / 2, h / 2, 1, 8, (0..w / 2 * h / 2).map(|i| if (i % (w / 2) >= w / 4) == right { 255 } else { 0 }).collect());
+ m
+ };
+ let mut ifd0 = IfdBuilder::new();
+ ifd0.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![1]));
+ ifd0.set(t::DNG_VERSION, Value::Byte(vec![1, 6, 0, 0]));
+ ifd0.set(t::MAKE, Value::Ascii("Apple".into()));
+ ifd0.set(t::ORIENTATION, Value::Short(vec![6]));
+ ifd0.set(t::COLOR_MATRIX_1, Value::SRational(vec![(1, 1), (0, 1), (0, 1), (0, 1), (1, 1), (0, 1), (0, 1), (0, 1), (1, 1)]));
+ ifd0.set(t::CALIBRATION_ILLUMINANT_1, Value::Short(vec![21]));
+ ifd0.set(t::PHOTOMETRIC, Value::Short(vec![photometric::RGB]));
+ image(&mut ifd0, 4, 2, 3, 8, vec![128; 24]);
+ ifd0.add_sub_ifd(raw);
+ ifd0.add_sub_ifd(matte("urn:com:apple:photo:2020:aux:semanticskymatte", true));
+ ifd0.add_sub_ifd(matte("urn:com:apple:photo:2020:aux:semanticsomethingelse", false));
+ let bytes = TiffWriter::new(ByteOrder::Little, false).write(&[ifd0]).unwrap();
+
+ let (img, info) = load_bytes(&bytes, 64).unwrap();
+ assert_eq!((img.width, img.height), (8, 16));
+ assert!(info.mattes.is_some());
+ let sky = Mask {
+ components: vec![MaskComponent { name: None, op: MaskOp::Add, invert: false, shape: MaskShape::Sky }],
+ adjust: LocalAdjustments { exposure: -3.0, ..Default::default() },
+ ..Default::default()
+ };
+ let s = DevelopSettings { masks: vec![sky], ..Default::default() };
+ let req = RenderRequest::fit(8, 16);
+ let green = |s: &DevelopSettings, info: &SourceInfo, y: usize| render(&img, info, s, &req).image.get(4, y)[1] as i32;
+ let plain = DevelopSettings::default();
+ assert_eq!(green(&s, &info, 2), green(&plain, &info, 2), "no sky at the top");
+ assert!(green(&s, &info, 13) + 40 < green(&plain, &info, 13), "the sky at the bottom is darkened");
+ // without the matte the heuristic looks at the top of the frame instead
+ let heuristic = SourceInfo { mattes: None, ..info.clone() };
+ assert!(green(&s, &heuristic, 2) < green(&plain, &heuristic, 2));
+ assert_eq!(green(&s, &heuristic, 13), green(&plain, &heuristic, 13));
+ }
+
/// Apple ProRAW: like Lightroom Classic, the default render ignores the file's
/// `ProfileGainTableMap` (Apple's local tone mapping); a map that would double every pixel
/// changes nothing.
diff --git a/crates/engine/src/media.rs b/crates/engine/src/media.rs
index 1ed89e473..56ad62d4a 100644
--- a/crates/engine/src/media.rs
+++ b/crates/engine/src/media.rs
@@ -143,7 +143,7 @@ impl DecodedSource {
/// What to render these pixels against: the decoder's facts, else `header` (the catalog's)
/// with any stored camera tone curve.
pub fn info_or(&self, header: SourceInfo) -> SourceInfo {
- self.info.unwrap_or(SourceInfo { camera_tone: self.camera_tone.or(header.camera_tone), ..header })
+ self.info.clone().unwrap_or_else(|| SourceInfo { camera_tone: self.camera_tone.or(header.camera_tone), ..header })
}
}
@@ -253,8 +253,9 @@ fn rendered_budget(share: usize) -> usize {
RENDERED_MEM_BYTES.min(share / 4)
}
-fn source_bytes(img: &Rgb32f) -> usize {
- img.data.len() * 12 + 64 + std::mem::size_of::<SourceInfo>()
+fn source_bytes(s: &DecodedSource) -> usize {
+ let mattes = s.info.as_ref().and_then(|i| i.mattes.as_ref()).map_or(0, |m| m.bytes());
+ s.image.data.len() * 12 + 64 + std::mem::size_of::<SourceInfo>() + mattes
}
impl MediaCache {
@@ -319,8 +320,8 @@ impl MediaCache {
let Some((tick, which)) = candidates.into_iter().flatten().min() else { break };
let freed = match which {
0 => self.thumbs.pop_oldest(),
- 1 => self.previews.iter().position(|e| e.2 == tick).map(|i| source_bytes(&self.previews.remove(i).1.image)),
- 2 => self.full.take().map(|e| source_bytes(&e.1.image)),
+ 1 => self.previews.iter().position(|e| e.2 == tick).map(|i| source_bytes(&self.previews.remove(i).1)),
+ 2 => self.full.take().map(|e| source_bytes(&e.1)),
_ => self.rendered.evict_oldest(),
};
match freed {
@@ -372,7 +373,7 @@ impl MediaCache {
let tick = lightcraft_preview::next_tick();
match level {
SourceLevel::Thumb => {
- let cost = source_bytes(&img.image);
+ let cost = source_bytes(&img);
self.thumbs.insert(id, img, cost);
}
SourceLevel::Preview => {
@@ -415,8 +416,8 @@ impl MediaCache {
/// Decoded sources held: (thumbnail level, preview level, full size).
pub fn usage(&self) -> (crate::memory::Usage, crate::memory::Usage, crate::memory::Usage) {
use crate::memory::Usage;
- let previews = Usage::new(self.previews.len(), self.previews.iter().map(|e| source_bytes(&e.1.image)).sum());
- let full = self.full.as_ref().map(|e| Usage::new(1, source_bytes(&e.1.image))).unwrap_or_default();
+ let previews = Usage::new(self.previews.len(), self.previews.iter().map(|e| source_bytes(&e.1)).sum());
+ let full = self.full.as_ref().map(|e| Usage::new(1, source_bytes(&e.1))).unwrap_or_default();
(Usage::new(self.thumbs.len(), self.thumbs.cost()), previews, full)
}
@@ -1210,20 +1211,21 @@ mod tests {
let mut s = crate::Session::with_demo();
let id = s.active().unwrap();
let mut job = s.render_job(id, 64, 64, false, true).unwrap();
+ let loaded_info = info.clone();
job.source = SourceRef::File {
path: "synthetic.arw".into(),
max_edge: 64,
loader: Some(Arc::new(move |_, _| {
let mut image = Rgb32f::new(64, 64);
image.data.fill([0.1; 3]);
- Ok((image, info))
+ Ok((image, loaded_info.clone()))
})),
fallback: None,
};
job.info = SourceInfo::default(); // Header facts cannot override decoder facts.
job.settings = Arc::new(DevelopSettings::default());
let r = job.clone().run();
- assert_eq!(r.loaded.as_ref().unwrap().info, Some(info));
+ assert_eq!(r.loaded.as_ref().unwrap().info.as_ref(), Some(&info));
let expected = lightcraft_pipeline::render(&r.loaded.as_ref().unwrap().image, &info, &job.settings, &job.request);
assert_eq!(r.rendered.as_ref().unwrap().image.data, expected.image.data);
s.accept(&r);
diff --git a/crates/gpu/src/render.rs b/crates/gpu/src/render.rs
index 60ffd8065..7e7ac8a3e 100644
--- a/crates/gpu/src/render.rs
+++ b/crates/gpu/src/render.rs
@@ -717,7 +717,7 @@ pub fn render(
Err(m) => {
let img = host.lin.get_or_insert_with(|| Arc::new(cx.read_rgb(&lin, w, h))).clone();
let l = host.log_l.get_or_insert_with(|| Arc::new(cx.read_plane(&prep.log_l, w, h))).clone();
- lightcraft_pipeline::masks::evaluate_one(m, &plan.frame, w, h, &img, &l, s.light.exposure as f32)
+ lightcraft_pipeline::masks::evaluate_one(m, &plan.frame, w, h, &img, &l, s.light.exposure as f32, plan.mattes.as_deref())
}
});
lightcraft_pipeline::visualize::apply(&mut image, req.overlay, &plan, overlay_mask.as_ref());
@@ -1018,7 +1018,7 @@ fn masks(cx: &mut Cx<'_>, lin: &Buf, prep: &Prep, plan: &Plan<'_>, host: &mut Ho
// no kernel: evaluate on the CPU
let img = host.lin.get_or_insert_with(|| Arc::new(cx.read_rgb(lin, w, h))).clone();
let l = host.log_l.get_or_insert_with(|| Arc::new(cx.read_plane(&prep.log_l, w, h))).clone();
- let mut v = lightcraft_pipeline::masks::shape_alpha(&comp.shape, frame, w, h, &img, &l, ev);
+ let mut v = lightcraft_pipeline::masks::shape_alpha(&comp.shape, frame, w, h, &img, &l, ev, plan.mattes.as_deref());
if comp.invert {
v.data.iter_mut().for_each(|x| *x = 1.0 - *x);
}
diff --git a/crates/pipeline/src/lib.rs b/crates/pipeline/src/lib.rs
index 057fb42fa..76e80ed22 100644
--- a/crates/pipeline/src/lib.rs
+++ b/crates/pipeline/src/lib.rs
@@ -55,7 +55,7 @@ use lightcraft_raster::{Histogram, Plane, Rgb32f, Rgba8, par_rows};
pub use tone::ToneMap;
/// Facts about the source the settings are interpreted against.
-#[derive(Clone, Copy, Debug, PartialEq)]
+#[derive(Clone, Debug, PartialEq)]
pub struct SourceInfo {
/// Lens corrections embedded in the file (DNG opcodes), relative to the EXIF-oriented source.
pub lens: Option<lightcraft_develop::EmbeddedLens>,
@@ -67,11 +67,13 @@ pub struct SourceInfo {
/// No measured camera illuminant: WB adjustments are relative to the camera's rendered look.
pub relative_wb: bool,
pub camera_tone: Option<tone::CameraTone>,
+ /// Segmentation mattes stored in the file (DNG semantic masks): AI masks use them.
+ pub mattes: Option<Arc<masks::Mattes>>,
}
impl Default for SourceInfo {
fn default() -> Self {
- Self { raw: false, as_shot_temp: 6500.0, as_shot_tint: 0.0, lens: None, relative_wb: false, camera_tone: None }
+ Self { raw: false, as_shot_temp: 6500.0, as_shot_tint: 0.0, lens: None, relative_wb: false, camera_tone: None, mattes: None }
}
}
@@ -332,6 +334,8 @@ pub struct Plan<'a> {
/// [`spots::window_for_reads`]); this is the requested window inside the rendered one, which
/// the result is cut to.
pub keep: Option<PixelWindow>,
+ /// The source's segmentation mattes ([`SourceInfo::mattes`]).
+ pub mattes: Option<Arc<masks::Mattes>>,
}
/// Long edge of the small render a windowed render estimates the whole frame's airlight from.
@@ -392,7 +396,7 @@ pub fn plan<'a>(src: &Rgb32f, info: &SourceInfo, s: &'a DevelopSettings, req: &R
[d.nr_luminance, d.nr_detail, d.nr_color, d.nr_color_detail, d.nr_color_smoothness].map(f64::to_bits),
src_long,
));
- Plan { settings, frame, w, h, px_per_long, src_long, geo, lin_key, eyes, fixed_air, keep }
+ Plan { settings, frame, w, h, px_per_long, src_long, geo, lin_key, eyes, fixed_air, keep, mattes: info.mattes.clone() }
}
/// Whether the scene-linear stage needs work only the CPU does (defringe, spot removal).
@@ -474,7 +478,7 @@ fn render_impl(src: Src<'_>, info: &SourceInfo, s: &DevelopSettings, req: &Rende
Some(p) if p.key == lin_key => p,
_ => local::Planes { key: lin_key, ..Default::default() },
};
- let mut prep = local::prepare(lin.clone(), s, frame, px_per_long, req.quality, &mut planes);
+ let mut prep = local::prepare(lin.clone(), s, frame, px_per_long, req.quality, &mut planes, plan.mattes.as_deref());
if let Some(a) = plan.fixed_air {
prep.air = a;
}
@@ -543,7 +547,7 @@ fn overlay_alpha(o: Overlay, plan: &Plan<'_>, prep: &Prepared) -> Option<Plane>
return Some(e.alpha.clone());
}
let ev = plan.settings.light.exposure as f32;
- Some(masks::evaluate_one(m, &plan.frame, plan.w, plan.h, &prep.img, &prep.log_l, ev))
+ Some(masks::evaluate_one(m, &plan.frame, plan.w, plan.h, &prep.img, &prep.log_l, ev, plan.mattes.as_deref()))
}
/// Convenience: render a before/after pair side by side is up to the UI; this renders "before"
diff --git a/crates/pipeline/src/local.rs b/crates/pipeline/src/local.rs
index 61ec84820..898e69cfd 100644
--- a/crates/pipeline/src/local.rs
+++ b/crates/pipeline/src/local.rs
@@ -279,7 +279,16 @@ pub fn plane_sigmas(s: &DevelopSettings, px_per_long: f64, q: Quality) -> PlaneS
/// The spatial planes the per-pixel stage needs for `img` (white-balanced, before exposure),
/// reusing whatever `planes` already holds for it. Missing planes are computed side by side (each
/// one alone scales poorly: the guided filters work on small subsampled grids).
-pub(crate) fn prepare(img: Arc<Rgb32f>, s: &DevelopSettings, frame: &Frame, px_per_long: f64, q: Quality, planes: &mut Planes) -> Prepared {
+#[allow(clippy::too_many_arguments)]
+pub(crate) fn prepare(
+ img: Arc<Rgb32f>,
+ s: &DevelopSettings,
+ frame: &Frame,
+ px_per_long: f64,
+ q: Quality,
+ planes: &mut Planes,
+ mattes: Option<&masks::Mattes>,
+) -> Prepared {
let log_l = planes.log_l.get_or_insert_with(|| Arc::new(timed("log_l", || img.map(log_lum)))).clone();
let PlaneSigmas { base: base_sigma, clarity: clarity_sigma, texture: texture_sigma, dark: dark_sigma, chroma: chroma_sigma } =
plane_sigmas(s, px_per_long, q);
@@ -324,7 +333,7 @@ pub(crate) fn prepare(img: Arc<Rgb32f>, s: &DevelopSettings, frame: &Frame, px_p
None => (None, 1.0),
};
let ev = s.light.exposure as f32;
- let masks = timed("masks", || masks::evaluate(&s.masks, frame, img.width, img.height, &img, &log_l, ev));
+ let masks = timed("masks", || masks::evaluate(&s.masks, frame, img.width, img.height, &img, &log_l, ev, mattes));
Prepared { img, log_l, base, clarity_blur, texture_blur, dark, chroma_blur, air, masks, px_per_long }
}
diff --git a/crates/pipeline/src/masks.rs b/crates/pipeline/src/masks.rs
index aae261c5a..4777c8782 100644
--- a/crates/pipeline/src/masks.rs
+++ b/crates/pipeline/src/masks.rs
@@ -2,14 +2,96 @@
//!
//! Shapes are defined in normalized oriented-image coordinates; sizes are in "long-edge units".
//! Components combine with Add (max), Subtract (a·(1−c)) and Intersect (a·c).
+//!
+//! AI shapes (Sky, Subject, Background, People) use the segmentation [`Mattes`] the source carries
+//! (DNG semantic masks, e.g. iPhone ProRAW's) when it has a matching one, else a heuristic.
use lightcraft_develop::{BrushStroke, LocalAdjustments, Mask, MaskOp, MaskShape, SegMask};
-use lightcraft_geom::Point;
-use lightcraft_raster::{Plane, Rgb32f};
+use lightcraft_geom::{Orientation, Point};
+use lightcraft_raster::{Image, Plane, Rgb32f};
use crate::for_rows;
use crate::geometry::Frame;
+/// What an embedded segmentation matte selects.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum MatteKind {
+ Sky,
+ /// A whole person (all people in the photo).
+ Person,
+ Skin,
+ Hair,
+ Teeth,
+ Glasses,
+}
+
+/// Segmentation mattes that came with the source: 0..255 alpha planes, each covering the whole
+/// (EXIF-oriented) source image at its own resolution. Several mattes of one kind (e.g. one per
+/// person) are combined with max.
+#[derive(Clone, Default, PartialEq)]
+pub struct Mattes {
+ mattes: Vec<(MatteKind, Image<u8>)>,
+}
+
+impl std::fmt::Debug for Mattes {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_list().entries(self.mattes.iter().map(|(k, m)| (k, m.width, m.height))).finish()
+ }
+}
+
+impl Mattes {
+ /// Add a matte (empty ones are ignored).
+ pub fn push(&mut self, kind: MatteKind, matte: Image<u8>) {
+ if matte.width > 0 && matte.height > 0 && matte.data.len() == matte.width * matte.height {
+ self.mattes.push((kind, matte));
+ }
+ }
+
+ pub fn is_empty(&self) -> bool {
+ self.mattes.is_empty()
+ }
+
+ /// Memory held (bytes).
+ pub fn bytes(&self) -> usize {
+ self.mattes.iter().map(|(_, m)| m.data.len()).sum()
+ }
+
+ fn of(&self, kinds: &[MatteKind]) -> Vec<&Image<u8>> {
+ self.mattes.iter().filter(|(k, _)| kinds.contains(k)).map(|(_, m)| m).collect()
+ }
+
+ /// The mattes a shape uses instead of its heuristic (empty: none fits).
+ fn for_shape(&self, shape: &MaskShape) -> Vec<&Image<u8>> {
+ use MatteKind::*;
+ // the whole person: a person matte, else the union of the parts
+ let person = || {
+ let p = self.of(&[Person]);
+ if p.is_empty() { self.of(&[Skin, Hair, Teeth, Glasses]) } else { p }
+ };
+ match shape {
+ MaskShape::Sky => self.of(&[Sky]),
+ MaskShape::Subject => person(),
+ MaskShape::People { parts, .. } => {
+ let wanted: Vec<MatteKind> = parts
+ .iter()
+ .filter_map(|p| {
+ let p = p.to_ascii_lowercase();
+ [("skin", Skin), ("hair", Hair), ("teeth", Teeth), ("glasses", Glasses)]
+ .into_iter()
+ .find(|(n, _)| p.contains(n))
+ .map(|(_, k)| k)
+ })
+ .collect();
+ let whole =
+ parts.is_empty() || parts.iter().any(|p| matches!(p.to_ascii_lowercase().as_str(), "person" | "entire person" | "entireperson"));
+ // parts without a matte (lips, clothes, …) keep the heuristic
+ if whole { person() } else { self.of(&wanted) }
+ }
+ _ => Vec::new(),
+ }
+ }
+}
+
pub struct Evaluated {
/// The mask's id ([`Mask::id`]).
pub id: u32,
@@ -23,22 +105,24 @@ fn smooth(e0: f32, e1: f32, x: f32) -> f32 {
t * t * (3.0 - 2.0 * t)
}
-/// The visible masks with components, evaluated in order.
-pub fn evaluate(masks: &[Mask], frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32) -> Vec<Evaluated> {
+/// The visible masks with components, evaluated in order (`mattes`: the source's, see [`Mattes`]).
+#[allow(clippy::too_many_arguments)]
+pub fn evaluate(masks: &[Mask], frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32, mattes: Option<&Mattes>) -> Vec<Evaluated> {
masks
.iter()
.filter(|m| m.visible && !m.components.is_empty())
- .map(|m| Evaluated { id: m.id, alpha: evaluate_one(m, frame, w, h, img, log_l, ev), adjust: m.adjust })
+ .map(|m| Evaluated { id: m.id, alpha: evaluate_one(m, frame, w, h, img, log_l, ev, mattes), adjust: m.adjust })
.collect()
}
/// The alpha plane of mask `m` (whether visible or not): its components combined, inverted and
/// scaled by its amount.
-pub fn evaluate_one(m: &Mask, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32) -> Plane {
+#[allow(clippy::too_many_arguments)]
+pub fn evaluate_one(m: &Mask, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32, mattes: Option<&Mattes>) -> Plane {
let mut alpha = Plane::new(w, h);
let mut first = true;
for comp in &m.components {
- let mut c = shape_alpha(&comp.shape, frame, w, h, img, log_l, ev);
+ let mut c = shape_alpha(&comp.shape, frame, w, h, img, log_l, ev, mattes);
if comp.invert {
c.data.iter_mut().for_each(|v| *v = 1.0 - *v);
}
@@ -88,7 +172,13 @@ fn for_each_pos(frame: &Frame, w: usize, h: usize, out: &mut Plane, f: impl Fn(P
});
}
-pub fn shape_alpha(shape: &MaskShape, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32) -> Plane {
+/// The alpha plane of one component shape. AI shapes use the source's matching `mattes`, if any.
+#[allow(clippy::too_many_arguments)]
+pub fn shape_alpha(shape: &MaskShape, frame: &Frame, w: usize, h: usize, img: &Rgb32f, log_l: &Plane, ev: f32, mattes: Option<&Mattes>) -> Plane {
+ let fitting = mattes.map(|m| m.for_shape(shape)).unwrap_or_default();
+ if !fitting.is_empty() {
+ return matte_alpha(&fitting, frame, w, h);
+ }
let mut out = Plane::new(w, h);
let to_long = |p: Point| frame.norm_to_long(p);
// `img`/`log_l` are before exposure: range masks select on the exposed values.
@@ -180,7 +270,7 @@ pub fn shape_alpha(shape: &MaskShape, frame: &Frame, w: usize, h: usize, img: &R
smooth_plane(&mut out, 0.015 * frame_px(frame, w));
}
MaskShape::Background => {
- let mut s = shape_alpha(&MaskShape::Subject, frame, w, h, img, log_l, ev);
+ let mut s = shape_alpha(&MaskShape::Subject, frame, w, h, img, log_l, ev, mattes);
s.data.iter_mut().for_each(|v| *v = 1.0 - *v);
out = s;
}
@@ -252,6 +342,71 @@ fn smooth_plane(p: &mut Plane, sigma: f32) {
*p = lightcraft_raster::blur::gaussian(p, sigma.max(0.5));
}
+/// The union (max) of `mattes` (0..255, over the EXIF-oriented source) at output resolution,
+/// sampled like the image: user orientation, lens / perspective warp, crop and flips; area-averaged
+/// first when the output is much smaller than the matte, bilinear otherwise.
+fn matte_alpha(mattes: &[&Image<u8>], frame: &Frame, w: usize, h: usize) -> Plane {
+ let mut out = Plane::new(w, h);
+ let o2t = frame.out_to_oriented(w, h);
+ let ppl = frame.px_per_long(w).max(1e-9);
+ for m in mattes {
+ let oriented =
+ if frame.orient == Orientation::Normal { std::borrow::Cow::Borrowed(*m) } else { std::borrow::Cow::Owned(m.oriented(frame.orient)) };
+ // matte px per output px
+ let k = oriented.width as f64 / frame.ow * frame.ow.max(frame.oh) / ppl;
+ let base = box_down(&oriented, if k >= 2.0 { (k.floor() as usize).min(64) } else { 1 });
+ let (sx, sy) = ((base.width as f64 / frame.ow) as f32, (base.height as f64 / frame.oh) as f32);
+ for_rows(&mut out.data, w, |y, row| {
+ for (x, v) in row.iter_mut().enumerate() {
+ let s = match &frame.warp {
+ Some(wp) => {
+ let Some((_, s)) = wp.frame(&o2t, x, y) else {
+ continue;
+ };
+ s
+ }
+ None => o2t.apply(Point::new(x as f64 + 0.5, y as f64 + 0.5)),
+ };
+ *v = v.max(bilinear(&base, s.x as f32 * sx, s.y as f32 * sy) / 255.0);
+ }
+ });
+ }
+ out
+}
+
+/// `m` area-averaged by `f × f` (0..255 values kept); `f == 1` converts only.
+fn box_down(m: &Image<u8>, f: usize) -> Plane {
+ let f = f.max(1);
+ let (bw, bh) = (m.width.div_ceil(f).max(1), m.height.div_ceil(f).max(1));
+ let mut out = Plane::new(bw, bh);
+ for_rows(&mut out.data, bw, |by, row| {
+ let rows = by * f..((by + 1) * f).min(m.height);
+ for (bx, v) in row.iter_mut().enumerate() {
+ let cols = bx * f..((bx + 1) * f).min(m.width);
+ let (mut sum, mut n) = (0u32, 0u32);
+ for y in rows.clone() {
+ for x in cols.clone() {
+ sum += m.data.get(y * m.width + x).copied().unwrap_or(0) as u32;
+ n += 1;
+ }
+ }
+ *v = sum as f32 / n.max(1) as f32;
+ }
+ });
+ out
+}
+
+/// Bilinear sample at continuous pixel coordinates (pixel centres at +0.5), edges extended.
+fn bilinear(p: &Plane, x: f32, y: f32) -> f32 {
+ let (fx, fy) = (x - 0.5, y - 0.5);
+ let (x0, y0) = (fx.floor(), fy.floor());
+ let (tx, ty) = (fx - x0, fy - y0);
+ let (x0, y0) = (x0 as isize, y0 as isize);
+ let a = p.get_clamped(x0, y0) + (p.get_clamped(x0 + 1, y0) - p.get_clamped(x0, y0)) * tx;
+ let b = p.get_clamped(x0, y0 + 1) + (p.get_clamped(x0 + 1, y0 + 1) - p.get_clamped(x0, y0 + 1)) * tx;
+ a + (b - a) * ty
+}
+
/// A rough display mapping for colour picking (so samples taken on screen match).
pub(crate) fn tonemap_for_select(c: [f32; 3]) -> [f32; 3] {
c.map(|v| v / (1.0 + v))
@@ -436,7 +591,7 @@ mod tests {
let f = frame(100, 50);
let img = Rgb32f::new(100, 50);
let l = Plane::new(100, 50);
- let a = shape_alpha(&MaskShape::Linear { start: Point::new(0.0, 0.0), end: Point::new(1.0, 0.0) }, &f, 100, 50, &img, &l, 0.0);
+ let a = shape_alpha(&MaskShape::Linear { start: Point::new(0.0, 0.0), end: Point::new(1.0, 0.0) }, &f, 100, 50, &img, &l, 0.0, None);
assert!(a.get(1, 25) > 0.99 && a.get(98, 25) < 0.01);
assert!((a.get(50, 25) - 0.5).abs() < 0.05);
}
@@ -452,7 +607,7 @@ mod tests {
MaskShape::Prompt { text: "left".into(), seg: Some(seg.clone()), detail: vec![], edge: 0.0 },
] {
for (w, h) in [(40, 20), (400, 200)] {
- let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
+ let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None);
assert!(a.get(1, h / 2) > 0.99 && a.get(w - 2, h / 2) < 0.01, "{w}×{h}");
if w >= 400 {
let soft = (0..w).filter(|x| (0.05..0.95).contains(&a.get(*x, h / 2))).count();
@@ -465,7 +620,7 @@ mod tests {
d.crop.geometry.rect = lightcraft_geom::Rect::from_center(Point::new(0.75, 0.5), 0.5, 1.0);
let f = Frame::new(40, 20, &d, true);
let shape = MaskShape::Prompt { text: "left".into(), seg: Some(seg), detail: vec![], edge: 0.0 };
- let a = shape_alpha(&shape, &f, 20, 20, &Rgb32f::new(20, 20), &Plane::new(20, 20), 0.0);
+ let a = shape_alpha(&shape, &f, 20, 20, &Rgb32f::new(20, 20), &Plane::new(20, 20), 0.0, None);
assert!(a.data.iter().all(|v| *v < 0.05));
}
@@ -478,7 +633,7 @@ mod tests {
let patch = SegMask::from_logits_in(side, &l, [0.5, 0.0, 1.0, 1.0]);
let shape = MaskShape::Object { hint: vec![Point::new(0.6, 0.5)], exclude: vec![], seg: Some(coarse), detail: vec![patch], edge: 0.0 };
let (w, h) = (200, 100);
- let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
+ let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None);
assert!(a.get(20, 50) < 0.01, "outside the patch: the coarse mask");
assert!(a.get(120, 50) > 0.99, "inside the patch, its selected half");
assert!(a.get(190, 50) < 0.01, "inside the patch, its unselected half");
@@ -494,7 +649,7 @@ mod tests {
let shape = MaskShape::Object { hint: vec![Point::new(0.2, 0.5)], exclude: vec![], seg: None, detail: vec![patch; 20_000], edge: 0.0 };
let (w, h) = (64, 32);
let t = std::time::Instant::now();
- let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
+ let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None);
assert!(a.get(5, 16) > 0.99 && a.get(60, 16) < 0.01);
assert!(t.elapsed() < std::time::Duration::from_secs(20), "{:?}", t.elapsed());
}
@@ -507,7 +662,7 @@ mod tests {
let (w, h) = (400, 40);
let soft_px = |edge: f64| {
let shape = MaskShape::Prompt { text: "x".into(), seg: Some(seg.clone()), detail: vec![], edge };
- let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
+ let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0, None);
assert!(a.get(2, h / 2) > 0.9 && a.get(w - 3, h / 2) < 0.1, "still the same selection at {edge}");
(0..w).filter(|x| (0.1..0.9).contains(&a.get(*x, h / 2))).count()
};
@@ -524,7 +679,7 @@ mod tests {
MaskShape::Prompt { text: "sky".into(), seg: None, detail: vec![], edge: 0.0 },
MaskShape::Prompt { text: "sky".into(), seg: Some(SegMask { side: 4, data: "damaged!".into(), rect: None }), detail: vec![], edge: 0.0 },
] {
- assert!(shape_alpha(&shape, &f, 30, 30, &img, &l, 0.0).data.iter().all(|v| *v == 0.0));
+ assert!(shape_alpha(&shape, &f, 30, 30, &img, &l, 0.0, None).data.iter().all(|v| *v == 0.0));
}
}
@@ -534,10 +689,10 @@ mod tests {
let img = Rgb32f::new(100, 100);
let l = Plane::new(100, 100);
let shape = MaskShape::Radial { center: Point::new(0.5, 0.5), rx: 0.2, ry: 0.2, angle: 0.0, feather: 20.0, invert: false };
- let a = shape_alpha(&shape, &f, 100, 100, &img, &l, 0.0);
+ let a = shape_alpha(&shape, &f, 100, 100, &img, &l, 0.0, None);
assert!(a.get(50, 50) > 0.99 && a.get(5, 5) < 0.01);
let m = Mask { components: vec![MaskComponent { name: None, op: MaskOp::Add, invert: true, shape }], ..Default::default() };
- let e = evaluate(&[m], &f, 100, 100, &img, &l, 0.0);
+ let e = evaluate(&[m], &f, 100, 100, &img, &l, 0.0, None);
assert!(e[0].alpha.get(50, 50) < 0.01);
}
@@ -552,7 +707,7 @@ mod tests {
components: vec![MaskComponent { name: None, op: MaskOp::Add, invert: false, shape: MaskShape::Brush { strokes: vec![stroke, erase] } }],
..Default::default()
};
- let e = evaluate(&[m], &f, 200, 100, &img, &l, 0.0);
+ let e = evaluate(&[m], &f, 200, 100, &img, &l, 0.0, None);
let a = &e[0].alpha;
assert!(a.get(40, 50) > 0.9, "{}", a.get(40, 50));
assert!(a.get(100, 50) < 0.05, "erased centre {}", a.get(100, 50));
@@ -574,8 +729,8 @@ mod tests {
..Default::default()
};
let comp = |auto| MaskShape::Brush { strokes: vec![stroke(auto)] };
- let plain = shape_alpha(&comp(false), &f, w, h, &img, &l, 0.0);
- let auto = shape_alpha(&comp(true), &f, w, h, &img, &l, 0.0);
+ let plain = shape_alpha(&comp(false), &f, w, h, &img, &l, 0.0, None);
+ let auto = shape_alpha(&comp(true), &f, w, h, &img, &l, 0.0, None);
// without Auto Mask the brush spills over the edge; with it, it stays on the dark side
assert!(plain.get(105, 50) > 0.9, "{}", plain.get(105, 50));
assert!(auto.get(105, 50) < 0.05, "spill {}", auto.get(105, 50));
@@ -585,8 +740,8 @@ mod tests {
// on a flat area Auto Mask paints like the plain brush
let flat = Rgb32f::from_fn(w, h, |_, _| [0.2; 3]);
let fl = flat.map(crate::local::log_lum);
- let a = shape_alpha(&comp(true), &f, w, h, &flat, &fl, 0.0);
- let b = shape_alpha(&comp(false), &f, w, h, &flat, &fl, 0.0);
+ let a = shape_alpha(&comp(true), &f, w, h, &flat, &fl, 0.0, None);
+ let b = shape_alpha(&comp(false), &f, w, h, &flat, &fl, 0.0, None);
assert!((a.get(94, 50) - b.get(94, 50)).abs() < 0.02, "{} vs {}", a.get(94, 50), b.get(94, 50));
}
@@ -609,7 +764,7 @@ mod tests {
};
let refined = Mask { refine: 100.0, ..soft.clone() };
let step = |m: &Mask| {
- let a = evaluate_one(m, &f, w, h, &img, &log_l, 0.0);
+ let a = evaluate_one(m, &f, w, h, &img, &log_l, 0.0, None);
a.data[40 * w + 63] - a.data[40 * w + 56]
};
let (plain, sharp) = (step(&soft), step(&refined));
@@ -640,7 +795,7 @@ mod tests {
}
let alpha = |refine| {
let shape = MaskShape::ColorRange { samples: vec![sample], refine };
- let a = shape_alpha(&shape, &f, w, h, &img, &log_l, 0.0);
+ let a = shape_alpha(&shape, &f, w, h, &img, &log_l, 0.0, None);
(a.get(30, 20), a.get(90, 20))
};
for refine in [0.0, 50.0, 85.0] {
@@ -653,4 +808,63 @@ mod tests {
let (l, r) = alpha(100.0);
assert!(l > 0.95 && r < 0.3, "refine 100: {l} vs {r}");
}
+
+ /// A half-resolution matte selecting the left `frac` of a `w × h` source.
+ fn left_matte(w: usize, h: usize, frac: f64) -> Image<u8> {
+ Image::from_fn(w / 2, h / 2, |x, _| if (x as f64 + 0.5) < frac * (w / 2) as f64 { 255 } else { 0 })
+ }
+
+ /// The Sky mask follows the source's sky matte where it has one (here: the bottom-left of a
+ /// uniformly bright blue photo, where the heuristic sees no sky, and not its top right, where
+ /// it does), through user orientation, at any output size; without one it keeps the heuristic.
+ #[test]
+ fn sky_uses_the_sources_matte() {
+ let (w, h) = (200, 100);
+ let img = Rgb32f::filled(w, h, [0.5, 0.7, 1.4]);
+ let l = img.map(crate::local::log_lum);
+ let mut mattes = Mattes::default();
+ mattes.push(MatteKind::Sky, left_matte(w, h, 0.5));
+ let f = frame(w, h);
+ let heuristic = shape_alpha(&MaskShape::Sky, &f, w, h, &img, &l, 0.0, None);
+ let matte = shape_alpha(&MaskShape::Sky, &f, w, h, &img, &l, 0.0, Some(&mattes));
+ assert!(heuristic.get(150, 5) > 0.9 && heuristic.get(25, 90) < 0.1, "{} {}", heuristic.get(150, 5), heuristic.get(25, 90));
+ assert!(matte.get(150, 5) < 0.01 && matte.get(25, 90) > 0.99, "{} {}", matte.get(150, 5), matte.get(25, 90));
+ // mattes of other kinds leave the heuristic alone
+ let mut hair = Mattes::default();
+ hair.push(MatteKind::Hair, left_matte(w, h, 0.5));
+ assert_eq!(shape_alpha(&MaskShape::Sky, &f, w, h, &img, &l, 0.0, Some(&hair)), heuristic);
+ // rotated 90° clockwise, the source's left half is the top half; a preview and a larger
+ // render agree
+ let s = lightcraft_develop::DevelopSettings { orientation: Orientation::Rotate90, ..Default::default() };
+ let f = Frame::new(w, h, &s, true);
+ for (ow, oh) in [(100, 200), (25, 50)] {
+ let small = Rgb32f::filled(ow, oh, [0.5, 0.7, 1.4]);
+ let a = shape_alpha(&MaskShape::Sky, &f, ow, oh, &small, &small.map(crate::local::log_lum), 0.0, Some(&mattes));
+ let at = |x: f64, y: f64| a.get((x * ow as f64) as usize, (y * oh as f64) as usize);
+ assert!(at(0.5, 0.2) > 0.99 && at(0.5, 0.8) < 0.01, "{ow}×{oh}: {} {}", at(0.5, 0.2), at(0.5, 0.8));
+ }
+ }
+
+ /// Subject, Background and People use person mattes: the whole person is the union of the
+ /// parts; a People part without a matte keeps the heuristic.
+ #[test]
+ fn people_and_subject_use_person_mattes() {
+ let (w, h) = (200, 100);
+ let img = Rgb32f::filled(w, h, [0.2; 3]);
+ let l = img.map(crate::local::log_lum);
+ let mut mattes = Mattes::default();
+ mattes.push(MatteKind::Skin, left_matte(w, h, 0.25));
+ let right: Image<u8> = Image::from_fn(w / 2, h / 2, |x, _| if x >= 75 { 255 } else { 0 });
+ mattes.push(MatteKind::Hair, right);
+ let f = frame(w, h);
+ let eval = |shape: &MaskShape| shape_alpha(shape, &f, w, h, &img, &l, 0.0, Some(&mattes));
+ let subject = eval(&MaskShape::Subject);
+ assert!(subject.get(10, 50) > 0.99 && subject.get(190, 50) > 0.99 && subject.get(100, 50) < 0.01);
+ let background = eval(&MaskShape::Background);
+ assert!(background.get(100, 50) > 0.99 && background.get(10, 50) < 0.01);
+ let hair = eval(&MaskShape::People { person: 0, parts: vec!["Hair".into()] });
+ assert!(hair.get(190, 50) > 0.99 && hair.get(10, 50) < 0.01);
+ let lips = MaskShape::People { person: 0, parts: vec!["Lips".into()] };
+ assert_eq!(eval(&lips), shape_alpha(&lips, &f, w, h, &img, &l, 0.0, None));
+ }
}
diff --git a/crates/raw/src/lib.rs b/crates/raw/src/lib.rs
index b9bc029ad..82f1bdf8d 100644
--- a/crates/raw/src/lib.rs
+++ b/crates/raw/src/lib.rs
@@ -2,7 +2,8 @@
//!
//! - [`probe`] recognises raw containers; [`decode`] turns a file into a [`RawImage`] (sensor data + everything
//! needed to render it: CFA, black/white levels, active area, default crop, orientation, DNG colour tags,
-//! opcode lists, [`Metadata`]); [`embedded_preview`] returns the largest embedded JPEG (or DNG 1.7 JPEG XL) preview.
+//! opcode lists, [`Metadata`]); [`embedded_preview`] returns the largest embedded JPEG (or DNG 1.7 JPEG XL) preview;
+//! [`semantic_masks`] reads a DNG's semantic masks (segmentation mattes, e.g. iPhone ProRAW's sky matte).
//! - [`RawImage::normalized`] subtracts black, scales white to 1.0 and crops to the active area (applying DNG
//! `OpcodeList1`/`OpcodeList2`); [`demosaic`] turns CFA data into camera-RGB [`Rgb32f`];
//! [`RawImage::develop`] does all of it plus `OpcodeList3` and the default crop; [`RawImage::develop_binned`]
@@ -36,6 +37,7 @@ pub mod ljpeg;
pub mod opcodes;
mod preview;
pub mod profile;
+pub mod semantic;
mod tiffraw;
mod unpack;
mod vendor;
@@ -48,6 +50,7 @@ pub use lightcraft_meta::Metadata;
pub use lightcraft_raster::Rgb32f;
pub use opcodes::{Opcode, OpcodeLists};
pub use preview::{PreviewColorSpace, embedded_preview, embedded_preview_color_space};
+pub use semantic::{SemanticMask, semantic_masks};
use lightcraft_color::Xy;
use lightcraft_tiff::{Tiff, TiffError};
diff --git a/crates/raw/src/semantic.rs b/crates/raw/src/semantic.rs
new file mode 100644
index 000000000..e9af87c08
--- /dev/null
+++ b/crates/raw/src/semantic.rs
@@ -0,0 +1,136 @@
+//! DNG semantic masks (DNG 1.6+, chapter "Semantic Masks"): single-channel segmentation mattes
+//! stored in their own IFDs (NewSubFileType 10004.H, PhotometricInterpretation 52527) with a
+//! `SemanticName`, an optional `SemanticInstanceID` and an optional `MaskSubArea`. iPhone ProRAW
+//! stores e.g. a sky matte this way.
+//!
+//! [`semantic_masks`] reads them separately from [`crate::decode`] (only callers that use them
+//! pay for them); malformed or unsupported mask IFDs are skipped.
+
+use crate::Rect;
+use crate::tiffraw::{Packing, read_image};
+use crate::{RawData, Result};
+use lightcraft_raster::Image;
+use lightcraft_tiff::Tiff;
+use lightcraft_tiff::tags::{self as t, photometric};
+
+/// At most this many mask IFDs are read from one file.
+pub const MAX_MASKS: usize = 16;
+/// Masks with more samples than this are skipped (a mask is a low-resolution matte; this is
+/// 8192 × 4096).
+pub const MAX_MASK_SAMPLES: usize = 1 << 25;
+
+/// Where a cropped mask sits in the uncropped mask (DNG `MaskSubArea`). The uncropped mask
+/// (`full_width × full_height`) spans the main image's active area.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct MaskSubArea {
+ pub top: u32,
+ pub left: u32,
+ pub full_width: u32,
+ pub full_height: u32,
+}
+
+/// One semantic mask: `width × height` values, 0 = not part of the attribute, 65535 = fully part.
+#[derive(Clone, Debug, PartialEq)]
+pub struct SemanticMask {
+ /// `SemanticName`, e.g. `urn:com:apple:photo:2020:aux:semanticskymatte`.
+ pub name: String,
+ pub instance_id: Option<String>,
+ pub width: usize,
+ pub height: usize,
+ /// `None`: the mask covers the whole active area.
+ pub sub_area: Option<MaskSubArea>,
+ pub data: Vec<u16>,
+}
+
+/// The semantic masks of a DNG, in file order (none for other files). Only the first
+/// [`MAX_MASKS`] mask IFDs are looked at, malformed ones included, so a file full of mask IFDs
+/// costs at most that many decodes.
+pub fn semantic_masks(bytes: &[u8]) -> Vec<SemanticMask> {
+ let Ok(tiff) = Tiff::parse(bytes) else { return Vec::new() };
+ tiff.all_ifds()
+ .into_iter()
+ .filter(|i| i.u16(t::PHOTOMETRIC) == Some(photometric::MASK) || i.u32(t::NEW_SUBFILE_TYPE) == Some(t::SUBFILE_SEMANTIC_MASK))
+ .take(MAX_MASKS)
+ .filter_map(|ifd| read_mask(bytes, &tiff, ifd).ok().flatten())
+ .collect()
+}
+
+fn read_mask(bytes: &[u8], tiff: &Tiff, ifd: &lightcraft_tiff::Ifd) -> Result<Option<SemanticMask>> {
+ let Some(name) = ifd.string(t::SEMANTIC_NAME).filter(|n| !n.is_empty()) else { return Ok(None) };
+ let info = ifd.image()?;
+ let (w, h) = (info.width as usize, info.height as usize);
+ if info.samples_per_pixel != 1 || w.checked_mul(h).is_none_or(|n| n > MAX_MASK_SAMPLES) {
+ return Ok(None);
+ }
+ let bits = info.bits() as u32;
+ let data: Vec<u16> = match read_image(bytes, &info, tiff.order, Packing::Msb)? {
+ RawData::U16(v) if (1..=16).contains(&bits) => {
+ let max = ((1u32 << bits) - 1) as f32;
+ if bits == 16 { v } else { v.into_iter().map(|s| (s as f32 / max * 65535.0).round().min(65535.0) as u16).collect() }
+ }
+ RawData::F32(v) => v.into_iter().map(|s| if s.is_finite() { (s.clamp(0.0, 1.0) * 65535.0).round() as u16 } else { 0 }).collect(),
+ RawData::U16(_) => return Ok(None),
+ };
+ if data.len() != w * h {
+ return Ok(None);
+ }
+ // the tag is ignored when the stored mask does not fit inside the uncropped one
+ let sub_area = match ifd.u64s(t::MASK_SUB_AREA).as_deref() {
+ Some(&[top, left, fw, fh]) => {
+ let fits = fw > 0 && fh > 0 && left.saturating_add(w as u64) <= fw && top.saturating_add(h as u64) <= fh && fw.max(fh) <= u32::MAX as u64;
+ fits.then_some(MaskSubArea { top: top as u32, left: left as u32, full_width: fw as u32, full_height: fh as u32 })
+ }
+ _ => None,
+ };
+ Ok(Some(SemanticMask { name, instance_id: ifd.string(t::SEMANTIC_INSTANCE_ID).filter(|s| !s.is_empty()), width: w, height: h, sub_area, data }))
+}
+
+impl SemanticMask {
+ /// The mask value (0..1) at a continuous position of the uncropped mask (pixel centres at
+ /// +0.5); 0 outside the stored (cropped) mask.
+ pub fn value_at(&self, x: f64, y: f64) -> f32 {
+ let (left, top) = self.sub_area.map_or((0.0, 0.0), |a| (a.left as f64, a.top as f64));
+ let (x, y) = (x - left, y - top);
+ if self.width == 0 || self.height == 0 || !(x >= 0.0 && y >= 0.0 && x <= self.width as f64 && y <= self.height as f64) {
+ return 0.0;
+ }
+ let (fx, fy) = ((x - 0.5).max(0.0), (y - 0.5).max(0.0));
+ let (x0, y0) = (fx.floor() as usize, fy.floor() as usize);
+ let (tx, ty) = ((fx - x0 as f64) as f32, (fy - y0 as f64) as f32);
+ let at = |x: usize, y: usize| {
+ let (x, y) = (x.min(self.width - 1), y.min(self.height - 1));
+ self.data.get(y * self.width + x).copied().unwrap_or(0) as f32 / 65535.0
+ };
+ let top = at(x0, y0) + (at(x0 + 1, y0) - at(x0, y0)) * tx;
+ let bottom = at(x0, y0 + 1) + (at(x0 + 1, y0 + 1) - at(x0, y0 + 1)) * tx;
+ top + (bottom - top) * ty
+ }
+
+ /// The mask over the developed image ([`crate::RawImage::develop`]: the default `crop` of the
+ /// `active_area`, before orientation), at the mask's own resolution, as 0..255 alpha. `None`
+ /// for an empty crop.
+ pub fn developed(&self, active_area: Rect, crop: Rect) -> Option<Image<u8>> {
+ let c = crop.clipped(active_area.width, active_area.height);
+ if c.width == 0 || c.height == 0 || self.width == 0 || self.height == 0 {
+ return None;
+ }
+ let (fw, fh) = self.sub_area.map_or((self.width, self.height), |a| (a.full_width as usize, a.full_height as usize));
+ // uncropped-mask px per active-area px
+ let (sx, sy) = (fw as f64 / active_area.width as f64, fh as f64 / active_area.height as f64);
+ let ow = ((c.width as f64 * sx).round() as usize).clamp(1, self.width.max(fw));
+ let oh = ((c.height as f64 * sy).round() as usize).clamp(1, self.height.max(fh));
+ if ow.checked_mul(oh).is_none_or(|n| n > MAX_MASK_SAMPLES) {
+ return None;
+ }
+ let (kx, ky) = (c.width as f64 / ow as f64, c.height as f64 / oh as f64);
+ let mut out = Image::new(ow, oh);
+ lightcraft_raster::par_rows(&mut out.data, ow, |y, row| {
+ let ay = c.y as f64 + (y as f64 + 0.5) * ky;
+ for (x, v) in row.iter_mut().enumerate() {
+ let ax = c.x as f64 + (x as f64 + 0.5) * kx;
+ *v = (self.value_at(ax * sx, ay * sy) * 255.0).round() as u8;
+ }
+ });
+ Some(out)
+ }
+}
diff --git a/crates/raw/tests/corpus.rs b/crates/raw/tests/corpus.rs
index 683381c40..53193f1f1 100644
--- a/crates/raw/tests/corpus.rs
+++ b/crates/raw/tests/corpus.rs
@@ -300,6 +300,33 @@ fn corpus_dngs_carry_profile_looks() {
eprintln!("{seen} DNGs with profile looks checked");
}
+/// iPhone ProRAW (`dng-apple-iphone12pro-proraw.dng`, iPhone 12 Pro, CC0 from raw.pixls.us, see
+/// `cargo xtask corpus`) carries a lossy-JPEG sky matte at half resolution as a DNG semantic mask:
+/// it decodes, and it is a sky. The matte is stored like the raw (landscape; the photo is shown
+/// rotated 90° clockwise), so the sky is on its left and the lake and shore on its right.
+#[test]
+fn corpus_proraw_sky_matte() {
+ let path = corpus_root().join("raw/dng-apple-iphone12pro-proraw.dng");
+ let Ok(bytes) = std::fs::read(&path) else {
+ eprintln!("skip: {} absent", path.display());
+ return;
+ };
+ let t0 = Instant::now();
+ let masks = lightcraft_raw::semantic_masks(&bytes);
+ eprintln!("semantic masks read in {:.1} ms", t0.elapsed().as_secs_f64() * 1e3);
+ let names: Vec<&str> = masks.iter().map(|m| m.name.as_str()).collect();
+ assert_eq!(names, ["urn:com:apple:photo:2020:aux:semanticskymatte"]);
+ let m = &masks[0];
+ assert_eq!((m.width, m.height, m.sub_area), (2016, 1512, None));
+ let cols = |x0: usize, x1: usize| {
+ let sum: f64 = (0..m.height).flat_map(|y| m.data[y * m.width + x0..y * m.width + x1].iter()).map(|&v| v as f64 / 65535.0).sum();
+ sum / ((x1 - x0) * m.height) as f64
+ };
+ let (left, right) = (cols(0, 100), cols(1916, 2016));
+ eprintln!("sky matte: left columns {left:.3}, right columns {right:.3}");
+ assert!(left > 0.5 && right < 0.02, "left {left}, right {right}");
+}
+
/// Issue #148: Sony ARWs from before ~2017 carry no plain white-balance, black-level or crop tags in the raw IFD.
/// White balance comes from the maker note's enciphered `Tag2010`, the black level from the encrypted `SR2SubIFD`
/// and the crop from `FullImageSize`; without them the RX100 III opened bright green. Expected values: the black
diff --git a/crates/raw/tests/dng.rs b/crates/raw/tests/dng.rs
index 9cbbb0ea3..02d4f82a3 100644
--- a/crates/raw/tests/dng.rs
+++ b/crates/raw/tests/dng.rs
@@ -824,3 +824,130 @@ fn gain_table_map_byte_orders() {
assert_eq!(img.color.profile.gain_table_map.map(|m| m.gains), read.then(|| vec![1.0, 2.0]), "{data:?} payload, {file:?} file");
}
}
+
+/// A semantic mask IFD (8-bit, uncompressed) named `name`.
+fn mask_ifd(w: usize, h: usize, name: &str, px: Vec<u8>) -> IfdBuilder {
+ let mut m = IfdBuilder::new();
+ m.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![t::SUBFILE_SEMANTIC_MASK]));
+ m.set(t::IMAGE_WIDTH, Value::Long(vec![w as u32]));
+ m.set(t::IMAGE_LENGTH, Value::Long(vec![h as u32]));
+ m.set(t::BITS_PER_SAMPLE, Value::Short(vec![8]));
+ m.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![1]));
+ m.set(t::PHOTOMETRIC, Value::Short(vec![photometric::MASK]));
+ m.set(t::COMPRESSION, Value::Short(vec![1]));
+ if !name.is_empty() {
+ m.set(t::SEMANTIC_NAME, Value::Ascii(name.into()));
+ }
+ m.set_image(ImageData::Strips { rows_per_strip: h as u32, strips: vec![px] });
+ m
+}
+
+/// Semantic masks: a cropped mask is placed by its `MaskSubArea` (zero outside it), mapped onto the
+/// developed image (default crop of the active area), and doesn't disturb the raw decode.
+#[test]
+fn semantic_masks_are_read_and_placed() {
+ use lightcraft_raw::semantic::MaskSubArea;
+ for order in [ByteOrder::Little, ByteOrder::Big] {
+ let bytes = profile_dng(order, |ifd0| {
+ // a 4 × 2 crop at (top 1, left 2) of an 8 × 4 mask over the 8 × 6 image
+ let mut sky = mask_ifd(4, 2, "sky", vec![10, 20, 30, 40, 50, 60, 70, 255]);
+ sky.set(t::MASK_SUB_AREA, Value::Long(vec![1, 2, 8, 4]));
+ ifd0.add_sub_ifd(sky);
+ let mut skin = mask_ifd(2, 2, "skin", vec![0, 255, 255, 0]);
+ skin.set(t::SEMANTIC_INSTANCE_ID, Value::Ascii("person_a".into()));
+ ifd0.add_sub_ifd(skin);
+ });
+ let raw = decode(&bytes).unwrap();
+ assert_eq!((raw.width, raw.height, raw.cpp), (8, 6, 3), "the masks are not the raw image");
+ let masks = lightcraft_raw::semantic_masks(&bytes);
+ assert_eq!(masks.len(), 2);
+ let (sky, skin) = (&masks[0], &masks[1]);
+ assert_eq!((sky.name.as_str(), sky.instance_id.as_deref(), sky.width, sky.height), ("sky", None, 4, 2));
+ assert_eq!(sky.sub_area, Some(MaskSubArea { top: 1, left: 2, full_width: 8, full_height: 4 }));
+ assert_eq!((sky.data[0], sky.data[7]), (10 * 257, 65535));
+ assert_eq!((skin.name.as_str(), skin.instance_id.as_deref(), skin.sub_area), ("skin", Some("person_a"), None));
+ // positions are in uncropped-mask pixels: zero outside the stored crop
+ assert!((sky.value_at(2.5, 1.5) - 10.0 / 255.0).abs() < 1e-4);
+ assert!((sky.value_at(5.5, 2.5) - 1.0).abs() < 1e-4);
+ assert_eq!((sky.value_at(1.5, 1.5), sky.value_at(2.5, 0.5), sky.value_at(6.5, 1.5)), (0.0, 0.0, 0.0));
+ // over the whole image: the uncropped mask's 8 × 4 grid
+ let full = sky.developed(Rect::new(0, 0, 8, 6), Rect::new(0, 0, 8, 6)).unwrap();
+ assert_eq!((full.width, full.height), (8, 4));
+ assert_eq!((full.get(2, 1), full.get(5, 2), full.get(1, 1), full.get(6, 2), full.get(2, 0)), (10, 255, 0, 0, 0));
+ // a default crop keeping columns 2..6 of the image: the stored crop fills it
+ let cropped = sky.developed(Rect::new(0, 0, 8, 6), Rect::new(2, 0, 4, 6)).unwrap();
+ assert_eq!((cropped.width, cropped.height), (4, 4));
+ assert_eq!((cropped.get(0, 1), cropped.get(3, 2), cropped.get(0, 0)), (10, 255, 0));
+ }
+}
+
+/// Malformed semantic masks are skipped (never a panic), and so is the `MaskSubArea` of a mask it
+/// doesn't fit; the raw image still decodes.
+#[test]
+fn malformed_semantic_masks_are_skipped() {
+ let bytes = profile_dng(ByteOrder::Little, |ifd0| {
+ ifd0.add_sub_ifd(mask_ifd(2, 2, "", vec![1, 2, 3, 4])); // no SemanticName
+ let mut rgb = mask_ifd(2, 2, "rgb", vec![0; 12]);
+ rgb.set(t::SAMPLES_PER_PIXEL, Value::Short(vec![3]));
+ rgb.set(t::BITS_PER_SAMPLE, Value::Short(vec![8, 8, 8]));
+ ifd0.add_sub_ifd(rgb);
+ let mut huge = mask_ifd(2, 2, "huge", vec![0; 4]);
+ huge.set(t::IMAGE_WIDTH, Value::Long(vec![100_000]));
+ huge.set(t::IMAGE_LENGTH, Value::Long(vec![100_000]));
+ ifd0.add_sub_ifd(huge);
+ let mut tiny = mask_ifd(2, 2, "tiny data", vec![0; 4]);
+ tiny.set(t::IMAGE_WIDTH, Value::Long(vec![4000]));
+ tiny.set(t::IMAGE_LENGTH, Value::Long(vec![3000]));
+ ifd0.add_sub_ifd(tiny);
+ let mut jxl = mask_ifd(2, 2, "unknown coding", vec![0; 4]);
+ jxl.set(t::COMPRESSION, Value::Short(vec![60_000]));
+ ifd0.add_sub_ifd(jxl);
+ let mut zero = mask_ifd(2, 2, "zero size", vec![0; 4]);
+ zero.set(t::IMAGE_WIDTH, Value::Long(vec![0]));
+ ifd0.add_sub_ifd(zero);
+ let mut short = mask_ifd(4, 4, "short strip", vec![200; 3]);
+ short.set(t::MASK_SUB_AREA, Value::Long(vec![u32::MAX, u32::MAX, u32::MAX, u32::MAX]));
+ ifd0.add_sub_ifd(short);
+ let mut outside = mask_ifd(2, 2, "sub area too small", vec![255; 4]);
+ outside.set(t::MASK_SUB_AREA, Value::Long(vec![3, 0, 2, 4]));
+ ifd0.add_sub_ifd(outside);
+ });
+ assert_eq!(decode(&bytes).unwrap().cpp, 3);
+ let masks = lightcraft_raw::semantic_masks(&bytes);
+ let names: Vec<&str> = masks.iter().map(|m| m.name.as_str()).collect();
+ assert_eq!(names, ["short strip", "sub area too small"]);
+ assert!(masks.iter().all(|m| m.sub_area.is_none()), "sub areas the mask doesn't fit are ignored");
+ for m in &masks {
+ let d = m.developed(Rect::new(0, 0, 8, 6), Rect::new(0, 0, 8, 6)).unwrap();
+ assert_eq!((d.width, d.height), (m.width, m.height));
+ assert!(m.developed(Rect::new(0, 0, 8, 6), Rect::new(8, 6, 0, 0)).is_none());
+ }
+ // not a TIFF at all
+ assert!(lightcraft_raw::semantic_masks(b"not a dng").is_empty());
+}
+
+/// Only the first `MAX_MASKS` mask IFDs are decoded, malformed ones included, so a file full of
+/// mask IFDs costs a bounded number of decodes: here the first `MAX_MASKS` can't be decoded, and
+/// the valid masks after them are never looked at.
+#[test]
+fn semantic_mask_ifds_past_the_cap_are_not_decoded() {
+ use lightcraft_raw::semantic::MAX_MASKS;
+ let with = |undecodable: usize, valid: usize| {
+ profile_dng(ByteOrder::Little, |ifd0| {
+ for i in 0..undecodable {
+ let mut m = mask_ifd(2, 2, &format!("undecodable {i}"), vec![0; 4]);
+ m.set(t::COMPRESSION, Value::Short(vec![60_000]));
+ ifd0.add_sub_ifd(m);
+ }
+ for i in 0..valid {
+ ifd0.add_sub_ifd(mask_ifd(2, 2, &format!("valid {i}"), vec![255; 4]));
+ }
+ })
+ };
+ let bytes = with(MAX_MASKS, 3 * MAX_MASKS);
+ assert_eq!(decode(&bytes).unwrap().cpp, 3);
+ assert!(lightcraft_raw::semantic_masks(&bytes).is_empty());
+ // one fewer undecodable mask leaves room for exactly one valid one
+ let names: Vec<String> = lightcraft_raw::semantic_masks(&with(MAX_MASKS - 1, 3 * MAX_MASKS)).into_iter().map(|m| m.name).collect();
+ assert_eq!(names, ["valid 0"]);
+}
diff --git a/crates/tiff/src/tags.rs b/crates/tiff/src/tags.rs
index acdc76697..5771316a5 100644
--- a/crates/tiff/src/tags.rs
+++ b/crates/tiff/src/tags.rs
@@ -180,6 +180,12 @@ pub const FORWARD_MATRIX_3: u16 = 52532;
pub const ILLUMINANT_DATA_1: u16 = 52533;
pub const ILLUMINANT_DATA_2: u16 = 52534;
pub const ILLUMINANT_DATA_3: u16 = 52535;
+// DNG 1.6 semantic masks (spec chapter 4, "Semantic Masks")
+pub const SEMANTIC_NAME: u16 = 52526;
+pub const SEMANTIC_INSTANCE_ID: u16 = 52528;
+pub const MASK_SUB_AREA: u16 = 52536;
+/// NewSubFileType of a semantic mask IFD.
+pub const SUBFILE_SEMANTIC_MASK: u32 = 0x10004;
pub const PROFILE_GAIN_TABLE_MAP_2: u16 = 52544;
// --- Compression codes (TIFF 6.0, DNG, and widely used registrations)
@@ -208,4 +214,6 @@ pub mod photometric {
pub const YCBCR: u16 = 6;
pub const CFA: u16 = 32803;
pub const LINEAR_RAW: u16 = 34892;
+ /// DNG 1.6 semantic mask.
+ pub const MASK: u16 = 52527;
}
diff --git a/docs/parity.md b/docs/parity.md
index 628cd8629..8e6c6f5a1 100644
--- a/docs/parity.md
+++ b/docs/parity.md
@@ -341,11 +341,11 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| Id | Feature | Tier | Status | Evidence | Missing / notes |
|---|---|---|---|---|---|
| LR-MASK-PANEL | Masks panel | P0 | ✅ | `cmd:panel.masking`, `cmd:mask.add`, `cmd:mask.select`, `cmd:mask.rename`, `cmd:mask.duplicate` (`invert`), `cmd:mask.move`, `cmd:mask.visible`, `cmd:mask.delete`, `crates/ui-egui/src/panels/masking.rs` | list with per-mask show/hide eye, double-click rename, right-click menu (duplicate, duplicate and invert, invert, hide, move up/down, rename, delete); fits any panel width (create tiles go to three columns, mask actions wrap, long component names are cut short); no drag-to-reorder |
-| LR-MASK-SUBJECT | Select subject | P2 | 🟡 | `cmd:mask.add` (`subject`), `crates/pipeline/src/masks.rs` | saliency heuristic, no segmentation model |
-| LR-MASK-SKY | Select sky | P2 | 🟡 | `cmd:mask.add` (`sky`) | heuristic |
-| LR-MASK-BACKGROUND | Select background | P2 | 🟡 | `cmd:mask.add` (`background`) | inverse of the subject heuristic |
+| LR-MASK-SUBJECT | Select subject | P2 | 🟡 | `cmd:mask.add` (`subject`), `crates/pipeline/src/masks.rs` | saliency heuristic, no segmentation model; a photo that carries person mattes (DNG semantic masks: iPhone ProRAW portrait / skin / hair / teeth / glasses mattes) uses them instead (synthetic files only: no corpus file carries person mattes) |
+| LR-MASK-SKY | Select sky | P2 | 🟡 | `cmd:mask.add` (`sky`), `crates/pipeline/src/masks.rs` (`Mattes`), `crates/raw/src/semantic.rs` | heuristic; a photo that carries a sky matte (DNG semantic mask, e.g. iPhone ProRAW) uses it instead, through orientation / crop / lens corrections, preview = export (checked on one CC0 iPhone 12 Pro ProRAW and synthetic files) |
+| LR-MASK-BACKGROUND | Select background | P2 | 🟡 | `cmd:mask.add` (`background`) | inverse of the subject mask (heuristic, or the photo's person mattes) |
| LR-MASK-OBJECTS | Object selection | P2 | 🟡 | `cmd:mask.add` (`object`, `prompt`), `cmd:mask.objectPoint`, `cmd:mask.refineDetail`, `cmd:segment.prepare`, `cmd:segment.model.status`, `cmd:segment.model.download`, `cmd:segment.model.cancel`, `crates/segment`, `crates/engine/src/segment/mod.rs` | SAM 3 in pure Rust (candle; Metal on macOS, CPU elsewhere): Object tile → click to include, ⌥-click to leave out; Describe tile → a text prompt selects every instance ("sky", "the red car"); both also as Add/Subtract/Intersect components; + / − under the selected mask; comma lists (`car, road`); a zoomed-in detail pass for 5–10× finer edges on small objects; per-selection Edge (hard ↔ soft). The model runs on its own worker thread (the UI never waits; panics become errors), is unloaded after 10 min idle, and is optional: the segmentation is stored with the mask (288² logits), so renders and exports never need it. Weights are never bundled (SAM License): without them the app offers a consented background download (mirrors, resume, timeouts, SHA-256) — **but no default download location is configured yet** (users set `LIGHTCRAFT_SAM3_MIRRORS` or install by hand: the dialog names the three files, shows the model folder and links docs/ai-masks.md). Not verified against Lightroom's Select Object; no brush/box object mode; the first click on a photo waits for its analysis (~4 s on an M4 Pro, much longer on CPU) |
-| LR-MASK-PEOPLE | People parts | P2 | ⬜ | | |
+| LR-MASK-PEOPLE | People parts | P2 | ⬜ | | shape exists (no UI): uses the photo's person / skin / hair / teeth / glasses mattes when it carries them (DNG semantic masks), else the subject heuristic |
| LR-MASK-LANDSCAPE | Landscape classes | P2 | ⬜ | | shape exists, evaluates empty |
| LR-MASK-BRUSH | Brush mask | P0 | ✅ | `cmd:tool.brush`, `cmd:mask.brushStroke` (`autoMask`), `crates/pipeline/src/masks.rs` | size/feather/flow/density/erase; Auto Mask: dabs weighted by similarity to the colour under the dab centre, refined by a guided filter on luminance (CPU + GPU); no A/B brushes, no pressure |
| LR-MASK-LINEAR | Linear gradient | P0 | ✅ | `cmd:tool.linear`, `cmd:mask.update` | |
diff --git a/docs/whats-new.md b/docs/whats-new.md
index f6361b048..ca5758bf4 100644
--- a/docs/whats-new.md
+++ b/docs/whats-new.md
@@ -22,6 +22,9 @@
embedded JPEG.
- Apple ProRAW's gain table map (its local tone mapping, `ProfileGainTableMap`) is now read and kept when a photo is
exported or converted to DNG. It is not applied to the render: Lightroom Classic renders ProRAW without it.
+- iPhone ProRAW and other DNGs that carry their own segmentation mattes (DNG semantic masks) use them for the Select
+ Sky, Subject and Background masks instead of our heuristics, so the sky is selected where the camera found it
+ (checked on one CC0 iPhone 12 Pro ProRAW). Photos without mattes are unchanged.
- Panasonic and Leica raws (RW2, RWL) are now corrected for lens distortion the way the camera corrects its own JPEG
(issue #256): the correction the camera records in the file is applied under Lens Corrections ▸ Enable Profile
Corrections, on by default for newly imported photos, with the same framing as the camera's JPEG. At 12 mm the