PdfCraft: An open-source PDF reader and editor written by scratch in pure Rust
M11: an indexed image's palette says which of its samples is dark, so it is averaged too
Grzegorz Martensc412093dc779
diff --git a/crates/optimize/README.md b/crates/optimize/README.mdindex af9b5385e..b8346a626 100644--- a/crates/optimize/README.md+++ b/crates/optimize/README.md@@ -59,12 +59,14 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_goingdownsampling it to JPEG.- **Downsampling:** one whose effective resolution is above `above_ppi` is averaged down to`target_ppi`: each new pixel takes the sample covering more of its area, and the dark one on a- tie, so a dark line half a new pixel wide stays: black in grey and the painted part of an image- mask (a 0, or a 1 with an inverting decode array). Only images whose dark sample is known so- are averaged: `DeviceGray` (grey and RGB packed to one bit among them) and image masks, with no- decode array or one of two different numbers, and not drawn or listed where `/DefaultGray` or- `/DefaultRGB` is or may be in force. Not a separation, a DeviceN, calibrated or ICC grey, or an- indexed image, whose palette isn't read. Not one with a soft mask, whose own size would have to+ tie, so a dark line half a new pixel wide stays: black in grey, the painted part of an image+ mask (a 0, or a 1 with an inverting decode array), and the darker of an indexed image's two+ palette entries. Only images whose dark sample is known so are averaged: `DeviceGray` (grey and+ RGB packed to one bit among them), image masks, and indexed images over `DeviceGray` or+ `DeviceRGB`, whose palette is read (Rec. 601 luma for RGB), with no decode array or one of two+ different numbers, and not drawn or listed where `/DefaultGray` or `/DefaultRGB` is or may be+ in force. Not a separation or a DeviceN, whose tint transform isn't read, calibrated or ICC+ grey, or an indexed image over another base or whose two entries look alike. Not one with a soft mask, whose own size would have tofollow; not a halftone or a dither, most of whose new pixels would cover both black and white,so that taking the sample covering more would turn its tones solid (a 50% screen black, a 25%one white), nor one with a square of four new pixels that each would, as a small screeneddiff --git a/crates/optimize/src/images.rs b/crates/optimize/src/images.rsindex 4c5cf5466..2929b9884 100644--- a/crates/optimize/src/images.rs+++ b/crates/optimize/src/images.rs@@ -911,8 +911,12 @@ fn packed_rows(samples: &[u8], row_len: usize, w: usize, h: usize, (n, bpc, whitSome(out)}-/// Resolve polarity only for explicitly understood samples. Indexed palettes and custom-/// colour spaces stay at their original resolution rather than guessing which entry is dark.+/// Whether sample 1 is the dark one of the 1-bit image `d` describes, after its decode array+/// (`[0 1]` when it has none; another must be two different numbers): black in grey is a 0, and so+/// is the painted part of an image mask. An indexed image's samples pick palette entries (its+/// decode array maps them to entries, rounded and clipped to the table, ISO 32000-2 §8.9.5.2), and+/// the palette says which is darker ([`lightness`]). `None` for anything else: separations and+/// DeviceN, whose tint transforms aren't read, other colour spaces, and two entries alike.fn dark_is_one(doc: &Document, d: &Dict) -> Option<bool> {let number = |v: Option<&Object>| v.and_then(|v| doc.resolve(v).as_f64()).filter(|v| v.is_finite());let (d0, d1) = match d.get(b"Decode").map(|v| doc.resolve(v)).as_deref() {@@ -928,10 +932,42 @@ fn dark_is_one(doc: &Document, d: &Dict) -> Option<bool> {}match d.get(b"ColorSpace").map(|cs| doc.resolve(cs)).as_deref() {Some(Object::Name(n)) if matches!(n.as_slice(), b"DeviceGray" | b"G") => Some(d1 < d0),+ Some(Object::Array(a)) if matches!(a.first().and_then(Object::as_name), Some(b"Indexed" | b"I")) => {+ let hival = a.get(2).and_then(|v| doc.resolve(v).as_int()).filter(|h| (0..=255).contains(h))?;+ // `hival` is 0 to 255, and the rounded value is clipped to it: the cast can't wrap.+ let entry = |v: f64| v.round().clamp(0.0, hival as f64) as usize;+ let (l0, l1) = (lightness(doc, a, entry(d0))?, lightness(doc, a, entry(d1))?);+ (l0 != l1).then_some(l1 < l0)+ }_ => None,}}+/// How light entry `i` of the palette of indexed colour space `a` (`[/Indexed base hival lookup]`)+/// is, from 0 (black) to 255,000 (white): its grey, or its luma (Rec. 601) over `DeviceRGB`. Only+/// those two bases count, where the palette's values are the colour: under `/DefaultGray` or+/// `/DefaultRGB` they aren't, and the caller asks for no polarity there; `None` for another base+/// (calibrated, CMYK, an ICC profile, a resource name), and for an entry the table doesn't hold.+fn lightness(doc: &Document, a: &[Object], i: usize) -> Option<u32> {+ let n = match doc.resolve(a.get(1)?).as_name()? {+ b"DeviceGray" => 1,+ b"DeviceRGB" => 3,+ _ => return None,+ };+ // At most 256 entries of three components: a stream decoding to more is no palette.+ let table = match &*doc.resolve(a.get(3)?) {+ Object::String(s) => s.bytes.clone(),+ Object::Stream(s) => doc.with_direct_parms(s).decoded_within(4096).ok()?,+ _ => return None,+ };+ let c: Vec<u32> = table.get(i.checked_mul(n)?..i.checked_mul(n)?.checked_add(n)?)?.iter().map(|v| u32::from(*v)).collect();+ Some(match c.as_slice() {+ [v] => v * 1000,+ [r, g, b] => 299 * r + 587 * g + 114 * b,+ _ => return None,+ })+}+/// `w` × `h` 1-bit rows (each starting on a new byte) averaged down to `nw` × `nh`, no larger:/// each new pixel takes the sample that covers more of its area, and the dark one when the two tie/// (1 if `dark_is_one`, see [`dark_is_one`]), so a dark line half a new pixel wide stays. `None`@@ -1003,12 +1039,13 @@ fn downsample_bits_checked(data: &[u8], w: u32, h: u32, nw: u32, nh: u32, dark_i/// `/DefaultGray` or `/DefaultRGB` is or may be in force. Also says whether it was resampled.////// - **Resampled** ([`downsample_bits_checked`]) when its resolution is above `above_ppi`, to `target_ppi`,-/// if its dark sample is known ([`dark_is_one`]: grey or an image mask, not under `defaults`),-/// unless it has a soft mask, whose own size would then have to follow, or is a halftone or a-/// dither: most of its new pixels would cover both black and white, and taking the sample that-/// covers more would turn its tones solid (a 50% screen black, a 25% one white). So would a-/// square of four new pixels that each cover both, as a small screened picture on a white page-/// has (and ragged scanned text, which is then left at its resolution too).+/// if its dark sample is known ([`dark_is_one`]: grey, an image mask, or an indexed image whose+/// palette says which entry is darker, not under `defaults`), unless it has a soft mask, whose+/// own size would then have to follow, or is a halftone or a dither: most of its new pixels would+/// cover both black and white, and taking the sample that covers more would turn its tones solid+/// (a 50% screen black, a 25% one white). So would a square of four new pixels that each cover+/// both, as a small screened picture on a white page has (and ragged scanned text, which is then+/// left at its resolution too)./// - **Stored** as CCITT Group 4 with `CcittG4`, or as 1-bit Flate when that is smaller, which only/// a resampled image or one packed from more bits may be: a 1-bit image that is only stored again/// becomes Group 4 or stays. Group 4 alone is only decoded to be resampled; Group 3 is stored@@ -1313,8 +1350,9 @@ mod downsample_tests {assert_eq!(downsample_bits(&[0b1100_0000], 3, 1, 2, 1, false).map(|(out, _)| out), Some(vec![0b1000_0000]));}- /// The dark sample: 0 in grey and of an image mask, 1 in a separation, each the other way round- /// with an inverting decode array; 0 for an indexed image, whatever its decode array.+ /// The dark sample: 0 in grey and of an image mask, the other way round with an inverting decode+ /// array; none known for a separation, nor for an indexed image without a palette (see the next+ /// test), nor with a decode array that isn't two numbers.#[test]fn the_dark_sample_follows_the_colour_space_and_decode_array() {let doc = Document::new_empty();@@ -1338,6 +1376,7 @@ mod downsample_tests {(vec![mask, inverted.clone()], Some(true)),(vec![separation.clone()], None),(vec![separation, inverted.clone()], None),+ // A palette that isn't there says nothing.(vec![indexed.clone()], None),(vec![indexed, inverted], None),(vec![("ColorSpace", Object::name("DeviceGray")), ("Decode", Object::Array(vec![Object::name("x"), 0.into()]))], None),@@ -1346,6 +1385,54 @@ mod downsample_tests {}}+ /// An indexed image's dark sample is the one picking the darker palette entry, after its decode+ /// array: over DeviceGray and DeviceRGB (by luma, each weight checked), with the table a string+ /// or a stream, directly or by reference. Another base (CMYK, calibrated, an ICC profile, a+ /// resource name), two entries alike, an entry the table doesn't hold, and a `hival` past 255+ /// say nothing.+ #[test]+ fn an_indexed_images_palette_says_which_sample_is_dark() {+ let mut doc = Document::new_empty();+ let string = |bytes: &[u8]| Object::String(pdfcraft_cos::PdfString::literal(bytes.to_vec()));+ let referenced = doc.add(string(&[255, 0]));+ let stream = doc.add(Object::Stream(pdfcraft_cos::Stream::flate(Dict::new(), &[0, 0, 0, 255, 255, 255])));+ let cal = |name: &str| Object::Array(vec![Object::name(name), Object::Dict(Dict::new())]);+ let indexed = |base: Object, hival: i64, table: Object| {+ let mut d = Dict::new();+ d.set(b"ColorSpace".to_vec(), Object::Array(vec![Object::name("Indexed"), base, hival.into(), table]));+ d+ };+ let inverted = |mut d: Dict| {+ d.set(b"Decode".to_vec(), Object::Array(vec![1.into(), 0.into()]));+ d+ };+ for (d, dark_one, what) in [+ (indexed(Object::name("DeviceGray"), 1, string(&[0, 255])), Some(false), "grey, black first"),+ (indexed(Object::name("DeviceGray"), 1, string(&[255, 0])), Some(true), "grey, white first"),+ (indexed(Object::name("DeviceGray"), 1, string(&[90, 89])), Some(true), "two greys a step apart"),+ (inverted(indexed(Object::name("DeviceGray"), 1, string(&[255, 0]))), Some(false), "white first, inverted"),+ (indexed(Object::name("DeviceGray"), 1, Object::Ref(referenced)), Some(true), "the table by reference"),+ (indexed(Object::name("DeviceRGB"), 1, Object::Ref(stream)), Some(false), "RGB in a stream"),+ (indexed(Object::name("DeviceRGB"), 1, string(&[255, 255, 0, 0, 0, 255])), Some(true), "yellow, then blue"),+ // Pure red (76,245) is lighter than pure blue (29,070), and darker than green (149,685).+ (indexed(Object::name("DeviceRGB"), 1, string(&[255, 0, 0, 0, 0, 255])), Some(true), "red, then blue"),+ (indexed(Object::name("DeviceRGB"), 1, string(&[255, 0, 0, 0, 255, 0])), Some(false), "red, then green"),+ (indexed(Object::name("DeviceRGB"), 1, string(&[0, 0, 255, 0, 255, 0])), Some(false), "blue, then green"),+ // Blue (29,070) is lighter than grey 20 (20,000): the blue weight decides it.+ (indexed(Object::name("DeviceRGB"), 1, string(&[0, 0, 255, 20, 20, 20])), Some(true), "blue, then dark grey"),+ (indexed(Object::name("DeviceCMYK"), 1, string(&[0, 0, 0, 0, 0, 0, 0, 255])), None, "CMYK"),+ (indexed(cal("CalGray"), 1, string(&[200, 20])), None, "calibrated grey"),+ (indexed(cal("ICCBased"), 1, string(&[0, 255])), None, "an ICC profile"),+ (indexed(Object::name("CS0"), 1, string(&[0, 255])), None, "a resource name"),+ (indexed(Object::name("DeviceGray"), 1, string(&[7, 7])), None, "two entries alike"),+ (indexed(Object::name("DeviceGray"), 1, string(&[0])), None, "a table of one entry"),+ (indexed(Object::name("DeviceGray"), 0, string(&[0, 255])), None, "hival 0: both samples pick entry 0"),+ (indexed(Object::name("DeviceGray"), 256, string(&[0, 255])), None, "hival past 255"),+ ] {+ assert_eq!(dark_is_one(&doc, &d), dark_one, "{what}");+ }+ }+#[test]fn what_it_cannot_resample_is_none() {assert_eq!(downsample_bits(&[0; 2], 4, 2, 5, 1, false).map(|(out, _)| out), None, "wider");diff --git a/crates/optimize/src/lib.rs b/crates/optimize/src/lib.rsindex 6e8988fd1..a4a0a810e 100644--- a/crates/optimize/src/lib.rs+++ b/crates/optimize/src/lib.rs@@ -13,7 +13,7 @@//! and `/ColorTransform` disagree). Black-and-white images (1-bit, image masks and CCITT fax,//! Group 3 among them; and eight-bit grey and RGB of only black and white, and indexed images//! that use only palette entries 0 and 1, packed to one bit) above a threshold are averaged down to a-//! target resolution when grey or image masks, halftones and dithers aside (see the README), and+//! target resolution when grey, image masks or indexed, halftones and dithers aside (see the README), and//! are stored as CCITT Group 4 or lossless JBIG2, or 1-bit//! Flate once resampled or packed, when that is smaller.//! - **Discard objects:** page thumbnails, alternate images, document tags (structure tree),diff --git a/crates/optimize/tests/optimize.rs b/crates/optimize/tests/optimize.rsindex 8f55d9021..92dc2e2d8 100644--- a/crates/optimize/tests/optimize.rs+++ b/crates/optimize/tests/optimize.rs@@ -3445,10 +3445,12 @@ fn lzw_streams_that_cannot_be_read_or_are_not_asked_for_stay_unchanged() {assert_eq!(stream(&doc, c).decoded_strict().unwrap(), ops);}-/// Unknown palette polarity is never guessed. Both palette orders and Decode directions-/// retain the original thin rule and exact samples through a full save and render.+/// An indexed image's palette says which sample is dark, so a thin rule survives averaging as it does+/// in grey: with the palette either way round, and the decode array either way, the averaged image+/// holds the averaged DeviceGray one's samples (flipped where its samples are), and looks the same+/// through a full save and render.#[test]-fn indexed_thin_rules_keep_their_samples_for_both_palette_orders_and_decodes() {+fn indexed_thin_rules_survive_averaging_for_both_palette_orders_and_decodes() {let (w, h) = (1200u32, 480u32);let stride = (w as usize).div_ceil(8);let mut gray_data = vec![0xFF; stride * h as usize];@@ -3460,6 +3462,7 @@ fn indexed_thin_rules_keep_their_samples_for_both_palette_orders_and_decodes() {page(&mut gray, &[("Im", gray_image)], "q 144 0 0 57.6 20 600 cm /Im Do Q");let gray_render = render(&gray);optimize(&mut gray, &Settings::default()).unwrap();+ let gray_after = render(&Document::open(Arc::new(write_full(&gray, &SaveOptions::default()).unwrap())).unwrap());let gray_stream = stream(&gray, gray_image);let gray_samples = gray.decoded_image(&gray_stream).unwrap();assert_eq!((gray_stream.dict.int(b"Width"), gray_stream.dict.int(b"Height")), (Some(600), Some(240)));@@ -3491,10 +3494,12 @@ fn indexed_thin_rules_keep_their_samples_for_both_palette_orders_and_decodes() {assert_eq!(before, gray_render, "the palette image is visually identical to DeviceGray");optimize(&mut doc, &Settings::default()).unwrap();let s = stream(&doc, image);- assert_eq!((s.dict.int(b"Width"), s.dict.int(b"Height")), (Some(w.into()), Some(h.into())));- assert_eq!(doc.decoded_image(&s).unwrap(), data);+ assert_eq!((s.dict.int(b"Width"), s.dict.int(b"Height")), (Some(600), Some(240)), "white-first={white_first}, inverted={inverted}");+ // The rows are 600 samples, whole bytes: flipping each byte flips each sample.+ let expected: Vec<u8> = gray_samples.iter().map(|b| if white_first != inverted { !b } else { *b }).collect();+ assert_eq!(doc.decoded_image(&s).unwrap(), expected, "white-first={white_first}, inverted={inverted}: the rule is dark");let saved = Document::open(Arc::new(write_full(&doc, &SaveOptions::default()).unwrap())).unwrap();- assert_eq!(render(&saved), before, "palette white-first={white_first}, inverted={inverted}");+ assert!(render(&saved) == gray_after, "palette white-first={white_first}, inverted={inverted}");}}}diff --git a/parity/acrobat-features.toml b/parity/acrobat-features.tomlindex 4878eac62..4430e5fac 100644--- a/parity/acrobat-features.toml+++ b/parity/acrobat-features.toml@@ -7384,8 +7384,8 @@ milestone = "M11"status = "partial"commands = ["optimize.advanced"]tools = ["doc_optimize"]-evidence = ["crates/optimize/tests/optimize.rs::images_are_measured_where_drawn_and_downsampled", "crates/optimize/tests/optimize.rs::settings_choose_what_happens", "crates/automation/tests/it/automation.rs::optimizing_through_tools", "crates/optimize/tests/optimize.rs::a_page_picture_printed_as_one_pixel_strips_is_merged_into_one_image", "crates/optimize/tests/optimize.rs::images_that_do_not_tile_a_picture_are_left_alone", "crates/optimize/tests/optimize.rs::a_deeply_nested_form_is_measured_at_its_largest_use", "crates/optimize/tests/optimize.rs::images_are_not_resampled_when_some_uses_could_not_be_measured", "crates/optimize/tests/optimize.rs::image_resolution_uses_the_pages_physical_unit", "crates/optimize/tests/optimize.rs::optimization_retains_an_image_below_threshold_on_a_scaled_page", "crates/optimize/tests/optimize.rs::shared_forms_are_measured_in_each_pages_physical_units", "crates/optimize/tests/optimize.rs::ambiguous_monochrome_text_is_recompressed_without_lossy_reduction", "crates/optimize/src/images.rs::it_matches_the_slow_way", "crates/automation/tests/it/automation.rs::doc_optimize_averages_black_and_white_scans_down", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy"]-notes = "colour and grayscale, bicubic, effective resolution measured where drawn (every use, with page-local UserUnit scaling, through nested forms up to 12 deep; when a use can't be measured, nothing is resampled); image fragments stacked edge to edge are merged first (side-by-side tiles not yet); monochrome grey images and image masks (1-bit, CCITT fax, and eight-bit black-and-white grey and RGB packed to one bit) averaged by area to 300 ppi above 450 ppi by default, a tie to the dark sample (black in grey, painted in a mask, after a decode array of two different numbers), then stored as Group 4 or 1-bit Flate (Monochrome Images ▸ Downsample to; doc_optimize mono); not separations, DeviceN, calibrated or ICC grey, or indexed images (their dark sample isn't read), images drawn or listed where /DefaultGray or /DefaultRGB is in force, images with a soft mask, halftones and dithers (most of their new pixels would cover both black and white, and their tones turn solid), images with a square of four such new pixels (a small screened picture on a white page; ragged scanned text has them along its strokes too, and keeps its resolution), or images wider than a fax row (1,048,576 samples), and Acrobat's bicubic and subsampling choices for monochrome aren't offered"+evidence = ["crates/optimize/tests/optimize.rs::images_are_measured_where_drawn_and_downsampled", "crates/optimize/tests/optimize.rs::settings_choose_what_happens", "crates/automation/tests/it/automation.rs::optimizing_through_tools", "crates/optimize/tests/optimize.rs::a_page_picture_printed_as_one_pixel_strips_is_merged_into_one_image", "crates/optimize/tests/optimize.rs::images_that_do_not_tile_a_picture_are_left_alone", "crates/optimize/tests/optimize.rs::a_deeply_nested_form_is_measured_at_its_largest_use", "crates/optimize/tests/optimize.rs::images_are_not_resampled_when_some_uses_could_not_be_measured", "crates/optimize/tests/optimize.rs::image_resolution_uses_the_pages_physical_unit", "crates/optimize/tests/optimize.rs::optimization_retains_an_image_below_threshold_on_a_scaled_page", "crates/optimize/tests/optimize.rs::shared_forms_are_measured_in_each_pages_physical_units", "crates/optimize/tests/optimize.rs::ambiguous_monochrome_text_is_recompressed_without_lossy_reduction", "crates/optimize/tests/optimize.rs::indexed_thin_rules_survive_averaging_for_both_palette_orders_and_decodes", "crates/optimize/src/images.rs::an_indexed_images_palette_says_which_sample_is_dark", "crates/optimize/src/images.rs::it_matches_the_slow_way", "crates/automation/tests/it/automation.rs::doc_optimize_averages_black_and_white_scans_down", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy"]+notes = "colour and grayscale, bicubic, effective resolution measured where drawn (every use, with page-local UserUnit scaling, through nested forms up to 12 deep; when a use can't be measured, nothing is resampled); image fragments stacked edge to edge are merged first (side-by-side tiles not yet); monochrome grey, image masks and indexed images (1-bit, CCITT fax, and eight-bit black-and-white grey and RGB packed to one bit) averaged by area to 300 ppi above 450 ppi by default, a tie to the dark sample (black in grey, painted in a mask, the darker palette entry over DeviceGray or DeviceRGB, after a decode array of two different numbers), then stored as Group 4 or 1-bit Flate (Monochrome Images ▸ Downsample to; doc_optimize mono); not separations or DeviceN (their tint transforms aren't read), calibrated or ICC grey, indexed images over another base, images drawn or listed where /DefaultGray or /DefaultRGB is in force, images with a soft mask, halftones and dithers (most of their new pixels would cover both black and white, and their tones turn solid), images with a square of four such new pixels (a small screened picture on a white page; ragged scanned text has them along its strokes too, and keeps its resolution), or images wider than a fax row (1,048,576 samples), and Acrobat's bicubic and subsampling choices for monochrome aren't offered"[[feature]]id = "optimize.recompress-images"