Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(autoindex): settle an outrunning GIF comment on the canvas, not on 0xFF (#379)

xerj-orgb3044cd0cc08
diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rs
index 24e36675c..61b4c0eab 100644
--- a/engine/crates/xerj-autoindex/src/sniff.rs
+++ b/engine/crates/xerj-autoindex/src/sniff.rs
@@ -421,7 +421,10 @@ fn accept(_prefix: &[u8]) -> bool {
///
/// * after `0x21`, the extension label. The spec defines four — Plain Text
/// `0x01`, Graphic Control `0xF9`, Comment `0xFE`, Application `0xFF` —
-/// three unprintable and the fourth a control character.
+/// three unprintable and the fourth a control character. Three of them also
+/// pin the first sub-block size; Comment does not, so its chain is walked,
+/// and a chain that outruns the 8 KiB buffer is settled on the canvas
+/// instead — see `canvas_dimension_text_cannot_write`.
/// * after `0x2C`, a 9-byte Image Descriptor. The frame must be non-empty and
/// must fit inside the canvas declared two fields earlier, which four
/// little-endian u16s of prose do not.
@@ -483,19 +486,24 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {
// there is no fixed byte to pin. Walk the sub-block chain instead:
// a real Comment terminates with a 0x00 and is followed by another
// introducer. Prose has to land that whole structure, not one byte.
- (Some(0xfe), Some(&first)) if first != 0 => {
+ //
+ // A zero FIRST size is the empty comment `21 FE 00`, which giflib
+ // writes for `EGifPutExtensionLeader(0xFE)` + `EGifPutExtensionTrailer`
+ // and which every decoder here accepts. It used to fall to `_ =>
+ // false`, which refused all six such files in the corpus even
+ // though every one of them decodes in both Pillow and giflib.
+ // The NUL is the evidence: it sits at a derived offset, and
+ // prose has no NUL to put there — so nothing further is checked,
+ // and in particular the byte AFTER it is not, because giflib emits
+ // its zero-length sub-block and its trailer as `21 FE 00 00`.
+ (Some(0xfe), Some(0)) => true,
+ (Some(0xfe), Some(_)) => {
// Walk the sub-block chain. Two outcomes are decisive and one
// is not, and conflating them is what broke real GIFs: a
// comment longer than the 8 KiB prefix simply cannot be walked
// to its end here, and "I ran out of buffer" is not evidence
// of text.
let mut at = block + 2;
- // Of the sizes we get to see before the buffer ends, every one
- // but the last must be a full 255. The last may legitimately be
- // a short remainder whose terminator sits just past the edge.
- let mut prior_were_full = true;
- let mut last: Option<u8> = None;
- let mut seen = 0usize;
loop {
match prefix.get(at) {
// Terminated inside the prefix: decisive. A real
@@ -505,20 +513,11 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {
Some(0) => {
return matches!(prefix.get(at + 1), None | Some(0x21 | 0x2c | 0x3b));
}
- Some(&len) => {
- if let Some(prev) = last {
- prior_were_full &= prev == 0xff;
- }
- last = Some(len);
- seen += 1;
- at += 1 + usize::from(len);
- }
- // Ran past the prefix. Every encoder packs a long
- // comment into maximal 255-byte sub-blocks, so every
- // size we got to see is 0xFF. Prose cannot do that:
- // its "sizes" are the text's own bytes, which are
- // printable ASCII, never 0xFF.
- None => return seen > 1 && prior_were_full,
+ Some(&len) => at += 1 + usize::from(len),
+ // Ran past the prefix — undecidable HERE, so decide on
+ // the Logical Screen Descriptor instead. See
+ // `canvas_dimension_text_cannot_write`.
+ None => return canvas_dimension_text_cannot_write(prefix),
}
}
}
@@ -576,6 +575,80 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {
}
}
+/// The tie-break for a Comment sub-block chain that runs past the 8 KiB
+/// `read_prefix` buffer, decided on the Logical Screen Descriptor because it
+/// CANNOT be decided on the chain.
+///
+/// The chain carries no evidence at that point, and this is the whole reason
+/// the arm above is written the way it is: every sub-block size in `1..=255` is
+/// legal, so "the sizes we managed to see" says nothing about whether the file
+/// is an image. Round 3 shipped `seen > 1 && every size seen is 0xFF`, on the
+/// reasoning that encoders pack long comments maximally. They do not. Measured
+/// against a 939-file ground-truth corpus (674 images / 265 text files, every
+/// image decode-checked twice — Pillow `open`+`load` and giflib
+/// `DGifOpenFileName`+`DGifSlurp`):
+///
+/// * 92 corpus images open on a Comment whose chain outruns the buffer. The
+/// 0xFF rule refuses 37 of them: 33 land in `TxtProse` and are sectioned
+/// into 2,519 junk records, 4 reach `binary`/`unknown` on the control-char
+/// ratio alone. giflib's streaming API (`EGifPutExtensionBlock`) writes ONE
+/// sub-block per call at whatever length the caller passed, so chains of 1,
+/// 2, 16, 32, 64, 100, 120, 127, 128, 192, 200 and 250..=254 bytes are all
+/// ordinary encoder output; ImageMagick writes 200/199/199… and
+/// Pillow-then-spliced files mix 255 with 10.
+/// * The eight real commented GIFs on the build machine carry sub-blocks of
+/// 17, 26, 45 and 49 bytes and not a single 255 (GIMP, ezgif.com and
+/// ajaxload.info output, in Lucene, Supabase and pdfjs-dist checkouts).
+/// They terminate inside the prefix so they never reach here — but they are
+/// what the 0xFF premise was measured against, and it is false.
+///
+/// So the decision moves to a field prose cannot forge. The canvas dimensions
+/// are two little-endian u16 at fixed offsets 6..10, always inside the prefix,
+/// which puts their HIGH bytes at 7 and 9. Every printable ASCII byte is
+/// `>= 0x20`, so all-printable text can only ever declare a dimension
+/// `>= 0x2000` (8192); a windows-1252 accent or a UTF-8 lead/continuation byte
+/// is `>= 0x80`, i.e. `>= 32768`. A dimension BELOW 8192 therefore requires a
+/// control byte at 7 or 9 — and the three control bytes text does legitimately
+/// carry (`\t`, `\n`, `\r`) are excluded, because they are exactly the ones it
+/// can. That exclusion is not theoretical: `"GIF89ax\ny…"` + `!` + 0xFE + French
+/// prose is junked as an image by a bare `< 8192` test and stays `TxtProse`
+/// with the exclusion — measured, and pinned by
+/// `prose_with_a_newline_in_the_canvas_is_not_a_gif`.
+///
+/// Cost on real images, measured over the same corpus: 673 of 674 declare a
+/// dimension under 8192, the exception being a JPEG someone had named `.gif`
+/// (17994x17993 read out of JFIF bytes, and it fails the `GIF8` magic anyway).
+/// The largest genuine canvas here is 2000x2000, so the rule has ~4x of margin,
+/// and no corpus image carries `\t`/`\n`/`\r` as a canvas high byte at all
+/// (high bytes observed: 0, 1, 2, 3, 4, 5, 7). The confusion matrix through
+/// `sniff()` for this rule is 666 images binary/gif, 6 binary/other, 2 text,
+/// against 257 of 265 text files still text — the 8 exceptions and the 2
+/// images being decided by OTHER arms, unchanged by this one.
+///
+/// What was tried and refused, with the measurement that refused it:
+///
+/// * `true` ("inconclusive means image"): keeps all 92, but junks 10 corpus
+/// text files as `binary (gif)` — prose only has to land `!`, a
+/// windows-1252 0xFE and one non-zero byte, which is #379 all over again.
+/// * `false` (round 2): refuses all 92; 88 of them become 3,063 prose records.
+/// * not matching `0xFE` at all: refuses every comment-first GIF, terminated
+/// or not — 420 of the 674 images still classify `gif` against 666 here,
+/// 177 land in the text path — and it saves not one text file over `false`.
+/// * the Global Colour Table flag (`packed & 0x80`), the other LSD field that
+/// looks unforgeable: it is NOT. A windows-1252 accented letter at offset
+/// 10 sets it, which is how the corpus's own `gctflag_*` fixtures were
+/// built — 8 of them are junked as images by that rule.
+/// * "the prefix is short, so the chain ran past EOF, so the file is
+/// truncated": true for the two corpus files it fires on (nmap's
+/// `pixel.gif`, which no decoder accepts), but `read_prefix` swallows the
+/// error on a gzip member (`read_to_end(..).ok()`), so a short buffer does
+/// not mean EOF for a `.gif.gz` and the rule would refuse it.
+fn canvas_dimension_text_cannot_write(prefix: &[u8]) -> bool {
+ // `prefix[..13]` is guaranteed by the length check in the only caller.
+ let high_byte_is_binary = |hi: u8| hi < 0x20 && !matches!(hi, b'\t' | b'\n' | b'\r');
+ high_byte_is_binary(prefix[7]) || high_byte_is_binary(prefix[9])
+}
+
/// Windows bitmap: `BM`, then a 14-byte BITMAPFILEHEADER, then a DIB header
/// that opens with its own size.
///
@@ -1938,9 +2011,12 @@ mod printable_magic_tests {
/// The first attempt treated "the sub-block chain did not terminate inside
/// the prefix" as evidence of text, so a real, decodable GIF carrying a
/// licence notice was handed to the prose extractor — #379 with the sign
- /// flipped. Encoders pack a long comment into maximal 255-byte sub-blocks,
- /// and prose cannot: its "sizes" are its own bytes, which are printable
- /// ASCII and never 0xFF. That is what separates the two here.
+ /// flipped.
+ ///
+ /// This fixture is the MAXIMALLY PACKED shape only. Round 3 then read that
+ /// property off this very test and shipped it as the rule ("every size seen
+ /// is 0xFF"); `a_streamed_comment_chain_is_not_packed_to_255` below is the
+ /// corpus answer to that, and the two must be read together.
#[test]
fn a_comment_longer_than_the_sniff_prefix_is_still_a_gif() {
/// GIF89a, 4x4, 2-entry global colour table, opening on a Comment
@@ -1982,6 +2058,165 @@ mod printable_magic_tests {
}
}
+ /// GIF89a, `canvas` px square, 2-entry global colour table, then a Comment
+ /// Extension whose sub-blocks are `sizes` cycled until the chain is longer
+ /// than the 8 KiB `sniff` prefix. Returns exactly what `read_prefix` would
+ /// hand `sniff_bytes` — the first 8192 bytes and no more.
+ fn streamed_comment_prefix(canvas: u16, sizes: &[u8]) -> Vec<u8> {
+ let mut g: Vec<u8> = b"GIF89a".to_vec();
+ g.extend_from_slice(&canvas.to_le_bytes());
+ g.extend_from_slice(&canvas.to_le_bytes());
+ g.push(0x80); // GCT present, 2 entries
+ g.push(0);
+ g.push(0);
+ g.extend_from_slice(&[0, 0, 0, 0xff, 0xff, 0xff]); // GCT
+ g.extend_from_slice(&[0x21, 0xfe]); // Comment Extension
+ for &n in sizes.iter().cycle() {
+ if g.len() > 9000 {
+ break;
+ }
+ g.push(n);
+ g.extend(std::iter::repeat_n(b'D', usize::from(n)));
+ }
+ g.truncate(8192);
+ g
+ }
+
+ /// Round 3's rule — "a chain that outruns the prefix is an image only if
+ /// every sub-block size seen is 0xFF" — measured against the ground-truth
+ /// corpus (939 files; 674 images, each decode-checked with Pillow AND
+ /// giflib `DGifSlurp`; 265 text files). It refuses 43 of the 92 images that
+ /// reach this arm, and 37 of those are sectioned into prose: 2,524 junk
+ /// records out of two and a half real image files.
+ ///
+ /// The premise is simply not how the encoders write. giflib's streaming
+ /// API emits one sub-block per `EGifPutExtensionBlock` call, at the length
+ /// the CALLER passed, so the corpus holds chains of 1, 2, 16, 32, 64, 100,
+ /// 120, 127, 128, 192, 200 and 250..=254 — `gstream_long_sz064_x160.gif`,
+ /// `prior_*_c001.gif` … `prior_*_c254.gif`. ImageMagick writes a first
+ /// sub-block at N and the rest at N-1 (`prior_*_imagick_im20k_w200.gif`:
+ /// 200, 199, 199, …). Splicing produces alternating 255/10 and 20-then-255
+ /// chains. Every size in `1..=255` is legal and all of these decode in both
+ /// decoders.
+ ///
+ /// The sizes below are the ones actually on disk in that corpus, not a
+ /// sweep invented here.
+ #[test]
+ fn a_streamed_comment_chain_is_not_packed_to_255() {
+ for (label, sizes) in [
+ ("giflib-stream-1", &[1u8][..]),
+ ("giflib-stream-2", &[2][..]),
+ ("giflib-stream-16", &[16][..]),
+ ("giflib-stream-32", &[32][..]),
+ ("giflib-stream-64", &[64][..]),
+ ("giflib-stream-100", &[100][..]),
+ ("giflib-stream-127", &[127][..]),
+ ("giflib-stream-128", &[128][..]),
+ ("giflib-stream-192", &[192][..]),
+ ("giflib-stream-250", &[250][..]),
+ ("giflib-stream-254", &[254][..]),
+ ("giflib-stream-255", &[255][..]),
+ ("imagemagick-200-then-199", &[200, 199, 199, 199][..]),
+ ("imagemagick-60-then-59", &[60, 59, 59, 59][..]),
+ ("spliced-255-then-10", &[255, 10][..]),
+ ("spliced-20-then-255", &[20, 255, 255, 255][..]),
+ ] {
+ let prefix = streamed_comment_prefix(48, sizes);
+ assert_eq!(prefix.len(), 8192, "{label}: fixture must fill the prefix");
+ let p = Path::new("frame.gif");
+ let sn = sniff_bytes(&prefix, p, p, false).unwrap();
+ assert_eq!(
+ (sn.family, sn.binary_kind.as_deref()),
+ (Family::Binary, Some("gif")),
+ "{label}: a real GIF whose comment sub-blocks are not 255 was \
+ refused and handed to the prose extractor — #379 inverted, and \
+ the shape 100% of the real commented GIFs on this machine have"
+ );
+ }
+ }
+
+ /// The empty Comment Extension, `21 FE 00`. giflib writes it whenever a
+ /// caller opens a comment and closes it without content
+ /// (`EGifPutExtensionLeader(0xFE)` then `EGifPutExtensionTrailer`), and the
+ /// corpus holds six such files — all six decode in Pillow AND giflib, and
+ /// all six were refused, because the arm's guard was `first != 0` and a
+ /// zero first size fell through to `_ => false`.
+ ///
+ /// Both on-disk shapes are here. `21 FE 00 2C` is the plain one;
+ /// `21 FE 00 00 2C` is giflib emitting a zero-length sub-block AND its
+ /// trailer, which is why the byte after the terminator is not checked.
+ #[test]
+ fn an_empty_comment_extension_is_still_a_gif() {
+ fn empty_comment_gif(tail: &[u8]) -> Vec<u8> {
+ let mut g: Vec<u8> = b"GIF89a".to_vec();
+ g.extend_from_slice(&8u16.to_le_bytes());
+ g.extend_from_slice(&8u16.to_le_bytes());
+ g.push(0x80);
+ g.push(0);
+ g.push(0);
+ g.extend_from_slice(&[0, 0, 0, 0xff, 0xff, 0xff]); // GCT
+ g.extend_from_slice(&[0x21, 0xfe, 0x00]); // empty comment
+ g.extend_from_slice(tail);
+ g
+ }
+ for (label, tail) in [
+ (
+ "21-fe-00-then-image",
+ &[0x2c, 0, 0, 0, 0, 8, 0, 8, 0, 0, 0x02][..],
+ ),
+ (
+ "21-fe-00-00-then-image",
+ &[0x00, 0x2c, 0, 0, 0, 0, 8, 0, 8, 0, 0, 0x02][..],
+ ),
+ ("21-fe-00-then-trailer", &[0x3b][..]),
+ ] {
+ let bytes = empty_comment_gif(tail);
+ let p = Path::new("empty.gif");
+ let sn = sniff_bytes(&bytes, p, p, false).unwrap();
+ assert_eq!(
+ (sn.family, sn.binary_kind.as_deref()),
+ (Family::Binary, Some("gif")),
+ "{label}: a decodable GIF carrying an empty comment was refused"
+ );
+ }
+ }
+
+ /// The tie-break for an outrunning chain is the canvas, and this is the
+ /// fixture that decides HOW it is read. The obvious form — "a canvas
+ /// dimension below 8192, which printable ASCII cannot declare" — is
+ /// forgeable, because `\t`, `\n` and `\r` are bytes below 0x20 that text
+ /// carries all the time: a newline at offset 7 declares a 2680 px width.
+ /// Measured through the real `sniff()`: with the bare `< 8192` test these
+ /// two files classify `binary (gif)`; with `\t\n\r` excluded they stay
+ /// `TxtProse`, which is what they are.
+ #[test]
+ fn prose_with_a_newline_in_the_canvas_is_not_a_gif() {
+ for (label, seven) in [("newline", b'\n'), ("tab", b'\t'), ("cr", b'\r')] {
+ let mut prose: Vec<u8> = b"GIF89ax".to_vec(); // 0..=6
+ prose.push(seven); // 7: canvas width high byte
+ prose.extend_from_slice(b"yze"); // 8,9 height; 10 packed (GCT clear)
+ prose.extend_from_slice(b"st"); // 11 background, 12 aspect
+ prose.push(b'!'); // 13: Extension introducer
+ prose.push(0xfe); // 14: Comment label
+ prose.push(b'L'); // 15: first sub-block size, 76
+ for _ in 0..400 {
+ prose.extend_from_slice(
+ "Le format GIF est tres ancien et reste partout sur le web. ".as_bytes(),
+ );
+ }
+ prose.truncate(8192);
+ let p = Path::new("notes.txt");
+ let sn = sniff_bytes(&prose, p, p, false).unwrap();
+ assert_eq!(
+ sn.family,
+ Family::TxtProse,
+ "{label}: prose was junked as {:?} — a canvas rule that admits \
+ the control characters text actually carries is not a rule",
+ sn.binary_kind
+ );
+ }
+ }
+
/// The other half of the same arm: prose must not reach it. `0xFE` is not
/// ASCII, so this needs a windows-1252 byte — which `decode()` reads back
/// as text, exactly the path that made the label-only check unsound.
diff --git a/engine/crates/xerj-autoindex/tests/sniff_printable_magic.rs b/engine/crates/xerj-autoindex/tests/sniff_printable_magic.rs
index cc2ba0a74..a7e8e91ba 100644
--- a/engine/crates/xerj-autoindex/tests/sniff_printable_magic.rs
+++ b/engine/crates/xerj-autoindex/tests/sniff_printable_magic.rs
@@ -88,6 +88,83 @@ fn a_real_bitmap_on_disk_is_still_binary() {
assert_eq!(sn.binary_kind.as_deref(), Some("bmp"));
}
+/// The comment arm on the path that actually loses data: a file on disk whose
+/// comment is longer than the 8 KiB `read_prefix` buffer, so the sub-block
+/// chain cannot be walked to its end and the terminator is never seen.
+///
+/// The unit fixtures in `sniff::printable_magic_tests` truncate to 8192 by
+/// hand; these are real files of 20 KB and 60 KB, cut by `read_prefix` itself.
+/// The chain shapes are the ones the ground-truth corpus found on disk —
+/// giflib's streaming API writes one sub-block per `EGifPutExtensionBlock` call
+/// at the caller's length, so 64-byte and 100-byte chains are ordinary output,
+/// and the "every size is 0xFF" rule that round 3 shipped refuses them: 43 of
+/// the corpus's 92 outrunning images, 37 of them into `TxtProse`.
+///
+/// The negative in the same shape is the file below it: prose only has to land
+/// `!` and a windows-1252 `0xFE`, and it must NOT be junked.
+#[test]
+fn a_long_commented_gif_on_disk_survives_the_prefix_cut() {
+ let dir = tempfile::tempdir().unwrap();
+
+ fn streamed(sub_block: u8, comment_len: usize) -> Vec<u8> {
+ let mut g: Vec<u8> = b"GIF89a".to_vec();
+ g.extend_from_slice(&48u16.to_le_bytes());
+ g.extend_from_slice(&48u16.to_le_bytes());
+ g.push(0x80); // GCT present, 2 entries
+ g.push(0);
+ g.push(0);
+ g.extend_from_slice(&[0, 0, 0, 0xff, 0xff, 0xff]);
+ g.extend_from_slice(&[0x21, 0xfe]);
+ let mut left = comment_len;
+ while left > 0 {
+ let take = left.min(usize::from(sub_block));
+ g.push(take as u8);
+ g.extend(std::iter::repeat_n(b'D', take));
+ left -= take;
+ }
+ g.push(0x00);
+ g.extend_from_slice(&[0x2c, 0, 0, 0, 0, 48, 0, 48, 0, 0, 0x02]);
+ g
+ }
+
+ for (name, sub_block, len) in [
+ ("stream64.gif", 64u8, 20_000usize),
+ ("stream100.gif", 100, 60_000),
+ ("stream255.gif", 255, 60_000),
+ ] {
+ let p = dir.path().join(name);
+ let bytes = streamed(sub_block, len);
+ assert!(bytes.len() > 8192, "{name}: fixture must outrun the prefix");
+ std::fs::write(&p, &bytes).unwrap();
+ let sn = sniff(&p).unwrap();
+ assert_eq!(
+ (sn.family, sn.binary_kind.as_deref()),
+ (Family::Binary, Some("gif")),
+ "{name}: a real GIF with a {sub_block}-byte comment chain classified \
+ {}/{:?} — `scan_file` hands that to the prose extractor",
+ sn.family.as_str(),
+ sn.binary_kind
+ );
+ }
+
+ // The same shape, in prose: `!` at 13, a windows-1252 0xFE at 14, and a
+ // chain of printable "sizes" that never terminates inside the prefix.
+ let mut notes: Vec<u8> = b"GIF89a fichie!".to_vec();
+ notes.push(0xfe);
+ for _ in 0..400 {
+ notes.extend_from_slice("Le format GIF reste partout sur le web. ".as_bytes());
+ }
+ let np = dir.path().join("notes.txt");
+ std::fs::write(&np, &notes).unwrap();
+ let sn = sniff(&np).unwrap();
+ assert_eq!(
+ sn.family,
+ Family::TxtProse,
+ "prose was junked as {:?} through the comment arm (#379)",
+ sn.binary_kind
+ );
+}
+
/// An actual GIF, checked into this repository, read off disk.
///
/// Every other fixture in this file and in `sniff::printable_magic_tests` is