Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(sniff): the trade costs 10 text files, not 8 — counted from the code path this time

xerj-orgcf60b6569cf9
diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rs
index 6ce54dfa4..15d653feb 100644
--- a/engine/crates/xerj-autoindex/src/sniff.rs
+++ b/engine/crates/xerj-autoindex/src/sniff.rs
@@ -564,10 +564,14 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {
// and 0.4% of the buffer respectively.
//
// The discriminating is therefore done by the header
- // gate above, not here: `0x21` at 13+GCT and `0xFE` at
- // 14+GCT, and 0xFE is neither ASCII nor valid UTF-8.
+ // gate above, not here: `0x21` at `block` and `0xFE`
+ // at `block + 1`, where `block = 13 + gct` and `gct` is
+ // 0 without a Global Colour Table — so the pair sits at
+ // one of 13, 19, 25, 37, 61, 109, 205, 397 or 781, NOT
+ // at a fixed offset. 0xFE is neither ASCII nor valid
+ // UTF-8, which is the part ordinary text lacks.
// Verified on disk — the same 1 MiB French-prose file
- // with 0x7E at offset 14 stays TxtProse and with 0xFE
+ // with 0x7E in that slot stays TxtProse and with 0xFE
// becomes binary/gif. All this arm decides is how much
// text has to accumulate behind that gate before it is
// admitted, which is why the budget is the whole rule
@@ -579,7 +583,7 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {
//
// * answering image unconditionally (round 2's
// rejected `true`) junks 10 corpus text files —
- // windows-1252 prose carrying `þ` at offset 14.
+ // windows-1252 prose carrying `þ` in that slot.
// * answering text on a short buffer sends a
// TRUNCATED real GIF to the prose extractor: 7 of 7
// cuts of real corpus GIFs at 300 KB..1,048,575 B go
@@ -2369,19 +2373,30 @@ mod printable_magic_tests {
/// accept, then `0x21 0xFE`. `0xFE` is `þ` in windows-1252 and not ASCII,
/// which is the only part ordinary text lacks — and NOT at a fixed offset:
/// when the packed byte at 10 has bit 7 set (any accented letter does it)
- /// the Global Colour Table shifts `block` to 19, 25, 37, 61, 109, 205, 397
- /// or 781. An earlier version of this comment said "offset 14" and was
- /// wrong by a factor of nine.
+ /// the Global Colour Table shifts `block` from 13 to one of 19, 25, 37, 61,
+ /// 109, 205, 397 or 781 — nine possible slots, of which 13 (no colour
+ /// table) is one. An earlier version of this comment said "offset 14" flat;
+ /// the correction that replaced it then dropped the no-table case, which is
+ /// how the count below was wrong.
///
/// So this is a measured trade against merged `main`, not a free win, and
/// both sides are pinned here:
///
- /// * `main` keeps 8 corpus text negatives (`gctflag_ext_fe_p*`) and junks
- /// 18 decodable GIFs whose canvas high bytes happen to be printable.
- /// * this rule keeps all 18 GIFs and junks those same 8 text files — but
- /// only above the budget. At their real 8,960 bytes every build keeps
- /// them as text; it is file LENGTH that flips them, which is exactly
- /// the blind spot the corpus could not show.
+ /// * `main` keeps 10 corpus text negatives and junks 18 decodable GIFs
+ /// whose canvas high bytes happen to be printable.
+ /// * this rule keeps all 18 GIFs and junks those same 10 text files — but
+ /// only above the budget. At their real sizes every build keeps them as
+ /// text; it is file LENGTH that flips them, which is exactly the blind
+ /// spot the corpus could not show.
+ ///
+ /// The 10 are the eight `gctflag_ext_fe_p*` (packed bit 7 set, `block` 19)
+ /// plus `label_fe_thorn.txt` and `label_fe_then_printable_size.txt` (no
+ /// colour table, `block` 13). An earlier version of this comment said
+ /// eight, because the class was enumerated from the filenames a reviewer
+ /// had supplied rather than from the code path — the same mistake this file
+ /// spends fifty lines warning about, made while correcting it. Above the
+ /// budget the shipped rule IS round 2's `true`, so the 10 named at that
+ /// bullet and the 10 here are necessarily the same set.
///
/// Chosen because the 18 are real artifacts that giflib and Pillow both
/// decode, while the 8 are fixtures that only reach this state when
@@ -2450,6 +2465,32 @@ mod printable_magic_tests {
assertion honest matters more than keeping it green: if a future \
change makes it text, re-measure the 18 GIFs on the other side"
);
+
+ // The other two, which the first version of this test did not cover:
+ // no Global Colour Table, so `block` stays at 13 and the `21 FE` pair
+ // sits at 13/14 — the literal "offset 14" shape. Reproduces
+ // `label_fe_thorn.txt`, and it is why the trade costs 10 files, not 8.
+ let mut flat: Vec<u8> = b"GIF89a fichie".to_vec(); // 0..13, packed at 10
+ assert_eq!(flat[10] & 0x80, 0, "no colour table, so block stays 13");
+ flat.push(0x21); // 13
+ flat.push(0xfe); // 14
+ flat.push(b'L'); // 15
+ while flat.len() < GIF_PREFIX + 4096 {
+ flat.extend_from_slice("Le format GIF reste partout sur le web. ".as_bytes());
+ }
+ assert_eq!(
+ sniff_bytes(&flat[..8_960], p, p, false).unwrap().family,
+ Family::TxtProse,
+ "at its corpus size the no-table shape is text on every build"
+ );
+ assert_eq!(
+ sniff_bytes(&flat[..GIF_PREFIX], p, p, false)
+ .unwrap()
+ .family,
+ Family::Binary,
+ "the no-table shape pays the same cost past the budget — if this is \
+ not asserted the trade gets counted from filenames again"
+ );
}
/// The other half of the same arm: prose must not reach it. `0xFE` is not