AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(sniff): state the trade against merged main, and pin the side that costs
xerj-orgd778f70bd1a0
diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rsindex 4a4c67c03..6ce54dfa4 100644--- a/engine/crates/xerj-autoindex/src/sniff.rs+++ b/engine/crates/xerj-autoindex/src/sniff.rs@@ -2363,17 +2363,32 @@ mod printable_magic_tests {/// is a recorded trade and not a later surprise.////// A file that outruns `GIF_PREFIX` with its chain intact is called an- /// image. Prose can reach that state, but only by opening with the whole- /// GIF header: `GIF89a`, eight bytes forming a Logical Screen Descriptor- /// the earlier checks accept, `!` at 13, and — the part ordinary text does- /// not have — a `0xFE` at offset 14, which is `þ` in windows-1252 and not- /// ASCII at all. It must then run for a megabyte without a NUL.+ /// image, and text reaches that state easily — it has no NUL, so the walk+ /// cannot terminate in it. The entry cost is the header alone: `GIF89a`,+ /// eight bytes forming a Logical Screen Descriptor the earlier checks+ /// accept, then `0x21 0xFE`. `0xFE` is `þ` in windows-1252 and not ASCII,+ /// which is the only part ordinary text lacks — and NOT at a fixed offset:+ /// when the packed byte at 10 has bit 7 set (any accented letter does it)+ /// the Global Colour Table shifts `block` to 19, 25, 37, 61, 109, 205, 397+ /// or 781. An earlier version of this comment said "offset 14" and was+ /// wrong by a factor of nine.///- /// The alternative was the round-4 canvas rule, and its cost is unbounded- /// in the other direction: it junks any real image whose canvas high bytes- /// are printable, which is 18 decodable GIFs on `main` today and grows with- /// canvas size. Measured over the 265-file text corpus, this rule changes- /// nothing — 257 stay text, the same 8 reach `Binary` by other arms.+ /// So this is a measured trade against merged `main`, not a free win, and+ /// both sides are pinned here:+ ///+ /// * `main` keeps 8 corpus text negatives (`gctflag_ext_fe_p*`) and junks+ /// 18 decodable GIFs whose canvas high bytes happen to be printable.+ /// * this rule keeps all 18 GIFs and junks those same 8 text files — but+ /// only above the budget. At their real 8,960 bytes every build keeps+ /// them as text; it is file LENGTH that flips them, which is exactly+ /// the blind spot the corpus could not show.+ ///+ /// Chosen because the 18 are real artifacts that giflib and Pillow both+ /// decode, while the 8 are fixtures that only reach this state when+ /// inflated 117x. A sweep of 403,716 real files found 66 opening with GIF+ /// magic and none carrying the comment shape at all, so neither class has a+ /// measured natural base rate — which is the honest reason to write the+ /// trade down rather than claim the rule is safe.#[test]fn prose_past_the_budget_is_called_an_image_and_prose_under_it_is_not() {let mut prose: Vec<u8> = b"GIF89ax".to_vec();@@ -2404,6 +2419,37 @@ mod printable_magic_tests {ever needs to change, change the budget or find evidence the chain \can carry, do not go back to guessing from a fixed offset");++ // The concrete cost, from the corpus itself: the #379 negatives built+ // to represent this exact shape are kept only by being short. This+ // reproduces `gctflag_ext_fe_p80.txt` — packed byte 0x80, so the GCT+ // moves `block` to 19 — and asserts both sides of the length boundary.+ let mut gct: Vec<u8> = b"GIF89a".to_vec();+ gct.extend_from_slice(b"pros"); // 6..10: canvas, four printable bytes+ gct.push(0x80); // 10: packed, GCT flag set by an accented letter+ gct.extend_from_slice(b"st"); // 11, 12+ gct.extend_from_slice(&[0, 0, 0, 0xff, 0xff, 0xff]); // 13..19: the GCT+ gct.push(0x21); // 19: extension introducer, shifted by the GCT+ gct.push(0xfe); // 20: comment label+ gct.push(b' '); // 21: first sub-block size+ while gct.len() < GIF_PREFIX + 4096 {+ gct.extend_from_slice("Le format GIF reste partout sur le web. ".as_bytes());+ }+ let short: Vec<u8> = gct.iter().copied().take(8_960).collect();+ assert_eq!(+ sniff_bytes(&short, p, p, false).unwrap().family,+ Family::TxtProse,+ "at the corpus's own size this file is text on every build"+ );+ let long: Vec<u8> = gct.iter().copied().take(GIF_PREFIX).collect();+ assert_eq!(+ sniff_bytes(&long, p, p, false).unwrap().family,+ Family::Binary,+ "the same bytes past the budget are junked as an image — this is the \+ cost of the rule, and merged main does NOT pay it. Keeping this \+ assertion honest matters more than keeping it green: if a future \+ change makes it text, re-measure the 18 GIFs on the other side"+ );}/// The other half of the same arm: prose must not reach it. `0xFE` is not