AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(sniff): correct the corpus claim, pin both canvas offsets, and put each doc back on its own test
xerj-orgff1949271d8e
diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rsindex 4a5a9e876..3c3f41a9c 100644--- a/engine/crates/xerj-autoindex/src/sniff.rs+++ b/engine/crates/xerj-autoindex/src/sniff.rs@@ -590,8 +590,10 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {// none, so ANY text file that clears the header gate// runs its chain to EOF and lands here — at 3 KB as// readily as at 3 MB. It is then called an image- // whenever `prefix[7]` or `prefix[9]` is a control byte- // other than \t, \n or \r. Measured: the same prose+ // whenever `prefix[7]` or `prefix[9]` is a byte below+ // 0x20 other than \t, \n or \r — 29 values, not "a+ // control byte": DEL (0x7F) and the C1 range do not+ // trigger it. Measured: the same prose// with 0x0B, 0x0C or 0x1B at offset 7 is Binary at// 3,000 / 20,000 / 600,000 bytes, and with 0x0A stays// TxtProse at all three.@@ -601,8 +603,17 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {// as "10 corpus text files above the budget" is the// full-buffer path's cost alone; the canvas path adds// prose with a control byte in a canvas high slot, at- // any size, and no corpus file exercises it because the- // negatives were built with `\n` there.+ // any size, and no corpus file exercises it — but not+ // for the reason an earlier version of this comment+ // gave. It said the negatives were built with `\n` in+ // that slot. They were not: of the 264 GIF-magic text+ // negatives, ZERO carry 0x0A at offset 7 or 9, and zero+ // carry any byte below 0x20 there at all. The canvas+ // high bytes are ordinary printable prose — 'b' x194,+ // 'e' x26 at offset 7; 'd' x180, 't' x26 at offset 9 —+ // so the corpus misses this path by having no control+ // byte anywhere near the canvas, which is a property of+ // English text and not of how the fixtures were built.//// That composition is measured against the budget test// alone, not assumed. Identical on every set that@@ -2373,6 +2384,11 @@ mod printable_magic_tests {/// `GIF_PREFIX` bounds the FULL-buffer path, so pin it to a fixed byte/// count rather than to itself.///+ /// This fixture reaches that path only because `\n` at offset 7 keeps it+ /// off the canvas path — an accident of the fixture, not a property of the+ /// rule. `the_canvas_path_junks_prose_at_any_size_and_the_budget_does_not_bound_it`+ /// pins the path this one cannot see, and says so there rather than here.+ ////// It is not "the whole rule" — the comment here used to say that, and/// `the_canvas_path_junks_prose_at_any_size_and_the_budget_does_not_bound_it`/// above is why it is not. This fixture reaches the budget path only@@ -2381,20 +2397,26 @@ mod printable_magic_tests {/// down here so the next person to widen the text cost does not read this/// test as a safety net it is not. Both fixtures above are written in terms of the constant/// and track any change to it; without this, `>= 300_000` passes the suite.- /// The canvas path's cost, which has NO size floor — pinned because the- /// two tests below are green only by accident of their fixtures.+ /// The canvas path's cost, which has NO size floor.+ ///+ /// The two tests below bound only the FULL-buffer path. They reach it+ /// because their fixtures put `\n` at offset 7 — one of the three bytes+ /// `canvas_dimension_text_cannot_write` excludes — so the canvas path never+ /// fires for them. That is an accident of those fixtures, not a property of+ /// the rule, and without this test nothing would pin the other path at all.///- /// Both use `\n` at offset 7, and `\n` is one of the three control bytes- /// `canvas_dimension_text_cannot_write` deliberately excludes, so neither- /// of them ever reaches the canvas path they were written to bound. Any- /// other control byte there makes the same prose an image at any size.+ /// Both offsets are exercised: the rule is `prefix[7] || prefix[9]`, and a+ /// test that varied only one would let a future edit drop the other half+ /// without failing anything.#[test]fn the_canvas_path_junks_prose_at_any_size_and_the_budget_does_not_bound_it() {let p = Path::new("notes.txt");- let build = |high: u8, size: usize| {- let mut v: Vec<u8> = b"GIF89ax".to_vec();- v.push(high);- v.extend_from_slice(b"yzest");+ // `slot` is 7 or 9 — the two canvas high bytes the rule reads. Varying+ // only one would let a future edit drop the other half of the+ // disjunction without failing anything.+ let build = |slot: usize, high: u8, size: usize| {+ let mut v: Vec<u8> = b"GIF89axyyzest".to_vec();+ v[slot] = high;v.push(b'!');v.push(0xfe);v.push(b'L');@@ -2405,24 +2427,37 @@ mod printable_magic_tests {v};for size in [3_000usize, 20_000, 600_000] {- for high in [0x0bu8, 0x0c, 0x1b] {+ for slot in [7usize, 9] {+ for high in [0x0bu8, 0x0c, 0x1b] {+ assert_eq!(+ sniff_bytes(&build(slot, high, size), p, p, false)+ .unwrap()+ .family,+ Family::Binary,+ "prose with {high:#04x} at offset {slot} must be called an \+ image at {size} bytes: the canvas path has no size floor, \+ so any statement of this rule's cost that says \"above \+ the budget\" describes only the other half"+ );+ }assert_eq!(- sniff_bytes(&build(high, size), p, p, false).unwrap().family,- Family::Binary,- "prose with control byte {high:#04x} at offset 7 is called an \- image at {size} bytes — the canvas path is not size-gated, \- and any statement of this rule's cost that says \"above the \- budget\" is describing only the other half"+ sniff_bytes(&build(slot, b'\n', size), p, p, false)+ .unwrap()+ .family,+ Family::TxtProse,+ "\\n at offset {slot} must stay text: it is excluded from the \+ canvas rule, and that exclusion is the only reason the two \+ budget tests below reach the path they claim to bound"+ );+ assert_eq!(+ sniff_bytes(&build(slot, 0x7f, size), p, p, false)+ .unwrap()+ .family,+ Family::TxtProse,+ "0x7F must stay text at offset {slot}: the rule is \"below \+ 0x20\", not \"a control byte\", and DEL is a control byte");}- assert_eq!(- sniff_bytes(&build(b'\n', size), p, p, false)- .unwrap()- .family,- Family::TxtProse,- "\\n at offset 7 is excluded, which is the ONLY reason the two \- tests below look like they bound anything"- );}}