Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(sniff): correct the corpus claim, pin both canvas offsets, and put each doc back on its own test

xerj-orgff1949271d8e
diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rs
index 4a5a9e876..3c3f41a9c 100644
--- a/engine/crates/xerj-autoindex/src/sniff.rs
+++ b/engine/crates/xerj-autoindex/src/sniff.rs
@@ -590,8 +590,10 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {
// none, so ANY text file that clears the header gate
// runs its chain to EOF and lands here — at 3 KB as
// readily as at 3 MB. It is then called an image
- // whenever `prefix[7]` or `prefix[9]` is a control byte
- // other than \t, \n or \r. Measured: the same prose
+ // whenever `prefix[7]` or `prefix[9]` is a byte below
+ // 0x20 other than \t, \n or \r — 29 values, not "a
+ // control byte": DEL (0x7F) and the C1 range do not
+ // trigger it. Measured: the same prose
// with 0x0B, 0x0C or 0x1B at offset 7 is Binary at
// 3,000 / 20,000 / 600,000 bytes, and with 0x0A stays
// TxtProse at all three.
@@ -601,8 +603,17 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {
// as "10 corpus text files above the budget" is the
// full-buffer path's cost alone; the canvas path adds
// prose with a control byte in a canvas high slot, at
- // any size, and no corpus file exercises it because the
- // negatives were built with `\n` there.
+ // any size, and no corpus file exercises it — but not
+ // for the reason an earlier version of this comment
+ // gave. It said the negatives were built with `\n` in
+ // that slot. They were not: of the 264 GIF-magic text
+ // negatives, ZERO carry 0x0A at offset 7 or 9, and zero
+ // carry any byte below 0x20 there at all. The canvas
+ // high bytes are ordinary printable prose — 'b' x194,
+ // 'e' x26 at offset 7; 'd' x180, 't' x26 at offset 9 —
+ // so the corpus misses this path by having no control
+ // byte anywhere near the canvas, which is a property of
+ // English text and not of how the fixtures were built.
//
// That composition is measured against the budget test
// alone, not assumed. Identical on every set that
@@ -2373,6 +2384,11 @@ mod printable_magic_tests {
/// `GIF_PREFIX` bounds the FULL-buffer path, so pin it to a fixed byte
/// count rather than to itself.
///
+ /// This fixture reaches that path only because `\n` at offset 7 keeps it
+ /// off the canvas path — an accident of the fixture, not a property of the
+ /// rule. `the_canvas_path_junks_prose_at_any_size_and_the_budget_does_not_bound_it`
+ /// pins the path this one cannot see, and says so there rather than here.
+ ///
/// It is not "the whole rule" — the comment here used to say that, and
/// `the_canvas_path_junks_prose_at_any_size_and_the_budget_does_not_bound_it`
/// above is why it is not. This fixture reaches the budget path only
@@ -2381,20 +2397,26 @@ mod printable_magic_tests {
/// down here so the next person to widen the text cost does not read this
/// test as a safety net it is not. Both fixtures above are written in terms of the constant
/// and track any change to it; without this, `>= 300_000` passes the suite.
- /// The canvas path's cost, which has NO size floor — pinned because the
- /// two tests below are green only by accident of their fixtures.
+ /// The canvas path's cost, which has NO size floor.
+ ///
+ /// The two tests below bound only the FULL-buffer path. They reach it
+ /// because their fixtures put `\n` at offset 7 — one of the three bytes
+ /// `canvas_dimension_text_cannot_write` excludes — so the canvas path never
+ /// fires for them. That is an accident of those fixtures, not a property of
+ /// the rule, and without this test nothing would pin the other path at all.
///
- /// Both use `\n` at offset 7, and `\n` is one of the three control bytes
- /// `canvas_dimension_text_cannot_write` deliberately excludes, so neither
- /// of them ever reaches the canvas path they were written to bound. Any
- /// other control byte there makes the same prose an image at any size.
+ /// Both offsets are exercised: the rule is `prefix[7] || prefix[9]`, and a
+ /// test that varied only one would let a future edit drop the other half
+ /// without failing anything.
#[test]
fn the_canvas_path_junks_prose_at_any_size_and_the_budget_does_not_bound_it() {
let p = Path::new("notes.txt");
- let build = |high: u8, size: usize| {
- let mut v: Vec<u8> = b"GIF89ax".to_vec();
- v.push(high);
- v.extend_from_slice(b"yzest");
+ // `slot` is 7 or 9 — the two canvas high bytes the rule reads. Varying
+ // only one would let a future edit drop the other half of the
+ // disjunction without failing anything.
+ let build = |slot: usize, high: u8, size: usize| {
+ let mut v: Vec<u8> = b"GIF89axyyzest".to_vec();
+ v[slot] = high;
v.push(b'!');
v.push(0xfe);
v.push(b'L');
@@ -2405,24 +2427,37 @@ mod printable_magic_tests {
v
};
for size in [3_000usize, 20_000, 600_000] {
- for high in [0x0bu8, 0x0c, 0x1b] {
+ for slot in [7usize, 9] {
+ for high in [0x0bu8, 0x0c, 0x1b] {
+ assert_eq!(
+ sniff_bytes(&build(slot, high, size), p, p, false)
+ .unwrap()
+ .family,
+ Family::Binary,
+ "prose with {high:#04x} at offset {slot} must be called an \
+ image at {size} bytes: the canvas path has no size floor, \
+ so any statement of this rule's cost that says \"above \
+ the budget\" describes only the other half"
+ );
+ }
assert_eq!(
- sniff_bytes(&build(high, size), p, p, false).unwrap().family,
- Family::Binary,
- "prose with control byte {high:#04x} at offset 7 is called an \
- image at {size} bytes — the canvas path is not size-gated, \
- and any statement of this rule's cost that says \"above the \
- budget\" is describing only the other half"
+ sniff_bytes(&build(slot, b'\n', size), p, p, false)
+ .unwrap()
+ .family,
+ Family::TxtProse,
+ "\\n at offset {slot} must stay text: it is excluded from the \
+ canvas rule, and that exclusion is the only reason the two \
+ budget tests below reach the path they claim to bound"
+ );
+ assert_eq!(
+ sniff_bytes(&build(slot, 0x7f, size), p, p, false)
+ .unwrap()
+ .family,
+ Family::TxtProse,
+ "0x7F must stay text at offset {slot}: the rule is \"below \
+ 0x20\", not \"a control byte\", and DEL is a control byte"
);
}
- assert_eq!(
- sniff_bytes(&build(b'\n', size), p, p, false)
- .unwrap()
- .family,
- Family::TxtProse,
- "\\n at offset 7 is excluded, which is the ONLY reason the two \
- tests below look like they bound anything"
- );
}
}