Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(sniff): a paragraph describing a rule three commits dead, and a third unmeasured universal

xerj-orgef2eee685029
diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rs
index 63700f71c..ef45eba5d 100644
--- a/engine/crates/xerj-autoindex/src/sniff.rs
+++ b/engine/crates/xerj-autoindex/src/sniff.rs
@@ -553,9 +553,14 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {
// is a NUL landing on a hop; every other byte 1..=255 is
// a legal sub-block size. So "the chain survived" means
// "no NUL landed on a hop" and nothing else. Text has no
- // NULs, so text survives with probability 1 — measured:
- // 265 of 265 corpus text negatives walk a full megabyte
- // unbroken when tiled past the budget. High-entropy
+ // NULs, so text survives with probability 1. Measured
+ // precisely: 0 of the 265 corpus text negatives contain
+ // a NUL anywhere, so no walk over any of them can
+ // terminate — which is the property this needs. (Only
+ // 10 of the 265 actually reach the walk; the rest fail
+ // the header gate first. An earlier draft said "265 of
+ // 265 walk a full megabyte", which overstated what was
+ // exercised even though the conclusion holds.) High-entropy
// binary is what the walk rejects: 0 of 200 uniform
// random megabytes survive, matching
// (255/256)^(2^20/128.5) = 1.4e-14. It is not even much
@@ -612,17 +617,25 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {
// there. The distribution is 'b' x194 / 'e' x26 at
// offset 7 and 'd' x180 / 't' x26 at offset 9, and 180
// of the files carry the literal bytes `abcd` at 6..9 —
- // because the generator splices `filler = a,b,c,d,…`
- // straight after the magic (build_text.py:199). So the
+ // because the generators splice `filler = a,b,c,d,…`
+ // straight after the magic — build_text.py:199 for 140
+ // of them, the same line duplicated at build_text2.py:62
+ // and :72 for the other 40. So the
// frequencies are the builder's alphabet, NOT "ordinary
// prose" as an earlier version of this comment claimed,
// and certainly not a property of English: the corpus
// deliberately includes Spanish, French, Russian,
// Chinese, Greek, Hebrew and Arabic negatives.
//
- // Every one of these files is synthetic — nothing in
- // real prose opens with GIF magic — and the builder
- // chose what follows it. No fixture family was ever
+ // Every one of these files was generated by
+ // build_text.py / build_text2.py, so the builder chose
+ // what follows the magic. (An earlier draft added
+ // "nothing in real prose opens with GIF magic" here.
+ // That is not among what was counted, and this file
+ // elsewhere rests on the opposite: the block-introducer
+ // arm exists because `"GIF89a header, the six bytes at
+ // the front of every GIF file"` is real prose that does.)
+ // No fixture family was ever
// built to put a control byte at 7 or 9, so the corpus
// has no power to see this path. That is a fact about
// its construction, and there is no further mechanism
@@ -748,12 +761,18 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {
// written down as a property of the format. The budget test is not that: a
// full buffer means a megabyte of sub-block chain was actually traversed.
//
-// The short-buffer answer costs real files and the cost is stated at the arm
-// itself: truncated GIFs, plain and gzipped alike, go to the prose extractor.
-// An earlier draft of this comment claimed it "can only mislabel a `.gif.gz`";
-// that was false — 7 of 7 plain truncated corpus GIFs flip too — and it is
-// recorded here because narrowing a known cost until it sounds acceptable is
-// the same failure as inventing a discriminator, one document later.
+// HISTORICAL, and left here because the mistake is instructive. For three
+// commits the short-buffer answer was plain `prefix.len() >= GIF_PREFIX`, and
+// this paragraph recorded its cost in the present tense: truncated GIFs, plain
+// and gzipped alike, going to the prose extractor — 7 of 7 plain cuts, after an
+// earlier draft had wrongly narrowed it to `.gif.gz` only. All true then.
+//
+// It stopped being true when the canvas fallback came back: with it, 13 of 13
+// curated cuts and 60 of 60 real-corpus cuts at 300 KB / 700 KB / 1,048,575 B
+// are Binary, and the arm now says so. The paragraph nonetheless survived four
+// commits and six reviews still asserting the opposite of the code twenty lines
+// above it — which is the very failure it was written to warn about, one
+// document later, in the document doing the warning.
fn canvas_dimension_text_cannot_write(prefix: &[u8]) -> bool {
// `prefix[..13]` is guaranteed by the length check in the only caller.