AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(sniff): a paragraph describing a rule three commits dead, and a third unmeasured universal
xerj-orgef2eee685029
diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rsindex 63700f71c..ef45eba5d 100644--- a/engine/crates/xerj-autoindex/src/sniff.rs+++ b/engine/crates/xerj-autoindex/src/sniff.rs@@ -553,9 +553,14 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {// is a NUL landing on a hop; every other byte 1..=255 is// a legal sub-block size. So "the chain survived" means// "no NUL landed on a hop" and nothing else. Text has no- // NULs, so text survives with probability 1 — measured:- // 265 of 265 corpus text negatives walk a full megabyte- // unbroken when tiled past the budget. High-entropy+ // NULs, so text survives with probability 1. Measured+ // precisely: 0 of the 265 corpus text negatives contain+ // a NUL anywhere, so no walk over any of them can+ // terminate — which is the property this needs. (Only+ // 10 of the 265 actually reach the walk; the rest fail+ // the header gate first. An earlier draft said "265 of+ // 265 walk a full megabyte", which overstated what was+ // exercised even though the conclusion holds.) High-entropy// binary is what the walk rejects: 0 of 200 uniform// random megabytes survive, matching// (255/256)^(2^20/128.5) = 1.4e-14. It is not even much@@ -612,17 +617,25 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {// there. The distribution is 'b' x194 / 'e' x26 at// offset 7 and 'd' x180 / 't' x26 at offset 9, and 180// of the files carry the literal bytes `abcd` at 6..9 —- // because the generator splices `filler = a,b,c,d,…`- // straight after the magic (build_text.py:199). So the+ // because the generators splice `filler = a,b,c,d,…`+ // straight after the magic — build_text.py:199 for 140+ // of them, the same line duplicated at build_text2.py:62+ // and :72 for the other 40. So the// frequencies are the builder's alphabet, NOT "ordinary// prose" as an earlier version of this comment claimed,// and certainly not a property of English: the corpus// deliberately includes Spanish, French, Russian,// Chinese, Greek, Hebrew and Arabic negatives.//- // Every one of these files is synthetic — nothing in- // real prose opens with GIF magic — and the builder- // chose what follows it. No fixture family was ever+ // Every one of these files was generated by+ // build_text.py / build_text2.py, so the builder chose+ // what follows the magic. (An earlier draft added+ // "nothing in real prose opens with GIF magic" here.+ // That is not among what was counted, and this file+ // elsewhere rests on the opposite: the block-introducer+ // arm exists because `"GIF89a header, the six bytes at+ // the front of every GIF file"` is real prose that does.)+ // No fixture family was ever// built to put a control byte at 7 or 9, so the corpus// has no power to see this path. That is a fact about// its construction, and there is no further mechanism@@ -748,12 +761,18 @@ fn gif_screen_descriptor(prefix: &[u8]) -> bool {// written down as a property of the format. The budget test is not that: a// full buffer means a megabyte of sub-block chain was actually traversed.//-// The short-buffer answer costs real files and the cost is stated at the arm-// itself: truncated GIFs, plain and gzipped alike, go to the prose extractor.-// An earlier draft of this comment claimed it "can only mislabel a `.gif.gz`";-// that was false — 7 of 7 plain truncated corpus GIFs flip too — and it is-// recorded here because narrowing a known cost until it sounds acceptable is-// the same failure as inventing a discriminator, one document later.+// HISTORICAL, and left here because the mistake is instructive. For three+// commits the short-buffer answer was plain `prefix.len() >= GIF_PREFIX`, and+// this paragraph recorded its cost in the present tense: truncated GIFs, plain+// and gzipped alike, going to the prose extractor — 7 of 7 plain cuts, after an+// earlier draft had wrongly narrowed it to `.gif.gz` only. All true then.+//+// It stopped being true when the canvas fallback came back: with it, 13 of 13+// curated cuts and 60 of 60 real-corpus cuts at 300 KB / 700 KB / 1,048,575 B+// are Binary, and the arm now says so. The paragraph nonetheless survived four+// commits and six reviews still asserting the opposite of the code twenty lines+// above it — which is the very failure it was written to warn about, one+// document later, in the document doing the warning.fn canvas_dimension_text_cannot_write(prefix: &[u8]) -> bool {// `prefix[..13]` is guaranteed by the length check in the only caller.