AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(autoindex): qualify the FBX magic, the last printable signature taken on faith
xerj-org4b6923ee7905
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex acf629183..1b6ed4461 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -147,17 +147,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0uncompressed PCM) still classify as text**, exactly as they do on `ca4d75a`;bounding sectioning memory is the fix for that, and is not attempted here.-- **The new magic-byte signatures now require structural confirmation.** PSD,- TIFF, RIFF, OGG, FLAC, MP3, FBX and EXR were added by the first reland- attempt as bare `starts_with` tests, and five of those signatures are- printable ASCII, so they matched text. Measured, same method as above: a CSV- whose first column header is `ID3` went `csv` -> `binary`, and prose whose- first word is `RIFF`, `OggS`, `fLaC` or `8BPS` went `txt-prose` -> `binary`.- Each printable signature is now qualified by the byte that follows it (PSD- version, RIFF FORM type, Ogg stream-structure version, FLAC block type, ID3v2- major version and synchsafe size), which is what Lucene's- `CodecUtil.checkHeader` does with `CODEC_MAGIC`. The true positives are- unchanged and still covered.+- **The new magic-byte signatures now require structural confirmation.** Nine+ signatures (PSD, both TIFF byte orders, RIFF, OGG, FLAC, MP3, FBX and EXR)+ were added by the first reland attempt as bare `starts_with` tests. **Six** of+ them are entirely printable ASCII — `8BPS`, `RIFF`, `OggS`, `fLaC`, `ID3` and+ the 18 letters `Kaydara FBX Binary` — so they matched ordinary text. Measured+ through the real `sniff()` on `ca4d75a` vs this branch with identical+ fixtures:++ | fixture | `ca4d75a` | first reland |+ | --- | --- | --- |+ | CSV whose first column header is `ID3` | `csv` | `binary`/`mp3` |+ | prose opening `RIFF is a container format used by WAV files. …` | `txt-prose` | `binary`/`riff` |+ | prose opening `OggS pages carry the packets of an Ogg stream …` | `txt-prose` | `binary`/`ogg` |+ | prose opening `fLaC is the four byte magic of a FLAC audio …` | `txt-prose` | `binary`/`flac` |+ | prose opening `8BPS is the magic of an Adobe Photoshop …` | `txt-prose` | `binary`/`psd` |+ | `.md` note opening `Kaydara FBX Binary is the 20-byte magic …` | `txt-prose` | `binary`/`fbx` |++ Each of the six is now qualified by what must follow it: PSD version, RIFF+ FORM type, Ogg stream-structure version, FLAC block type, ID3v2 major version+ and synchsafe size, and for FBX the full 23-byte header (`Kaydara FBX+ Binary`, two spaces, NUL, `0x1A`, `0x00`). The FBX case was missed by the+ first pass at this fix and caught in review — at 18 printable characters it is+ the likeliest of the six to open a real sentence, and likeliest precisely+ inside the Unity/3D-asset corpus this feature targets. Every fixture in the+ table above now classifies exactly as it does on `ca4d75a`. The true positives+ are unchanged and still covered; the FBX true-positive fixture was corrected+ to the real 23-byte header, which is what any Autodesk tool emits.++ This is what Lucene's `CodecUtil.checkHeader` does with `CODEC_MAGIC`+ (`lucene/core/src/java/org/apache/lucene/codecs/CodecUtil.java:183`, which+ hands straight to `checkHeaderNoMagic` at `:202` and refuses the file unless+ the codec name *and* a version in range follow — the magic alone is never+ taken as proof). Apache-2.0, same licence as XERJ; adapted, not copied.++ **Not fixed here, deliberately:** `GIF8` and `BM` are printable-ASCII+ signatures with exactly the same defect, but they are **pre-existing** — they+ are unchanged from `ca4d75a`, and prose opening `GIF8`/`GIF89a`, a CSV whose+ first column header is `GIF8`, and prose opening `BM` were each measured to+ classify as `binary`/`gif` and `binary`/`bmp` on `ca4d75a` **and** on this+ branch, identically. Fixing them changes behaviour unrelated to Unity, so it+ is filed as a follow-up rather than smuggled in here. `GIF8` is listed+ alongside `BM` so the next reader does not conclude `BM` is the only one.- **A junk entry could overwrite a successfully indexed file's catalog row.**When phase B met a record group phase A never sampled, the worker pushed a@@ -182,6 +213,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0documented "which scenes use this script?" query is not impacted. Fixing#332 is an engine-side change to the FTS writer input type.+- **`GIF8` and `BM` are still unqualified printable-ASCII magic signatures**, so+ a text file whose first characters are `GIF8`, `GIF89a` or `BM` is classified+ `binary` and junked. This is **pre-existing, not introduced here** — measured+ identical on `ca4d75a` and on this branch (`binary`/`gif` and `binary`/`bmp`+ in both) — and is left unchanged because fixing it is a behaviour change with+ no Unity content. Both names are recorded here on purpose: `BM` is the+ obvious one and `GIF8` is the one a reader would otherwise miss. Follow-up.+## [1.0.0-rc.16] - 2026-08-13Both rc.15 known issues (the progress-stream forgery and the outer-`.gitignore`diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rsindex 66d87cc7e..bcd4ac129 100644--- a/engine/crates/xerj-autoindex/src/sniff.rs+++ b/engine/crates/xerj-autoindex/src/sniff.rs@@ -183,11 +183,22 @@ fn sniff_bytes(// ~100x its size in RAM through sectioning. Magic bytes are the reliable// signal.//- // Every signature whose bytes are PRINTABLE ASCII carries a `qualify`- // check as well, because "starts with these four letters" is also true of- // ordinary text: a CSV whose first column header is `ID3` and prose whose- // first word is `RIFF`, `OggS`, `fLaC` or `8BPS` were all junked as media- // by the unqualified table (measured — see the tests below). This is the+ // Every signature ADDED BY THIS BRANCH whose bytes are printable ASCII+ // carries a `qualify` check as well, because "starts with these letters"+ // is also true of ordinary text: a CSV whose first column header is `ID3`,+ // and prose whose first word is `RIFF`, `OggS`, `fLaC`, `8BPS` or+ // `Kaydara FBX Binary`, were all junked as media by the unqualified table+ // (measured — see the tests below). That is six of the nine signatures+ // added here; the other three (`II*\x00`, `MM\x00*`, the EXR magic) carry+ // a byte text cannot contain.+ //+ // Two printable signatures remain unqualified — `GIF8` and `BM` — and both+ // have the same defect. They are NOT new: they are unchanged from+ // `ca4d75a` and were measured to classify identically on both trees, so+ // fixing them is a separate behaviour change and is filed as a follow-up+ // rather than smuggled into a Unity PR.+ //+ // Qualifying a magic number is the// rule Lucene applies to its own containers: `CodecUtil.checkHeader`// (`lucene/core/src/java/org/apache/lucene/codecs/CodecUtil.java:183-201`,// Apache-2.0) matches `CODEC_MAGIC` and then hands straight to@@ -217,7 +228,7 @@ fn sniff_bytes((&b"Kaydara FBX Binary"[..],"fbx",- accept as fn(&[u8]) -> bool,+ fbx_header as fn(&[u8]) -> bool,),(&b"\x76\x2f\x31\x01"[..], "exr", accept as fn(&[u8]) -> bool),] {@@ -360,8 +371,19 @@ fn sniff_bytes(/// the bytes are text — the binary heuristics key off this exact string.pub const WINDOWS_1252_LOSSY: &str = "windows-1252 (lossy)";-/// Magic-byte signature that needs no structural confirmation: its bytes are-/// not printable ASCII, so no text file can begin with them by accident.+/// Magic-byte signature taken as sufficient on its own.+///+/// For most callers that is because the signature contains a byte text cannot+/// contain (`\x89PNG`, `\xff\xd8\xff`, `\x7fELF`, `\x00\x00\x01\x00`, the NUL+/// in `II*\x00` / `MM\x00*`, the `\x01` in the EXR magic).+///+/// It is NOT true of the two remaining callers: `GIF8` and `BM` are printable+/// ASCII, so prose or a CSV header beginning with those characters is junked+/// as an image. Both predate this branch — they are unchanged from `ca4d75a`+/// and measured identical on both trees — so they are left alone here rather+/// than widened into an unrelated behaviour change, and are tracked as a+/// follow-up. Do not read this function as a proof that its callers are safe;+/// it is only a statement that no *further* check is performed.fn accept(_prefix: &[u8]) -> bool {true}@@ -396,6 +418,20 @@ fn flac_metadata_block(prefix: &[u8]) -> bool {prefix.len() >= 5 && prefix[4] & 0x7f == 0}+/// Autodesk FBX, binary flavour: the header is the 18 letters `Kaydara FBX+/// Binary`, two spaces, a NUL, then `0x1A 0x00` — 23 bytes before the uint32+/// version field. The three unprintable bytes are the entire discriminator;+/// the letters on their own are ordinary English, and they are *more* likely+/// to appear as prose inside this PR's own target corpus, a Unity/3D-asset+/// tree, than anywhere else. Measured through `sniff()`: a `.md` note opening+/// "Kaydara FBX Binary is the 20-byte magic that opens every binary FBX file+/// exported by Autodesk tools. " was `TxtProse` on `ca4d75a` and+/// `Binary`/`fbx` on this branch until this check existed, and `scan_file`+/// turns `Family::Binary` into "junk: binary content (fbx)" — never indexed.+fn fbx_header(prefix: &[u8]) -> bool {+ prefix.starts_with(b"Kaydara FBX Binary \x00\x1a\x00")+}+/// ID3v2 tag header: `ID3`, a major version (2, 3 and 4 are the versions that/// exist), a revision that is not the reserved 0xFF, a flags byte, and a/// four-byte SYNCHSAFE size whose bytes each have the top bit clear.@@ -846,31 +882,46 @@ mod unity_sniff_tests {/// bytes must win before any text heuristic runs.#[test]fn media_containers_are_binary_by_magic_not_heuristics() {- for (name, head) in [- ("t.psd", &b"8BPS\x00\x01"[..]),- ("t.tif", &b"II*\x00\x08\x00"[..]),- ("t.tif2", &b"MM\x00*\x00\x08"[..]),- ("t.wav", &b"RIFF\x24\x08\x00\x00WAVE"[..]),- ("t.ogg", &b"OggS\x00\x02"[..]),- ("t.flac", &b"fLaC\x00\x00\x00\x22"[..]),- ("t.mp3", &b"ID3\x03\x00\x00\x00\x00\x0f\x76"[..]),- ("t.fbx", &b"Kaydara FBX Binary \x00"[..]),+ for (name, head, kind) in [+ ("t.psd", &b"8BPS\x00\x01"[..], "psd"),+ ("t.tif", &b"II*\x00\x08\x00"[..], "tiff"),+ ("t.tif2", &b"MM\x00*\x00\x08"[..], "tiff"),+ ("t.wav", &b"RIFF\x24\x08\x00\x00WAVE"[..], "riff"),+ ("t.ogg", &b"OggS\x00\x02"[..], "ogg"),+ ("t.flac", &b"fLaC\x00\x00\x00\x22"[..], "flac"),+ ("t.mp3", &b"ID3\x03\x00\x00\x00\x00\x0f\x76"[..], "mp3"),+ // The REAL 23-byte binary-FBX header: the 18 letters, two spaces,+ // NUL, 0x1A, 0x00. The previous fixture stopped at the NUL, which+ // is not a header any Autodesk tool emits and which no longer+ // satisfies `fbx_header`.+ ("t.fbx", &b"Kaydara FBX Binary \x00\x1a\x00"[..], "fbx"),] {let mut bytes = head.to_vec();// A printable tail that WOULD pass the prose heuristics.bytes.extend(b"lorem ipsum dolor sit amet, consectetur adipiscing elit. ".repeat(20));let sn = sniff_bytes(&bytes, Path::new(name), Path::new(name), false).unwrap();assert_eq!(sn.family, Family::Binary, "{name} must be binary");+ assert_eq!(+ sn.binary_kind.as_deref(),+ Some(kind),+ "{name} must be recognised as {kind}, not fall through to the \+ heuristics and land on `unknown`"+ );}}- /// Regression (round 2): the printable-ASCII signatures added by this- /// branch matched TEXT. Measured through `sniff()` on `ca4d75a` vs this- /// branch with identical fixtures, a CSV whose first column header is- /// `ID3` went `Csv` -> `Binary`, and prose whose first word is `RIFF`,- /// `OggS`, `fLaC` or `8BPS` went `TxtProse` -> `Binary`. `scan_file` turns- /// `Family::Binary` into "junk: binary content (unknown)", so each of- /// those files stopped being indexed at all.+ /// Regression: the printable-ASCII signatures added by this branch matched+ /// TEXT. Measured through `sniff()` on `ca4d75a` vs this branch with+ /// identical fixtures, a CSV whose first column header is `ID3` went `Csv`+ /// -> `Binary`, and prose whose first word is `RIFF`, `OggS`, `fLaC`,+ /// `8BPS` or `Kaydara FBX Binary` went `TxtProse` -> `Binary`.+ /// `scan_file` turns `Family::Binary` into "junk: binary content (...)",+ /// so each of those files stopped being indexed at all.+ ///+ /// The FBX case was missed in round 2 and found in review: it was the one+ /// signature in the table still using `accept`, and at 18 printable+ /// characters it is the likeliest of all of them to open a real sentence+ /// — in a Unity/3D-asset corpus above all, which is this PR's target.////// The true positives above must keep passing, which is the whole point:/// the signature is necessary, it is just not sufficient.@@ -901,6 +952,11 @@ mod unity_sniff_tests {"psd","8BPS is the magic of an Adobe Photoshop document header. ",),+ (+ "fbx",+ "Kaydara FBX Binary is the 20-byte magic that opens every binary \+ FBX file exported by Autodesk tools. ",+ ),] {let text = opener.repeat(30);let sn = sniff_bytes(