Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(autoindex): qualify the FBX magic, the last printable signature taken on faith

xerj-org4b6923ee7905
diff --git a/CHANGELOG.md b/CHANGELOG.md
index acf629183..1b6ed4461 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -147,17 +147,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
uncompressed PCM) still classify as text**, exactly as they do on `ca4d75a`;
bounding sectioning memory is the fix for that, and is not attempted here.
-- **The new magic-byte signatures now require structural confirmation.** PSD,
- TIFF, RIFF, OGG, FLAC, MP3, FBX and EXR were added by the first reland
- attempt as bare `starts_with` tests, and five of those signatures are
- printable ASCII, so they matched text. Measured, same method as above: a CSV
- whose first column header is `ID3` went `csv` -> `binary`, and prose whose
- first word is `RIFF`, `OggS`, `fLaC` or `8BPS` went `txt-prose` -> `binary`.
- Each printable signature is now qualified by the byte that follows it (PSD
- version, RIFF FORM type, Ogg stream-structure version, FLAC block type, ID3v2
- major version and synchsafe size), which is what Lucene's
- `CodecUtil.checkHeader` does with `CODEC_MAGIC`. The true positives are
- unchanged and still covered.
+- **The new magic-byte signatures now require structural confirmation.** Nine
+ signatures (PSD, both TIFF byte orders, RIFF, OGG, FLAC, MP3, FBX and EXR)
+ were added by the first reland attempt as bare `starts_with` tests. **Six** of
+ them are entirely printable ASCII — `8BPS`, `RIFF`, `OggS`, `fLaC`, `ID3` and
+ the 18 letters `Kaydara FBX Binary` — so they matched ordinary text. Measured
+ through the real `sniff()` on `ca4d75a` vs this branch with identical
+ fixtures:
+
+ | fixture | `ca4d75a` | first reland |
+ | --- | --- | --- |
+ | CSV whose first column header is `ID3` | `csv` | `binary`/`mp3` |
+ | prose opening `RIFF is a container format used by WAV files. …` | `txt-prose` | `binary`/`riff` |
+ | prose opening `OggS pages carry the packets of an Ogg stream …` | `txt-prose` | `binary`/`ogg` |
+ | prose opening `fLaC is the four byte magic of a FLAC audio …` | `txt-prose` | `binary`/`flac` |
+ | prose opening `8BPS is the magic of an Adobe Photoshop …` | `txt-prose` | `binary`/`psd` |
+ | `.md` note opening `Kaydara FBX Binary is the 20-byte magic …` | `txt-prose` | `binary`/`fbx` |
+
+ Each of the six is now qualified by what must follow it: PSD version, RIFF
+ FORM type, Ogg stream-structure version, FLAC block type, ID3v2 major version
+ and synchsafe size, and for FBX the full 23-byte header (`Kaydara FBX
+ Binary`, two spaces, NUL, `0x1A`, `0x00`). The FBX case was missed by the
+ first pass at this fix and caught in review — at 18 printable characters it is
+ the likeliest of the six to open a real sentence, and likeliest precisely
+ inside the Unity/3D-asset corpus this feature targets. Every fixture in the
+ table above now classifies exactly as it does on `ca4d75a`. The true positives
+ are unchanged and still covered; the FBX true-positive fixture was corrected
+ to the real 23-byte header, which is what any Autodesk tool emits.
+
+ This is what Lucene's `CodecUtil.checkHeader` does with `CODEC_MAGIC`
+ (`lucene/core/src/java/org/apache/lucene/codecs/CodecUtil.java:183`, which
+ hands straight to `checkHeaderNoMagic` at `:202` and refuses the file unless
+ the codec name *and* a version in range follow — the magic alone is never
+ taken as proof). Apache-2.0, same licence as XERJ; adapted, not copied.
+
+ **Not fixed here, deliberately:** `GIF8` and `BM` are printable-ASCII
+ signatures with exactly the same defect, but they are **pre-existing** — they
+ are unchanged from `ca4d75a`, and prose opening `GIF8`/`GIF89a`, a CSV whose
+ first column header is `GIF8`, and prose opening `BM` were each measured to
+ classify as `binary`/`gif` and `binary`/`bmp` on `ca4d75a` **and** on this
+ branch, identically. Fixing them changes behaviour unrelated to Unity, so it
+ is filed as a follow-up rather than smuggled in here. `GIF8` is listed
+ alongside `BM` so the next reader does not conclude `BM` is the only one.
- **A junk entry could overwrite a successfully indexed file's catalog row.**
When phase B met a record group phase A never sampled, the worker pushed a
@@ -182,6 +213,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
documented "which scenes use this script?" query is not impacted. Fixing
#332 is an engine-side change to the FTS writer input type.
+- **`GIF8` and `BM` are still unqualified printable-ASCII magic signatures**, so
+ a text file whose first characters are `GIF8`, `GIF89a` or `BM` is classified
+ `binary` and junked. This is **pre-existing, not introduced here** — measured
+ identical on `ca4d75a` and on this branch (`binary`/`gif` and `binary`/`bmp`
+ in both) — and is left unchanged because fixing it is a behaviour change with
+ no Unity content. Both names are recorded here on purpose: `BM` is the
+ obvious one and `GIF8` is the one a reader would otherwise miss. Follow-up.
+
## [1.0.0-rc.16] - 2026-08-13
Both rc.15 known issues (the progress-stream forgery and the outer-`.gitignore`
diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rs
index 66d87cc7e..bcd4ac129 100644
--- a/engine/crates/xerj-autoindex/src/sniff.rs
+++ b/engine/crates/xerj-autoindex/src/sniff.rs
@@ -183,11 +183,22 @@ fn sniff_bytes(
// ~100x its size in RAM through sectioning. Magic bytes are the reliable
// signal.
//
- // Every signature whose bytes are PRINTABLE ASCII carries a `qualify`
- // check as well, because "starts with these four letters" is also true of
- // ordinary text: a CSV whose first column header is `ID3` and prose whose
- // first word is `RIFF`, `OggS`, `fLaC` or `8BPS` were all junked as media
- // by the unqualified table (measured — see the tests below). This is the
+ // Every signature ADDED BY THIS BRANCH whose bytes are printable ASCII
+ // carries a `qualify` check as well, because "starts with these letters"
+ // is also true of ordinary text: a CSV whose first column header is `ID3`,
+ // and prose whose first word is `RIFF`, `OggS`, `fLaC`, `8BPS` or
+ // `Kaydara FBX Binary`, were all junked as media by the unqualified table
+ // (measured — see the tests below). That is six of the nine signatures
+ // added here; the other three (`II*\x00`, `MM\x00*`, the EXR magic) carry
+ // a byte text cannot contain.
+ //
+ // Two printable signatures remain unqualified — `GIF8` and `BM` — and both
+ // have the same defect. They are NOT new: they are unchanged from
+ // `ca4d75a` and were measured to classify identically on both trees, so
+ // fixing them is a separate behaviour change and is filed as a follow-up
+ // rather than smuggled into a Unity PR.
+ //
+ // Qualifying a magic number is the
// rule Lucene applies to its own containers: `CodecUtil.checkHeader`
// (`lucene/core/src/java/org/apache/lucene/codecs/CodecUtil.java:183-201`,
// Apache-2.0) matches `CODEC_MAGIC` and then hands straight to
@@ -217,7 +228,7 @@ fn sniff_bytes(
(
&b"Kaydara FBX Binary"[..],
"fbx",
- accept as fn(&[u8]) -> bool,
+ fbx_header as fn(&[u8]) -> bool,
),
(&b"\x76\x2f\x31\x01"[..], "exr", accept as fn(&[u8]) -> bool),
] {
@@ -360,8 +371,19 @@ fn sniff_bytes(
/// the bytes are text — the binary heuristics key off this exact string.
pub const WINDOWS_1252_LOSSY: &str = "windows-1252 (lossy)";
-/// Magic-byte signature that needs no structural confirmation: its bytes are
-/// not printable ASCII, so no text file can begin with them by accident.
+/// Magic-byte signature taken as sufficient on its own.
+///
+/// For most callers that is because the signature contains a byte text cannot
+/// contain (`\x89PNG`, `\xff\xd8\xff`, `\x7fELF`, `\x00\x00\x01\x00`, the NUL
+/// in `II*\x00` / `MM\x00*`, the `\x01` in the EXR magic).
+///
+/// It is NOT true of the two remaining callers: `GIF8` and `BM` are printable
+/// ASCII, so prose or a CSV header beginning with those characters is junked
+/// as an image. Both predate this branch — they are unchanged from `ca4d75a`
+/// and measured identical on both trees — so they are left alone here rather
+/// than widened into an unrelated behaviour change, and are tracked as a
+/// follow-up. Do not read this function as a proof that its callers are safe;
+/// it is only a statement that no *further* check is performed.
fn accept(_prefix: &[u8]) -> bool {
true
}
@@ -396,6 +418,20 @@ fn flac_metadata_block(prefix: &[u8]) -> bool {
prefix.len() >= 5 && prefix[4] & 0x7f == 0
}
+/// Autodesk FBX, binary flavour: the header is the 18 letters `Kaydara FBX
+/// Binary`, two spaces, a NUL, then `0x1A 0x00` — 23 bytes before the uint32
+/// version field. The three unprintable bytes are the entire discriminator;
+/// the letters on their own are ordinary English, and they are *more* likely
+/// to appear as prose inside this PR's own target corpus, a Unity/3D-asset
+/// tree, than anywhere else. Measured through `sniff()`: a `.md` note opening
+/// "Kaydara FBX Binary is the 20-byte magic that opens every binary FBX file
+/// exported by Autodesk tools. " was `TxtProse` on `ca4d75a` and
+/// `Binary`/`fbx` on this branch until this check existed, and `scan_file`
+/// turns `Family::Binary` into "junk: binary content (fbx)" — never indexed.
+fn fbx_header(prefix: &[u8]) -> bool {
+ prefix.starts_with(b"Kaydara FBX Binary \x00\x1a\x00")
+}
+
/// ID3v2 tag header: `ID3`, a major version (2, 3 and 4 are the versions that
/// exist), a revision that is not the reserved 0xFF, a flags byte, and a
/// four-byte SYNCHSAFE size whose bytes each have the top bit clear.
@@ -846,31 +882,46 @@ mod unity_sniff_tests {
/// bytes must win before any text heuristic runs.
#[test]
fn media_containers_are_binary_by_magic_not_heuristics() {
- for (name, head) in [
- ("t.psd", &b"8BPS\x00\x01"[..]),
- ("t.tif", &b"II*\x00\x08\x00"[..]),
- ("t.tif2", &b"MM\x00*\x00\x08"[..]),
- ("t.wav", &b"RIFF\x24\x08\x00\x00WAVE"[..]),
- ("t.ogg", &b"OggS\x00\x02"[..]),
- ("t.flac", &b"fLaC\x00\x00\x00\x22"[..]),
- ("t.mp3", &b"ID3\x03\x00\x00\x00\x00\x0f\x76"[..]),
- ("t.fbx", &b"Kaydara FBX Binary \x00"[..]),
+ for (name, head, kind) in [
+ ("t.psd", &b"8BPS\x00\x01"[..], "psd"),
+ ("t.tif", &b"II*\x00\x08\x00"[..], "tiff"),
+ ("t.tif2", &b"MM\x00*\x00\x08"[..], "tiff"),
+ ("t.wav", &b"RIFF\x24\x08\x00\x00WAVE"[..], "riff"),
+ ("t.ogg", &b"OggS\x00\x02"[..], "ogg"),
+ ("t.flac", &b"fLaC\x00\x00\x00\x22"[..], "flac"),
+ ("t.mp3", &b"ID3\x03\x00\x00\x00\x00\x0f\x76"[..], "mp3"),
+ // The REAL 23-byte binary-FBX header: the 18 letters, two spaces,
+ // NUL, 0x1A, 0x00. The previous fixture stopped at the NUL, which
+ // is not a header any Autodesk tool emits and which no longer
+ // satisfies `fbx_header`.
+ ("t.fbx", &b"Kaydara FBX Binary \x00\x1a\x00"[..], "fbx"),
] {
let mut bytes = head.to_vec();
// A printable tail that WOULD pass the prose heuristics.
bytes.extend(b"lorem ipsum dolor sit amet, consectetur adipiscing elit. ".repeat(20));
let sn = sniff_bytes(&bytes, Path::new(name), Path::new(name), false).unwrap();
assert_eq!(sn.family, Family::Binary, "{name} must be binary");
+ assert_eq!(
+ sn.binary_kind.as_deref(),
+ Some(kind),
+ "{name} must be recognised as {kind}, not fall through to the \
+ heuristics and land on `unknown`"
+ );
}
}
- /// Regression (round 2): the printable-ASCII signatures added by this
- /// branch matched TEXT. Measured through `sniff()` on `ca4d75a` vs this
- /// branch with identical fixtures, a CSV whose first column header is
- /// `ID3` went `Csv` -> `Binary`, and prose whose first word is `RIFF`,
- /// `OggS`, `fLaC` or `8BPS` went `TxtProse` -> `Binary`. `scan_file` turns
- /// `Family::Binary` into "junk: binary content (unknown)", so each of
- /// those files stopped being indexed at all.
+ /// Regression: the printable-ASCII signatures added by this branch matched
+ /// TEXT. Measured through `sniff()` on `ca4d75a` vs this branch with
+ /// identical fixtures, a CSV whose first column header is `ID3` went `Csv`
+ /// -> `Binary`, and prose whose first word is `RIFF`, `OggS`, `fLaC`,
+ /// `8BPS` or `Kaydara FBX Binary` went `TxtProse` -> `Binary`.
+ /// `scan_file` turns `Family::Binary` into "junk: binary content (...)",
+ /// so each of those files stopped being indexed at all.
+ ///
+ /// The FBX case was missed in round 2 and found in review: it was the one
+ /// signature in the table still using `accept`, and at 18 printable
+ /// characters it is the likeliest of all of them to open a real sentence
+ /// — in a Unity/3D-asset corpus above all, which is this PR's target.
///
/// The true positives above must keep passing, which is the whole point:
/// the signature is necessary, it is just not sufficient.
@@ -901,6 +952,11 @@ mod unity_sniff_tests {
"psd",
"8BPS is the magic of an Adobe Photoshop document header. ",
),
+ (
+ "fbx",
+ "Kaydara FBX Binary is the 20-byte magic that opens every binary \
+ FBX file exported by Autodesk tools. ",
+ ),
] {
let text = opener.repeat(30);
let sn = sniff_bytes(