Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge origin/main into fix/issue-328-vector-term-dict

xerj-orgc23bb4ef775c
diff --cc CHANGELOG.md
index 1b314655c,4c45bf407..fd3c37d15
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -153,235 -178,211 +178,440 @@@ and this project adheres to [Semantic V
from the moment it is allowlisted. Seed the target with snapshot/restore
first if it needs the existing documents.
+### Performance
+
+- **A field mapped `dense_vector` — and the undeclared `_chunks` companion the
+ passage pipeline writes beside it — no longer gets a full-text term
+ dictionary** ([#328](https://github.com/xerj-org/xerj/issues/328)). Both
+ halves are named in that headline because the companion is where most of the
+ bytes are: of the 39,796,380 B removed below, **26,518,005 B (67%) is
+ `<seg>.emb_chunks.fst`**, and in that corpus `emb_chunks` is not in the
+ mapping at all — it is matched by NAME, not by anyone declaring it
+ `dense_vector`. The field was fed
+ through the lexical indexing path and given an FST + postings that no query
+ path can read: kNN is served by `hnsw/graph.bin`, `exists` from
+ `_source`/doc values, and `_field_caps` and highlighting never open the
+ field's term dictionary. `extract_field_text`'s array arm joined the
+ components with spaces, so a 128-dim vector became one enormous
+ decimal-string term per document.
+
+ Measured on a 5,000-doc × 128-dim corpus (text + keyword + long +
+ `dense_vector` + its `_chunks` companion), after `force_merge(1)`, summing
+ every file in the data directory except the write-ahead log:
+ **54,068,549 B → 14,251,975 B (−73.6%, 3.79×)**. Per file, `<seg>.emb.fst`
+ 13,256,659 B → 0, `<seg>.emb.post` 10,834 B → 0, `<seg>.emb.norms` 48 B → 0,
+ `<seg>.emb_chunks.fst` 26,518,005 B → 0, `<seg>.emb_chunks.post` 10,834 B → 0,
+ while the three lexical `.fst` files (`body` 278 B, `cat` 53 B, `n` 209 B) are
+ byte-identical on both sides.
+
+ `.wal` is excluded from that total because its tail is reclaimed
+ asynchronously and it is not reproducible: on the 300-doc fixture the
+ committed regression test uses, runs put the *whole directory* anywhere
+ between 858,147 B and 1,105,001 B while the durable bytes moved by ~500 B. The
+ test prints both and asserts on the durable one. It reports
+ **3,249,323 B → 857,265–857,769 B** for that fixture (before-side measured by
+ reverting `index.rs` and `memtable.rs` to the merge base `ca4d75a` on the same
+ fixture), which is the same −73.6% at 1/17th the corpus. Eight runs of the
+ committed test on this revision printed 857,265 / 857,529 / 857,533 /
+ 857,545 / 857,549 / 857,657 / 857,657 / 857,769 B, a 504 B spread; the ceiling
+ it asserts is 1,500,000 B, three orders of magnitude clear of that jitter and
+ still 1.7 MB below the before-side.
+
+ A `dense_vector` nested under an object mapping is covered too — its
+ components were landing in the *parent's* term dictionary, because the segment
+ builder flattens the whole object into one text field. On the nested fixture
+ `<seg>.passages.fst` goes 398,558 B → 45 B, for both mapping shapes that reach
+ `FieldType::Vector` (a dotted top-level `passages.vec`, and a `vec`
+ sub-mapping under a `passages` object).
+
+ The exclusion covers the whole family a vector field generates, not just the
+ base name: `<field>_chunks` (the per-document multi-vector) and
+ `__xerj_passage_meta__<field>` are excluded on the same walk that decides
+ which fields get an HNSW graph. The companion is the bigger half —
+ 26,518,005 B against `emb.fst`'s 13,256,659 B, 49% of the whole pre-change
+ index — so a base-name-only exclusion would leave more behind than it removes.
+
+ **Two rules, and they do not have the same strength.** The base field is
+ excluded BY DECLARED TYPE (`dense_vector`, unconditionally). The two
+ companions are excluded BY NAME, because they arrive in `_source` with no
+ mapping entry behind them, so there is no declared type to read — and a
+ name-based rule YIELDS to any declaration it finds. **If you have mapped your
+ own field called `<vector>_chunks` or `__xerj_passage_meta__<vector>` as
+ anything at all other than `dense_vector` — `text`, `keyword`, `long`,
+ `integer`, `short`, `byte`, `unsigned_long`, `double`, `float`, `half_float`,
+ `scaled_float`, `date`, `date_nanos`, `boolean`, `ip`, `geo_point`, `binary`,
+ `object`, `nested`, or a type string XERJ does not recognise — nothing about
+ it changes**: it keeps its term dictionary, every query naming it keeps
+ answering exactly what it answered before, and only the `dense_vector` beside
+ it loses its postings. The same holds for a field of that name XERJ mapped for
+ you dynamically, whatever type it inferred, as long as the value is not itself
+ a multi-vector. Measured on
+ `{text body, keyword emb_chunks, dense_vector emb}`, 50 docs, half `tenant-a`
+ (before → after): `term {emb_chunks:"tenant-a"}` 25 → 25,
+ `terms` 25 → 25, `bool{filter:[term emb_chunks]}` 25 → 25,
+ `bool{must:[match body],filter:[term …]}` 25 → 25,
+ `constant_score{filter:term}` 25 → 25,
+ `bool{must_not:[term emb_chunks]}` 25 → 25, `<seg>.emb_chunks.fst`
+ 54 B → 54 B, `<seg>.emb.fst` 16,713 B → 0. Same with a `text` mapping:
+ `match` / `match_phrase` / `multi_match` / `simple_query_string` / `prefix`
+ all 25 → 25 and `<seg>.emb_chunks.fst` 71 B → 71 B. A `long` mapping keeps a
+ 44 B `.fst` and a `double` mapping a 48 B one, both 25 → 25 on `term`,
+ `terms`, `range`, `bool{filter}` and `bool{must_not}`. A field of that name
+ XERJ mapped for you *dynamically* is exempt on the same rule — measured
+ 25 → 25 for a `text` one (54 B `.fst`) and 25 → 25 for a `long` one
+ (44 B `.fst`).
+
+ **What makes "yields to any declaration" affordable** is that the companion
+ is stopped from ever acquiring one. `<vector>_chunks` is user-supplied
+ `_source` — an array of float arrays — so dynamic mapping used to register it
+ like any other unmapped key and infer `double` from it, producing a mapping
+ entry indistinguishable from one you wrote. Dynamic mapping now declines to
+ register a `<vector>_chunks` key when a `dense_vector` named `<vector>` is
+ already mapped *and* the value is a rectangular, non-empty array of non-empty
+ all-numeric arrays — the exact multi-vector shape, and nothing else. A scalar,
+ a string, a flat array or a ragged one under that name is your field and is
+ mapped as one. Measured on `{text body, dense_vector emb}` with `emb_chunks`
+ left out of the mapping, 50 docs: an `[[…],[…]]` value gets no mapping entry,
+ `<seg>.emb_chunks.fst` = 0 B and a lexical clause on it answers 0; a `"tenant-a"`
+ value is mapped `text`, keeps a 54 B `.fst` and answers 25; a `7` value is
+ mapped `long`, keeps a 44 B `.fst` and answers 25.
+
+ Both halves were measured on their own, by disabling each in the tree and
+ re-running the committed tests:
+
+ | | `emb_chunks.fst` (300-doc) | durable index | declared `long` sibling |
+ |---|---:|---:|---|
+ | this release | **0 B** | 857,265–857,769 B | 44 B `.fst`, answers 25 |
+ | yield restored to `long`/`double`-excluded | 0 B | 857,445 B | **0 B, answers 0, `must_not` returns 50** |
+ | dynamic-mapping refusal disabled | **1,592,118 B** | **2,451,336 B** | 44 B, answers 25 |
+
+ The middle row is the behaviour rc.16 would have shipped; the bottom row is
+ the 67% of the saving that an unconditional "any declaration wins" gives up.
+ Neither cost is paid here.
+
+ A lexical clause that NAMES a `dense_vector` (or one of its companions) is now
+ lowered to `match_none` at plan time rather than falling through to the
+ stored-doc scan. That is the correctness half of the change, not only the
+ speed half: with the postings removed and no lowering,
+ `{"multi_match":{"fields":["emb"],"query":"0"}}` returns **every document**
+ instead of none, because the scan renders the float array to text and every
+ component contains a `0`. Measured on the corpus above, first call for each
+ shape (`main` → postings-gone-without-lowering → this release):
+
+ | query | `main` | no lowering | this release |
+ |---|---|---|---|
+ | `multi_match {fields:["emb"], query:"0"}` | 0 hits, 0.409 ms | **5,000 hits**, 123.5 ms | 0 hits, 0.012 ms |
+ | `multi_match {fields:["emb_chunks"], query:"0"}` | 0 hits, 2.24 ms | **5,000 hits**, 138.5 ms | 0 hits, 0.014 ms |
+ | `term {emb: "<component>"}` | 0 hits, 89.2 ms | 0 hits, 186.8 ms | 0 hits, 0.061 ms |
+ | `range {emb: {gte:-2, lte:2}}` | 0 hits, 119.8 ms | 0 hits, 134.5 ms | 0 hits, 0.009 ms |
+ | `simple_query_string {fields:["emb"]}` | 0 hits, 0.033 ms | 0 hits, 114.5 ms | 0 hits, 0.015 ms |
+ | `constant_score {filter: term emb}` | 0 hits, 83.0 ms | 0 hits, 110.3 ms | 0 hits, 0.012 ms |
+ | `exists {field: "emb"}` | 5,000 hits, 83.0 ms | 5,000 hits, 108.0 ms | 5,000 hits, 88.5 ms |
+ | `match {body: "liquidity"}` | 5,000 hits, 5.18 ms | 5,000 hits, 5.96 ms | 5,000 hits, 4.95 ms |
+ | `term {cat: "even"}` | 2,500 hits, 2.32 ms | 2,500 hits, 2.87 ms | 2,500 hits, 2.27 ms |
+
+ `exists`, `knn` and `semantic` on the field are untouched, a mixed `fields`
+ list keeps its lexical members, and a `nested` query is deliberately not
+ rewritten (its field names resolve against the element, not the root).
+
+ **Four behaviour changes to know about:**
+
+ - **A `dense_vector` (and its undeclared `_chunks` companion) now has no
+ lexical surface at all: every lexical leaf naming one *outright* answers 0
+ hits.** "Outright" excludes one shape, measured rather than assumed: a
+ `fields` entry that is a PATTERN (`["emb.*"]`) is NOT lowered, because
+ resolving a pattern needs the expansion universe — dynamic fields included —
+ and lowering one that would have expanded onto a real lexical field is the
+ silent-zero failure this release goes out of its way to avoid. So
+ `{"multi_match":{"query":"0","fields":["emb.*"]}}` still answers 300 of 300
+ on the 300-doc fixture, exactly as on `main`: the unprojectable clause falls
+ to the stored-doc scan, which resolves the pattern against the source's
+ field names and matches the rendered decimals. Unchanged in both columns, so
+ not a regression — but not closed either. `["emb"]`, `["emb*"]` and `["*"]`
+ all answer 0. Before this release
+ such a clause fell through to the stored-doc scan, which renders the float
+ array back to decimal text and matches against *that*, so whether a shape
+ answered 0 or answered most of the corpus depended entirely on whether the
+ rendered decimals happened to contain the probe. This is a CLASS of changed
+ answers, not one row. Measured on the 300-doc × 128-dim regression fixture,
+ `main` → this release:
+
+ | query | before | after |
+ |---|---:|---:|
+ | `wildcard {emb: "0*"}` | **300** (every document) | 0 |
+ | `fuzzy {emb: {value:"0", fuzziness:2}}` | **300** (every document) | 0 |
+ | `wildcard {emb_chunks: "0*"}` | **300** (every document) | 0 |
+ | `fuzzy {emb_chunks: {value:"0", fuzziness:2}}` | **300** | 0 |
+ | `prefix {emb: "0"}` | 146 | 0 |
+ | `prefix {emb_chunks: "0"}` | 146 | 0 |
+ | `match_phrase_prefix {emb: "0"}` | 50 | 0 |
+ | `term {emb: <exact component>}` numeric | 1 | 0 |
+ | `terms {emb: [<exact component>]}` numeric | 1 | 0 |
+
+ Shapes whose probe text never appeared in a rendered float (`term` with a
+ string, `match`, `match_phrase`, `regexp`, `range`, `simple_query_string`,
+ `query_string`) answered 0 on both sides. ES rejects these queries on a
+ `dense_vector` outright and Lucene gives a field with no terms an empty
+ scorer, so 0 is the answer this release adopts for all of them. The
+ regression test pins the whole class rather than sampling it.
+ - **A `bool` that mentions a `dense_vector` can change `_score` and
+ `max_score` without changing its hit count.** Dropping the dead clause makes
+ the surviving bool projectable onto the inverted index, so BM25 scores it
+ where the stored-doc scan used to. Measured on the same fixture:
+
+ | query | hits | `_score` | `max_score` |
+ |---|---|---|---|
+ | `bool{should:[term emb, term cat:"even"]}` | 150 → 150 | 0.008402659 → 0.6931471 | 1.6931472 → 0.6931471 |
+ | `bool{must:[match body], must_not:[term emb]}` | 300 → 300 | 0.008402659 (unchanged) | 1.6931472 → 0.008402659 |
+
+ Both moves are toward consistency — on `main` these queries reported a
+ `max_score` that **no returned hit carried**, and the first now scores
+ exactly like `{"term":{"cat":"even"}}` on its own, which is what it reduces
+ to. Absolute scores are not part of XERJ's compatibility surface, but
+ anything comparing scores across a version boundary or asserting on
+ `max_score` for a bool that touches a vector field will see this. Plain
+ lexical queries are unaffected; the blast radius is bools with a
+ `dense_vector` clause in them.
+ - **`GET /<index>/_mapping` no longer lists a `<vector>_chunks` field that
+ XERJ mapped for you.** The multi-vector companion beside a declared
+ `dense_vector` used to be registered by dynamic mapping as `double` (an
+ array of float arrays walks to its first scalar). It is not registered at
+ all now, so it disappears from `_mapping` and `_field_caps` — the same
+ treatment `__xerj_passage_meta__<vector>` already had. Nothing that reads
+ `_source` changes; the values are stored and returned exactly as before, and
+ `knn` and `semantic` on the vector itself are untouched. A field of that
+ name that you mapped yourself, or that XERJ mapped from a non-multi-vector
+ value, is unaffected and still listed.
+
+ One consequence worth knowing: aggregating or sorting on that companion
+ stops working, because it no longer has a mapping entry to aggregate
+ through. On a real multi-vector this was never a meaningful operation, but
+ it was a `200` before and is not now.
+ - **Existing indices keep their bloat until a merge rewrites them.** This is a
+ write-side rule. Segments written by an earlier release keep their `.emb.*`
+ / `.emb_chunks.*` files, and the per-segment `fts_has_field` gate reads
+ whichever shape it finds. Run an explicit
+ `POST /<index>/_forcemerge?max_num_segments=1` to reclaim the bytes on an
+ index that already exists.
+
+ The companion rule has an ordering limit of the same kind, and it fails in
+ the safe direction: XERJ can only decline to map a companion it can
+ recognise, and recognising one needs the `dense_vector` to be mapped
+ *before* the documents arrive. Index first and add the mapping afterwards
+ and `<vector>_chunks` already carries a dynamic `double` entry, which the
+ by-name rule then yields to — so that index keeps the companion's term
+ dictionary (measured 9,467 B on a 50-doc × 16-dim fixture) instead of
+ reclaiming it. It costs bytes, not answers.
+
+ *Not changed:* ES rejects `term`/`match`/`range` on a `dense_vector` outright,
+ where XERJ answers `200`. This release removes bytes and adopts Lucene's
+ zero-hit answer; it does not add the rejection.
+
+ ### Fixed
+
+ - **Unity assets were sampled through a 4 MiB cap, silently junking whole
+ object classes.** Unity YAML is a grouped family — each `unity_class` is its
+ own dataset and a class's first document can sit anywhere in a scene that
+ `extract/unity.rs` itself says can exceed 200 MB. A class first appearing
+ past the cap was never sampled, got no entry in the plan, and phase B then
+ had nowhere to route its records: they became `file_junk` with nothing said.
+ The cap is now 512 MiB for this family, **and** any record whose group was
+ never sampled is now reported by name and count instead of silently counted
+ as junk.
+
+ - **A `script_guid` that failed to resolve produced no counter, no warning and
+ no report line.** `build_unity_guid_map` discarded `extract_meta`'s `Result`,
+ so an unreadable `.meta` was indistinguishable from a script nothing
+ references — on the feature's own headline query. Unreadable sidecars,
+ sidecars carrying no usable guid, and unresolvable `script_guid`s are now
+ each reported.
+
+ - **`script_path`/`script_class` were mapped only when the phase-A sample
+ happened to contain a `script_guid`**, while phase B stamps them whenever a
+ guid resolves. A cluster whose sampled window held no `m_Script` therefore
+ got them dynamic-mapped at index time, feeding the field-budget overshoot in
+ #312. They are now registered for every Unity cluster, and the enrichment
+ runs *before* `coerce_record` rather than after it, so the fields it stamps
+ are validated like every other field instead of bypassing coercion.
+
+ - **`build_unity_guid_map` ran serially, unmetered, on the critical path of
+ every run** — including a resumed no-op incremental, which otherwise has no
+ work to do. A real Unity project has 10k-500k `.meta` sidecars. It now runs
+ on `crate::pool` under the progress meter (the unattributed-stretch pattern
+ of #241).
+
+ - **A `--stub` file was named from the content path**, which under durable
+ preparation is a content-addressed blob — so every stub's one and only field
+ would have been titled after a blob ordinal (the #294 failure class).
+ It now uses `Sniffed::logical_name`.
+
+ - **`order::band_from_family_str` disagreed with `order::band`.** The string
+ form's catch-all ranked `binary` as `Bulk` where the enum form ranks it
+ `Vendored`, so a resumed run could order work differently from the run that
+ planned it. Both now agree for every family, pinned by a test that iterates
+ the whole enum.
+
+ - **The autoindex use-case README documented three flags and behaviours that
+ do not exist.** `--no-default-excludes` and `--no-gitignore` are spelled
+ `--no-default-ignores` and `--no-ignore`, and `cli::parse` hard-errors on an
+ unknown argument — so a reader who copied the documented invocation got an
+ error instead of an index. The marker-gated pruning of `node_modules/` and
+ `target/` was described but had been dropped from the branch. A test now
+ parses every `--flag` named in that README and fails if the CLI would reject
+ it.
+
+ - **`stub_matcher_tests::an_invalid_pattern_fails_loudly_at_startup` did not
+ test its own name** — it asserted only that a *valid* pattern compiles.
+ `glob_to_regex` escapes every metacharacter, so no glob can produce a
+ syntactically invalid regex; the one reachable failure is the compiled-size
+ limit, which `?` and `**` reach at ~10^5 characters. Over-long patterns are
+ now rejected explicitly with a message naming the flag and the limit, and the
+ test exercises that path.
+
+ ### Changed
+
+ - **Two proposed byte-statistics binary heuristics were dropped before they
+ shipped; raw TGA is detected by its header instead.** Both were attempts to
+ recognise a raw texture that decodes into printable characters, and both
+ turned out to be tests for "not written in Latin script".
+
+ The first, from PR #274, classified any text over 4 KiB with under 5%
+ whitespace as binary. `nonblank` is built from `text.lines()`, so newlines
+ are already stripped and only intra-line whitespace counts — which means
+ Chinese, Japanese, Korean, Thai, Lao, Khmer and Burmese prose, base64 blobs,
+ FASTA sequences and minified single-line files all score 0%. This repo's own
+ `failure_resume_http_tests::legacy_key_collision_fails_before_visibility_with_scoped_guidance`
+ builds a 65,537-byte fixture of one repeated ASCII letter; with that guard
+ reinstated the test fails with `left: 3, right: 0` — the run exits 3 instead
+ of 0.
+
+ The second was introduced by the first attempt to reland #274 and is
+ **removed here**: "decoded via lossy windows-1252 AND over 30% non-ASCII is
+ pixel soup". windows-1252 is the fallback every legacy 8-bit codepage decodes
+ through, so it is not a test for image data. Measured through `sniff()` on
+ `ca4d75a` versus that branch, with identical fixtures:
+
+ | fixture | `ca4d75a` | with the guard |
+ |---|---|---|
+ | windows-1251 Russian, 13,000 B | `txt-prose` | `binary` |
+ | KOI8-R Russian, 13,000 B | `txt-prose` | `binary` |
+ | windows-1253 Greek, 12,200 B | `txt-prose` | `binary` |
+ | windows-1255 Hebrew, 11,000 B | `txt-prose` | `binary` |
+ | windows-1256 Arabic, 10,800 B | `txt-prose` | `binary` |
+ | Shift-JIS Japanese, 22,401 B (byte pad 1; likewise pad 3) | `txt-prose` | `binary` |
+ | Shift-JIS Japanese + ASCII code, 13,360 B | `txt-lines` | `binary` |
+ | Shift-JIS Japanese + ASCII code, 66,800 B | `txt-lines` | `binary` |
+
+ `scan_file` turns `Family::Binary` into `junk: binary content (unknown)`, so
+ each of those files stopped being indexed. A `looks_like_legacy_cjk` escape
+ hatch shipped with it and could not carry the weight: single-byte codepages
+ never form valid Shift-JIS/GBK/Big5/EUC-KR double-byte pairs, so Cyrillic,
+ Greek, Hebrew and Arabic were never rescued at all; it required a *lossless*
+ trial decode while `sniff()` sees only the first 8192 bytes, so the same
+ Japanese document was text at byte-offset pads 0 and 2 and binary at pads 1
+ and 3; and a realistic Japanese technical document (prose around ASCII code
+ fences) sits below its 30% ideograph floor while sitting above the guard's
+ 30% non-ASCII ceiling. Every fixture in the table above now classifies
+ exactly as it does on `ca4d75a`.
+
+ What replaces it is `looks_like_tga_header`: TGA has no magic number, but its
+ 18-byte header is constrained enough that bytes 1 and 2 of the file must both
+ be control characters. **Headerless raw payloads (`.raw`, `.bytes`,
+ uncompressed PCM) still classify as text**, exactly as they do on `ca4d75a`.
+ Bounding what a magic-less binary costs is the fix for that and is not
+ attempted here — see [#381](https://github.com/xerj-org/xerj/issues/381);
+ `for_each_section` already streams, so the unbounded quantity is the record
+ count, not resident memory.
+
+ - **The new magic-byte signatures now require structural confirmation.** Nine
+ signatures (PSD, both TIFF byte orders, RIFF, OGG, FLAC, MP3, FBX and EXR)
+ were added by the first reland attempt as bare `starts_with` tests. **Six** of
+ them are entirely printable ASCII — `8BPS`, `RIFF`, `OggS`, `fLaC`, `ID3` and
+ the 18 letters `Kaydara FBX Binary` — so they matched ordinary text. Measured
+ through the real `sniff()` on `ca4d75a` vs this branch with identical
+ fixtures:
+
+ | fixture | `ca4d75a` | first reland |
+ | --- | --- | --- |
+ | CSV whose first column header is `ID3` | `csv` | `binary`/`mp3` |
+ | prose opening `RIFF is a container format used by WAV files. …` | `txt-prose` | `binary`/`riff` |
+ | prose opening `OggS pages carry the packets of an Ogg stream …` | `txt-prose` | `binary`/`ogg` |
+ | prose opening `fLaC is the four byte magic of a FLAC audio …` | `txt-prose` | `binary`/`flac` |
+ | prose opening `8BPS is the magic of an Adobe Photoshop …` | `txt-prose` | `binary`/`psd` |
+ | `.md` note opening `Kaydara FBX Binary is the 20-byte magic …` | `txt-prose` | `binary`/`fbx` |
+
+ Each of the six is now qualified by what must follow it: PSD version, RIFF
+ FORM type, Ogg stream-structure version, FLAC block type, ID3v2 major version
+ and synchsafe size, and for FBX the full 23-byte header (`Kaydara FBX
+ Binary`, two spaces, NUL, `0x1A`, `0x00`). The FBX case was missed by the
+ first pass at this fix and caught in review — at 18 printable characters it is
+ the likeliest of the six to open a real sentence, and likeliest precisely
+ inside the Unity/3D-asset corpus this feature targets. Every fixture in the
+ table above now classifies exactly as it does on `ca4d75a`. The true positives
+ are unchanged and still covered; the FBX true-positive fixture was corrected
+ to the real 23-byte header, which is what any Autodesk tool emits.
+
+ This is what Lucene's `CodecUtil.checkHeader` does with `CODEC_MAGIC`
+ (`lucene/core/src/java/org/apache/lucene/codecs/CodecUtil.java:183`, which
+ hands straight to `checkHeaderNoMagic` at `:202` and refuses the file unless
+ the codec name *and* a version in range follow — the magic alone is never
+ taken as proof). Apache-2.0, same licence as XERJ; adapted, not copied.
+
+ **Not fixed here, deliberately:** `GIF8` and `BM` are printable-ASCII
+ signatures with exactly the same defect, but they are **pre-existing** — they
+ are unchanged from `ca4d75a`, and prose opening `GIF8`/`GIF89a`, a CSV whose
+ first column header is `GIF8`, and prose opening `BM` were each measured to
+ classify as `binary`/`gif` and `binary`/`bmp` on `ca4d75a` **and** on this
+ branch, identically. Fixing them changes behaviour unrelated to Unity, so it
+ is filed as **#380** rather than smuggled in here, and today's wrong answers
+ are pinned by `gif8_and_bm_are_still_taken_on_faith` so #380 has to flip that
+ assertion deliberately. `GIF8` is listed alongside `BM` so the next reader
+ does not conclude `BM` is the only one.
+
+ - **A junk entry could overwrite a successfully indexed file's catalog row.**
+ When phase B met a record group phase A never sampled, the worker pushed a
+ whole-file junk entry while leaving `send_err` unset — so the file also
+ reached `journal.file_done`, and both passes wrote `catalog::file_doc` under
+ the same `file:{file_key}` id into the same bulk. The junk document landed
+ second and won: a file that indexed records was reported in the catalog as
+ status `junk` with `records: 0`, `files_junk` counted it, and
+ `CodeCoverage::observe` ran for it twice. Reachable with no Unity involved,
+ via a >64 MiB SQL dump whose first row for some table starts past
+ `SQLDUMP_SAMPLE_LIMIT`. The unsampled group is now reported on the progress
+ meter instead, the dropped records stay on that file's own completion where
+ they already were, and the disjointness the code claimed in a comment is now
+ enforced at the one place that holds both sets (`shadowed_junk_entries`).
+
+ ### Known issues
+
+ - `ref_guids` (Unity) and `joints` (BVH) are multi-valued keyword arrays, and
+ #332 means array elements are joined into one FTS token, so a post-flush
+ `term`/`match` on a single element does not hit. `script_guid`,
+ `script_path` and `script_class` are single-valued and unaffected, so the
+ documented "which scenes use this script?" query is not impacted. Fixing
+ #332 is an engine-side change to the FTS writer input type.
+
+ - **`GIF8` and `BM` are still unqualified printable-ASCII magic signatures**, so
+ a text file whose first characters are `GIF8`, `GIF89a` or `BM` is classified
+ `binary` and junked. This is **pre-existing, not introduced here** — measured
+ identical on `ca4d75a` and on this branch (`binary`/`gif` and `binary`/`bmp`
+ in both) — and is left unchanged because fixing it is a behaviour change with
+ no Unity content. Both names are recorded here on purpose: `BM` is the
+ obvious one and `GIF8` is the one a reader would otherwise miss. Tracked as
+ **#380**; `gif8_and_bm_are_still_taken_on_faith` pins the current behaviour.
+
+ - **A magic-less binary is still sectioned and indexed in full.** The two
+ byte-statistics guards this branch removed (they junked non-Latin text) also
+ caught NUL-free binary payloads that decode printable under the
+ windows-1252 fallback. Nothing replaced them except `looks_like_tga_header`,
+ which covers TGA only. Measured on this branch: a 4,194,495-byte printable
+ NUL-free blob named `texture.bytes` sniffs `txt-prose` and expands into 2048
+ indexed records. This is **not** the peak-RSS bug from #239 —
+ `for_each_section` is still streaming and per-file memory is still bounded;
+ the unbounded quantity is the record COUNT. Removing the guards was still
+ right: they deleted CJK, Cyrillic, Greek, Hebrew and Arabic documents
+ worldwide. Tracked as **#381**.
+
## [1.0.0-rc.16] - 2026-08-13
Both rc.15 known issues (the progress-stream forgery and the outer-`.gitignore`