AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge origin/main into fix/issue-328-vector-term-dict
xerj-orgc23bb4ef775c
diff --cc CHANGELOG.mdindex 1b314655c,4c45bf407..fd3c37d15--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -153,235 -178,211 +178,440 @@@ and this project adheres to [Semantic Vfrom the moment it is allowlisted. Seed the target with snapshot/restorefirst if it needs the existing documents.+### Performance++- **A field mapped `dense_vector` — and the undeclared `_chunks` companion the+ passage pipeline writes beside it — no longer gets a full-text term+ dictionary** ([#328](https://github.com/xerj-org/xerj/issues/328)). Both+ halves are named in that headline because the companion is where most of the+ bytes are: of the 39,796,380 B removed below, **26,518,005 B (67%) is+ `<seg>.emb_chunks.fst`**, and in that corpus `emb_chunks` is not in the+ mapping at all — it is matched by NAME, not by anyone declaring it+ `dense_vector`. The field was fed+ through the lexical indexing path and given an FST + postings that no query+ path can read: kNN is served by `hnsw/graph.bin`, `exists` from+ `_source`/doc values, and `_field_caps` and highlighting never open the+ field's term dictionary. `extract_field_text`'s array arm joined the+ components with spaces, so a 128-dim vector became one enormous+ decimal-string term per document.++ Measured on a 5,000-doc × 128-dim corpus (text + keyword + long ++ `dense_vector` + its `_chunks` companion), after `force_merge(1)`, summing+ every file in the data directory except the write-ahead log:+ **54,068,549 B → 14,251,975 B (−73.6%, 3.79×)**. Per file, `<seg>.emb.fst`+ 13,256,659 B → 0, `<seg>.emb.post` 10,834 B → 0, `<seg>.emb.norms` 48 B → 0,+ `<seg>.emb_chunks.fst` 26,518,005 B → 0, `<seg>.emb_chunks.post` 10,834 B → 0,+ while the three lexical `.fst` files (`body` 278 B, `cat` 53 B, `n` 209 B) are+ byte-identical on both sides.++ `.wal` is excluded from that total because its tail is reclaimed+ asynchronously and it is not reproducible: on the 300-doc fixture the+ committed regression test uses, runs put the *whole directory* anywhere+ between 858,147 B and 1,105,001 B while the durable bytes moved by ~500 B. The+ test prints both and asserts on the durable one. It reports+ **3,249,323 B → 857,265–857,769 B** for that fixture (before-side measured by+ reverting `index.rs` and `memtable.rs` to the merge base `ca4d75a` on the same+ fixture), which is the same −73.6% at 1/17th the corpus. Eight runs of the+ committed test on this revision printed 857,265 / 857,529 / 857,533 /+ 857,545 / 857,549 / 857,657 / 857,657 / 857,769 B, a 504 B spread; the ceiling+ it asserts is 1,500,000 B, three orders of magnitude clear of that jitter and+ still 1.7 MB below the before-side.++ A `dense_vector` nested under an object mapping is covered too — its+ components were landing in the *parent's* term dictionary, because the segment+ builder flattens the whole object into one text field. On the nested fixture+ `<seg>.passages.fst` goes 398,558 B → 45 B, for both mapping shapes that reach+ `FieldType::Vector` (a dotted top-level `passages.vec`, and a `vec`+ sub-mapping under a `passages` object).++ The exclusion covers the whole family a vector field generates, not just the+ base name: `<field>_chunks` (the per-document multi-vector) and+ `__xerj_passage_meta__<field>` are excluded on the same walk that decides+ which fields get an HNSW graph. The companion is the bigger half —+ 26,518,005 B against `emb.fst`'s 13,256,659 B, 49% of the whole pre-change+ index — so a base-name-only exclusion would leave more behind than it removes.++ **Two rules, and they do not have the same strength.** The base field is+ excluded BY DECLARED TYPE (`dense_vector`, unconditionally). The two+ companions are excluded BY NAME, because they arrive in `_source` with no+ mapping entry behind them, so there is no declared type to read — and a+ name-based rule YIELDS to any declaration it finds. **If you have mapped your+ own field called `<vector>_chunks` or `__xerj_passage_meta__<vector>` as+ anything at all other than `dense_vector` — `text`, `keyword`, `long`,+ `integer`, `short`, `byte`, `unsigned_long`, `double`, `float`, `half_float`,+ `scaled_float`, `date`, `date_nanos`, `boolean`, `ip`, `geo_point`, `binary`,+ `object`, `nested`, or a type string XERJ does not recognise — nothing about+ it changes**: it keeps its term dictionary, every query naming it keeps+ answering exactly what it answered before, and only the `dense_vector` beside+ it loses its postings. The same holds for a field of that name XERJ mapped for+ you dynamically, whatever type it inferred, as long as the value is not itself+ a multi-vector. Measured on+ `{text body, keyword emb_chunks, dense_vector emb}`, 50 docs, half `tenant-a`+ (before → after): `term {emb_chunks:"tenant-a"}` 25 → 25,+ `terms` 25 → 25, `bool{filter:[term emb_chunks]}` 25 → 25,+ `bool{must:[match body],filter:[term …]}` 25 → 25,+ `constant_score{filter:term}` 25 → 25,+ `bool{must_not:[term emb_chunks]}` 25 → 25, `<seg>.emb_chunks.fst`+ 54 B → 54 B, `<seg>.emb.fst` 16,713 B → 0. Same with a `text` mapping:+ `match` / `match_phrase` / `multi_match` / `simple_query_string` / `prefix`+ all 25 → 25 and `<seg>.emb_chunks.fst` 71 B → 71 B. A `long` mapping keeps a+ 44 B `.fst` and a `double` mapping a 48 B one, both 25 → 25 on `term`,+ `terms`, `range`, `bool{filter}` and `bool{must_not}`. A field of that name+ XERJ mapped for you *dynamically* is exempt on the same rule — measured+ 25 → 25 for a `text` one (54 B `.fst`) and 25 → 25 for a `long` one+ (44 B `.fst`).++ **What makes "yields to any declaration" affordable** is that the companion+ is stopped from ever acquiring one. `<vector>_chunks` is user-supplied+ `_source` — an array of float arrays — so dynamic mapping used to register it+ like any other unmapped key and infer `double` from it, producing a mapping+ entry indistinguishable from one you wrote. Dynamic mapping now declines to+ register a `<vector>_chunks` key when a `dense_vector` named `<vector>` is+ already mapped *and* the value is a rectangular, non-empty array of non-empty+ all-numeric arrays — the exact multi-vector shape, and nothing else. A scalar,+ a string, a flat array or a ragged one under that name is your field and is+ mapped as one. Measured on `{text body, dense_vector emb}` with `emb_chunks`+ left out of the mapping, 50 docs: an `[[…],[…]]` value gets no mapping entry,+ `<seg>.emb_chunks.fst` = 0 B and a lexical clause on it answers 0; a `"tenant-a"`+ value is mapped `text`, keeps a 54 B `.fst` and answers 25; a `7` value is+ mapped `long`, keeps a 44 B `.fst` and answers 25.++ Both halves were measured on their own, by disabling each in the tree and+ re-running the committed tests:++ | | `emb_chunks.fst` (300-doc) | durable index | declared `long` sibling |+ |---|---:|---:|---|+ | this release | **0 B** | 857,265–857,769 B | 44 B `.fst`, answers 25 |+ | yield restored to `long`/`double`-excluded | 0 B | 857,445 B | **0 B, answers 0, `must_not` returns 50** |+ | dynamic-mapping refusal disabled | **1,592,118 B** | **2,451,336 B** | 44 B, answers 25 |++ The middle row is the behaviour rc.16 would have shipped; the bottom row is+ the 67% of the saving that an unconditional "any declaration wins" gives up.+ Neither cost is paid here.++ A lexical clause that NAMES a `dense_vector` (or one of its companions) is now+ lowered to `match_none` at plan time rather than falling through to the+ stored-doc scan. That is the correctness half of the change, not only the+ speed half: with the postings removed and no lowering,+ `{"multi_match":{"fields":["emb"],"query":"0"}}` returns **every document**+ instead of none, because the scan renders the float array to text and every+ component contains a `0`. Measured on the corpus above, first call for each+ shape (`main` → postings-gone-without-lowering → this release):++ | query | `main` | no lowering | this release |+ |---|---|---|---|+ | `multi_match {fields:["emb"], query:"0"}` | 0 hits, 0.409 ms | **5,000 hits**, 123.5 ms | 0 hits, 0.012 ms |+ | `multi_match {fields:["emb_chunks"], query:"0"}` | 0 hits, 2.24 ms | **5,000 hits**, 138.5 ms | 0 hits, 0.014 ms |+ | `term {emb: "<component>"}` | 0 hits, 89.2 ms | 0 hits, 186.8 ms | 0 hits, 0.061 ms |+ | `range {emb: {gte:-2, lte:2}}` | 0 hits, 119.8 ms | 0 hits, 134.5 ms | 0 hits, 0.009 ms |+ | `simple_query_string {fields:["emb"]}` | 0 hits, 0.033 ms | 0 hits, 114.5 ms | 0 hits, 0.015 ms |+ | `constant_score {filter: term emb}` | 0 hits, 83.0 ms | 0 hits, 110.3 ms | 0 hits, 0.012 ms |+ | `exists {field: "emb"}` | 5,000 hits, 83.0 ms | 5,000 hits, 108.0 ms | 5,000 hits, 88.5 ms |+ | `match {body: "liquidity"}` | 5,000 hits, 5.18 ms | 5,000 hits, 5.96 ms | 5,000 hits, 4.95 ms |+ | `term {cat: "even"}` | 2,500 hits, 2.32 ms | 2,500 hits, 2.87 ms | 2,500 hits, 2.27 ms |++ `exists`, `knn` and `semantic` on the field are untouched, a mixed `fields`+ list keeps its lexical members, and a `nested` query is deliberately not+ rewritten (its field names resolve against the element, not the root).++ **Four behaviour changes to know about:**++ - **A `dense_vector` (and its undeclared `_chunks` companion) now has no+ lexical surface at all: every lexical leaf naming one *outright* answers 0+ hits.** "Outright" excludes one shape, measured rather than assumed: a+ `fields` entry that is a PATTERN (`["emb.*"]`) is NOT lowered, because+ resolving a pattern needs the expansion universe — dynamic fields included —+ and lowering one that would have expanded onto a real lexical field is the+ silent-zero failure this release goes out of its way to avoid. So+ `{"multi_match":{"query":"0","fields":["emb.*"]}}` still answers 300 of 300+ on the 300-doc fixture, exactly as on `main`: the unprojectable clause falls+ to the stored-doc scan, which resolves the pattern against the source's+ field names and matches the rendered decimals. Unchanged in both columns, so+ not a regression — but not closed either. `["emb"]`, `["emb*"]` and `["*"]`+ all answer 0. Before this release+ such a clause fell through to the stored-doc scan, which renders the float+ array back to decimal text and matches against *that*, so whether a shape+ answered 0 or answered most of the corpus depended entirely on whether the+ rendered decimals happened to contain the probe. This is a CLASS of changed+ answers, not one row. Measured on the 300-doc × 128-dim regression fixture,+ `main` → this release:++ | query | before | after |+ |---|---:|---:|+ | `wildcard {emb: "0*"}` | **300** (every document) | 0 |+ | `fuzzy {emb: {value:"0", fuzziness:2}}` | **300** (every document) | 0 |+ | `wildcard {emb_chunks: "0*"}` | **300** (every document) | 0 |+ | `fuzzy {emb_chunks: {value:"0", fuzziness:2}}` | **300** | 0 |+ | `prefix {emb: "0"}` | 146 | 0 |+ | `prefix {emb_chunks: "0"}` | 146 | 0 |+ | `match_phrase_prefix {emb: "0"}` | 50 | 0 |+ | `term {emb: <exact component>}` numeric | 1 | 0 |+ | `terms {emb: [<exact component>]}` numeric | 1 | 0 |++ Shapes whose probe text never appeared in a rendered float (`term` with a+ string, `match`, `match_phrase`, `regexp`, `range`, `simple_query_string`,+ `query_string`) answered 0 on both sides. ES rejects these queries on a+ `dense_vector` outright and Lucene gives a field with no terms an empty+ scorer, so 0 is the answer this release adopts for all of them. The+ regression test pins the whole class rather than sampling it.+ - **A `bool` that mentions a `dense_vector` can change `_score` and+ `max_score` without changing its hit count.** Dropping the dead clause makes+ the surviving bool projectable onto the inverted index, so BM25 scores it+ where the stored-doc scan used to. Measured on the same fixture:++ | query | hits | `_score` | `max_score` |+ |---|---|---|---|+ | `bool{should:[term emb, term cat:"even"]}` | 150 → 150 | 0.008402659 → 0.6931471 | 1.6931472 → 0.6931471 |+ | `bool{must:[match body], must_not:[term emb]}` | 300 → 300 | 0.008402659 (unchanged) | 1.6931472 → 0.008402659 |++ Both moves are toward consistency — on `main` these queries reported a+ `max_score` that **no returned hit carried**, and the first now scores+ exactly like `{"term":{"cat":"even"}}` on its own, which is what it reduces+ to. Absolute scores are not part of XERJ's compatibility surface, but+ anything comparing scores across a version boundary or asserting on+ `max_score` for a bool that touches a vector field will see this. Plain+ lexical queries are unaffected; the blast radius is bools with a+ `dense_vector` clause in them.+ - **`GET /<index>/_mapping` no longer lists a `<vector>_chunks` field that+ XERJ mapped for you.** The multi-vector companion beside a declared+ `dense_vector` used to be registered by dynamic mapping as `double` (an+ array of float arrays walks to its first scalar). It is not registered at+ all now, so it disappears from `_mapping` and `_field_caps` — the same+ treatment `__xerj_passage_meta__<vector>` already had. Nothing that reads+ `_source` changes; the values are stored and returned exactly as before, and+ `knn` and `semantic` on the vector itself are untouched. A field of that+ name that you mapped yourself, or that XERJ mapped from a non-multi-vector+ value, is unaffected and still listed.++ One consequence worth knowing: aggregating or sorting on that companion+ stops working, because it no longer has a mapping entry to aggregate+ through. On a real multi-vector this was never a meaningful operation, but+ it was a `200` before and is not now.+ - **Existing indices keep their bloat until a merge rewrites them.** This is a+ write-side rule. Segments written by an earlier release keep their `.emb.*`+ / `.emb_chunks.*` files, and the per-segment `fts_has_field` gate reads+ whichever shape it finds. Run an explicit+ `POST /<index>/_forcemerge?max_num_segments=1` to reclaim the bytes on an+ index that already exists.++ The companion rule has an ordering limit of the same kind, and it fails in+ the safe direction: XERJ can only decline to map a companion it can+ recognise, and recognising one needs the `dense_vector` to be mapped+ *before* the documents arrive. Index first and add the mapping afterwards+ and `<vector>_chunks` already carries a dynamic `double` entry, which the+ by-name rule then yields to — so that index keeps the companion's term+ dictionary (measured 9,467 B on a 50-doc × 16-dim fixture) instead of+ reclaiming it. It costs bytes, not answers.++ *Not changed:* ES rejects `term`/`match`/`range` on a `dense_vector` outright,+ where XERJ answers `200`. This release removes bytes and adopts Lucene's+ zero-hit answer; it does not add the rejection.++ ### Fixed++ - **Unity assets were sampled through a 4 MiB cap, silently junking whole+ object classes.** Unity YAML is a grouped family — each `unity_class` is its+ own dataset and a class's first document can sit anywhere in a scene that+ `extract/unity.rs` itself says can exceed 200 MB. A class first appearing+ past the cap was never sampled, got no entry in the plan, and phase B then+ had nowhere to route its records: they became `file_junk` with nothing said.+ The cap is now 512 MiB for this family, **and** any record whose group was+ never sampled is now reported by name and count instead of silently counted+ as junk.++ - **A `script_guid` that failed to resolve produced no counter, no warning and+ no report line.** `build_unity_guid_map` discarded `extract_meta`'s `Result`,+ so an unreadable `.meta` was indistinguishable from a script nothing+ references — on the feature's own headline query. Unreadable sidecars,+ sidecars carrying no usable guid, and unresolvable `script_guid`s are now+ each reported.++ - **`script_path`/`script_class` were mapped only when the phase-A sample+ happened to contain a `script_guid`**, while phase B stamps them whenever a+ guid resolves. A cluster whose sampled window held no `m_Script` therefore+ got them dynamic-mapped at index time, feeding the field-budget overshoot in+ #312. They are now registered for every Unity cluster, and the enrichment+ runs *before* `coerce_record` rather than after it, so the fields it stamps+ are validated like every other field instead of bypassing coercion.++ - **`build_unity_guid_map` ran serially, unmetered, on the critical path of+ every run** — including a resumed no-op incremental, which otherwise has no+ work to do. A real Unity project has 10k-500k `.meta` sidecars. It now runs+ on `crate::pool` under the progress meter (the unattributed-stretch pattern+ of #241).++ - **A `--stub` file was named from the content path**, which under durable+ preparation is a content-addressed blob — so every stub's one and only field+ would have been titled after a blob ordinal (the #294 failure class).+ It now uses `Sniffed::logical_name`.++ - **`order::band_from_family_str` disagreed with `order::band`.** The string+ form's catch-all ranked `binary` as `Bulk` where the enum form ranks it+ `Vendored`, so a resumed run could order work differently from the run that+ planned it. Both now agree for every family, pinned by a test that iterates+ the whole enum.++ - **The autoindex use-case README documented three flags and behaviours that+ do not exist.** `--no-default-excludes` and `--no-gitignore` are spelled+ `--no-default-ignores` and `--no-ignore`, and `cli::parse` hard-errors on an+ unknown argument — so a reader who copied the documented invocation got an+ error instead of an index. The marker-gated pruning of `node_modules/` and+ `target/` was described but had been dropped from the branch. A test now+ parses every `--flag` named in that README and fails if the CLI would reject+ it.++ - **`stub_matcher_tests::an_invalid_pattern_fails_loudly_at_startup` did not+ test its own name** — it asserted only that a *valid* pattern compiles.+ `glob_to_regex` escapes every metacharacter, so no glob can produce a+ syntactically invalid regex; the one reachable failure is the compiled-size+ limit, which `?` and `**` reach at ~10^5 characters. Over-long patterns are+ now rejected explicitly with a message naming the flag and the limit, and the+ test exercises that path.++ ### Changed++ - **Two proposed byte-statistics binary heuristics were dropped before they+ shipped; raw TGA is detected by its header instead.** Both were attempts to+ recognise a raw texture that decodes into printable characters, and both+ turned out to be tests for "not written in Latin script".++ The first, from PR #274, classified any text over 4 KiB with under 5%+ whitespace as binary. `nonblank` is built from `text.lines()`, so newlines+ are already stripped and only intra-line whitespace counts — which means+ Chinese, Japanese, Korean, Thai, Lao, Khmer and Burmese prose, base64 blobs,+ FASTA sequences and minified single-line files all score 0%. This repo's own+ `failure_resume_http_tests::legacy_key_collision_fails_before_visibility_with_scoped_guidance`+ builds a 65,537-byte fixture of one repeated ASCII letter; with that guard+ reinstated the test fails with `left: 3, right: 0` — the run exits 3 instead+ of 0.++ The second was introduced by the first attempt to reland #274 and is+ **removed here**: "decoded via lossy windows-1252 AND over 30% non-ASCII is+ pixel soup". windows-1252 is the fallback every legacy 8-bit codepage decodes+ through, so it is not a test for image data. Measured through `sniff()` on+ `ca4d75a` versus that branch, with identical fixtures:++ | fixture | `ca4d75a` | with the guard |+ |---|---|---|+ | windows-1251 Russian, 13,000 B | `txt-prose` | `binary` |+ | KOI8-R Russian, 13,000 B | `txt-prose` | `binary` |+ | windows-1253 Greek, 12,200 B | `txt-prose` | `binary` |+ | windows-1255 Hebrew, 11,000 B | `txt-prose` | `binary` |+ | windows-1256 Arabic, 10,800 B | `txt-prose` | `binary` |+ | Shift-JIS Japanese, 22,401 B (byte pad 1; likewise pad 3) | `txt-prose` | `binary` |+ | Shift-JIS Japanese + ASCII code, 13,360 B | `txt-lines` | `binary` |+ | Shift-JIS Japanese + ASCII code, 66,800 B | `txt-lines` | `binary` |++ `scan_file` turns `Family::Binary` into `junk: binary content (unknown)`, so+ each of those files stopped being indexed. A `looks_like_legacy_cjk` escape+ hatch shipped with it and could not carry the weight: single-byte codepages+ never form valid Shift-JIS/GBK/Big5/EUC-KR double-byte pairs, so Cyrillic,+ Greek, Hebrew and Arabic were never rescued at all; it required a *lossless*+ trial decode while `sniff()` sees only the first 8192 bytes, so the same+ Japanese document was text at byte-offset pads 0 and 2 and binary at pads 1+ and 3; and a realistic Japanese technical document (prose around ASCII code+ fences) sits below its 30% ideograph floor while sitting above the guard's+ 30% non-ASCII ceiling. Every fixture in the table above now classifies+ exactly as it does on `ca4d75a`.++ What replaces it is `looks_like_tga_header`: TGA has no magic number, but its+ 18-byte header is constrained enough that bytes 1 and 2 of the file must both+ be control characters. **Headerless raw payloads (`.raw`, `.bytes`,+ uncompressed PCM) still classify as text**, exactly as they do on `ca4d75a`.+ Bounding what a magic-less binary costs is the fix for that and is not+ attempted here — see [#381](https://github.com/xerj-org/xerj/issues/381);+ `for_each_section` already streams, so the unbounded quantity is the record+ count, not resident memory.++ - **The new magic-byte signatures now require structural confirmation.** Nine+ signatures (PSD, both TIFF byte orders, RIFF, OGG, FLAC, MP3, FBX and EXR)+ were added by the first reland attempt as bare `starts_with` tests. **Six** of+ them are entirely printable ASCII — `8BPS`, `RIFF`, `OggS`, `fLaC`, `ID3` and+ the 18 letters `Kaydara FBX Binary` — so they matched ordinary text. Measured+ through the real `sniff()` on `ca4d75a` vs this branch with identical+ fixtures:++ | fixture | `ca4d75a` | first reland |+ | --- | --- | --- |+ | CSV whose first column header is `ID3` | `csv` | `binary`/`mp3` |+ | prose opening `RIFF is a container format used by WAV files. …` | `txt-prose` | `binary`/`riff` |+ | prose opening `OggS pages carry the packets of an Ogg stream …` | `txt-prose` | `binary`/`ogg` |+ | prose opening `fLaC is the four byte magic of a FLAC audio …` | `txt-prose` | `binary`/`flac` |+ | prose opening `8BPS is the magic of an Adobe Photoshop …` | `txt-prose` | `binary`/`psd` |+ | `.md` note opening `Kaydara FBX Binary is the 20-byte magic …` | `txt-prose` | `binary`/`fbx` |++ Each of the six is now qualified by what must follow it: PSD version, RIFF+ FORM type, Ogg stream-structure version, FLAC block type, ID3v2 major version+ and synchsafe size, and for FBX the full 23-byte header (`Kaydara FBX+ Binary`, two spaces, NUL, `0x1A`, `0x00`). The FBX case was missed by the+ first pass at this fix and caught in review — at 18 printable characters it is+ the likeliest of the six to open a real sentence, and likeliest precisely+ inside the Unity/3D-asset corpus this feature targets. Every fixture in the+ table above now classifies exactly as it does on `ca4d75a`. The true positives+ are unchanged and still covered; the FBX true-positive fixture was corrected+ to the real 23-byte header, which is what any Autodesk tool emits.++ This is what Lucene's `CodecUtil.checkHeader` does with `CODEC_MAGIC`+ (`lucene/core/src/java/org/apache/lucene/codecs/CodecUtil.java:183`, which+ hands straight to `checkHeaderNoMagic` at `:202` and refuses the file unless+ the codec name *and* a version in range follow — the magic alone is never+ taken as proof). Apache-2.0, same licence as XERJ; adapted, not copied.++ **Not fixed here, deliberately:** `GIF8` and `BM` are printable-ASCII+ signatures with exactly the same defect, but they are **pre-existing** — they+ are unchanged from `ca4d75a`, and prose opening `GIF8`/`GIF89a`, a CSV whose+ first column header is `GIF8`, and prose opening `BM` were each measured to+ classify as `binary`/`gif` and `binary`/`bmp` on `ca4d75a` **and** on this+ branch, identically. Fixing them changes behaviour unrelated to Unity, so it+ is filed as **#380** rather than smuggled in here, and today's wrong answers+ are pinned by `gif8_and_bm_are_still_taken_on_faith` so #380 has to flip that+ assertion deliberately. `GIF8` is listed alongside `BM` so the next reader+ does not conclude `BM` is the only one.++ - **A junk entry could overwrite a successfully indexed file's catalog row.**+ When phase B met a record group phase A never sampled, the worker pushed a+ whole-file junk entry while leaving `send_err` unset — so the file also+ reached `journal.file_done`, and both passes wrote `catalog::file_doc` under+ the same `file:{file_key}` id into the same bulk. The junk document landed+ second and won: a file that indexed records was reported in the catalog as+ status `junk` with `records: 0`, `files_junk` counted it, and+ `CodeCoverage::observe` ran for it twice. Reachable with no Unity involved,+ via a >64 MiB SQL dump whose first row for some table starts past+ `SQLDUMP_SAMPLE_LIMIT`. The unsampled group is now reported on the progress+ meter instead, the dropped records stay on that file's own completion where+ they already were, and the disjointness the code claimed in a comment is now+ enforced at the one place that holds both sets (`shadowed_junk_entries`).++ ### Known issues++ - `ref_guids` (Unity) and `joints` (BVH) are multi-valued keyword arrays, and+ #332 means array elements are joined into one FTS token, so a post-flush+ `term`/`match` on a single element does not hit. `script_guid`,+ `script_path` and `script_class` are single-valued and unaffected, so the+ documented "which scenes use this script?" query is not impacted. Fixing+ #332 is an engine-side change to the FTS writer input type.++ - **`GIF8` and `BM` are still unqualified printable-ASCII magic signatures**, so+ a text file whose first characters are `GIF8`, `GIF89a` or `BM` is classified+ `binary` and junked. This is **pre-existing, not introduced here** — measured+ identical on `ca4d75a` and on this branch (`binary`/`gif` and `binary`/`bmp`+ in both) — and is left unchanged because fixing it is a behaviour change with+ no Unity content. Both names are recorded here on purpose: `BM` is the+ obvious one and `GIF8` is the one a reader would otherwise miss. Tracked as+ **#380**; `gif8_and_bm_are_still_taken_on_faith` pins the current behaviour.++ - **A magic-less binary is still sectioned and indexed in full.** The two+ byte-statistics guards this branch removed (they junked non-Latin text) also+ caught NUL-free binary payloads that decode printable under the+ windows-1252 fallback. Nothing replaced them except `looks_like_tga_header`,+ which covers TGA only. Measured on this branch: a 4,194,495-byte printable+ NUL-free blob named `texture.bytes` sniffs `txt-prose` and expands into 2048+ indexed records. This is **not** the peak-RSS bug from #239 —+ `for_each_section` is still streaming and per-file memory is still bounded;+ the unbounded quantity is the record COUNT. Removing the guards was still+ right: they deleted CJK, Cyrillic, Greek, Hebrew and Arabic documents+ worldwide. Tracked as **#381**.+## [1.0.0-rc.16] - 2026-08-13Both rc.15 known issues (the progress-stream forgery and the outer-`.gitignore`