AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(autoindex): stop the reland's sniffer from junking non-Latin text
xerj-org71714fb39df5
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5564d15d0..acf629183 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -97,27 +97,81 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Changed-- **A proposed whitespace-density binary heuristic was dropped before it- shipped.** PR #274 classified any text over 4 KiB with under 5% whitespace as- binary, to catch raw TGA payloads. `nonblank` is built from `text.lines()`,- so newlines are already stripped and only intra-line whitespace counts —- which means Chinese, Japanese, Korean, Thai, Lao, Khmer and Burmese prose,- base64 blobs, FASTA sequences and minified single-line files all scored 0%- and would have been junked as "binary content (unknown)", worldwide, in every- corpus. This repo's own+- **Two proposed byte-statistics binary heuristics were dropped before they+ shipped; raw TGA is detected by its header instead.** Both were attempts to+ recognise a raw texture that decodes into printable characters, and both+ turned out to be tests for "not written in Latin script".++ The first, from PR #274, classified any text over 4 KiB with under 5%+ whitespace as binary. `nonblank` is built from `text.lines()`, so newlines+ are already stripped and only intra-line whitespace counts — which means+ Chinese, Japanese, Korean, Thai, Lao, Khmer and Burmese prose, base64 blobs,+ FASTA sequences and minified single-line files all score 0%. This repo's own`failure_resume_http_tests::legacy_key_collision_fails_before_visibility_with_scoped_guidance`- builds a 65,537-byte fixture of one repeated ASCII letter; with the guard in- place that run exits 3 instead of 0. High-byte payloads (what a real TGA is)- are still caught by the windows-1252 non-ASCII test, which is additionally- now skipped for byte sequences that decode losslessly as Shift-JIS/GBK/Big5/- EUC-KR with a substantially ideographic result — legacy-encoded CJK prose is- ~100% high-byte and was being junked by it. That probe deliberately reports- only "this is text", not *which* encoding: those four share most of their- valid double-byte space, and picking one without a statistical language model- would replace a Chinese document with plausible-looking Japanese mojibake.-- New magic-byte signatures (PSD, TIFF, RIFF, OGG, FLAC, MP3, FBX, EXR) are- kept — those are precise.+ builds a 65,537-byte fixture of one repeated ASCII letter; with that guard+ reinstated the test fails with `left: 3, right: 0` — the run exits 3 instead+ of 0.++ The second was introduced by the first attempt to reland #274 and is+ **removed here**: "decoded via lossy windows-1252 AND over 30% non-ASCII is+ pixel soup". windows-1252 is the fallback every legacy 8-bit codepage decodes+ through, so it is not a test for image data. Measured through `sniff()` on+ `ca4d75a` versus that branch, with identical fixtures:++ | fixture | `ca4d75a` | with the guard |+ |---|---|---|+ | windows-1251 Russian, 13,000 B | `txt-prose` | `binary` |+ | KOI8-R Russian, 13,000 B | `txt-prose` | `binary` |+ | windows-1253 Greek, 12,200 B | `txt-prose` | `binary` |+ | windows-1255 Hebrew, 11,000 B | `txt-prose` | `binary` |+ | windows-1256 Arabic, 10,800 B | `txt-prose` | `binary` |+ | Shift-JIS Japanese, 22,401 B (byte pad 1; likewise pad 3) | `txt-prose` | `binary` |+ | Shift-JIS Japanese + ASCII code, 13,360 B | `txt-lines` | `binary` |+ | Shift-JIS Japanese + ASCII code, 66,800 B | `txt-lines` | `binary` |++ `scan_file` turns `Family::Binary` into `junk: binary content (unknown)`, so+ each of those files stopped being indexed. A `looks_like_legacy_cjk` escape+ hatch shipped with it and could not carry the weight: single-byte codepages+ never form valid Shift-JIS/GBK/Big5/EUC-KR double-byte pairs, so Cyrillic,+ Greek, Hebrew and Arabic were never rescued at all; it required a *lossless*+ trial decode while `sniff()` sees only the first 8192 bytes, so the same+ Japanese document was text at byte-offset pads 0 and 2 and binary at pads 1+ and 3; and a realistic Japanese technical document (prose around ASCII code+ fences) sits below its 30% ideograph floor while sitting above the guard's+ 30% non-ASCII ceiling. Every fixture in the table above now classifies+ exactly as it does on `ca4d75a`.++ What replaces it is `looks_like_tga_header`: TGA has no magic number, but its+ 18-byte header is constrained enough that bytes 1 and 2 of the file must both+ be control characters. **Headerless raw payloads (`.raw`, `.bytes`,+ uncompressed PCM) still classify as text**, exactly as they do on `ca4d75a`;+ bounding sectioning memory is the fix for that, and is not attempted here.++- **The new magic-byte signatures now require structural confirmation.** PSD,+ TIFF, RIFF, OGG, FLAC, MP3, FBX and EXR were added by the first reland+ attempt as bare `starts_with` tests, and five of those signatures are+ printable ASCII, so they matched text. Measured, same method as above: a CSV+ whose first column header is `ID3` went `csv` -> `binary`, and prose whose+ first word is `RIFF`, `OggS`, `fLaC` or `8BPS` went `txt-prose` -> `binary`.+ Each printable signature is now qualified by the byte that follows it (PSD+ version, RIFF FORM type, Ogg stream-structure version, FLAC block type, ID3v2+ major version and synchsafe size), which is what Lucene's+ `CodecUtil.checkHeader` does with `CODEC_MAGIC`. The true positives are+ unchanged and still covered.++- **A junk entry could overwrite a successfully indexed file's catalog row.**+ When phase B met a record group phase A never sampled, the worker pushed a+ whole-file junk entry while leaving `send_err` unset — so the file also+ reached `journal.file_done`, and both passes wrote `catalog::file_doc` under+ the same `file:{file_key}` id into the same bulk. The junk document landed+ second and won: a file that indexed records was reported in the catalog as+ status `junk` with `records: 0`, `files_junk` counted it, and+ `CodeCoverage::observe` ran for it twice. Reachable with no Unity involved,+ via a >64 MiB SQL dump whose first row for some table starts past+ `SQLDUMP_SAMPLE_LIMIT`. The unsampled group is now reported on the progress+ meter instead, the dropped records stay on that file's own completion where+ they already were, and the disjointness the code claimed in a comment is now+ enforced at the one place that holds both sets (`shadowed_junk_entries`).### Known issuesdiff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rsindex 94eb3e0e4..e2fd7d5f4 100644--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs@@ -3251,3 +3251,139 @@ fn junk_scan_drops_its_spool_before_indexable_spools_are_retained() {drop(retained);assert_eq!(drops.load(Ordering::SeqCst), 2);}++/// A Unity scene whose SECOND `unity_class` starts past `after_bytes`, so a+/// phase A capped at that many bytes never samples it and phase B has nowhere+/// to route its records.+fn scene_with_a_late_class(after_bytes: usize) -> String {+ let mut s = String::from("%YAML 1.1\n%TAG !u! tag:unity3d.com,2011:\n");+ let mut id = 1;+ while s.len() < after_bytes {+ s.push_str(&format!(+ "--- !u!1 &{id}\nGameObject:\n m_Name: Pad{id:05}\n \+ m_TagString: Untagged\n m_IsActive: 1\n"+ ));+ id += 1;+ }+ s.push_str(+ "--- !u!114 &900001\nMonoBehaviour:\n m_Name: LateBehaviour\n \+ m_Enabled: 1\n speed: 4\n",+ );+ s+}++/// Blocker: phase B meeting a group phase A never sampled pushed a `JunkFile`+/// for the whole file while leaving `send_err` unset — so the file ALSO+/// reached `journal.file_done`, and both passes wrote `catalog::file_doc`+/// under the same `file:{file_key}` id into the same bulk. The junk document+/// landed second and won: a file that indexed real records was reported in the+/// catalog as status "junk" with records 0, and `files_junk` counted it.+///+/// Reachable with no Unity involved (a >64 MiB SQL dump whose first row for+/// some table starts past `SQLDUMP_SAMPLE_LIMIT`); Unity is used here only+/// because the override lets the fixture be 3 KB instead of 64 MB.+#[test]+fn an_unsampled_group_does_not_overwrite_the_files_indexed_catalog_row() {+ let _guard = FAILPOINT_TEST_LOCK+ .lock()+ .unwrap_or_else(|p| p.into_inner());+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK+ .lock()+ .unwrap_or_else(|p| p.into_inner());+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ // Scoped to this corpus, so running the suite multi-threaded cannot let+ // the cap reach an unrelated test's fixtures.+ let _limit = crate::SampleLimitOverride::set(corpus.path(), 2048);+ fs::create_dir_all(corpus.path().join("Assets/Scenes")).unwrap();+ fs::write(+ corpus.path().join("Assets/Scenes/Main.unity"),+ scene_with_a_late_class(2048),+ )+ .unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ // 3 = completed with junk: records WERE dropped, and that must still be+ // reported. What must not happen is the whole file being called junk.+ let (code, report) = run_index_report(config).unwrap();+ let report = report.unwrap();+ assert_eq!(code, 3, "{report}");++ let locked = endpoint.state.lock().unwrap();+ let files: Vec<&Value> = locked+ .catalog_docs+ .values()+ .filter(|doc| doc.get("doc_kind").and_then(Value::as_str) == Some("file"))+ .collect();+ assert_eq!(+ files.len(),+ 1,+ "one source file, one catalog row: {files:?}"+ );+ let file = files[0];+ assert_eq!(file["path"], "Assets/Scenes/Main.unity", "{file}");+ // The preconditions. Without these the test would pass on a fixture where+ // nothing was ever dropped, which is the failure mode it exists to catch.+ assert!(+ file["junk"].as_u64().unwrap_or(0) > 0,+ "precondition: the unsampled group's records must actually have been \+ dropped, or this fixture proves nothing: {file}"+ );+ assert!(+ file["records"].as_u64().unwrap_or(0) > 0,+ "precondition: the sampled group's records must actually have been \+ indexed: {file}"+ );+ assert_eq!(+ file["status"], "indexed",+ "a file that indexed records is not junk: {file}"+ );+ assert_eq!(+ report["files_junk"], 0,+ "the file completed; only some of its records were dropped: {report}"+ );+ assert!(+ locked.unexpected_catalog_actions.is_empty(),+ "{:?}",+ locked.unexpected_catalog_actions+ );+}++#[test]+fn a_junk_entry_for_a_completed_file_is_never_turned_into_a_catalog_row() {+ let entries = [+ state::JunkFile {+ file_key: "k-completed".into(),+ rel: "a.unity".into(),+ format: "unity".into(),+ status: "junk".into(),+ reason: "some group was never sampled".into(),+ bytes: 10,+ },+ state::JunkFile {+ file_key: "k-junked".into(),+ rel: "b.bin".into(),+ format: "binary".into(),+ status: "junk".into(),+ reason: "binary content".into(),+ bytes: 20,+ },+ ];+ let borrowed: Vec<&state::JunkFile> = entries.iter().collect();+ let completed: std::collections::HashSet<String> =+ ["k-completed".to_string()].into_iter().collect();+ let shadowed = crate::shadowed_junk_entries(&borrowed, &completed);+ assert_eq!(+ shadowed+ .iter()+ .map(|jf| jf.rel.as_str())+ .collect::<Vec<_>>(),+ vec!["a.unity"],+ "an entry naming a completed file collides with that file's own \+ catalog id and must be reported"+ );+ assert!(+ crate::shadowed_junk_entries(&borrowed, &std::collections::HashSet::new()).is_empty(),+ "with no completions nothing is shadowed"+ );+}diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex 785ca9a39..503ec5367 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -34,7 +34,7 @@ pub mod walk;use anyhow::{Context, Result};use serde::Serialize;use serde_json::{json, Map, Value};-use std::collections::HashMap;+use std::collections::{HashMap, HashSet};use std::io::{BufRead, BufReader, Seek, Write};use std::path::Path;use std::sync::atomic::{AtomicU64, Ordering};@@ -654,6 +654,35 @@ pub fn run_cli() -> i32 {}const GB: u64 = 1 << 30;++/// Junk entries that name a file which also reached `file_done`.+///+/// The catalog holds ONE document per file: `catalog::file_doc` derives its+/// `_id` from the file key alone (`catalog::file_id`). The completion pass and+/// the junk pass both write into the same bulk body, so an entry that appears+/// in both is not two rows, it is one row written twice — and the junk write,+/// which lands second, is the one that survives. The observable damage is that+/// a file which indexed N records is reported as status "junk" with records 0,+/// plus a double count in `junk_file_count` and a double+/// `CodeCoverage::observe`.+///+/// Producers are required to keep the two sets disjoint: a worker that gives up+/// on a file sets `send_err`, which suppresses the completion, and a worker+/// that merely wants to report a partial problem notes it on the progress meter+/// instead. This returns the violations rather than assuming there are none,+/// because the caller is the only place that holds both sets and because the+/// failure is silent everywhere else.+fn shadowed_junk_entries<'a>(+ all_junk: &[&'a state::JunkFile],+ completed_keys: &HashSet<String>,+) -> Vec<&'a state::JunkFile> {+ all_junk+ .iter()+ .filter(|jf| completed_keys.contains(&jf.file_key))+ .copied()+ .collect()+}+/// How many entries a human-facing listing prints before it summarises the/// rest. These lists are bounded by the corpus, not by the fault: unmounting a/// bind mount under an indexed root makes every content group vanish at once,@@ -676,6 +705,73 @@ const SQLDUMP_SAMPLE_LIMIT: u64 = 64 << 20;/// them to an anonymous counter.const UNITY_SAMPLE_LIMIT: u64 = 512 << 20;+/// How many bytes phase A reads from a file of this family before it stops+/// sampling. `None` means the family's extractor caps itself.+///+/// Split out of `scan_file` so tests can shrink it. The consequence of a+/// GROUPED family hitting this cap is not "a slightly thinner sample", it is a+/// whole `unity_class`/SQL table with no dataset to route to in phase B — and+/// the only fixture that reaches it naturally is a half-gigabyte file, which+/// is why that path shipped untested. `SampleLimitOverride` gives the suite a+/// fixture it can afford.+fn sample_limit_bytes(family: Family, path: &Path) -> Option<u64> {+ // Only the test override reads the path; the shipped caps are per-family.+ #[cfg(not(test))]+ let _ = path;+ #[cfg(test)]+ {+ // Scoped to ONE corpus root, not process-global. `cargo test` runs this+ // binary multi-threaded, phase A itself runs on `crate::pool`, and a+ // bare global would silently re-cap sampling for every unrelated test+ // that happened to overlap — a flake that would look like anything but+ // its cause.+ if let Some((root, bytes)) = SAMPLE_LIMIT_OVERRIDE+ .lock()+ .unwrap_or_else(|p| p.into_inner())+ .as_ref()+ {+ if path.starts_with(root) {+ return Some(*bytes);+ }+ }+ }+ match family {+ Family::SqlDump => Some(SQLDUMP_SAMPLE_LIMIT),+ Family::UnityYaml => Some(UNITY_SAMPLE_LIMIT),+ Family::Jsonl | Family::Logs | Family::Csv | Family::TxtLines => Some(SAMPLE_LIMIT_BYTES),+ Family::Sqlite => Some(1), // signals per-table row cap inside the extractor+ _ => None, // whole-file extractors cap themselves+ }+}++/// Test-only phase-A byte cap: `(corpus root, bytes)`, applied only to paths+/// under that root. `None` = off.+#[cfg(test)]+static SAMPLE_LIMIT_OVERRIDE: Mutex<Option<(std::path::PathBuf, u64)>> = Mutex::new(None);++/// Caps phase-A sampling under `root` for the lifetime of the guard.+#[cfg(test)]+pub(crate) struct SampleLimitOverride;++#[cfg(test)]+impl SampleLimitOverride {+ pub(crate) fn set(root: &Path, bytes: u64) -> Self {+ *SAMPLE_LIMIT_OVERRIDE+ .lock()+ .unwrap_or_else(|p| p.into_inner()) = Some((root.to_owned(), bytes));+ Self+ }+}++#[cfg(test)]+impl Drop for SampleLimitOverride {+ fn drop(&mut self) {+ *SAMPLE_LIMIT_OVERRIDE+ .lock()+ .unwrap_or_else(|p| p.into_inner()) = None;+ }+}+#[cfg(test)]static REPLACEMENT_FAILPOINT: std::sync::atomic::AtomicU8 = std::sync::atomic::AtomicU8::new(0);@@ -1086,13 +1182,7 @@ fn scan_file(out.sniffed = Some(sn);return out;}- let limit = match sn.family {- Family::SqlDump => Some(SQLDUMP_SAMPLE_LIMIT),- Family::UnityYaml => Some(UNITY_SAMPLE_LIMIT),- Family::Jsonl | Family::Logs | Family::Csv | Family::TxtLines => Some(SAMPLE_LIMIT_BYTES),- Family::Sqlite => Some(1), // signals per-table row cap inside the extractor- _ => None, // whole-file extractors cap themselves- };+ let limit = sample_limit_bytes(sn.family, path);type GroupAcc = (HashMap<String, infer::FieldAcc>,u64,@@ -4479,17 +4569,26 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {} else {String::new()};- extra_junk.lock().unwrap().push(JunkFile {- file_key: key.clone(),- rel: f.rel.clone(),- format: format_str(Some(&sn)),- status: "junk".into(),- reason: format!(- "{total} record(s) dropped: group(s) never sampled in \- phase A, so they have no dataset — {named}{suffix}"- ),- bytes: f.size,- });+ // A NOTE, never an `extra_junk` entry. This file+ // still reaches `journal.file_done` — `send_err`+ // is untouched — and every junk entry is turned+ // into a catalog document under the same+ // `file:{file_key}` id as that completion+ // (`catalog::file_doc`, `catalog.rs`). Pushing one+ // here put two documents with one id in the same+ // bulk, and the later one wins: a file that+ // indexed N records was reported as status "junk"+ // with records 0, `junk_file_count` counted it+ // twice, and `code_coverage.observe` ran for it+ // twice. The dropped records are already carried+ // on that file's own completion as `junk`+ // (`file_junk` below), which is where a per-file+ // drop count belongs.+ pr.note(&format!(+ "{}: {total} record(s) dropped: group(s) never sampled in \+ phase A, so they have no dataset — {named}{suffix}",+ f.rel+ ));}}// Raw-source href pass: the HTML extractor strips markup@@ -5062,9 +5161,11 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// the journal's completions are the corpus, not just this invocation's// slice, so a resume reports what is live rather than what it re-parsed.let mut code_coverage = CodeCoverage::default();+ let mut completed_keys: HashSet<String> = HashSet::new();{let j = journal_mx.lock().unwrap();for fd in j.done.values() {+ completed_keys.insert(fd.file_key.clone());let current_path = plan.files.get(&fd.file_key)@@ -5104,13 +5205,43 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {.collect();all_junk.extend(extra.iter());all_junk.extend(new_unplanned.iter());+ // Disjointness from the journal completions above is ENFORCED here, not+ // merely asserted in a comment: `catalog::file_doc` derives its `_id` from+ // the file key alone, so a junk entry for a file that also reached+ // `file_done` is a second document with an id the completion already used+ // — and the bulk applies them in order, so the junk one wins. A file that+ // indexed N records would be reported in the catalog as status "junk" with+ // records 0. See `shadowed_junk_entries`.+ let shadowed = shadowed_junk_entries(&all_junk, &completed_keys);+ // Debug builds stop on the producer's defect; release builds keep the+ // truthful document and say what they dropped.+ debug_assert!(+ shadowed.is_empty(),+ "junk entry for a file that reached file_done: {:?}",+ shadowed.iter().map(|jf| &jf.rel).collect::<Vec<_>>()+ );+ if !shadowed.is_empty() {+ // Not fatal: the completion document is the correct one and it is+ // already staged, so dropping the junk entry restores the truth. It is+ // still a defect in whichever producer emitted it, so say so.+ pr.note(&format!(+ "internal: {} junk entr(ies) named a file that also completed and \+ were dropped so the catalog keeps the indexed record: {}",+ shadowed.len(),+ shadowed+ .iter()+ .take(REFUSAL_LIST_CAP)+ .map(|jf| jf.rel.as_str())+ .collect::<Vec<_>>()+ .join(", ")+ ));+ all_junk.retain(|jf| !completed_keys.contains(&jf.file_key));+ }// Counted now: every later reader of this number outlives the borrows// `all_junk` holds on `plan` and `new_unplanned`, which the durable// junk-plan update below mutates.let junk_file_count = all_junk.len();for jf in &all_junk {- // Disjoint from the journal completions above: a file that reached- // `file_done` never appears here, so nothing is counted twice.code_coverage.observe(&jf.format, 0);let (id, doc) = catalog::file_doc(&jf.file_key,diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rsindex 9dc937816..66d87cc7e 100644--- a/engine/crates/xerj-autoindex/src/sniff.rs+++ b/engine/crates/xerj-autoindex/src/sniff.rs@@ -182,29 +182,62 @@ fn sniff_bytes(// to something printable), and a multi-MB PSD misread as prose costs// ~100x its size in RAM through sectioning. Magic bytes are the reliable// signal.- for (magic, kind) in [- (&b"\x89PNG"[..], "png"),- (&b"GIF8"[..], "gif"),- (&b"\xff\xd8\xff"[..], "jpeg"),- (&b"\x7fELF"[..], "elf"),- (&b"BM"[..], "bmp"),- (&b"\x00\x00\x01\x00"[..], "ico"),- (&b"8BPS"[..], "psd"),- (&b"II*\x00"[..], "tiff"),- (&b"MM\x00*"[..], "tiff"),- (&b"RIFF"[..], "riff"),- (&b"OggS"[..], "ogg"),- (&b"fLaC"[..], "flac"),- (&b"ID3"[..], "mp3"),- (&b"Kaydara FBX Binary"[..], "fbx"),- (&b"\x76\x2f\x31\x01"[..], "exr"),+ //+ // Every signature whose bytes are PRINTABLE ASCII carries a `qualify`+ // check as well, because "starts with these four letters" is also true of+ // ordinary text: a CSV whose first column header is `ID3` and prose whose+ // first word is `RIFF`, `OggS`, `fLaC` or `8BPS` were all junked as media+ // by the unqualified table (measured — see the tests below). This is the+ // rule Lucene applies to its own containers: `CodecUtil.checkHeader`+ // (`lucene/core/src/java/org/apache/lucene/codecs/CodecUtil.java:183-201`,+ // Apache-2.0) matches `CODEC_MAGIC` and then hands straight to+ // `checkHeaderNoMagic` (`CodecUtil.java:202-246`), which refuses the file+ // unless the codec name AND a version inside an accepted range follow —+ // the magic alone is never taken as proof. Adapted, not copied: Lucene is+ // validating files it wrote itself, this is declining to delete somebody+ // else's prose.+ for (magic, kind, qualify) in [+ (&b"\x89PNG"[..], "png", accept as fn(&[u8]) -> bool),+ (&b"GIF8"[..], "gif", accept as fn(&[u8]) -> bool),+ (&b"\xff\xd8\xff"[..], "jpeg", accept as fn(&[u8]) -> bool),+ (&b"\x7fELF"[..], "elf", accept as fn(&[u8]) -> bool),+ (&b"BM"[..], "bmp", accept as fn(&[u8]) -> bool),+ (&b"\x00\x00\x01\x00"[..], "ico", accept as fn(&[u8]) -> bool),+ (&b"8BPS"[..], "psd", psd_version as fn(&[u8]) -> bool),+ (&b"II*\x00"[..], "tiff", accept as fn(&[u8]) -> bool),+ (&b"MM\x00*"[..], "tiff", accept as fn(&[u8]) -> bool),+ (&b"RIFF"[..], "riff", riff_form as fn(&[u8]) -> bool),+ (&b"OggS"[..], "ogg", ogg_page as fn(&[u8]) -> bool),+ (+ &b"fLaC"[..],+ "flac",+ flac_metadata_block as fn(&[u8]) -> bool,+ ),+ (&b"ID3"[..], "mp3", id3v2_header as fn(&[u8]) -> bool),+ (+ &b"Kaydara FBX Binary"[..],+ "fbx",+ accept as fn(&[u8]) -> bool,+ ),+ (&b"\x76\x2f\x31\x01"[..], "exr", accept as fn(&[u8]) -> bool),] {- if prefix.starts_with(magic) {+ if prefix.starts_with(magic) && qualify(prefix) {let mut s = mk(Family::Binary);s.binary_kind = Some(kind.into());return Ok(s);}}+ // Truevision TGA has NO magic number — the file opens straight into its+ // 18-byte header — so a raw texture is the one image format that reaches+ // the text heuristics on its own bytes. Its header is nevertheless highly+ // constrained, which is what makes this safe: byte 1 must be 0 or 1 and+ // byte 2 one of six small values, i.e. the second and third bytes of the+ // file must BOTH be control characters, which text is not.+ if looks_like_tga_header(prefix) {+ let mut s = mk(Family::Binary);+ s.binary_kind = Some("tga".into());+ return Ok(s);+ }// 2. Binary vs text: decode UTF-8, fall back windows-1252.let (text, encoding) = decode(prefix);@@ -224,26 +257,34 @@ fn sniff_bytes(s.binary_kind = Some("unknown".into());return Ok(s);}- // A prefix that only decoded via LOSSY windows-1252 and is majority- // high-byte is pixel/float soup, not prose in a legacy encoding: real- // windows-1252 text (accented European prose) runs well under 30%- // non-ASCII, while raw image channels run ~50%. Without this, a large- // TGA classified as txt-prose is amplified ~100x in RAM by sectioning.+ // NOTE — a "decoded via lossy windows-1252 AND over 30% non-ASCII is+ // pixel soup" guard was here, to catch a raw TGA. It is gone, and must+ // not come back in that form: the windows-1252 fallback is what EVERY+ // legacy 8-bit codepage and every legacy CJK encoding decodes through, so+ // the test was not "is this an image", it was "is this written in a script+ // that is not Latin". Measured through `sniff()` against `ca4d75a` with+ // identical fixtures, it changed windows-1251 Russian, KOI8-R Russian,+ // windows-1253 Greek, windows-1255 Hebrew and windows-1256 Arabic prose+ // from `TxtProse` to `Binary`, and `scan_file` turns `Binary` into+ // "junk: binary content (unknown)" — the file is never indexed and the+ // report says only that something unreadable was skipped.//- // Gated on the windows-1252 fallback SPECIFICALLY, and skipped when the- // bytes are plausibly legacy CJK: Shift-JIS/GBK/Big5/EUC-KR prose is- // ~100% high-byte, so the unqualified test junked every legacy-encoded- // CJK document in the corpus.- let legacy_cjk = encoding == WINDOWS_1252_LOSSY && looks_like_legacy_cjk(prefix);- if encoding == WINDOWS_1252_LOSSY && !legacy_cjk {- let total = text.chars().count().max(1);- let non_ascii = text.chars().filter(|c| !c.is_ascii()).count();- if non_ascii * 10 > total * 3 {- let mut s = mk(Family::Binary);- s.binary_kind = Some("unknown".into());- return Ok(s);- }- }+ // A `looks_like_legacy_cjk` escape hatch was tried and could not carry the+ // weight, for two reasons worth recording so it is not retried:+ // * single-byte codepages never form valid SHIFT_JIS/GBK/BIG5/EUC_KR+ // double-byte pairs, so Cyrillic/Greek/Hebrew/Arabic were never+ // rescued by it at all;+ // * it required a LOSSLESS trial decode, and `sniff()` only ever sees+ // `read_prefix(path, gzip, 8192)`. A double-byte character straddling+ // that cut makes every trial decode report `had_errors`, so the same+ // Shift-JIS document was text or binary depending on its byte length+ // — measured `TxtProse` at pad 0 and 2, `Binary` at pad 1 and 3.+ //+ // TGA is now caught by its header above, which is evidence of pixel data+ // rather than evidence of a non-Latin script. Headerless raw payloads+ // (`.raw`, `.bytes`, uncompressed PCM) still classify as text, exactly as+ // they do on `main`; bounding sectioning memory is the fix for that, not+ // byte statistics that cannot tell a texture from Cyrillic.// 2b. Unity text serialization, detected by content (never extension):// scenes/prefabs/assets open with `%YAML` and declare the Unity tag@@ -319,89 +360,91 @@ fn sniff_bytes(/// the bytes are text — the binary heuristics key off this exact string.pub const WINDOWS_1252_LOSSY: &str = "windows-1252 (lossy)";-/// Scripts written without spaces between words ("scriptio continua").-///-/// Whitespace density cannot be used to tell prose from byte soup for these:-/// Chinese, Japanese, Korean, Thai, Lao, Khmer and Burmese prose runs at-/// essentially 0% intra-line whitespace, the same as random bytes. Their-/// presence is instead POSITIVE evidence of text — neither pixel channels nor-/// float dumps produce ideographs.-///-/// The script set is the one Lucene singles out for exactly this reason:-/// `CJKBigramFilter` (`lucene/analysis/common/src/java/org/apache/lucene/-/// analysis/cjk/CJKBigramFilter.java:52-118`, Apache-2.0) defines `HAN`,-/// `HIRAGANA`, `KATAKANA` and `HANGUL` flags because a whitespace tokenizer-/// cannot segment them, and its default constructor-/// (`CJKBigramFilter.java:119-126`) enables all four together. Thai, Lao,-/// Khmer and Myanmar are added here on the same principle — Lucene reaches-/// for `ICUTokenizer`/dictionary breaking for those rather than a bigram-/// filter, but the property this function tests (no inter-word spaces) is-/// identical. Adapted, not copied: Lucene is deciding how to TOKENIZE known-/// text; this only decides whether the bytes are text at all.-fn is_scriptio_continua(c: char) -> bool {- matches!(c as u32,- 0x1100..=0x11FF // Hangul Jamo- | 0x0E00..=0x0EFF // Thai, Lao- | 0x1000..=0x109F // Myanmar- | 0x1780..=0x17FF // Khmer- | 0x3000..=0x303F // CJK symbols and punctuation (、。「」)- | 0x3040..=0x30FF // Hiragana, Katakana- | 0x3400..=0x4DBF // CJK Unified Ideographs Extension A- | 0x4E00..=0x9FFF // CJK Unified Ideographs- | 0xAC00..=0xD7AF // Hangul Syllables- | 0xF900..=0xFAFF // CJK Compatibility Ideographs- | 0xFF00..=0xFFEF // Halfwidth and Fullwidth Forms- | 0x20000..=0x2A6DF // CJK Unified Ideographs Extension B- )+/// Magic-byte signature that needs no structural confirmation: its bytes are+/// not printable ASCII, so no text file can begin with them by accident.+fn accept(_prefix: &[u8]) -> bool {+ true}-/// Share of `text` that is scriptio-continua, in percent.-fn continua_percent(text: &str) -> usize {- let mut total = 0usize;- let mut hits = 0usize;- for c in text.chars() {- total += 1;- if is_scriptio_continua(c) {- hits += 1;- }- }- if total == 0 {- return 0;- }- hits * 100 / total+/// Adobe Photoshop: `8BPS` is followed by a big-endian version, 1 for PSD and+/// 2 for PSB. Nothing else is defined, so a text file whose first word is+/// `8BPS` fails here on its fifth byte.+fn psd_version(prefix: &[u8]) -> bool {+ prefix.len() >= 6 && prefix[4] == 0 && matches!(prefix[5], 1 | 2)+}++/// RIFF container: `RIFF`, a little-endian chunk size, then a four-character+/// FORM type naming what the container actually holds. Requiring a known FORM+/// is what separates a WAV from a sentence that opens with the word "RIFF".+fn riff_form(prefix: &[u8]) -> bool {+ prefix.len() >= 12+ && matches!(+ &prefix[8..12],+ b"WAVE" | b"AVI " | b"WEBP" | b"RMID" | b"ANI " | b"CDDA" | b"PAL " | b"RDIB"+ )}-/// Whether `bytes` are plausibly text in a legacy CJK encoding.+/// Ogg page header: `OggS` then the stream structure version, which has been+/// 0 for the life of the format (RFC 3533 §6.1).+fn ogg_page(prefix: &[u8]) -> bool {+ prefix.len() >= 5 && prefix[4] == 0+}++/// FLAC stream: `fLaC` then a METADATA_BLOCK_HEADER whose block type must be+/// STREAMINFO (0) for the first block; the top bit is the last-block flag.+fn flac_metadata_block(prefix: &[u8]) -> bool {+ prefix.len() >= 5 && prefix[4] & 0x7f == 0+}++/// ID3v2 tag header: `ID3`, a major version (2, 3 and 4 are the versions that+/// exist), a revision that is not the reserved 0xFF, a flags byte, and a+/// four-byte SYNCHSAFE size whose bytes each have the top bit clear.///-/// This is a TEXT-NESS test and nothing more. It deliberately does NOT return-/// which encoding: Shift-JIS, GBK, Big5 and EUC-KR share most of the same-/// valid double-byte space, so a GBK document also decodes losslessly as-/// Shift-JIS (and vice versa) with a high ideograph share either way. Telling-/// them apart needs a statistical language model — what `chardet`/ICU do —-/// which this crate does not have, and guessing would silently replace a-/// Chinese document with plausible-looking Japanese mojibake. Tracked as a-/// follow-up; see CHANGELOG.+/// Unqualified, this signature is three ASCII letters: a CSV whose first+/// column header is `ID3` classified `Csv` on `ca4d75a` and `Binary` on this+/// branch until the version byte was checked.+fn id3v2_header(prefix: &[u8]) -> bool {+ prefix.len() >= 10+ && matches!(prefix[3], 2..=4)+ && prefix[4] != 0xff+ && prefix[6..10].iter().all(|b| *b < 0x80)+}++/// Truevision TGA, which has no magic number at all — every byte of its+/// 18-byte header is data. Detection is therefore by CONSTRAINT: an image type+/// from the defined set, a pixel depth from the defined set, a colour-map+/// descriptor that must be all-zero when there is no colour map, and non-zero+/// dimensions.///-/// What it IS sound for is the only question asked here: are these bytes a-/// pixel/float payload, or somebody's prose? The pixel-soup guards below key-/// off high-byte density and whitespace density, and legacy CJK prose looks-/// exactly like soup under both — ~100% high-byte and ~0% whitespace — so-/// without this they junk the file outright. Requiring a LOSSLESS decode-/// (one undecodable pair disqualifies it) plus a substantially ideographic-/// result keeps European windows-1252 prose out: its high bytes sit isolated-/// among ASCII, so most form no valid double-byte pair at all.-fn looks_like_legacy_cjk(bytes: &[u8]) -> bool {- [- encoding_rs::SHIFT_JIS,- encoding_rs::GBK,- encoding_rs::BIG5,- encoding_rs::EUC_KR,- ]- .into_iter()- .any(|enc| {- let (s, _, had_errors) = enc.decode(bytes);- !had_errors && continua_percent(&s) >= 30- })+/// The reason this is safe on text is byte 1 and byte 2: `color_map_type` is 0+/// or 1 and `image_type` is one of six values below 12, so both the second and+/// the third byte of the file have to be control characters. Prose, CSV, JSON+/// and source code fail on byte 1.+fn looks_like_tga_header(prefix: &[u8]) -> bool {+ if prefix.len() < 18 {+ return false;+ }+ let color_map_type = prefix[1];+ if color_map_type > 1 {+ return false;+ }+ // 1/2/3 uncompressed colour-mapped/true-colour/greyscale, 9/10/11 the+ // run-length encoded counterparts. Type 0 ("no image data") is excluded+ // deliberately: it carries no pixels, so nothing is lost by letting such a+ // file fall through to the text heuristics, and admitting it would make an+ // all-zero prefix a TGA.+ if !matches!(prefix[2], 1 | 2 | 3 | 9 | 10 | 11) {+ return false;+ }+ if !matches!(prefix[16], 8 | 15 | 16 | 24 | 32) {+ return false;+ }+ if color_map_type == 0 && prefix[3..8].iter().any(|b| *b != 0) {+ return false;+ }+ let width = u16::from_le_bytes([prefix[12], prefix[13]]);+ let height = u16::from_le_bytes([prefix[14], prefix[15]]);+ width > 0 && height > 0}fn decode(bytes: &[u8]) -> (String, &'static str) {@@ -564,10 +607,14 @@ fn txt_kind(nonblank: &[&str]) -> Family {// binary and the run exits 3 instead of 0 — the false positive is// reachable from this repo's own suite.//- // High-byte payloads (which is what a real TGA is) are still caught, by- // the windows-1252 non-ASCII test in `sniff_bytes`. Bounding sectioning- // memory is the right fix for the RAM concern; silently deleting files- // that look unusual is not. Tracked as a follow-up.+ // A real TGA is caught by `looks_like_tga_header` in `sniff_bytes`, on the+ // structure of its 18-byte header. The byte-statistics version of that+ // check ("mostly non-ASCII under the windows-1252 fallback") was tried and+ // removed in the same breath as this one and for the same reason: it+ // junked Cyrillic, Greek, Hebrew, Arabic and mixed Japanese prose.+ // Bounding sectioning memory is the right fix for the RAM concern;+ // silently deleting files that look unusual is not. Tracked as a+ // follow-up.let avg_len = nonblank.iter().map(|l| l.len()).sum::<usize>() as f64 / nonblank.len() as f64;if avg_len > 60.0 {return Family::TxtProse;@@ -805,6 +852,8 @@ mod unity_sniff_tests {("t.tif2", &b"MM\x00*\x00\x08"[..]),("t.wav", &b"RIFF\x24\x08\x00\x00WAVE"[..]),("t.ogg", &b"OggS\x00\x02"[..]),+ ("t.flac", &b"fLaC\x00\x00\x00\x22"[..]),+ ("t.mp3", &b"ID3\x03\x00\x00\x00\x00\x0f\x76"[..]),("t.fbx", &b"Kaydara FBX Binary \x00"[..]),] {let mut bytes = head.to_vec();@@ -815,6 +864,99 @@ mod unity_sniff_tests {}}+ /// Regression (round 2): the printable-ASCII signatures added by this+ /// branch matched TEXT. Measured through `sniff()` on `ca4d75a` vs this+ /// branch with identical fixtures, a CSV whose first column header is+ /// `ID3` went `Csv` -> `Binary`, and prose whose first word is `RIFF`,+ /// `OggS`, `fLaC` or `8BPS` went `TxtProse` -> `Binary`. `scan_file` turns+ /// `Family::Binary` into "junk: binary content (unknown)", so each of+ /// those files stopped being indexed at all.+ ///+ /// The true positives above must keep passing, which is the whole point:+ /// the signature is necessary, it is just not sufficient.+ #[test]+ fn a_printable_ascii_signature_alone_does_not_make_a_file_binary() {+ let csv = b"ID3,name,value\n1,alpha,2\n3,beta,4\n5,gamma,6\n7,delta,8\n";+ assert_eq!(+ sniff_bytes(csv, Path::new("t.csv"), Path::new("t.csv"), false)+ .unwrap()+ .family,+ Family::Csv,+ "a column header may legitimately be the three letters ID3"+ );+ for (label, opener) in [+ (+ "riff",+ "RIFF is a container format used by WAV files. It stores chunks. ",+ ),+ (+ "oggs",+ "OggS pages carry the packets of an Ogg stream in order. ",+ ),+ (+ "flac",+ "fLaC is the four byte magic of a FLAC audio stream header. ",+ ),+ (+ "psd",+ "8BPS is the magic of an Adobe Photoshop document header. ",+ ),+ ] {+ let text = opener.repeat(30);+ let sn = sniff_bytes(+ text.as_bytes(),+ Path::new("notes.txt"),+ Path::new("notes.txt"),+ false,+ )+ .unwrap();+ assert_ne!(+ sn.family,+ Family::Binary,+ "{label}: prose that opens with the signature word is still prose"+ );+ }+ }++ /// TGA is the format the removed byte-statistics guard existed for: it has+ /// no magic number, so a raw texture reaches the text heuristics on its own+ /// bytes. It is caught on the structure of its 18-byte header instead.+ #[test]+ fn a_raw_tga_texture_is_binary_by_header_structure() {+ // id_len=0, no colour map, uncompressed true-colour, 64x64, 24bpp.+ let mut tga: Vec<u8> = vec![0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 0, 64, 0, 24, 0];+ tga.extend((0..8192u32).map(|i| (0x80 + (i * 37) % 0x7f) as u8));+ let sn = sniff_bytes(+ &tga,+ Path::new("texture.tga"),+ Path::new("texture.tga"),+ false,+ )+ .unwrap();+ assert_eq!(sn.family, Family::Binary, "a real TGA header must be seen");+ assert_eq!(sn.binary_kind.as_deref(), Some("tga"));++ // And the constraint must be tight enough that text never satisfies+ // it. Every one of these is >= 18 bytes, so length is not what saves+ // them.+ for (label, text) in [+ (+ "prose",+ "The quick brown fox jumps over the lazy dog again.",+ ),+ ("csv", "id,name,value\n1,alpha,2\n3,beta,4\n"),+ ("json", "{\"id\": 1, \"name\": \"alpha\", \"value\": 2}"),+ ("code", "fn main() { println!(\"hello, world\"); }"),+ ("yaml", "name: alpha\nvalue: 2\nnested:\n a: 1\n"),+ ] {+ assert!(text.len() >= 18, "{label}: fixture too short to prove it");+ assert!(+ !looks_like_tga_header(text.as_bytes()),+ "{label}: text must not satisfy the TGA header constraints"+ );+ }+ }+/// Whitespace density must NOT be used to junk text. This pins the/// counter-examples that make the rejected guard unsafe: a long run of one/// ASCII letter (the shape of this repo's own 65,537-byte resume-key@@ -902,33 +1044,65 @@ mod unity_sniff_tests {}}- /// Regression: the sibling guard keyed off `encoding != "utf-8"`, and a- /// legacy CJK encoding decodes as lossy windows-1252 into ~100%- /// high-byte mojibake — so every Shift-JIS/GBK/Big5/EUC-KR document was- /// junked before any family heuristic ran.+ /// Classify a byte buffer through the REAL read path: a file on disk, read+ /// by `sniff()`, which sees only `read_prefix(path, gzip, 8192)`.+ ///+ /// Every legacy-encoding regression below has to go through this and not+ /// through `sniff_bytes` with the whole buffer. The escape hatch that was+ /// supposed to protect legacy CJK required a LOSSLESS trial decode, and a+ /// double-byte character straddling the 8192-byte cut makes every trial+ /// decode fail — so a test that hands over the complete buffer cannot+ /// observe half of the bug it is written for.+ fn sniff_file(bytes: &[u8], name: &str) -> Family {+ let dir = tempfile::tempdir().unwrap();+ let path = dir.path().join(name);+ std::fs::write(&path, bytes).unwrap();+ sniff(&path).unwrap().family+ }++ /// Regression (round 2): a guard on "lossy windows-1252 AND over 30%+ /// non-ASCII" is not a test for pixel data, it is a test for "not written+ /// in Latin script". windows-1252 is the fallback EVERY legacy 8-bit+ /// codepage decodes through.+ ///+ /// Measured through `sniff()` with these exact fixtures: on `ca4d75a` all+ /// five are `TxtProse`; with the guard present all five are `Binary`, and+ /// `scan_file` turns that into "junk: binary content (unknown)".///- /// The assertion is deliberately only "not binary". This crate cannot- /// tell these encodings apart (they share most of their valid- /// double-byte space) and does not pretend to: the file is still decoded- /// as windows-1252, exactly as it was before this branch, which is- /// mojibake — but mojibake that is INDEXED rather than deleted.+ /// The assertion is deliberately only "not binary" — this crate still+ /// decodes the bytes as windows-1252 and still produces mojibake, because+ /// telling legacy codepages apart needs a statistical language model it+ /// does not have. Mojibake that is INDEXED beats prose that is deleted.#[test]- fn legacy_encoded_cjk_prose_is_not_junked_as_binary() {+ fn legacy_single_byte_prose_is_not_junked_by_the_real_read_path() {for (label, enc, unit) in [(- "shift_jis",- encoding_rs::SHIFT_JIS,- "この文書はシステムの設計について説明します。",+ "windows-1251-ru",+ encoding_rs::WINDOWS_1251,+ "Этот документ описывает архитектуру системы и детали реализации. ",),- ("gbk", encoding_rs::GBK, "本文档描述了系统的架构设计。"),- ("big5", encoding_rs::BIG5, "本文件描述系統的架構設計。"),(- "euc-kr",- encoding_rs::EUC_KR,- "이문서는시스템설계를설명합니다",+ "koi8-r-ru",+ encoding_rs::KOI8_R,+ "Этот документ описывает архитектуру системы и детали реализации. ",+ ),+ (+ "windows-1253-el",+ encoding_rs::WINDOWS_1253,+ "Το έγγραφο αυτό περιγράφει την αρχιτεκτονική του συστήματος. ",+ ),+ (+ "windows-1255-he",+ encoding_rs::WINDOWS_1255,+ "מסמך זה מתאר את ארכיטקטורת המערכת ואת פרטי היישום שלה. ",+ ),+ (+ "windows-1256-ar",+ encoding_rs::WINDOWS_1256,+ "تصف هذه الوثيقة بنية النظام وتفاصيل تنفيذه بشكل كامل. ",),] {- let text = unit.repeat(4096 / unit.chars().count() + 2);+ let text = unit.repeat(200);let (bytes, _, had_errors) = enc.encode(&text);assert!(!had_errors, "{label}: fixture must encode cleanly");assert!(@@ -936,52 +1110,120 @@ mod unity_sniff_tests {"{label}: fixture must not be valid UTF-8, or it proves nothing");assert!(- looks_like_legacy_cjk(&bytes),- "{label}: text-ness probe must recognise this as prose"+ bytes.len() > 8192,+ "{label}: fixture must exceed the sniff prefix"+ );+ assert_ne!(+ sniff_file(&bytes, "doc.txt"),+ Family::Binary,+ "{label}: legacy-encoded prose must not be junked as binary");- let sn =- sniff_bytes(&bytes, Path::new("doc.txt"), Path::new("doc.txt"), false).unwrap();- assert_ne!(sn.family, Family::Binary, "{label}: must not be junked");}}- /// The legacy-CJK probe must not capture European text — that would- /// replace real windows-1252 prose with plausible-looking ideographs.+ /// Regression (round 2): the legacy-CJK escape hatch was alignment+ /// dependent. `sniff()` classifies from the first 8192 bytes only, so a+ /// double-byte character split by that cut made the trial decode report+ /// `had_errors` and the rescue declined. Measured on the branch that+ /// carried it: the SAME Shift-JIS document was `TxtProse` at byte pads 0+ /// and 2 and `Binary` at pads 1 and 3 — a coin flip per file.+ ///+ /// Sweeping the alignment is the point of this test; a single-length+ /// fixture passes half the time by luck.#[test]- fn european_windows_1252_prose_is_not_claimed_as_cjk() {+ fn legacy_cjk_prose_survives_every_prefix_alignment() {+ for (label, enc, unit) in [+ (+ "shift_jis",+ encoding_rs::SHIFT_JIS,+ "この文書はシステムの設計と実装について詳しく説明します。",+ ),+ (+ "gbk",+ encoding_rs::GBK,+ "本文档描述了系统的架构设计与实现。",+ ),+ (+ "big5",+ encoding_rs::BIG5,+ "本文件描述系統的架構設計與實作。",+ ),+ (+ "euc-kr",+ encoding_rs::EUC_KR,+ "이문서는시스템의설계와구현을설명합니다",+ ),+ ] {+ let text = unit.repeat(400);+ let (body, _, had_errors) = enc.encode(&text);+ assert!(!had_errors, "{label}: fixture must encode cleanly");+ assert!(+ body.len() > 8192,+ "{label}: fixture must exceed the sniff prefix, or alignment \+ cannot matter"+ );+ for pad in 0..4usize {+ let mut bytes = vec![b'a'; pad];+ bytes.extend_from_slice(&body);+ assert_ne!(+ sniff_file(&bytes, "doc.txt"),+ Family::Binary,+ "{label}: junked at byte-offset pad {pad}"+ );+ }+ }+ }++ /// Regression (round 2): a realistic Japanese technical document is not+ /// 100% ideographs — it is prose around ASCII code fences, identifiers and+ /// URLs. That shape sat under the 30% scriptio-continua threshold the+ /// rescue needed while sitting over the 30% non-ASCII threshold the guard+ /// junked on, so the ordinary shape of real CJK documentation was the case+ /// that lost. Measured: `TxtLines` on `ca4d75a`, `Binary` on the branch,+ /// at both 13 KB and 67 KB.+ #[test]+ fn mixed_script_documentation_is_not_junked() {+ let unit = "この関数は入力データを検証してから処理を実行し、結果を呼び出し元に返します。\n\+ エラーが発生した場合は、詳細な理由を含む診断情報を記録します。\n\+ ```\nfn process(d: &[u8]) -> Result<Vec<u8>, ProcessError> {\n \+ let checked = validate(d)?;\n Ok(checked.to_vec())\n}\n```\n\+ See docs/process.md and the API reference for the full parameter list.\n";+ let ideographic =+ unit.chars().filter(|c| !c.is_ascii()).count() * 100 / unit.chars().count().max(1);+ assert!(+ (20..30).contains(&ideographic),+ "fixture must sit in the band that loses — between the rescue's \+ 30% ideograph floor and the guard's 30% non-ASCII ceiling; got \+ {ideographic}%"+ );+ for reps in [40usize, 200] {+ let doc = unit.repeat(reps);+ let (bytes, _, _) = encoding_rs::SHIFT_JIS.encode(&doc);+ assert_ne!(+ sniff_file(&bytes, "doc.txt"),+ Family::Binary,+ "mixed Japanese/ASCII documentation ({} bytes) must not be junked",+ bytes.len()+ );+ }+ }++ /// Accented European prose is the case the removed guard claimed to be+ /// safe for. It must stay safe now that the guard is gone.+ #[test]+ fn european_windows_1252_prose_is_still_prose() {let text = "Le système décrit ici gère les données réservées à \l'opérateur, prêtes à être exportées. ";- let long = text.repeat(80);+ let long = text.repeat(200);let (bytes, _, _) = encoding_rs::WINDOWS_1252.encode(&long);assert!(std::str::from_utf8(&bytes).is_err());let (_, enc) = decode(&bytes);assert_eq!(enc, WINDOWS_1252_LOSSY, "must stay windows-1252");- assert!(- !looks_like_legacy_cjk(&bytes),- "accented Latin must not be claimed as CJK"+ assert_ne!(+ sniff_file(&bytes, "note.txt"),+ Family::Binary,+ "accented prose is still prose");- let sn = sniff_bytes(&bytes, Path::new("note.txt"), Path::new("note.txt"), false).unwrap();- assert_ne!(sn.family, Family::Binary, "accented prose is still prose");- }-- /// The pixel-soup protection the guards exist for must survive the fix:- /// high-byte float/pixel payloads still have to be caught.- #[test]- fn high_byte_pixel_soup_is_still_binary() {- // Bytes that are not valid UTF-8, not valid in any legacy CJK- // encoding as a whole, and majority high-byte — a raw texture.- let soup: Vec<u8> = (0..8192u32)- .map(|i| (0x80 + (i * 37) % 0x7f) as u8)- .collect();- assert!(std::str::from_utf8(&soup).is_err());- let sn = sniff_bytes(- &soup,- Path::new("texture.tga"),- Path::new("texture.tga"),- false,- )- .unwrap();- assert_eq!(sn.family, Family::Binary, "pixel soup must stay binary");}#[test]