Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(autoindex): stop the reland's sniffer from junking non-Latin text

xerj-org71714fb39df5
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5564d15d0..acf629183 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -97,27 +97,81 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
-- **A proposed whitespace-density binary heuristic was dropped before it
- shipped.** PR #274 classified any text over 4 KiB with under 5% whitespace as
- binary, to catch raw TGA payloads. `nonblank` is built from `text.lines()`,
- so newlines are already stripped and only intra-line whitespace counts —
- which means Chinese, Japanese, Korean, Thai, Lao, Khmer and Burmese prose,
- base64 blobs, FASTA sequences and minified single-line files all scored 0%
- and would have been junked as "binary content (unknown)", worldwide, in every
- corpus. This repo's own
+- **Two proposed byte-statistics binary heuristics were dropped before they
+ shipped; raw TGA is detected by its header instead.** Both were attempts to
+ recognise a raw texture that decodes into printable characters, and both
+ turned out to be tests for "not written in Latin script".
+
+ The first, from PR #274, classified any text over 4 KiB with under 5%
+ whitespace as binary. `nonblank` is built from `text.lines()`, so newlines
+ are already stripped and only intra-line whitespace counts — which means
+ Chinese, Japanese, Korean, Thai, Lao, Khmer and Burmese prose, base64 blobs,
+ FASTA sequences and minified single-line files all score 0%. This repo's own
`failure_resume_http_tests::legacy_key_collision_fails_before_visibility_with_scoped_guidance`
- builds a 65,537-byte fixture of one repeated ASCII letter; with the guard in
- place that run exits 3 instead of 0. High-byte payloads (what a real TGA is)
- are still caught by the windows-1252 non-ASCII test, which is additionally
- now skipped for byte sequences that decode losslessly as Shift-JIS/GBK/Big5/
- EUC-KR with a substantially ideographic result — legacy-encoded CJK prose is
- ~100% high-byte and was being junked by it. That probe deliberately reports
- only "this is text", not *which* encoding: those four share most of their
- valid double-byte space, and picking one without a statistical language model
- would replace a Chinese document with plausible-looking Japanese mojibake.
-
- New magic-byte signatures (PSD, TIFF, RIFF, OGG, FLAC, MP3, FBX, EXR) are
- kept — those are precise.
+ builds a 65,537-byte fixture of one repeated ASCII letter; with that guard
+ reinstated the test fails with `left: 3, right: 0` — the run exits 3 instead
+ of 0.
+
+ The second was introduced by the first attempt to reland #274 and is
+ **removed here**: "decoded via lossy windows-1252 AND over 30% non-ASCII is
+ pixel soup". windows-1252 is the fallback every legacy 8-bit codepage decodes
+ through, so it is not a test for image data. Measured through `sniff()` on
+ `ca4d75a` versus that branch, with identical fixtures:
+
+ | fixture | `ca4d75a` | with the guard |
+ |---|---|---|
+ | windows-1251 Russian, 13,000 B | `txt-prose` | `binary` |
+ | KOI8-R Russian, 13,000 B | `txt-prose` | `binary` |
+ | windows-1253 Greek, 12,200 B | `txt-prose` | `binary` |
+ | windows-1255 Hebrew, 11,000 B | `txt-prose` | `binary` |
+ | windows-1256 Arabic, 10,800 B | `txt-prose` | `binary` |
+ | Shift-JIS Japanese, 22,401 B (byte pad 1; likewise pad 3) | `txt-prose` | `binary` |
+ | Shift-JIS Japanese + ASCII code, 13,360 B | `txt-lines` | `binary` |
+ | Shift-JIS Japanese + ASCII code, 66,800 B | `txt-lines` | `binary` |
+
+ `scan_file` turns `Family::Binary` into `junk: binary content (unknown)`, so
+ each of those files stopped being indexed. A `looks_like_legacy_cjk` escape
+ hatch shipped with it and could not carry the weight: single-byte codepages
+ never form valid Shift-JIS/GBK/Big5/EUC-KR double-byte pairs, so Cyrillic,
+ Greek, Hebrew and Arabic were never rescued at all; it required a *lossless*
+ trial decode while `sniff()` sees only the first 8192 bytes, so the same
+ Japanese document was text at byte-offset pads 0 and 2 and binary at pads 1
+ and 3; and a realistic Japanese technical document (prose around ASCII code
+ fences) sits below its 30% ideograph floor while sitting above the guard's
+ 30% non-ASCII ceiling. Every fixture in the table above now classifies
+ exactly as it does on `ca4d75a`.
+
+ What replaces it is `looks_like_tga_header`: TGA has no magic number, but its
+ 18-byte header is constrained enough that bytes 1 and 2 of the file must both
+ be control characters. **Headerless raw payloads (`.raw`, `.bytes`,
+ uncompressed PCM) still classify as text**, exactly as they do on `ca4d75a`;
+ bounding sectioning memory is the fix for that, and is not attempted here.
+
+- **The new magic-byte signatures now require structural confirmation.** PSD,
+ TIFF, RIFF, OGG, FLAC, MP3, FBX and EXR were added by the first reland
+ attempt as bare `starts_with` tests, and five of those signatures are
+ printable ASCII, so they matched text. Measured, same method as above: a CSV
+ whose first column header is `ID3` went `csv` -> `binary`, and prose whose
+ first word is `RIFF`, `OggS`, `fLaC` or `8BPS` went `txt-prose` -> `binary`.
+ Each printable signature is now qualified by the byte that follows it (PSD
+ version, RIFF FORM type, Ogg stream-structure version, FLAC block type, ID3v2
+ major version and synchsafe size), which is what Lucene's
+ `CodecUtil.checkHeader` does with `CODEC_MAGIC`. The true positives are
+ unchanged and still covered.
+
+- **A junk entry could overwrite a successfully indexed file's catalog row.**
+ When phase B met a record group phase A never sampled, the worker pushed a
+ whole-file junk entry while leaving `send_err` unset — so the file also
+ reached `journal.file_done`, and both passes wrote `catalog::file_doc` under
+ the same `file:{file_key}` id into the same bulk. The junk document landed
+ second and won: a file that indexed records was reported in the catalog as
+ status `junk` with `records: 0`, `files_junk` counted it, and
+ `CodeCoverage::observe` ran for it twice. Reachable with no Unity involved,
+ via a >64 MiB SQL dump whose first row for some table starts past
+ `SQLDUMP_SAMPLE_LIMIT`. The unsampled group is now reported on the progress
+ meter instead, the dropped records stay on that file's own completion where
+ they already were, and the disjointness the code claimed in a comment is now
+ enforced at the one place that holds both sets (`shadowed_junk_entries`).
### Known issues
diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
index 94eb3e0e4..e2fd7d5f4 100644
--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
@@ -3251,3 +3251,139 @@ fn junk_scan_drops_its_spool_before_indexable_spools_are_retained() {
drop(retained);
assert_eq!(drops.load(Ordering::SeqCst), 2);
}
+
+/// A Unity scene whose SECOND `unity_class` starts past `after_bytes`, so a
+/// phase A capped at that many bytes never samples it and phase B has nowhere
+/// to route its records.
+fn scene_with_a_late_class(after_bytes: usize) -> String {
+ let mut s = String::from("%YAML 1.1\n%TAG !u! tag:unity3d.com,2011:\n");
+ let mut id = 1;
+ while s.len() < after_bytes {
+ s.push_str(&format!(
+ "--- !u!1 &{id}\nGameObject:\n m_Name: Pad{id:05}\n \
+ m_TagString: Untagged\n m_IsActive: 1\n"
+ ));
+ id += 1;
+ }
+ s.push_str(
+ "--- !u!114 &900001\nMonoBehaviour:\n m_Name: LateBehaviour\n \
+ m_Enabled: 1\n speed: 4\n",
+ );
+ s
+}
+
+/// Blocker: phase B meeting a group phase A never sampled pushed a `JunkFile`
+/// for the whole file while leaving `send_err` unset — so the file ALSO
+/// reached `journal.file_done`, and both passes wrote `catalog::file_doc`
+/// under the same `file:{file_key}` id into the same bulk. The junk document
+/// landed second and won: a file that indexed real records was reported in the
+/// catalog as status "junk" with records 0, and `files_junk` counted it.
+///
+/// Reachable with no Unity involved (a >64 MiB SQL dump whose first row for
+/// some table starts past `SQLDUMP_SAMPLE_LIMIT`); Unity is used here only
+/// because the override lets the fixture be 3 KB instead of 64 MB.
+#[test]
+fn an_unsampled_group_does_not_overwrite_the_files_indexed_catalog_row() {
+ let _guard = FAILPOINT_TEST_LOCK
+ .lock()
+ .unwrap_or_else(|p| p.into_inner());
+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK
+ .lock()
+ .unwrap_or_else(|p| p.into_inner());
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ // Scoped to this corpus, so running the suite multi-threaded cannot let
+ // the cap reach an unrelated test's fixtures.
+ let _limit = crate::SampleLimitOverride::set(corpus.path(), 2048);
+ fs::create_dir_all(corpus.path().join("Assets/Scenes")).unwrap();
+ fs::write(
+ corpus.path().join("Assets/Scenes/Main.unity"),
+ scene_with_a_late_class(2048),
+ )
+ .unwrap();
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ // 3 = completed with junk: records WERE dropped, and that must still be
+ // reported. What must not happen is the whole file being called junk.
+ let (code, report) = run_index_report(config).unwrap();
+ let report = report.unwrap();
+ assert_eq!(code, 3, "{report}");
+
+ let locked = endpoint.state.lock().unwrap();
+ let files: Vec<&Value> = locked
+ .catalog_docs
+ .values()
+ .filter(|doc| doc.get("doc_kind").and_then(Value::as_str) == Some("file"))
+ .collect();
+ assert_eq!(
+ files.len(),
+ 1,
+ "one source file, one catalog row: {files:?}"
+ );
+ let file = files[0];
+ assert_eq!(file["path"], "Assets/Scenes/Main.unity", "{file}");
+ // The preconditions. Without these the test would pass on a fixture where
+ // nothing was ever dropped, which is the failure mode it exists to catch.
+ assert!(
+ file["junk"].as_u64().unwrap_or(0) > 0,
+ "precondition: the unsampled group's records must actually have been \
+ dropped, or this fixture proves nothing: {file}"
+ );
+ assert!(
+ file["records"].as_u64().unwrap_or(0) > 0,
+ "precondition: the sampled group's records must actually have been \
+ indexed: {file}"
+ );
+ assert_eq!(
+ file["status"], "indexed",
+ "a file that indexed records is not junk: {file}"
+ );
+ assert_eq!(
+ report["files_junk"], 0,
+ "the file completed; only some of its records were dropped: {report}"
+ );
+ assert!(
+ locked.unexpected_catalog_actions.is_empty(),
+ "{:?}",
+ locked.unexpected_catalog_actions
+ );
+}
+
+#[test]
+fn a_junk_entry_for_a_completed_file_is_never_turned_into_a_catalog_row() {
+ let entries = [
+ state::JunkFile {
+ file_key: "k-completed".into(),
+ rel: "a.unity".into(),
+ format: "unity".into(),
+ status: "junk".into(),
+ reason: "some group was never sampled".into(),
+ bytes: 10,
+ },
+ state::JunkFile {
+ file_key: "k-junked".into(),
+ rel: "b.bin".into(),
+ format: "binary".into(),
+ status: "junk".into(),
+ reason: "binary content".into(),
+ bytes: 20,
+ },
+ ];
+ let borrowed: Vec<&state::JunkFile> = entries.iter().collect();
+ let completed: std::collections::HashSet<String> =
+ ["k-completed".to_string()].into_iter().collect();
+ let shadowed = crate::shadowed_junk_entries(&borrowed, &completed);
+ assert_eq!(
+ shadowed
+ .iter()
+ .map(|jf| jf.rel.as_str())
+ .collect::<Vec<_>>(),
+ vec!["a.unity"],
+ "an entry naming a completed file collides with that file's own \
+ catalog id and must be reported"
+ );
+ assert!(
+ crate::shadowed_junk_entries(&borrowed, &std::collections::HashSet::new()).is_empty(),
+ "with no completions nothing is shadowed"
+ );
+}
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index 785ca9a39..503ec5367 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -34,7 +34,7 @@ pub mod walk;
use anyhow::{Context, Result};
use serde::Serialize;
use serde_json::{json, Map, Value};
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
use std::io::{BufRead, BufReader, Seek, Write};
use std::path::Path;
use std::sync::atomic::{AtomicU64, Ordering};
@@ -654,6 +654,35 @@ pub fn run_cli() -> i32 {
}
const GB: u64 = 1 << 30;
+
+/// Junk entries that name a file which also reached `file_done`.
+///
+/// The catalog holds ONE document per file: `catalog::file_doc` derives its
+/// `_id` from the file key alone (`catalog::file_id`). The completion pass and
+/// the junk pass both write into the same bulk body, so an entry that appears
+/// in both is not two rows, it is one row written twice — and the junk write,
+/// which lands second, is the one that survives. The observable damage is that
+/// a file which indexed N records is reported as status "junk" with records 0,
+/// plus a double count in `junk_file_count` and a double
+/// `CodeCoverage::observe`.
+///
+/// Producers are required to keep the two sets disjoint: a worker that gives up
+/// on a file sets `send_err`, which suppresses the completion, and a worker
+/// that merely wants to report a partial problem notes it on the progress meter
+/// instead. This returns the violations rather than assuming there are none,
+/// because the caller is the only place that holds both sets and because the
+/// failure is silent everywhere else.
+fn shadowed_junk_entries<'a>(
+ all_junk: &[&'a state::JunkFile],
+ completed_keys: &HashSet<String>,
+) -> Vec<&'a state::JunkFile> {
+ all_junk
+ .iter()
+ .filter(|jf| completed_keys.contains(&jf.file_key))
+ .copied()
+ .collect()
+}
+
/// How many entries a human-facing listing prints before it summarises the
/// rest. These lists are bounded by the corpus, not by the fault: unmounting a
/// bind mount under an indexed root makes every content group vanish at once,
@@ -676,6 +705,73 @@ const SQLDUMP_SAMPLE_LIMIT: u64 = 64 << 20;
/// them to an anonymous counter.
const UNITY_SAMPLE_LIMIT: u64 = 512 << 20;
+/// How many bytes phase A reads from a file of this family before it stops
+/// sampling. `None` means the family's extractor caps itself.
+///
+/// Split out of `scan_file` so tests can shrink it. The consequence of a
+/// GROUPED family hitting this cap is not "a slightly thinner sample", it is a
+/// whole `unity_class`/SQL table with no dataset to route to in phase B — and
+/// the only fixture that reaches it naturally is a half-gigabyte file, which
+/// is why that path shipped untested. `SampleLimitOverride` gives the suite a
+/// fixture it can afford.
+fn sample_limit_bytes(family: Family, path: &Path) -> Option<u64> {
+ // Only the test override reads the path; the shipped caps are per-family.
+ #[cfg(not(test))]
+ let _ = path;
+ #[cfg(test)]
+ {
+ // Scoped to ONE corpus root, not process-global. `cargo test` runs this
+ // binary multi-threaded, phase A itself runs on `crate::pool`, and a
+ // bare global would silently re-cap sampling for every unrelated test
+ // that happened to overlap — a flake that would look like anything but
+ // its cause.
+ if let Some((root, bytes)) = SAMPLE_LIMIT_OVERRIDE
+ .lock()
+ .unwrap_or_else(|p| p.into_inner())
+ .as_ref()
+ {
+ if path.starts_with(root) {
+ return Some(*bytes);
+ }
+ }
+ }
+ match family {
+ Family::SqlDump => Some(SQLDUMP_SAMPLE_LIMIT),
+ Family::UnityYaml => Some(UNITY_SAMPLE_LIMIT),
+ Family::Jsonl | Family::Logs | Family::Csv | Family::TxtLines => Some(SAMPLE_LIMIT_BYTES),
+ Family::Sqlite => Some(1), // signals per-table row cap inside the extractor
+ _ => None, // whole-file extractors cap themselves
+ }
+}
+
+/// Test-only phase-A byte cap: `(corpus root, bytes)`, applied only to paths
+/// under that root. `None` = off.
+#[cfg(test)]
+static SAMPLE_LIMIT_OVERRIDE: Mutex<Option<(std::path::PathBuf, u64)>> = Mutex::new(None);
+
+/// Caps phase-A sampling under `root` for the lifetime of the guard.
+#[cfg(test)]
+pub(crate) struct SampleLimitOverride;
+
+#[cfg(test)]
+impl SampleLimitOverride {
+ pub(crate) fn set(root: &Path, bytes: u64) -> Self {
+ *SAMPLE_LIMIT_OVERRIDE
+ .lock()
+ .unwrap_or_else(|p| p.into_inner()) = Some((root.to_owned(), bytes));
+ Self
+ }
+}
+
+#[cfg(test)]
+impl Drop for SampleLimitOverride {
+ fn drop(&mut self) {
+ *SAMPLE_LIMIT_OVERRIDE
+ .lock()
+ .unwrap_or_else(|p| p.into_inner()) = None;
+ }
+}
+
#[cfg(test)]
static REPLACEMENT_FAILPOINT: std::sync::atomic::AtomicU8 = std::sync::atomic::AtomicU8::new(0);
@@ -1086,13 +1182,7 @@ fn scan_file(
out.sniffed = Some(sn);
return out;
}
- let limit = match sn.family {
- Family::SqlDump => Some(SQLDUMP_SAMPLE_LIMIT),
- Family::UnityYaml => Some(UNITY_SAMPLE_LIMIT),
- Family::Jsonl | Family::Logs | Family::Csv | Family::TxtLines => Some(SAMPLE_LIMIT_BYTES),
- Family::Sqlite => Some(1), // signals per-table row cap inside the extractor
- _ => None, // whole-file extractors cap themselves
- };
+ let limit = sample_limit_bytes(sn.family, path);
type GroupAcc = (
HashMap<String, infer::FieldAcc>,
u64,
@@ -4479,17 +4569,26 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
} else {
String::new()
};
- extra_junk.lock().unwrap().push(JunkFile {
- file_key: key.clone(),
- rel: f.rel.clone(),
- format: format_str(Some(&sn)),
- status: "junk".into(),
- reason: format!(
- "{total} record(s) dropped: group(s) never sampled in \
- phase A, so they have no dataset — {named}{suffix}"
- ),
- bytes: f.size,
- });
+ // A NOTE, never an `extra_junk` entry. This file
+ // still reaches `journal.file_done` — `send_err`
+ // is untouched — and every junk entry is turned
+ // into a catalog document under the same
+ // `file:{file_key}` id as that completion
+ // (`catalog::file_doc`, `catalog.rs`). Pushing one
+ // here put two documents with one id in the same
+ // bulk, and the later one wins: a file that
+ // indexed N records was reported as status "junk"
+ // with records 0, `junk_file_count` counted it
+ // twice, and `code_coverage.observe` ran for it
+ // twice. The dropped records are already carried
+ // on that file's own completion as `junk`
+ // (`file_junk` below), which is where a per-file
+ // drop count belongs.
+ pr.note(&format!(
+ "{}: {total} record(s) dropped: group(s) never sampled in \
+ phase A, so they have no dataset — {named}{suffix}",
+ f.rel
+ ));
}
}
// Raw-source href pass: the HTML extractor strips markup
@@ -5062,9 +5161,11 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// the journal's completions are the corpus, not just this invocation's
// slice, so a resume reports what is live rather than what it re-parsed.
let mut code_coverage = CodeCoverage::default();
+ let mut completed_keys: HashSet<String> = HashSet::new();
{
let j = journal_mx.lock().unwrap();
for fd in j.done.values() {
+ completed_keys.insert(fd.file_key.clone());
let current_path = plan
.files
.get(&fd.file_key)
@@ -5104,13 +5205,43 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
.collect();
all_junk.extend(extra.iter());
all_junk.extend(new_unplanned.iter());
+ // Disjointness from the journal completions above is ENFORCED here, not
+ // merely asserted in a comment: `catalog::file_doc` derives its `_id` from
+ // the file key alone, so a junk entry for a file that also reached
+ // `file_done` is a second document with an id the completion already used
+ // — and the bulk applies them in order, so the junk one wins. A file that
+ // indexed N records would be reported in the catalog as status "junk" with
+ // records 0. See `shadowed_junk_entries`.
+ let shadowed = shadowed_junk_entries(&all_junk, &completed_keys);
+ // Debug builds stop on the producer's defect; release builds keep the
+ // truthful document and say what they dropped.
+ debug_assert!(
+ shadowed.is_empty(),
+ "junk entry for a file that reached file_done: {:?}",
+ shadowed.iter().map(|jf| &jf.rel).collect::<Vec<_>>()
+ );
+ if !shadowed.is_empty() {
+ // Not fatal: the completion document is the correct one and it is
+ // already staged, so dropping the junk entry restores the truth. It is
+ // still a defect in whichever producer emitted it, so say so.
+ pr.note(&format!(
+ "internal: {} junk entr(ies) named a file that also completed and \
+ were dropped so the catalog keeps the indexed record: {}",
+ shadowed.len(),
+ shadowed
+ .iter()
+ .take(REFUSAL_LIST_CAP)
+ .map(|jf| jf.rel.as_str())
+ .collect::<Vec<_>>()
+ .join(", ")
+ ));
+ all_junk.retain(|jf| !completed_keys.contains(&jf.file_key));
+ }
// Counted now: every later reader of this number outlives the borrows
// `all_junk` holds on `plan` and `new_unplanned`, which the durable
// junk-plan update below mutates.
let junk_file_count = all_junk.len();
for jf in &all_junk {
- // Disjoint from the journal completions above: a file that reached
- // `file_done` never appears here, so nothing is counted twice.
code_coverage.observe(&jf.format, 0);
let (id, doc) = catalog::file_doc(
&jf.file_key,
diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rs
index 9dc937816..66d87cc7e 100644
--- a/engine/crates/xerj-autoindex/src/sniff.rs
+++ b/engine/crates/xerj-autoindex/src/sniff.rs
@@ -182,29 +182,62 @@ fn sniff_bytes(
// to something printable), and a multi-MB PSD misread as prose costs
// ~100x its size in RAM through sectioning. Magic bytes are the reliable
// signal.
- for (magic, kind) in [
- (&b"\x89PNG"[..], "png"),
- (&b"GIF8"[..], "gif"),
- (&b"\xff\xd8\xff"[..], "jpeg"),
- (&b"\x7fELF"[..], "elf"),
- (&b"BM"[..], "bmp"),
- (&b"\x00\x00\x01\x00"[..], "ico"),
- (&b"8BPS"[..], "psd"),
- (&b"II*\x00"[..], "tiff"),
- (&b"MM\x00*"[..], "tiff"),
- (&b"RIFF"[..], "riff"),
- (&b"OggS"[..], "ogg"),
- (&b"fLaC"[..], "flac"),
- (&b"ID3"[..], "mp3"),
- (&b"Kaydara FBX Binary"[..], "fbx"),
- (&b"\x76\x2f\x31\x01"[..], "exr"),
+ //
+ // Every signature whose bytes are PRINTABLE ASCII carries a `qualify`
+ // check as well, because "starts with these four letters" is also true of
+ // ordinary text: a CSV whose first column header is `ID3` and prose whose
+ // first word is `RIFF`, `OggS`, `fLaC` or `8BPS` were all junked as media
+ // by the unqualified table (measured — see the tests below). This is the
+ // rule Lucene applies to its own containers: `CodecUtil.checkHeader`
+ // (`lucene/core/src/java/org/apache/lucene/codecs/CodecUtil.java:183-201`,
+ // Apache-2.0) matches `CODEC_MAGIC` and then hands straight to
+ // `checkHeaderNoMagic` (`CodecUtil.java:202-246`), which refuses the file
+ // unless the codec name AND a version inside an accepted range follow —
+ // the magic alone is never taken as proof. Adapted, not copied: Lucene is
+ // validating files it wrote itself, this is declining to delete somebody
+ // else's prose.
+ for (magic, kind, qualify) in [
+ (&b"\x89PNG"[..], "png", accept as fn(&[u8]) -> bool),
+ (&b"GIF8"[..], "gif", accept as fn(&[u8]) -> bool),
+ (&b"\xff\xd8\xff"[..], "jpeg", accept as fn(&[u8]) -> bool),
+ (&b"\x7fELF"[..], "elf", accept as fn(&[u8]) -> bool),
+ (&b"BM"[..], "bmp", accept as fn(&[u8]) -> bool),
+ (&b"\x00\x00\x01\x00"[..], "ico", accept as fn(&[u8]) -> bool),
+ (&b"8BPS"[..], "psd", psd_version as fn(&[u8]) -> bool),
+ (&b"II*\x00"[..], "tiff", accept as fn(&[u8]) -> bool),
+ (&b"MM\x00*"[..], "tiff", accept as fn(&[u8]) -> bool),
+ (&b"RIFF"[..], "riff", riff_form as fn(&[u8]) -> bool),
+ (&b"OggS"[..], "ogg", ogg_page as fn(&[u8]) -> bool),
+ (
+ &b"fLaC"[..],
+ "flac",
+ flac_metadata_block as fn(&[u8]) -> bool,
+ ),
+ (&b"ID3"[..], "mp3", id3v2_header as fn(&[u8]) -> bool),
+ (
+ &b"Kaydara FBX Binary"[..],
+ "fbx",
+ accept as fn(&[u8]) -> bool,
+ ),
+ (&b"\x76\x2f\x31\x01"[..], "exr", accept as fn(&[u8]) -> bool),
] {
- if prefix.starts_with(magic) {
+ if prefix.starts_with(magic) && qualify(prefix) {
let mut s = mk(Family::Binary);
s.binary_kind = Some(kind.into());
return Ok(s);
}
}
+ // Truevision TGA has NO magic number — the file opens straight into its
+ // 18-byte header — so a raw texture is the one image format that reaches
+ // the text heuristics on its own bytes. Its header is nevertheless highly
+ // constrained, which is what makes this safe: byte 1 must be 0 or 1 and
+ // byte 2 one of six small values, i.e. the second and third bytes of the
+ // file must BOTH be control characters, which text is not.
+ if looks_like_tga_header(prefix) {
+ let mut s = mk(Family::Binary);
+ s.binary_kind = Some("tga".into());
+ return Ok(s);
+ }
// 2. Binary vs text: decode UTF-8, fall back windows-1252.
let (text, encoding) = decode(prefix);
@@ -224,26 +257,34 @@ fn sniff_bytes(
s.binary_kind = Some("unknown".into());
return Ok(s);
}
- // A prefix that only decoded via LOSSY windows-1252 and is majority
- // high-byte is pixel/float soup, not prose in a legacy encoding: real
- // windows-1252 text (accented European prose) runs well under 30%
- // non-ASCII, while raw image channels run ~50%. Without this, a large
- // TGA classified as txt-prose is amplified ~100x in RAM by sectioning.
+ // NOTE — a "decoded via lossy windows-1252 AND over 30% non-ASCII is
+ // pixel soup" guard was here, to catch a raw TGA. It is gone, and must
+ // not come back in that form: the windows-1252 fallback is what EVERY
+ // legacy 8-bit codepage and every legacy CJK encoding decodes through, so
+ // the test was not "is this an image", it was "is this written in a script
+ // that is not Latin". Measured through `sniff()` against `ca4d75a` with
+ // identical fixtures, it changed windows-1251 Russian, KOI8-R Russian,
+ // windows-1253 Greek, windows-1255 Hebrew and windows-1256 Arabic prose
+ // from `TxtProse` to `Binary`, and `scan_file` turns `Binary` into
+ // "junk: binary content (unknown)" — the file is never indexed and the
+ // report says only that something unreadable was skipped.
//
- // Gated on the windows-1252 fallback SPECIFICALLY, and skipped when the
- // bytes are plausibly legacy CJK: Shift-JIS/GBK/Big5/EUC-KR prose is
- // ~100% high-byte, so the unqualified test junked every legacy-encoded
- // CJK document in the corpus.
- let legacy_cjk = encoding == WINDOWS_1252_LOSSY && looks_like_legacy_cjk(prefix);
- if encoding == WINDOWS_1252_LOSSY && !legacy_cjk {
- let total = text.chars().count().max(1);
- let non_ascii = text.chars().filter(|c| !c.is_ascii()).count();
- if non_ascii * 10 > total * 3 {
- let mut s = mk(Family::Binary);
- s.binary_kind = Some("unknown".into());
- return Ok(s);
- }
- }
+ // A `looks_like_legacy_cjk` escape hatch was tried and could not carry the
+ // weight, for two reasons worth recording so it is not retried:
+ // * single-byte codepages never form valid SHIFT_JIS/GBK/BIG5/EUC_KR
+ // double-byte pairs, so Cyrillic/Greek/Hebrew/Arabic were never
+ // rescued by it at all;
+ // * it required a LOSSLESS trial decode, and `sniff()` only ever sees
+ // `read_prefix(path, gzip, 8192)`. A double-byte character straddling
+ // that cut makes every trial decode report `had_errors`, so the same
+ // Shift-JIS document was text or binary depending on its byte length
+ // — measured `TxtProse` at pad 0 and 2, `Binary` at pad 1 and 3.
+ //
+ // TGA is now caught by its header above, which is evidence of pixel data
+ // rather than evidence of a non-Latin script. Headerless raw payloads
+ // (`.raw`, `.bytes`, uncompressed PCM) still classify as text, exactly as
+ // they do on `main`; bounding sectioning memory is the fix for that, not
+ // byte statistics that cannot tell a texture from Cyrillic.
// 2b. Unity text serialization, detected by content (never extension):
// scenes/prefabs/assets open with `%YAML` and declare the Unity tag
@@ -319,89 +360,91 @@ fn sniff_bytes(
/// the bytes are text — the binary heuristics key off this exact string.
pub const WINDOWS_1252_LOSSY: &str = "windows-1252 (lossy)";
-/// Scripts written without spaces between words ("scriptio continua").
-///
-/// Whitespace density cannot be used to tell prose from byte soup for these:
-/// Chinese, Japanese, Korean, Thai, Lao, Khmer and Burmese prose runs at
-/// essentially 0% intra-line whitespace, the same as random bytes. Their
-/// presence is instead POSITIVE evidence of text — neither pixel channels nor
-/// float dumps produce ideographs.
-///
-/// The script set is the one Lucene singles out for exactly this reason:
-/// `CJKBigramFilter` (`lucene/analysis/common/src/java/org/apache/lucene/
-/// analysis/cjk/CJKBigramFilter.java:52-118`, Apache-2.0) defines `HAN`,
-/// `HIRAGANA`, `KATAKANA` and `HANGUL` flags because a whitespace tokenizer
-/// cannot segment them, and its default constructor
-/// (`CJKBigramFilter.java:119-126`) enables all four together. Thai, Lao,
-/// Khmer and Myanmar are added here on the same principle — Lucene reaches
-/// for `ICUTokenizer`/dictionary breaking for those rather than a bigram
-/// filter, but the property this function tests (no inter-word spaces) is
-/// identical. Adapted, not copied: Lucene is deciding how to TOKENIZE known
-/// text; this only decides whether the bytes are text at all.
-fn is_scriptio_continua(c: char) -> bool {
- matches!(c as u32,
- 0x1100..=0x11FF // Hangul Jamo
- | 0x0E00..=0x0EFF // Thai, Lao
- | 0x1000..=0x109F // Myanmar
- | 0x1780..=0x17FF // Khmer
- | 0x3000..=0x303F // CJK symbols and punctuation (、。「」)
- | 0x3040..=0x30FF // Hiragana, Katakana
- | 0x3400..=0x4DBF // CJK Unified Ideographs Extension A
- | 0x4E00..=0x9FFF // CJK Unified Ideographs
- | 0xAC00..=0xD7AF // Hangul Syllables
- | 0xF900..=0xFAFF // CJK Compatibility Ideographs
- | 0xFF00..=0xFFEF // Halfwidth and Fullwidth Forms
- | 0x20000..=0x2A6DF // CJK Unified Ideographs Extension B
- )
+/// Magic-byte signature that needs no structural confirmation: its bytes are
+/// not printable ASCII, so no text file can begin with them by accident.
+fn accept(_prefix: &[u8]) -> bool {
+ true
}
-/// Share of `text` that is scriptio-continua, in percent.
-fn continua_percent(text: &str) -> usize {
- let mut total = 0usize;
- let mut hits = 0usize;
- for c in text.chars() {
- total += 1;
- if is_scriptio_continua(c) {
- hits += 1;
- }
- }
- if total == 0 {
- return 0;
- }
- hits * 100 / total
+/// Adobe Photoshop: `8BPS` is followed by a big-endian version, 1 for PSD and
+/// 2 for PSB. Nothing else is defined, so a text file whose first word is
+/// `8BPS` fails here on its fifth byte.
+fn psd_version(prefix: &[u8]) -> bool {
+ prefix.len() >= 6 && prefix[4] == 0 && matches!(prefix[5], 1 | 2)
+}
+
+/// RIFF container: `RIFF`, a little-endian chunk size, then a four-character
+/// FORM type naming what the container actually holds. Requiring a known FORM
+/// is what separates a WAV from a sentence that opens with the word "RIFF".
+fn riff_form(prefix: &[u8]) -> bool {
+ prefix.len() >= 12
+ && matches!(
+ &prefix[8..12],
+ b"WAVE" | b"AVI " | b"WEBP" | b"RMID" | b"ANI " | b"CDDA" | b"PAL " | b"RDIB"
+ )
}
-/// Whether `bytes` are plausibly text in a legacy CJK encoding.
+/// Ogg page header: `OggS` then the stream structure version, which has been
+/// 0 for the life of the format (RFC 3533 §6.1).
+fn ogg_page(prefix: &[u8]) -> bool {
+ prefix.len() >= 5 && prefix[4] == 0
+}
+
+/// FLAC stream: `fLaC` then a METADATA_BLOCK_HEADER whose block type must be
+/// STREAMINFO (0) for the first block; the top bit is the last-block flag.
+fn flac_metadata_block(prefix: &[u8]) -> bool {
+ prefix.len() >= 5 && prefix[4] & 0x7f == 0
+}
+
+/// ID3v2 tag header: `ID3`, a major version (2, 3 and 4 are the versions that
+/// exist), a revision that is not the reserved 0xFF, a flags byte, and a
+/// four-byte SYNCHSAFE size whose bytes each have the top bit clear.
///
-/// This is a TEXT-NESS test and nothing more. It deliberately does NOT return
-/// which encoding: Shift-JIS, GBK, Big5 and EUC-KR share most of the same
-/// valid double-byte space, so a GBK document also decodes losslessly as
-/// Shift-JIS (and vice versa) with a high ideograph share either way. Telling
-/// them apart needs a statistical language model — what `chardet`/ICU do —
-/// which this crate does not have, and guessing would silently replace a
-/// Chinese document with plausible-looking Japanese mojibake. Tracked as a
-/// follow-up; see CHANGELOG.
+/// Unqualified, this signature is three ASCII letters: a CSV whose first
+/// column header is `ID3` classified `Csv` on `ca4d75a` and `Binary` on this
+/// branch until the version byte was checked.
+fn id3v2_header(prefix: &[u8]) -> bool {
+ prefix.len() >= 10
+ && matches!(prefix[3], 2..=4)
+ && prefix[4] != 0xff
+ && prefix[6..10].iter().all(|b| *b < 0x80)
+}
+
+/// Truevision TGA, which has no magic number at all — every byte of its
+/// 18-byte header is data. Detection is therefore by CONSTRAINT: an image type
+/// from the defined set, a pixel depth from the defined set, a colour-map
+/// descriptor that must be all-zero when there is no colour map, and non-zero
+/// dimensions.
///
-/// What it IS sound for is the only question asked here: are these bytes a
-/// pixel/float payload, or somebody's prose? The pixel-soup guards below key
-/// off high-byte density and whitespace density, and legacy CJK prose looks
-/// exactly like soup under both — ~100% high-byte and ~0% whitespace — so
-/// without this they junk the file outright. Requiring a LOSSLESS decode
-/// (one undecodable pair disqualifies it) plus a substantially ideographic
-/// result keeps European windows-1252 prose out: its high bytes sit isolated
-/// among ASCII, so most form no valid double-byte pair at all.
-fn looks_like_legacy_cjk(bytes: &[u8]) -> bool {
- [
- encoding_rs::SHIFT_JIS,
- encoding_rs::GBK,
- encoding_rs::BIG5,
- encoding_rs::EUC_KR,
- ]
- .into_iter()
- .any(|enc| {
- let (s, _, had_errors) = enc.decode(bytes);
- !had_errors && continua_percent(&s) >= 30
- })
+/// The reason this is safe on text is byte 1 and byte 2: `color_map_type` is 0
+/// or 1 and `image_type` is one of six values below 12, so both the second and
+/// the third byte of the file have to be control characters. Prose, CSV, JSON
+/// and source code fail on byte 1.
+fn looks_like_tga_header(prefix: &[u8]) -> bool {
+ if prefix.len() < 18 {
+ return false;
+ }
+ let color_map_type = prefix[1];
+ if color_map_type > 1 {
+ return false;
+ }
+ // 1/2/3 uncompressed colour-mapped/true-colour/greyscale, 9/10/11 the
+ // run-length encoded counterparts. Type 0 ("no image data") is excluded
+ // deliberately: it carries no pixels, so nothing is lost by letting such a
+ // file fall through to the text heuristics, and admitting it would make an
+ // all-zero prefix a TGA.
+ if !matches!(prefix[2], 1 | 2 | 3 | 9 | 10 | 11) {
+ return false;
+ }
+ if !matches!(prefix[16], 8 | 15 | 16 | 24 | 32) {
+ return false;
+ }
+ if color_map_type == 0 && prefix[3..8].iter().any(|b| *b != 0) {
+ return false;
+ }
+ let width = u16::from_le_bytes([prefix[12], prefix[13]]);
+ let height = u16::from_le_bytes([prefix[14], prefix[15]]);
+ width > 0 && height > 0
}
fn decode(bytes: &[u8]) -> (String, &'static str) {
@@ -564,10 +607,14 @@ fn txt_kind(nonblank: &[&str]) -> Family {
// binary and the run exits 3 instead of 0 — the false positive is
// reachable from this repo's own suite.
//
- // High-byte payloads (which is what a real TGA is) are still caught, by
- // the windows-1252 non-ASCII test in `sniff_bytes`. Bounding sectioning
- // memory is the right fix for the RAM concern; silently deleting files
- // that look unusual is not. Tracked as a follow-up.
+ // A real TGA is caught by `looks_like_tga_header` in `sniff_bytes`, on the
+ // structure of its 18-byte header. The byte-statistics version of that
+ // check ("mostly non-ASCII under the windows-1252 fallback") was tried and
+ // removed in the same breath as this one and for the same reason: it
+ // junked Cyrillic, Greek, Hebrew, Arabic and mixed Japanese prose.
+ // Bounding sectioning memory is the right fix for the RAM concern;
+ // silently deleting files that look unusual is not. Tracked as a
+ // follow-up.
let avg_len = nonblank.iter().map(|l| l.len()).sum::<usize>() as f64 / nonblank.len() as f64;
if avg_len > 60.0 {
return Family::TxtProse;
@@ -805,6 +852,8 @@ mod unity_sniff_tests {
("t.tif2", &b"MM\x00*\x00\x08"[..]),
("t.wav", &b"RIFF\x24\x08\x00\x00WAVE"[..]),
("t.ogg", &b"OggS\x00\x02"[..]),
+ ("t.flac", &b"fLaC\x00\x00\x00\x22"[..]),
+ ("t.mp3", &b"ID3\x03\x00\x00\x00\x00\x0f\x76"[..]),
("t.fbx", &b"Kaydara FBX Binary \x00"[..]),
] {
let mut bytes = head.to_vec();
@@ -815,6 +864,99 @@ mod unity_sniff_tests {
}
}
+ /// Regression (round 2): the printable-ASCII signatures added by this
+ /// branch matched TEXT. Measured through `sniff()` on `ca4d75a` vs this
+ /// branch with identical fixtures, a CSV whose first column header is
+ /// `ID3` went `Csv` -> `Binary`, and prose whose first word is `RIFF`,
+ /// `OggS`, `fLaC` or `8BPS` went `TxtProse` -> `Binary`. `scan_file` turns
+ /// `Family::Binary` into "junk: binary content (unknown)", so each of
+ /// those files stopped being indexed at all.
+ ///
+ /// The true positives above must keep passing, which is the whole point:
+ /// the signature is necessary, it is just not sufficient.
+ #[test]
+ fn a_printable_ascii_signature_alone_does_not_make_a_file_binary() {
+ let csv = b"ID3,name,value\n1,alpha,2\n3,beta,4\n5,gamma,6\n7,delta,8\n";
+ assert_eq!(
+ sniff_bytes(csv, Path::new("t.csv"), Path::new("t.csv"), false)
+ .unwrap()
+ .family,
+ Family::Csv,
+ "a column header may legitimately be the three letters ID3"
+ );
+ for (label, opener) in [
+ (
+ "riff",
+ "RIFF is a container format used by WAV files. It stores chunks. ",
+ ),
+ (
+ "oggs",
+ "OggS pages carry the packets of an Ogg stream in order. ",
+ ),
+ (
+ "flac",
+ "fLaC is the four byte magic of a FLAC audio stream header. ",
+ ),
+ (
+ "psd",
+ "8BPS is the magic of an Adobe Photoshop document header. ",
+ ),
+ ] {
+ let text = opener.repeat(30);
+ let sn = sniff_bytes(
+ text.as_bytes(),
+ Path::new("notes.txt"),
+ Path::new("notes.txt"),
+ false,
+ )
+ .unwrap();
+ assert_ne!(
+ sn.family,
+ Family::Binary,
+ "{label}: prose that opens with the signature word is still prose"
+ );
+ }
+ }
+
+ /// TGA is the format the removed byte-statistics guard existed for: it has
+ /// no magic number, so a raw texture reaches the text heuristics on its own
+ /// bytes. It is caught on the structure of its 18-byte header instead.
+ #[test]
+ fn a_raw_tga_texture_is_binary_by_header_structure() {
+ // id_len=0, no colour map, uncompressed true-colour, 64x64, 24bpp.
+ let mut tga: Vec<u8> = vec![0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 0, 64, 0, 24, 0];
+ tga.extend((0..8192u32).map(|i| (0x80 + (i * 37) % 0x7f) as u8));
+ let sn = sniff_bytes(
+ &tga,
+ Path::new("texture.tga"),
+ Path::new("texture.tga"),
+ false,
+ )
+ .unwrap();
+ assert_eq!(sn.family, Family::Binary, "a real TGA header must be seen");
+ assert_eq!(sn.binary_kind.as_deref(), Some("tga"));
+
+ // And the constraint must be tight enough that text never satisfies
+ // it. Every one of these is >= 18 bytes, so length is not what saves
+ // them.
+ for (label, text) in [
+ (
+ "prose",
+ "The quick brown fox jumps over the lazy dog again.",
+ ),
+ ("csv", "id,name,value\n1,alpha,2\n3,beta,4\n"),
+ ("json", "{\"id\": 1, \"name\": \"alpha\", \"value\": 2}"),
+ ("code", "fn main() { println!(\"hello, world\"); }"),
+ ("yaml", "name: alpha\nvalue: 2\nnested:\n a: 1\n"),
+ ] {
+ assert!(text.len() >= 18, "{label}: fixture too short to prove it");
+ assert!(
+ !looks_like_tga_header(text.as_bytes()),
+ "{label}: text must not satisfy the TGA header constraints"
+ );
+ }
+ }
+
/// Whitespace density must NOT be used to junk text. This pins the
/// counter-examples that make the rejected guard unsafe: a long run of one
/// ASCII letter (the shape of this repo's own 65,537-byte resume-key
@@ -902,33 +1044,65 @@ mod unity_sniff_tests {
}
}
- /// Regression: the sibling guard keyed off `encoding != "utf-8"`, and a
- /// legacy CJK encoding decodes as lossy windows-1252 into ~100%
- /// high-byte mojibake — so every Shift-JIS/GBK/Big5/EUC-KR document was
- /// junked before any family heuristic ran.
+ /// Classify a byte buffer through the REAL read path: a file on disk, read
+ /// by `sniff()`, which sees only `read_prefix(path, gzip, 8192)`.
+ ///
+ /// Every legacy-encoding regression below has to go through this and not
+ /// through `sniff_bytes` with the whole buffer. The escape hatch that was
+ /// supposed to protect legacy CJK required a LOSSLESS trial decode, and a
+ /// double-byte character straddling the 8192-byte cut makes every trial
+ /// decode fail — so a test that hands over the complete buffer cannot
+ /// observe half of the bug it is written for.
+ fn sniff_file(bytes: &[u8], name: &str) -> Family {
+ let dir = tempfile::tempdir().unwrap();
+ let path = dir.path().join(name);
+ std::fs::write(&path, bytes).unwrap();
+ sniff(&path).unwrap().family
+ }
+
+ /// Regression (round 2): a guard on "lossy windows-1252 AND over 30%
+ /// non-ASCII" is not a test for pixel data, it is a test for "not written
+ /// in Latin script". windows-1252 is the fallback EVERY legacy 8-bit
+ /// codepage decodes through.
+ ///
+ /// Measured through `sniff()` with these exact fixtures: on `ca4d75a` all
+ /// five are `TxtProse`; with the guard present all five are `Binary`, and
+ /// `scan_file` turns that into "junk: binary content (unknown)".
///
- /// The assertion is deliberately only "not binary". This crate cannot
- /// tell these encodings apart (they share most of their valid
- /// double-byte space) and does not pretend to: the file is still decoded
- /// as windows-1252, exactly as it was before this branch, which is
- /// mojibake — but mojibake that is INDEXED rather than deleted.
+ /// The assertion is deliberately only "not binary" — this crate still
+ /// decodes the bytes as windows-1252 and still produces mojibake, because
+ /// telling legacy codepages apart needs a statistical language model it
+ /// does not have. Mojibake that is INDEXED beats prose that is deleted.
#[test]
- fn legacy_encoded_cjk_prose_is_not_junked_as_binary() {
+ fn legacy_single_byte_prose_is_not_junked_by_the_real_read_path() {
for (label, enc, unit) in [
(
- "shift_jis",
- encoding_rs::SHIFT_JIS,
- "この文書はシステムの設計について説明します。",
+ "windows-1251-ru",
+ encoding_rs::WINDOWS_1251,
+ "Этот документ описывает архитектуру системы и детали реализации. ",
),
- ("gbk", encoding_rs::GBK, "本文档描述了系统的架构设计。"),
- ("big5", encoding_rs::BIG5, "本文件描述系統的架構設計。"),
(
- "euc-kr",
- encoding_rs::EUC_KR,
- "이문서는시스템설계를설명합니다",
+ "koi8-r-ru",
+ encoding_rs::KOI8_R,
+ "Этот документ описывает архитектуру системы и детали реализации. ",
+ ),
+ (
+ "windows-1253-el",
+ encoding_rs::WINDOWS_1253,
+ "Το έγγραφο αυτό περιγράφει την αρχιτεκτονική του συστήματος. ",
+ ),
+ (
+ "windows-1255-he",
+ encoding_rs::WINDOWS_1255,
+ "מסמך זה מתאר את ארכיטקטורת המערכת ואת פרטי היישום שלה. ",
+ ),
+ (
+ "windows-1256-ar",
+ encoding_rs::WINDOWS_1256,
+ "تصف هذه الوثيقة بنية النظام وتفاصيل تنفيذه بشكل كامل. ",
),
] {
- let text = unit.repeat(4096 / unit.chars().count() + 2);
+ let text = unit.repeat(200);
let (bytes, _, had_errors) = enc.encode(&text);
assert!(!had_errors, "{label}: fixture must encode cleanly");
assert!(
@@ -936,52 +1110,120 @@ mod unity_sniff_tests {
"{label}: fixture must not be valid UTF-8, or it proves nothing"
);
assert!(
- looks_like_legacy_cjk(&bytes),
- "{label}: text-ness probe must recognise this as prose"
+ bytes.len() > 8192,
+ "{label}: fixture must exceed the sniff prefix"
+ );
+ assert_ne!(
+ sniff_file(&bytes, "doc.txt"),
+ Family::Binary,
+ "{label}: legacy-encoded prose must not be junked as binary"
);
- let sn =
- sniff_bytes(&bytes, Path::new("doc.txt"), Path::new("doc.txt"), false).unwrap();
- assert_ne!(sn.family, Family::Binary, "{label}: must not be junked");
}
}
- /// The legacy-CJK probe must not capture European text — that would
- /// replace real windows-1252 prose with plausible-looking ideographs.
+ /// Regression (round 2): the legacy-CJK escape hatch was alignment
+ /// dependent. `sniff()` classifies from the first 8192 bytes only, so a
+ /// double-byte character split by that cut made the trial decode report
+ /// `had_errors` and the rescue declined. Measured on the branch that
+ /// carried it: the SAME Shift-JIS document was `TxtProse` at byte pads 0
+ /// and 2 and `Binary` at pads 1 and 3 — a coin flip per file.
+ ///
+ /// Sweeping the alignment is the point of this test; a single-length
+ /// fixture passes half the time by luck.
#[test]
- fn european_windows_1252_prose_is_not_claimed_as_cjk() {
+ fn legacy_cjk_prose_survives_every_prefix_alignment() {
+ for (label, enc, unit) in [
+ (
+ "shift_jis",
+ encoding_rs::SHIFT_JIS,
+ "この文書はシステムの設計と実装について詳しく説明します。",
+ ),
+ (
+ "gbk",
+ encoding_rs::GBK,
+ "本文档描述了系统的架构设计与实现。",
+ ),
+ (
+ "big5",
+ encoding_rs::BIG5,
+ "本文件描述系統的架構設計與實作。",
+ ),
+ (
+ "euc-kr",
+ encoding_rs::EUC_KR,
+ "이문서는시스템의설계와구현을설명합니다",
+ ),
+ ] {
+ let text = unit.repeat(400);
+ let (body, _, had_errors) = enc.encode(&text);
+ assert!(!had_errors, "{label}: fixture must encode cleanly");
+ assert!(
+ body.len() > 8192,
+ "{label}: fixture must exceed the sniff prefix, or alignment \
+ cannot matter"
+ );
+ for pad in 0..4usize {
+ let mut bytes = vec![b'a'; pad];
+ bytes.extend_from_slice(&body);
+ assert_ne!(
+ sniff_file(&bytes, "doc.txt"),
+ Family::Binary,
+ "{label}: junked at byte-offset pad {pad}"
+ );
+ }
+ }
+ }
+
+ /// Regression (round 2): a realistic Japanese technical document is not
+ /// 100% ideographs — it is prose around ASCII code fences, identifiers and
+ /// URLs. That shape sat under the 30% scriptio-continua threshold the
+ /// rescue needed while sitting over the 30% non-ASCII threshold the guard
+ /// junked on, so the ordinary shape of real CJK documentation was the case
+ /// that lost. Measured: `TxtLines` on `ca4d75a`, `Binary` on the branch,
+ /// at both 13 KB and 67 KB.
+ #[test]
+ fn mixed_script_documentation_is_not_junked() {
+ let unit = "この関数は入力データを検証してから処理を実行し、結果を呼び出し元に返します。\n\
+ エラーが発生した場合は、詳細な理由を含む診断情報を記録します。\n\
+ ```\nfn process(d: &[u8]) -> Result<Vec<u8>, ProcessError> {\n \
+ let checked = validate(d)?;\n Ok(checked.to_vec())\n}\n```\n\
+ See docs/process.md and the API reference for the full parameter list.\n";
+ let ideographic =
+ unit.chars().filter(|c| !c.is_ascii()).count() * 100 / unit.chars().count().max(1);
+ assert!(
+ (20..30).contains(&ideographic),
+ "fixture must sit in the band that loses — between the rescue's \
+ 30% ideograph floor and the guard's 30% non-ASCII ceiling; got \
+ {ideographic}%"
+ );
+ for reps in [40usize, 200] {
+ let doc = unit.repeat(reps);
+ let (bytes, _, _) = encoding_rs::SHIFT_JIS.encode(&doc);
+ assert_ne!(
+ sniff_file(&bytes, "doc.txt"),
+ Family::Binary,
+ "mixed Japanese/ASCII documentation ({} bytes) must not be junked",
+ bytes.len()
+ );
+ }
+ }
+
+ /// Accented European prose is the case the removed guard claimed to be
+ /// safe for. It must stay safe now that the guard is gone.
+ #[test]
+ fn european_windows_1252_prose_is_still_prose() {
let text = "Le système décrit ici gère les données réservées à \
l'opérateur, prêtes à être exportées. ";
- let long = text.repeat(80);
+ let long = text.repeat(200);
let (bytes, _, _) = encoding_rs::WINDOWS_1252.encode(&long);
assert!(std::str::from_utf8(&bytes).is_err());
let (_, enc) = decode(&bytes);
assert_eq!(enc, WINDOWS_1252_LOSSY, "must stay windows-1252");
- assert!(
- !looks_like_legacy_cjk(&bytes),
- "accented Latin must not be claimed as CJK"
+ assert_ne!(
+ sniff_file(&bytes, "note.txt"),
+ Family::Binary,
+ "accented prose is still prose"
);
- let sn = sniff_bytes(&bytes, Path::new("note.txt"), Path::new("note.txt"), false).unwrap();
- assert_ne!(sn.family, Family::Binary, "accented prose is still prose");
- }
-
- /// The pixel-soup protection the guards exist for must survive the fix:
- /// high-byte float/pixel payloads still have to be caught.
- #[test]
- fn high_byte_pixel_soup_is_still_binary() {
- // Bytes that are not valid UTF-8, not valid in any legacy CJK
- // encoding as a whole, and majority high-byte — a raw texture.
- let soup: Vec<u8> = (0..8192u32)
- .map(|i| (0x80 + (i * 37) % 0x7f) as u8)
- .collect();
- assert!(std::str::from_utf8(&soup).is_err());
- let sn = sniff_bytes(
- &soup,
- Path::new("texture.tga"),
- Path::new("texture.tga"),
- false,
- )
- .unwrap();
- assert_eq!(sn.family, Family::Binary, "pixel soup must stay binary");
}
#[test]