Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(fts): disclose Thai tokenizer does run-isolation, not word segmentation

xerj-org67b56e6cef72
diff --git a/engine/crates/xerj-fts/src/analyzer.rs b/engine/crates/xerj-fts/src/analyzer.rs
index 0e0c19f44..9924d9f9b 100644
--- a/engine/crates/xerj-fts/src/analyzer.rs
+++ b/engine/crates/xerj-fts/src/analyzer.rs
@@ -776,11 +776,23 @@ impl Tokenizer for CjkTokenizer {
}
}
-/// Thai tokenizer.
+/// Thai tokenizer — **Thai-run isolation only, NOT dictionary word segmentation.**
///
-/// Splits Thai text on non-Thai character boundaries (spaces, ASCII, etc.).
-/// Thai word segmentation is complex (no spaces between words); this simple
-/// implementation at least separates Thai runs from non-Thai text.
+/// This tokenizer performs *script-run isolation*, not linguistic word breaking:
+/// - Each maximal contiguous run of Thai-script characters is emitted as **one
+/// token**, verbatim (Thai has no case, so no lowercasing is applied).
+/// - Non-Thai runs are split on whitespace and lowercased, like the other
+/// Latin-oriented tokenizers.
+///
+/// Example: `"สวัสดีabc def"` → `["สวัสดี", "abc", "def"]`.
+///
+/// Thai is written without spaces between words, so an entire Thai phrase or
+/// sentence collapses into a single token here. This is a deliberate,
+/// recall-limited simplification: Elasticsearch's `thai` analyzer uses an ICU
+/// `BreakIterator` (dictionary-based word segmentation) that would split that
+/// same run into multiple word tokens. Queries relying on matching individual
+/// Thai words within a run will therefore under-recall against XERJ compared
+/// with ES. Full dictionary word segmentation is out of scope for this tokenizer.
pub struct ThaiTokenizer;
impl Tokenizer for ThaiTokenizer {
@@ -1859,6 +1871,26 @@ mod tests {
assert_eq!(out[0].text, "hello");
}
+ #[test]
+ fn thai_tokenizer_isolates_runs_not_words() {
+ // Documents the honest behaviour: the ThaiTokenizer performs
+ // Thai-script-run ISOLATION, not dictionary word segmentation.
+ // A contiguous Thai run is emitted verbatim as a single token
+ // (no word breaking, no lowercasing — Thai has no case), while
+ // non-Thai runs are split on whitespace and lowercased.
+ let tok = ThaiTokenizer;
+ let tokens = tok.tokenize("สวัสดีabc def");
+ let texts: Vec<&str> = tokens.iter().map(|t| t.text.as_str()).collect();
+ assert_eq!(texts, vec!["สวัสดี", "abc", "def"]);
+
+ // The whole Thai run stays one token — it is NOT segmented into
+ // the individual words "สวัสดี" + "ครับ" that ES's ICU BreakIterator
+ // would produce. This asserts the recall-limited contract on purpose.
+ let joined = tok.tokenize("สวัสดีครับ");
+ assert_eq!(joined.len(), 1);
+ assert_eq!(joined[0].text, "สวัสดีครับ");
+ }
+
#[test]
fn stemmer_filter_english() {
let filter = StemmerFilter::english();