AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs(fts): disclose Thai tokenizer does run-isolation, not word segmentation
xerj-org67b56e6cef72
diff --git a/engine/crates/xerj-fts/src/analyzer.rs b/engine/crates/xerj-fts/src/analyzer.rsindex 0e0c19f44..9924d9f9b 100644--- a/engine/crates/xerj-fts/src/analyzer.rs+++ b/engine/crates/xerj-fts/src/analyzer.rs@@ -776,11 +776,23 @@ impl Tokenizer for CjkTokenizer {}}-/// Thai tokenizer.+/// Thai tokenizer — **Thai-run isolation only, NOT dictionary word segmentation.**///-/// Splits Thai text on non-Thai character boundaries (spaces, ASCII, etc.).-/// Thai word segmentation is complex (no spaces between words); this simple-/// implementation at least separates Thai runs from non-Thai text.+/// This tokenizer performs *script-run isolation*, not linguistic word breaking:+/// - Each maximal contiguous run of Thai-script characters is emitted as **one+/// token**, verbatim (Thai has no case, so no lowercasing is applied).+/// - Non-Thai runs are split on whitespace and lowercased, like the other+/// Latin-oriented tokenizers.+///+/// Example: `"สวัสดีabc def"` → `["สวัสดี", "abc", "def"]`.+///+/// Thai is written without spaces between words, so an entire Thai phrase or+/// sentence collapses into a single token here. This is a deliberate,+/// recall-limited simplification: Elasticsearch's `thai` analyzer uses an ICU+/// `BreakIterator` (dictionary-based word segmentation) that would split that+/// same run into multiple word tokens. Queries relying on matching individual+/// Thai words within a run will therefore under-recall against XERJ compared+/// with ES. Full dictionary word segmentation is out of scope for this tokenizer.pub struct ThaiTokenizer;impl Tokenizer for ThaiTokenizer {@@ -1859,6 +1871,26 @@ mod tests {assert_eq!(out[0].text, "hello");}+ #[test]+ fn thai_tokenizer_isolates_runs_not_words() {+ // Documents the honest behaviour: the ThaiTokenizer performs+ // Thai-script-run ISOLATION, not dictionary word segmentation.+ // A contiguous Thai run is emitted verbatim as a single token+ // (no word breaking, no lowercasing — Thai has no case), while+ // non-Thai runs are split on whitespace and lowercased.+ let tok = ThaiTokenizer;+ let tokens = tok.tokenize("สวัสดีabc def");+ let texts: Vec<&str> = tokens.iter().map(|t| t.text.as_str()).collect();+ assert_eq!(texts, vec!["สวัสดี", "abc", "def"]);++ // The whole Thai run stays one token — it is NOT segmented into+ // the individual words "สวัสดี" + "ครับ" that ES's ICU BreakIterator+ // would produce. This asserts the recall-limited contract on purpose.+ let joined = tok.tokenize("สวัสดีครับ");+ assert_eq!(joined.len(), 1);+ assert_eq!(joined[0].text, "สวัสดีครับ");+ }+#[test]fn stemmer_filter_english() {let filter = StemmerFilter::english();