Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

feat(text-crusher): CJK-aware segmentation + relevance via ICU (#1504)

Zhenjia ZHOU4035c041879e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8f3cac956..4780ed3f3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -117,6 +117,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* **learn:** weight loops in `headroom learn`. A new loop detector (`headroom/learn/loops.py`) recognizes repeated tool-call patterns — including RTK re-fetch loops, where RTK's output truncation makes the agent re-run larger-limit variants of a *successful* command — collapses output-limit variants to one signature, measures the wasted tokens, surfaces loops as a highest-priority digest section, and weights loop guardrails above one-off rules by their measured waste. Previously loops had no special weight and a no-failure re-fetch loop was skipped entirely. Adds an RTK-loop eval (`benchmarks/rtk_loop_learn_eval.py`) that reproduces a loop, runs it through Learn, and asserts the generated guardrail ranks first and prevents re-triggering.
* **learn:** write per-project learnings to the personal, gitignored `CLAUDE.local.md` by default instead of the team-shared `CLAUDE.md`, matching Claude Code's memory convention so machine-specific paths and tool-discovery byproducts no longer pollute the shared file. Adds a `--target` flag to override the destination (e.g. `--target CLAUDE.md` to opt back into the shared file, or any custom path), and auto-migrates a stale learned-patterns block out of an existing `CLAUDE.md` into `CLAUDE.local.md` with a warning ([#1072](https://github.com/chopratejas/headroom/issues/1072)).
* **proxy/transforms:** take large cold-start contexts off the synchronous kompress path — the root cause behind the `compression_first_stage` 30s-timeout + leaked-thread → executor-saturation cascade ([#1171](https://github.com/chopratejas/headroom/issues/1171)). A token size-gate inside the ML boundary routes oversized text away from ModernBERT (`HEADROOM_KOMPRESS_MAX_TOKENS`); a cooperative chunk-deadline bounds any kompress run that does proceed (`HEADROOM_COMPRESSION_DEADLINE_MS`); an opt-in off-path mode forwards uncompressed immediately and compresses in a single per-process background drain so the request never blocks on ML (`HEADROOM_BACKGROUND_COMPRESSION`); and a new native `TextCrusher` — a fast deterministic extractive prose compressor in `headroom._core` that reuses the shared BM25 relevance scorer — is the fast alternative to ModernBERT for large plain text (`HEADROOM_TEXT_CRUSHER`). All default off and fail-open. On a SQuAD answer-retention eval (requires the SQuAD dev set) TextCrusher keeps ~94% of buried answers at 30% size vs ~36% for truncate/random, and runs in one O(n) pass -- sub-second where ModernBERT takes minutes (self-contained speed benchmark in `benchmarks/text_crusher_quality_eval.py`).
+* **proxy/transforms:** `TextCrusher` now compresses CJK (Chinese/Japanese/Korean) text ([#1171](https://github.com/chopratejas/headroom/issues/1171)). CJK has no spaces or ASCII sentence terminators, so the prior ASCII splitter/tokenizer collapsed a whole CJK paragraph into one segment/one token and passed it through near-uncompressed. CJK-bearing input now takes an ICU (`icu_segmenter`, UAX#29 + dictionary) sentence/word segmentation path with a local BM25 relevance over the ICU tokens; pure-ASCII text is byte-identical to before, and the shared BM25 scorer is untouched. On real CMRC2018 Chinese QA, answer-retention under compression rises from 34% to ~91%; end-to-end aggregate savings on real CJK content rise from 16% to 40%.
* **proxy:** measure and surface rolling and current token throughput metrics (active/wall-clock input, compression, effective forward, and streamed generation) in `headroom perf` CLI and the dashboard ([#959](https://github.com/chopratejas/headroom/issues/959)).
* **vibe:** add Mistral Vibe CLI support with `headroom wrap vibe`.
* **proxy:** per-project savings breakdown on the dashboard for all wrapped agents — Claude Code, Codex, aider, Copilot, and Cursor ([#802](https://github.com/chopratejas/headroom/issues/802)). `headroom wrap claude`/`codex` tag requests with an `X-Headroom-Project` header (launch-directory name); `wrap aider`/`copilot`/`cursor` — whose clients cannot send custom headers — use a `/p/<name>` base-URL prefix the proxy strips. Savings are aggregated per project (persisted, schema v3 with transparent v2 migration), exposed as `savings.per_project` in `/stats` and `projects` in `/stats-history`, and shown in a Per-Project Savings dashboard table.
diff --git a/Cargo.lock b/Cargo.lock
index 5bebfaaa3..de43db646 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -1113,6 +1113,15 @@ version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
+[[package]]
+name = "core_maths"
+version = "0.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "77745e017f5edba1a9c1d854f6f3a52dac8a12dd5af5d2f54aecf61e43d80d30"
+dependencies = [
+ "libm",
+]
+
[[package]]
name = "cpufeatures"
version = "0.2.17"
@@ -1866,6 +1875,7 @@ dependencies = [
"flate2",
"hf-hub 0.4.3",
"http 1.4.2",
+ "icu_segmenter",
"magika",
"md-5",
"ort",
@@ -2225,6 +2235,21 @@ dependencies = [
"zerovec",
]
+[[package]]
+name = "icu_locale"
+version = "2.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d5a396343c7208121dc86e35623d3dfe19814a7613cfd14964994cdc9c9a2e26"
+dependencies = [
+ "icu_collections",
+ "icu_locale_core",
+ "icu_locale_data",
+ "icu_provider",
+ "potential_utf",
+ "tinystr",
+ "zerovec",
+]
+
[[package]]
name = "icu_locale_core"
version = "2.2.0"
@@ -2233,11 +2258,18 @@ checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29"
dependencies = [
"displaydoc",
"litemap",
+ "serde",
"tinystr",
"writeable",
"zerovec",
]
+[[package]]
+name = "icu_locale_data"
+version = "2.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d5fdcc9ac77c6d74ff5cf6e65ef3181d6af32003b16fce3a77fb451d2f695993"
+
[[package]]
name = "icu_normalizer"
version = "2.2.0"
@@ -2286,6 +2318,8 @@ checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421"
dependencies = [
"displaydoc",
"icu_locale_core",
+ "serde",
+ "stable_deref_trait",
"writeable",
"yoke",
"zerofrom",
@@ -2293,6 +2327,28 @@ dependencies = [
"zerovec",
]
+[[package]]
+name = "icu_segmenter"
+version = "2.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5c0794db0b1a86193ac9c48768d0e6c52c54448e0870ad87907d456ee0dac964"
+dependencies = [
+ "core_maths",
+ "icu_collections",
+ "icu_locale",
+ "icu_provider",
+ "icu_segmenter_data",
+ "potential_utf",
+ "utf8_iter",
+ "zerovec",
+]
+
+[[package]]
+name = "icu_segmenter_data"
+version = "2.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e4a2c462a4d927d512f5f882a033ddd62f33a05bb9f230d98f736ac3dc85938f"
+
[[package]]
name = "ident_case"
version = "1.0.1"
@@ -2522,6 +2578,12 @@ dependencies = [
"windows-link",
]
+[[package]]
+name = "libm"
+version = "0.2.16"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
+
[[package]]
name = "libredox"
version = "0.1.17"
@@ -3106,6 +3168,8 @@ version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564"
dependencies = [
+ "serde_core",
+ "writeable",
"zerovec",
]
@@ -4215,6 +4279,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d"
dependencies = [
"displaydoc",
+ "serde_core",
"zerovec",
]
@@ -5318,6 +5383,7 @@ dependencies = [
"displaydoc",
"yoke",
"zerofrom",
+ "zerovec",
]
[[package]]
@@ -5326,6 +5392,7 @@ version = "0.11.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239"
dependencies = [
+ "serde",
"yoke",
"zerofrom",
"zerovec-derive",
diff --git a/benchmarks/i18n_compression_eval.py b/benchmarks/i18n_compression_eval.py
new file mode 100644
index 000000000..a130b69ba
--- /dev/null
+++ b/benchmarks/i18n_compression_eval.py
@@ -0,0 +1,282 @@
+#!/usr/bin/env python3
+"""i18n compression-quality eval (zh/ja/ko): does extractive compression keep
+the answer-bearing content in CJK? No LLM/API calls -- fully local.
+
+Part C -- our own DETERMINISTIC needle answer-retention (zh/ja/ko): the always-
+runs regression gate. A distinctive needle sentence is buried (in the middle) in
+language-matched distractor sentences; compress query-aware; assert the needle
+survives. No external data. TextCrusher (query-aware) vs truncate (keep-recent)
+vs random baselines.
+
+Part B -- real-transcript fidelity with CJK-aware salient: optional, anonymized.
+
+Part A -- natural-data answer-retention on alexandrainst/multi-wiki-qa
+(zh-cn/ja/ko): optional, via the [evals] datasets extra, skipped if absent.
+
+Usage: python benchmarks/i18n_compression_eval.py [transcript.jsonl]
+"""
+
+from __future__ import annotations
+
+import glob
+import os
+import random
+import re
+import sys
+import time
+
+from headroom.transforms.text_crusher import TextCrusher
+
+_REDACT = [
+ (re.compile(r"/Users/[^/\s]+"), "/Users/USER"),
+ (re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"), "EMAIL"),
+ (re.compile(r"\b(?:sk|pk|ghp|gho|xox[baprs])-[A-Za-z0-9_-]{10,}\b"), "TOKEN"),
+ (re.compile(r"\b[A-Fa-f0-9]{40,}\b"), "HEX"),
+]
+# Split on ASCII and full-width CJK terminators so baselines segment CJK too.
+_SEG = re.compile(r"(?<=[.!?。!?])\s*|\n+")
+_CJK_RUN = re.compile(r"[㐀-鿿぀-ヿ가-힯]+")
+
+
+def anon(t: str) -> str:
+ for rx, rep in _REDACT:
+ t = rx.sub(rep, t)
+ return t
+
+
+def norm(s: str) -> str:
+ # CJK has no spaces; drop all whitespace so substring match is robust.
+ return re.sub(r"\s+", "", s.lower())
+
+
+def _segs(text: str) -> list[str]:
+ return [s for s in _SEG.split(text) if s.strip()]
+
+
+def truncate_keep_last(text: str, ratio: float) -> str:
+ segs = _segs(text)
+ budget = int(sum(len(s) for s in segs) * ratio)
+ kept: list[str] = []
+ c = 0
+ for s in reversed(segs):
+ if c >= budget:
+ break
+ kept.append(s)
+ c += len(s)
+ return "".join(reversed(kept))
+
+
+def random_keep(text: str, ratio: float, seed: int) -> str:
+ segs = _segs(text)
+ idx = list(range(len(segs)))
+ random.Random(seed).shuffle(idx)
+ budget = int(sum(len(s) for s in segs) * ratio)
+ kept: set[int] = set()
+ c = 0
+ for i in idx:
+ if c >= budget:
+ break
+ kept.add(i)
+ c += len(segs[i])
+ return "".join(segs[i] for i in sorted(kept))
+
+
+# --- Part C: deterministic needle retention (zh / ja / ko) ---------------------
+
+# Each needle carries a distinctive verbatim KEY that must survive. Distractors
+# are generated (deterministic, distinct, topic-unrelated to the query) so the
+# haystack is large enough to FORCE real compression -- the needle only survives
+# under TextCrusher because it is query-relevant, not because of passthrough.
+_NEEDLES = {
+ "zh": {
+ "query": "认证令牌缓存淘汰策略",
+ "key": "最近最少使用淘汰",
+ "needle": "认证令牌的缓存采用最近最少使用淘汰算法来管理过期条目。",
+ "distractor": lambda i: f"第{i}号监控服务器的日志显示子系统{i}今天运行平稳没有出现异常。",
+ },
+ "ja": {
+ "query": "認証トークン キャッシュ 破棄 アルゴリズム",
+ "key": "最長未使用",
+ "needle": "認証トークンのキャッシュは最長未使用アルゴリズムで管理される。",
+ "distractor": lambda i: (
+ f"{i}番目の監視サーバーのログには{i}番のサブシステムが本日も正常に稼働したと記録されている。"
+ ),
+ },
+ "ko": {
+ "query": "인증 토큰 캐시 제거 알고리즘",
+ "key": "최근 최소 사용",
+ "needle": "인증 토큰 캐시는 최근 최소 사용 알고리즘으로 관리된다.",
+ "distractor": lambda i: (
+ f"{i}번 모니터링 서버의 로그에는 {i}번 하위 시스템이 오늘도 정상 작동했다고 기록되어 있다."
+ ),
+ },
+}
+
+
+def _haystack(spec: dict, n_distract: int = 24) -> str:
+ half = n_distract // 2
+ before = [spec["distractor"](i) for i in range(half)]
+ after = [spec["distractor"](i) for i in range(half, n_distract)]
+ # needle in the MIDDLE so keep-recent (truncate) reliably misses it.
+ return "".join(before + [spec["needle"]] + after)
+
+
+def retention_synthetic(lang: str, ratio: float = 0.3, seed: int = 0) -> dict[str, bool]:
+ spec = _NEEDLES[lang]
+ hay = _haystack(spec)
+ key = norm(spec["key"])
+ tc = TextCrusher()
+ out_tc = tc.compress(hay, spec["query"], ratio).compressed
+ return {
+ "text_crusher": key in norm(out_tc),
+ "truncate": key in norm(truncate_keep_last(hay, ratio)),
+ "random": key in norm(random_keep(hay, ratio, seed)),
+ }
+
+
+def eval_synthetic(ratio: float = 0.3) -> None:
+ print(f"\n=== Part C: synthetic needle retention (zh/ja/ko, target_ratio={ratio}) ===")
+ print(f" {'lang':5} {'text_crusher':>13} {'truncate':>9} {'random':>7}")
+ for lang in ("zh", "ja", "ko"):
+ r = retention_synthetic(lang, ratio)
+ print(
+ f" {lang:5} {str(r['text_crusher']):>13} {str(r['truncate']):>9} {str(r['random']):>7}"
+ )
+ print(" (needle must survive under TextCrusher; baselines are the contrast)")
+
+
+# --- Part B: real CJK transcript fidelity (CJK-aware salient) ------------------
+
+# ASCII salient (identifiers/numbers/errors) STILL matters in CJK coding context.
+_SALIENT_ASCII = re.compile(
+ r"\b(?:error|exception|fail(?:ed|ure)?|warning|traceback|assert|todo|fixme)\b"
+ r"|\b[A-Z]{2,}\b|\b[A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*\b|\b\d+\b"
+)
+
+
+def _cjk_hapax(text: str) -> set[str]:
+ # distinctive CJK content = char-bigrams occurring exactly once (rare = must-keep)
+ grams: dict[str, int] = {}
+ for run in _CJK_RUN.findall(text):
+ for i in range(len(run) - 1):
+ g = run[i : i + 2]
+ grams[g] = grams.get(g, 0) + 1
+ return {g for g, c in grams.items() if c == 1}
+
+
+def salient_set(text: str) -> set[str]:
+ return set(_SALIENT_ASCII.findall(text)) | _cjk_hapax(text)
+
+
+def _block_texts(jsonl_path: str, min_chars: int, limit: int) -> list[str]:
+ import json
+
+ out: list[str] = []
+ with open(jsonl_path, encoding="utf-8") as fh:
+ for line in fh:
+ try:
+ o = json.loads(line)
+ except json.JSONDecodeError:
+ continue
+ c = (o.get("message") or {}).get("content")
+ parts = (
+ [c]
+ if isinstance(c, str)
+ else [
+ p["text"] for p in c if isinstance(p, dict) and isinstance(p.get("text"), str)
+ ]
+ if isinstance(c, list)
+ else []
+ )
+ for t in parts:
+ if len(t) >= min_chars and _CJK_RUN.search(t): # CJK-bearing only
+ out.append(anon(t))
+ if len(out) >= limit:
+ break
+ return out[:limit]
+
+
+def eval_transcript(
+ jsonl_path: str, ratio: float = 0.4, min_chars: int = 600, limit: int = 40
+) -> None:
+ blocks = _block_texts(jsonl_path, min_chars, limit)
+ if not blocks:
+ print(
+ f"\n=== Part B: no CJK blocks >= {min_chars} chars in {os.path.basename(jsonl_path)} ==="
+ )
+ return
+ tc = TextCrusher()
+ ratios: list[float] = []
+ times: list[float] = []
+ retentions: list[float] = []
+ for b in blocks:
+ sal_before = salient_set(b)
+ t0 = time.perf_counter()
+ out = tc.compress(b, "", ratio).compressed
+ times.append((time.perf_counter() - t0) * 1000)
+ retentions.append(len(sal_before & salient_set(out)) / max(1, len(sal_before)))
+ ratios.append(len(out) / max(1, len(b)))
+ n = len(blocks)
+ print(
+ f"\n=== Part B: real CJK transcript fidelity (n={n}, anonymized, target_ratio={ratio}) ==="
+ )
+ print(f" mean char-ratio kept: {sum(ratios) / n:.2f}")
+ print(f" mean speed: {sum(times) / n:.1f} ms/block")
+ print(f" CJK-aware salient retention: {sum(retentions) / n:.1%}")
+
+
+# --- Part A: optional natural-data retention (multi-wiki-qa zh/ja/ko) ----------
+# Schema verified: row = {id, title, context, question, answers:{text:[...]}}.
+# Answers are guaranteed verbatim substrings of the (long) context; CC-BY-NC-SA.
+
+
+def eval_multiwiki(
+ langs=("zh-cn", "ja", "ko"), n: int = 80, ratio: float = 0.3, seed: int = 0
+) -> None:
+ try:
+ from datasets import load_dataset
+ except ImportError:
+ print(
+ "\n=== Part A: `datasets` not installed; skipping (pip install headroom-ai[evals]) ==="
+ )
+ return
+ tc = TextCrusher()
+ print(f"\n=== Part A: multi-wiki-qa answer-retention (n={n}/lang, target_ratio={ratio}) ===")
+ print(f" {'lang':6} {'text_crusher':>13} {'truncate':>9} {'random':>7}")
+ for lang in langs:
+ try:
+ ds = load_dataset("alexandrainst/multi-wiki-qa", lang, split=f"train[:{n * 2}]")
+ except Exception as e: # noqa: BLE001 -- optional path, fail-open
+ print(f" {lang}: load failed ({e}); skipping")
+ continue
+ ex = []
+ for r in ds:
+ ans = r.get("answers")
+ a = ans["text"][0] if isinstance(ans, dict) and ans.get("text") else None
+ if r.get("context") and r.get("question") and a:
+ ex.append((r["context"], r["question"], a))
+ random.Random(seed).shuffle(ex)
+ ex = ex[:n]
+ hit = {"text_crusher": 0, "truncate": 0, "random": 0}
+ for ctx, q, ans in ex:
+ a = norm(ans)
+ hit["text_crusher"] += a in norm(tc.compress(ctx, q, ratio).compressed)
+ hit["truncate"] += a in norm(truncate_keep_last(ctx, ratio))
+ hit["random"] += a in norm(random_keep(ctx, ratio, seed))
+ m = max(1, len(ex))
+ print(
+ f" {lang:6} {hit['text_crusher'] / m:>12.0%} {hit['truncate'] / m:>9.0%} {hit['random'] / m:>7.0%}"
+ )
+
+
+if __name__ == "__main__":
+ eval_synthetic()
+ tx = sys.argv[1] if len(sys.argv) > 1 else None
+ if tx is None:
+ found = glob.glob(os.path.expanduser("~/.claude/projects/*headroom*/*.jsonl"))
+ tx = max(found, key=os.path.getsize) if found else None
+ if tx and os.path.exists(tx):
+ eval_transcript(tx)
+ else:
+ print("\nno transcript jsonl found; skipping Part B")
+ eval_multiwiki()
diff --git a/crates/headroom-core/Cargo.toml b/crates/headroom-core/Cargo.toml
index 198ace6f0..2b6442ecf 100644
--- a/crates/headroom-core/Cargo.toml
+++ b/crates/headroom-core/Cargo.toml
@@ -37,6 +37,22 @@ dashmap = "6"
# `regex` is already a transitive dep of tokenizers; depend on it directly so
# our hunk-header parser and priority-pattern matcher have a stable surface.
regex = "1"
+# CJK sentence + word segmentation for TextCrusher (#1171). CJK has no spaces or
+# ASCII terminators, so the default ASCII splitter/tokenizer collapses a whole
+# CJK paragraph into one segment/one token -> 0% compression. ICU4X's UAX#29
+# sentence + dictionary word segmenters fix this. Chosen over a hand-rolled
+# char-bigram (benchmarked on real CMRC2018 Chinese QA: 92.5% vs 91% answer-
+# retention) and over jieba/lindera (ZH-only / tens-to-hundreds of MB dicts).
+# - Why this version: 2.x is the stabilized ICU4X API (1.x used a different data
+# provider model); floored at 2.2 (Cargo.lock pins the exact patch) since
+# segmenter boundaries are observable in output -- bumps should be deliberate.
+# - Install surface: ~13 new crates, all pure Rust, no build scripts, no native
+# code, no build/runtime network. `compiled_data` bundles locale data at
+# compile time (hermetic). Maintained by the official unicode-org.
+# - `compiled_data` only (no `auto`/`lstm`): LSTM models cover SE-Asian scripts
+# (Thai/Lao), not CJK -- CJK uses the dictionary, so `auto` would pull in libm
+# for nothing. Required for CJK; pure-ASCII paths are unchanged.
+icu_segmenter = { version = "2.2", features = ["compiled_data"] }
# `flate2` for `_validate_with_zlib` in `adaptive_sizer`. Python's adaptive
# sizing pipeline uses `zlib.compress(..., level=1)` to validate the chosen
# K against compression-ratio diversity. We use the default `miniz_oxide`
diff --git a/crates/headroom-core/src/transforms/text_crusher/crusher.rs b/crates/headroom-core/src/transforms/text_crusher/crusher.rs
index 5a35cd061..e75a3c722 100644
--- a/crates/headroom-core/src/transforms/text_crusher/crusher.rs
+++ b/crates/headroom-core/src/transforms/text_crusher/crusher.rs
@@ -6,15 +6,58 @@
//! target ratio. Output is extractive: the kept sentences are verbatim words
//! (each segment trimmed, re-joined with `\n`) -- no invented words, no rewrite.
//!
-//! The relevance term REUSES the shared [`BM25Scorer`](crate::relevance) rather
-//! than reimplementing BM25 -- only the prose-specific splitting + selection
-//! lives here.
+//! The relevance term REUSES the shared [`BM25Scorer`](crate::relevance) for
+//! non-CJK text. CJK has no spaces/ASCII terminators, so CJK-bearing input takes
+//! an ICU (UAX#29 + dictionary) sentence/word segmentation path with a local
+//! BM25 over the ICU tokens; pure-ASCII text is byte-identical to before.
use std::cmp::Ordering;
use std::collections::HashSet;
+use std::sync::LazyLock;
use super::config::TextCrusherConfig;
use crate::relevance::{BM25Scorer, RelevanceScorer};
+use icu_segmenter::{
+ SentenceSegmenter, SentenceSegmenterBorrowed, WordSegmenter, WordSegmenterBorrowed,
+};
+
+// ICU segmenters resolved ONCE and reused (compiled_data is static, so the
+// borrowed view is 'static). A fresh segmenter per call would dominate this
+// compressor's request-path budget -- compress() tokenizes every segment.
+static SENTENCE_SEGMENTER: LazyLock<SentenceSegmenterBorrowed<'static>> =
+ LazyLock::new(|| SentenceSegmenter::new(Default::default()));
+static WORD_SEGMENTER: LazyLock<WordSegmenterBorrowed<'static>> =
+ LazyLock::new(|| WordSegmenter::new_dictionary(Default::default()));
+
+/// True for CJK ideographs, kana, Hangul, plus CJK punctuation (。、「」) and
+/// half/full-width forms — scripts/marks without ASCII spaces or terminators,
+/// which the default ASCII splitter/tokenizer can't segment. CJK-bearing text
+/// takes the ICU path; pure-ASCII text is byte-identical to before.
+fn is_cjk(c: char) -> bool {
+ matches!(
+ c as u32,
+ 0x3000..=0x303F // CJK symbols & punctuation (。、「」【】)
+ | 0x3040..=0x30FF // Hiragana + Katakana
+ | 0x3400..=0x4DBF // CJK Ext A
+ | 0x4E00..=0x9FFF // CJK Unified
+ | 0xAC00..=0xD7AF // Hangul syllables
+ | 0xF900..=0xFAFF // CJK Compatibility ideographs
+ | 0xFF00..=0xFFEF // half/full-width forms (!? カナ)
+ | 0x20000..=0x2FA1F // CJK Ext B–F + Compat Supplement
+ )
+}
+
+/// Token count for the reported ratio: ASCII whitespace words for non-CJK
+/// (unchanged), CJK-aware tokens when CJK is present. Whitespace-splitting would
+/// count a space-free CJK string as ONE token, making compression_ratio nonsense
+/// (a newline-joined N-segment output looks like N tokens vs a 1-token input).
+fn count_tokens(s: &str) -> usize {
+ if s.chars().any(is_cjk) {
+ tokens(s).len()
+ } else {
+ s.split_whitespace().count()
+ }
+}
const KEYWORDS: [&str; 10] = [
"error",
@@ -59,7 +102,7 @@ impl TextCrusher {
}
fn passthrough(content: &str, n_segments: usize) -> TextCrusherResult {
- let toks = content.split_whitespace().count();
+ let toks = count_tokens(content);
TextCrusherResult {
compressed: content.to_string(),
original_tokens: toks,
@@ -90,19 +133,37 @@ impl TextCrusher {
// admit nothing and silently fall back to a 100% passthrough).
let target_chars = ((total_chars as f64 * ratio) as usize).max(1);
- // Relevance via the shared BM25 scorer (already [0, 1]).
- let seg_refs: Vec<&str> = segments.iter().map(|s| s.as_str()).collect();
- let relevance = self.scorer.score_batch(&seg_refs, context);
-
let seg_tokens: Vec<Vec<String>> = segments.iter().map(|s| tokens(s)).collect();
+ // CJK content: relevance via a local BM25 over the ICU word tokens (the
+ // shared ASCII BM25Scorer scores zero terms for CJK). Dispatch on the
+ // CONTENT only -- pure-ASCII content keeps the shared scorer even when
+ // the query is CJK, so English output stays byte-identical.
+ let relevance: Vec<f64> = if segments.iter().any(|s| s.chars().any(is_cjk)) {
+ relevance_cjk(&seg_tokens, context)
+ } else {
+ let seg_refs: Vec<&str> = segments.iter().map(|s| s.as_str()).collect();
+ self.scorer
+ .score_batch(&seg_refs, context)
+ .iter()
+ .map(|r| r.score)
+ .collect()
+ };
+
let mut scores = vec![0.0f64; n];
for i in 0..n {
let recency = (i as f64 + 1.0) / n as f64;
- let rel = relevance.get(i).map(|r| r.score).unwrap_or(0.0);
- let words: Vec<&str> = segments[i].split_whitespace().collect();
- let salient = words.iter().filter(|w| is_salient(w)).count();
- let salience = salient as f64 / (words.len() as f64 + 1.0);
+ let rel = relevance.get(i).copied().unwrap_or(0.0);
+ // CJK segments have no spaces, so split_whitespace yields one giant
+ // "word" and zero salience; use the already-computed ICU tokens.
+ let (salient, word_count) = if segments[i].chars().any(is_cjk) {
+ let s = seg_tokens[i].iter().filter(|w| is_salient(w)).count();
+ (s, seg_tokens[i].len())
+ } else {
+ let words: Vec<&str> = segments[i].split_whitespace().collect();
+ (words.iter().filter(|w| is_salient(w)).count(), words.len())
+ };
+ let salience = salient as f64 / (word_count as f64 + 1.0);
let mut score =
cfg.w_recency * recency + cfg.w_relevance * rel + cfg.w_salience * salience;
if segments[i].len() < cfg.min_segment_chars {
@@ -153,8 +214,8 @@ impl TextCrusher {
.map(|i| segments[i].as_str())
.collect::<Vec<_>>()
.join("\n");
- let orig_tok = content.split_whitespace().count();
- let comp_tok = compressed.split_whitespace().count();
+ let orig_tok = count_tokens(content);
+ let comp_tok = count_tokens(&compressed);
TextCrusherResult {
compression_ratio: if orig_tok > 0 {
comp_tok as f64 / orig_tok as f64
@@ -170,9 +231,18 @@ impl TextCrusher {
}
}
-/// Split into sentence/line segments: on newlines, and after `.`/`!`/`?`
-/// followed by whitespace. Byte-faithful (kept segments are joined verbatim).
+/// Sentence/line segmentation, dispatched on content: ICU for CJK-bearing text,
+/// the original ASCII splitter otherwise (byte-identical to before).
fn split_segments(text: &str) -> Vec<String> {
+ if text.chars().any(is_cjk) {
+ split_segments_icu(text)
+ } else {
+ split_segments_ascii(text)
+ }
+}
+
+/// ASCII path (unchanged): on newlines, and after `.`/`!`/`?` + whitespace.
+fn split_segments_ascii(text: &str) -> Vec<String> {
let mut segs = Vec::new();
for line in text.split('\n') {
let trimmed = line.trim();
@@ -202,7 +272,79 @@ fn split_segments(text: &str) -> Vec<String> {
segs
}
+/// CJK path: ICU/UAX#29 sentence boundaries per line, then the length fallback.
+fn split_segments_icu(text: &str) -> Vec<String> {
+ let seg = *SENTENCE_SEGMENTER;
+ let mut out = Vec::new();
+ for line in text.split('\n') {
+ let trimmed = line.trim();
+ if trimmed.is_empty() {
+ continue;
+ }
+ let mut prev = 0usize;
+ for b in seg.segment_str(trimmed) {
+ if b > prev {
+ let s = trimmed[prev..b].trim();
+ if !s.is_empty() {
+ out.push(s.to_string());
+ }
+ prev = b;
+ }
+ }
+ if prev < trimmed.len() {
+ let s = trimmed[prev..].trim();
+ if !s.is_empty() {
+ out.push(s.to_string());
+ }
+ }
+ }
+ apply_length_fallback(out)
+}
+
+/// Mandatory terminator-sparse fallback: split any over-long CJK-bearing segment
+/// on whitespace / CJK secondary punctuation, then a hard char cap.
+fn apply_length_fallback(segs: Vec<String>) -> Vec<String> {
+ let cap = 60usize;
+ let hard = 40usize;
+ let mut out = Vec::new();
+ for s in segs {
+ if s.chars().count() <= cap || !s.chars().any(is_cjk) {
+ out.push(s);
+ continue;
+ }
+ let mut piece = String::new();
+ for c in s.chars() {
+ piece.push(c);
+ let n = piece.chars().count();
+ let soft = c.is_whitespace() || matches!(c, '、' | ',' | ';' | ':' | '·' | '…');
+ if (soft && n >= hard / 2) || n >= hard {
+ let t = piece.trim();
+ if !t.is_empty() {
+ out.push(t.to_string());
+ }
+ piece.clear();
+ }
+ }
+ let t = piece.trim();
+ if !t.is_empty() {
+ out.push(t.to_string());
+ }
+ }
+ out
+}
+
+/// Word-unit tokenization for shingles/relevance, dispatched on content: ICU
+/// word segmentation for CJK, the original ASCII alnum-run tokenizer otherwise.
fn tokens(text: &str) -> Vec<String> {
+ if text.chars().any(is_cjk) {
+ tokens_icu(text)
+ } else {
+ tokens_ascii(text)
+ }
+}
+
+/// ASCII path (unchanged): lowercased alphanumeric/underscore runs.
+fn tokens_ascii(text: &str) -> Vec<String> {
let mut out = Vec::new();
let mut cur = String::new();
for c in text.chars() {
@@ -220,6 +362,77 @@ fn tokens(text: &str) -> Vec<String> {
out
}
+/// CJK path: ICU WordSegmenter (dictionary) word units; alnum-bearing, lowercased.
+fn tokens_icu(text: &str) -> Vec<String> {
+ let seg = *WORD_SEGMENTER;
+ let mut out = Vec::new();
+ let mut prev = 0usize;
+ for b in seg.segment_str(text) {
+ if b > prev {
+ let w = text[prev..b].trim();
+ if !w.is_empty() && w.chars().any(|c| c.is_alphanumeric()) {
+ out.push(w.to_lowercase());
+ }
+ prev = b;
+ }
+ }
+ out
+}
+
+/// BM25 relevance over the segments' ICU word tokens. This is INTENTIONALLY a
+/// separate scorer from the shared [`BM25Scorer`](crate::relevance): that one is
+/// parity-locked to Python and tokenizes with an ASCII-only regex, so it scores
+/// zero terms for CJK and cannot be reused here. This variant takes pre-computed
+/// ICU word tokens and uses textbook BM25 (k1=1.2, b=0.75) -- deliberately NOT
+/// BM25Scorer's ASCII-tuned k1=1.5 + long-identifier bonus, which don't transfer
+/// to CJK words. The `+1` inside the idf log keeps it non-negative; output is
+/// max-normalized to [0, 1] to match the shared scorer's range in the weighting.
+fn relevance_cjk(seg_tokens: &[Vec<String>], context: &str) -> Vec<f64> {
+ use std::collections::HashMap;
+ let n = seg_tokens.len();
+ let qtokens: HashSet<String> = tokens(context).into_iter().collect();
+ if n == 0 || qtokens.is_empty() {
+ return vec![0.0; n];
+ }
+ let mut df: HashMap<&str, usize> = HashMap::new();
+ for toks in seg_tokens {
+ let uniq: HashSet<&str> = toks.iter().map(|s| s.as_str()).collect();
+ for t in uniq {
+ *df.entry(t).or_insert(0) += 1;
+ }
+ }
+ let nf = n as f64;
+ let idf = |t: &str| -> f64 {
+ let d = *df.get(t).unwrap_or(&0) as f64;
+ (((nf - d + 0.5) / (d + 0.5)) + 1.0).ln()
+ };
+ let (k1, b) = (1.2_f64, 0.75_f64);
+ let avgdl = (seg_tokens.iter().map(|t| t.len()).sum::<usize>() as f64 / nf).max(1.0);
+ let mut out = vec![0.0f64; n];
+ for (i, toks) in seg_tokens.iter().enumerate() {
+ let mut tf: HashMap<&str, usize> = HashMap::new();
+ for t in toks {
+ *tf.entry(t.as_str()).or_insert(0) += 1;
+ }
+ let dl = toks.len() as f64;
+ let mut score = 0.0;
+ for q in &qtokens {
+ if let Some(&f) = tf.get(q.as_str()) {
+ let f = f as f64;
+ score += idf(q) * (f * (k1 + 1.0)) / (f + k1 * (1.0 - b + b * dl / avgdl));
+ }
+ }
+ out[i] = score;
+ }
+ let max = out.iter().cloned().fold(0.0f64, f64::max);
+ if max > 0.0 {
+ for s in &mut out {
+ *s /= max;
+ }
+ }
+ out
+}
+
fn shingles(words: &[String], k: usize) -> HashSet<String> {
let mut set = HashSet::new();
if words.is_empty() {
@@ -314,4 +527,118 @@ mod tests {
let r = TextCrusher::default().compress("one. two. three.", "", None);
assert_eq!(r.compression_ratio, 1.0);
}
+
+ #[test]
+ fn cjk_splits_on_full_width_terminators() {
+ let zh = "今天天气很好。我们去公园散步。然后回家吃饭。下午还要开会。晚上看电影。";
+ let segs = split_segments(zh);
+ assert!(
+ segs.len() >= 4,
+ "expected multiple CJK sentences, got {segs:?}"
+ );
+ for s in &segs {
+ assert!(zh.contains(s.as_str()), "segment not verbatim: {s}");
+ }
+ }
+
+ #[test]
+ fn cjk_terminator_sparse_length_fallback() {
+ // a long flowing CJK run with NO terminators must STILL split (the fallback)
+ let zh = "机器学习模型从数据中学习特征并识别模式进行预测的系统会不断地调整参数\
+ 以最小化误差从而提升准确率这是一段很长的没有任何标点的中文用来测试兜底\
+ 切分是否生效以及能否产生多个段落供后续打分与去重使用确保不会整段透传";
+ assert!(
+ split_segments(zh).len() >= 2,
+ "terminator-sparse CJK must still split into multiple segments"
+ );
+ }
+
+ #[test]
+ fn cjk_tokens_not_one_giant_token() {
+ // the old ASCII tokenizer collapsed a whole Han run into ONE token
+ assert!(
+ tokens("数据库连接失败重试三次").len() >= 3,
+ "CJK run must yield multiple word-ish tokens"
+ );
+ }
+
+ #[test]
+ fn cjk_relevance_keeps_query_match() {
+ let needle = "认证令牌的缓存策略采用最近最少使用淘汰算法来管理过期。";
+ let filler = "今天天气很好。我们去公园散步。然后回家吃饭。下午还要开会。\
+ 晚上看电影。明天继续工作。周末去爬山。后天有个会议。";
+ let doc = format!("{filler}{needle}{filler}");
+ let r = TextCrusher::default().compress(&doc, "认证令牌缓存策略", Some(0.3));
+ assert!(r.compressed_tokens < r.original_tokens, "should compress");
+ assert!(
+ r.compressed.contains("认证令牌"),
+ "query-relevant CJK sentence must survive selection: {}",
+ r.compressed
+ );
+ }
+
+ #[test]
+ fn mixed_cjk_latin_keeps_ascii_terms_and_compresses() {
+ let content = "系统启动失败。认证模块超时。\nERROR: connection refused at host.\n\
+ 数据库连接池耗尽。重试机制触发。服务降级处理完成。请检查日志。";
+ let r = TextCrusher::default().compress(content, "ERROR connection", Some(0.5));
+ assert!(r.compression_ratio < 1.0, "mixed content must compress");
+ assert!(r.original_tokens > 5, "must not collapse CJK to one token");
+ assert!(
+ r.compressed.contains("ERROR"),
+ "ASCII term relevant to the query must survive: {}",
+ r.compressed
+ );
+ }
+
+ #[test]
+ fn korean_tokenizes_and_splits() {
+ let ko = "인증 토큰의 캐시 전략은 최근 최소 사용 알고리즘으로 관리된다。\
+ 세션은 만료 시간에 따라 자동으로 정리된다。";
+ assert!(
+ tokens(ko).len() >= 4,
+ "Korean must yield multiple tokens via ICU"
+ );
+ assert!(split_segments(ko).len() >= 2, "Korean sentences must split");
+ }
+
+ #[test]
+ fn japanese_no_space_tokenizes_via_dictionary() {
+ // Japanese has no spaces; ICU dictionary segmentation must still split
+ // this into multiple word tokens (whitespace-splitting would give one).
+ let ja = "認証トークンのキャッシュ戦略は最近最少使用アルゴリズムで管理される。\
+ セッションは有効期限に従って自動的に整理される。";
+ assert!(
+ tokens(ja).len() >= 5,
+ "Japanese must split into multiple ICU tokens"
+ );
+ assert!(
+ split_segments(ja).len() >= 2,
+ "Japanese sentences must split on 。"
+ );
+ }
+
+ #[test]
+ fn cjk_token_count_and_ratio_are_sane() {
+ let zh = "系统架构遵循微服务模式。每个服务拥有自己的数据存储和接口。".repeat(20);
+ let r = TextCrusher::default().compress(&zh, "", Some(0.4));
+ assert!(
+ r.original_tokens > 10,
+ "CJK token count must not collapse to 1"
+ );
+ assert!(r.compressed_tokens < r.original_tokens);
+ assert!(r.compression_ratio > 0.0 && r.compression_ratio <= 1.0);
+ }
+
+ #[test]
+ fn ascii_content_unchanged_even_with_cjk_query() {
+ // C-fix: dispatch is on CONTENT, not query. A CJK query against pure-ASCII
+ // content must take the unchanged ASCII path (shared BM25Scorer).
+ let content = doc(40);
+ let with_ascii_q = TextCrusher::default().compress(&content, "topic 7", Some(0.3));
+ let with_cjk_q = TextCrusher::default().compress(&content, "主题 七", Some(0.3));
+ // ASCII content compresses identically regardless of the query script
+ assert_eq!(with_ascii_q.total_segments, with_cjk_q.total_segments);
+ assert!(with_cjk_q.compressed_tokens < with_cjk_q.original_tokens);
+ }
}
diff --git a/tests/parity/fixtures/text_crusher/unicode_d46b3fb761b1.json b/tests/parity/fixtures/text_crusher/unicode_d46b3fb761b1.json
index 511f413b2..66ff1396a 100644
--- a/tests/parity/fixtures/text_crusher/unicode_d46b3fb761b1.json
+++ b/tests/parity/fixtures/text_crusher/unicode_d46b3fb761b1.json
@@ -7,12 +7,12 @@
"target_ratio": 0.4
},
"output": {
- "compressed": "句子 0 描述了系统在主题 0 上的行为细节。 句子 1 描述了系统在主题 1 上的行为细节。 句子 2 描述了系统在主题 2 上的行为细节。 句子 3 描述了系统在主题 3 上的行为细节。 句子 4 描述了系统在主题 4 上的行为细节。 句子 5 描述了系统在主题 5 上的行为细节。 句子 6 描述了系统在主题 6 上的行为细节。 句子 7 描述了系统在主题 7 上的行为细节。 句子 8 描述了系统在主题 8 上的行为细节。 句子 9 描述了系统在主题 9 上的行为细节。 句子 10 描述了系统在主题 10 上的行为细节。 句子 11 描述了系统在主题 11 上的行为细节。",
- "original_tokens": 60,
+ "compressed": "句子 7 描述了系统在主题 7 上的行为细节。\n句子 8 描述了系统在主题 8 上的行为细节。\n句子 9 描述了系统在主题 9 上的行为细节。\n句子 10 描述了系统在主题 10 上的行为细节。\n句子 11 描述了系统在主题 11 上的行为细节。",
+ "original_tokens": 144,
"compressed_tokens": 60,
- "compression_ratio": 1.0,
- "kept_segments": 1,
- "total_segments": 1
+ "compression_ratio": 0.4166666666666667,
+ "kept_segments": 5,
+ "total_segments": 12
},
"input_sha256": "d46b3fb761b16a7771144b388721042e322655450fb9f10e80459d118aabf600"
}
\ No newline at end of file
diff --git a/tests/test_transforms/test_text_crusher_cjk_eval.py b/tests/test_transforms/test_text_crusher_cjk_eval.py
new file mode 100644
index 000000000..a60444e9b
--- /dev/null
+++ b/tests/test_transforms/test_text_crusher_cjk_eval.py
@@ -0,0 +1,28 @@
+"""CI regression gate for CJK (zh/ja/ko) compression answer-retention.
+
+Deterministic: a query-relevant needle buried among distractors must survive
+query-aware compression (TextCrusher) and must do at least as well as the
+keep-recent / random baselines. Guards the #1171/#1504 CJK TextCrusher path.
+"""
+
+import os
+import sys
+
+import pytest
+
+# benchmarks/ is not a package on the import path by default; add the repo root.
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
+
+from benchmarks.i18n_compression_eval import retention_synthetic # noqa: E402
+
+
[email protected]("lang", ["zh", "ja", "ko"])
+def test_cjk_needle_survives_compression(lang: str) -> None:
+ r = retention_synthetic(lang, ratio=0.3, seed=0)
+ assert r["text_crusher"], f"{lang}: query-relevant needle dropped by TextCrusher"
+
+
[email protected]("lang", ["zh", "ja", "ko"])
+def test_text_crusher_beats_or_ties_baselines(lang: str) -> None:
+ r = retention_synthetic(lang, ratio=0.3, seed=0)
+ assert r["text_crusher"] >= max(r["truncate"], r["random"])