Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

perf(lang): skip lines too short to be detected in the per-leaf scan

Adeshina Lasisiba2d973c7c9c
diff --git a/laya/lang.py b/laya/lang.py
index d457940b..87a06a1b 100644
--- a/laya/lang.py
+++ b/laya/lang.py
@@ -550,6 +550,16 @@ def _leaf_non_english(leaf: str) -> Optional[Dict[str, object]]:
best_n = -1
best: Optional[Dict[str, object]] = None
for line in leaf.split("\n"):
+ # A line this short cannot be selected, so skip it before the slice, `strip`,
+ # `_CODE_LINE` and a full `_analyse_text` pass. Each branch below needs four `_WORD`
+ # tokens or NON_LATIN_MIN_LETTERS letters. `_WORD` matches maximal runs of letters and
+ # letter-like numerals, so four tokens need three separators between them: seven
+ # characters, and ten letters need ten. What makes seven safe rather than six is that
+ # every one of those counts is taken on the raw line -- `İ` lowers to two code points, so
+ # counting on `sample.lower()` would let a four-character line reach four tokens, which is
+ # why `latin_profile` replaces `İ` before lowering.
+ if len(line) < 7:
+ continue
sample = line[:4000]
if not sample.strip() or _CODE_LINE.search(sample):
continue
diff --git a/tests/test_router.py b/tests/test_router.py
index 558d0091..d243aafa 100644
--- a/tests/test_router.py
+++ b/tests/test_router.py
@@ -197,6 +197,70 @@ _de_long = {
check("route/dict german after long english note", _r_de.route(_de_long, {}).model, "multilingual")
check("route/dict german after long english note names de",
_r_de.route(_de_long, {})["detection"]["language"], "de")
+# The leaf scan #384 added runs a full `_analyse_text` pass on every line of every value. A line
+# too short to be selected is now skipped before that happens. The #384 checks above pin the
+# answers; these add the cost side, the constant itself, and the invariant a cheaper scan must
+# not trade away.
+#
+# Ratios to prose of the same size, never wall-clock, so a loaded runner inflates both sides.
+# Warm-up then best of nine: the numerators are ~2 ms windows against a ~24 ms denominator, and
+# at five reps a single descheduled run moved the 10 000-field ratio to 0.385. Nine reps holds it.
+# Previous / ceiling / current, each ceiling the geometric mean of the pair it separates:
+# one-char lines, 50 000 chars 2.85 / 0.80 / 0.23
+# six-char lines, 50 000 chars 1.24 / 0.35 / 0.10
+# 10 000 six-character fields 1.85 / 0.68 / 0.25
+# Worst current ratio over six trials was 0.232 / 0.098 / 0.249 idle and 0.231 / 0.097 / 0.246
+# with twelve busy processes on the box, so every ceiling keeps at least 2.7x of room under load;
+# reverting gives 3.19 / 1.25 / 2.27 under that same load, 3.3x or more the other way. The
+# measured ratio goes in the check name, so a red CI log says what it was, not only "got False".
+def _ms(state, reps=9):
+ analyse(state)
+ analyse(state)
+ best = float("inf")
+ for _ in range(reps):
+ t = _time.perf_counter()
+ analyse(state)
+ best = min(best, _time.perf_counter() - t)
+ return best * 1000
+
+
+# Prose LINES, not one long line: `_leaf_non_english` slices each line to 4000 characters, so a
+# 50 000-character one-liner does a fraction of the work and would be a meaningless denominator
+# (2.2 ms against 24.1 ms for the same characters as lines). Every state here is a dict on
+# purpose: `analyse` returns a plain string's verdict before the leaf scan runs.
+_PROSE_50K = {"body": "\n".join(["The customer was billed twice and wants a refund."] * 1020)[:50_000]}
+_prose_ms = _ms(_PROSE_50K)
+for _name, _state, _ceiling in (
+ ("one-char lines", {"body": ("a\n" * 25_000)[:50_000]}, 0.80),
+ ("six-char lines", {"body": ("abcdef\n" * 7_142)[:50_000]}, 0.35),
+ ("10k six-char fields", {"f%d" % i: "abcdef" for i in range(10_000)}, 0.68),
+):
+ _ratio = _ms(_state) / _prose_ms
+ check("leaf scan/%s vs prose = %.2f, ceiling %.2f" % (_name, _ratio, _ceiling),
+ _ratio < _ceiling, True)
+
+# The constant. Seven is the largest sound threshold: each branch of `_leaf_non_english` needs
+# four `_WORD` tokens -- maximal runs of letters and letter-like numerals, so four need three
+# separators between them -- or ten letters. "é à ü ö" is exactly seven characters and four
+# tokens, and must still be read: this is the check that pins the threshold, and raising it to 8
+# turns that check red. "é à üö" is six characters and three tokens; no six-character line can be
+# selected at any threshold, so that check is a behaviour pin rather than a second bound -- it
+# passes on unguarded code too, and is here to catch a future change that makes short lines
+# selectable. Both values sit past 4000 characters of English, which is what makes the leaf scan
+# the code under test: before that the segment scan has already answered.
+_EN_PAST_SEGMENT_CAP = "The customer was billed twice and wants a refund. " * 120
+check("leaf scan/a 7-character foreign line is still read",
+ analyse({"note": _EN_PAST_SEGMENT_CAP, "msg": "é à ü ö"})["is_english"], False)
+check("leaf scan/a 6-character line stays English",
+ analyse({"note": _EN_PAST_SEGMENT_CAP, "msg": "é à üö"})["is_english"], True)
+
+# #384's invariant, which a budget shared across the state would break: a long earlier value must
+# not stop a later one from being read. Cheap to keep, and it is the one way a future attempt to
+# bound this scan by total characters would go wrong silently -- the state below would route to
+# the English checkpoint, which BENCHMARKS.md shows collapsing off English.
+check("leaf scan/a later value is still read after a 50k earlier one",
+ analyse({"pad": "x " * 25_000, "msg": "我们三月份被重复收费了两次。"})["is_english"], False)
+
from collections import UserDict # noqa: E402
from types import MappingProxyType # noqa: E402
check("route/userdict german", _r_de.route(UserDict({"message": _DE}), {}).model, "multilingual")