Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(router): a lang code that names no language must not pick a checkpoint

PerryLinkc15b97dbb2b3
diff --git a/README.md b/README.md
index 507e230e..754591e3 100644
--- a/README.md
+++ b/README.md
@@ -415,7 +415,7 @@ router.predict(state, questions, lang_guess=lambda s: my_lid(s))
router = Router(preload=True, lang_guess=my_lid)
```
-The hint only decides *English or not*: a code whose primary subtag is `en`, `eng` or `english` routes to the English checkpoint and everything else routes to the multilingual one. `"en_US"` and `"en_US.UTF-8"` are read as English, so `$LANG` can be passed straight through. Returning `None`, or an empty code, makes it abstain and the built-in detector decides as before — so a LID model that is unsure does not force a checkpoint. An explicit `model=`, `task=` or `lang=` still wins, and the default path is unchanged.
+The hint only decides *English or not*: a code whose primary subtag is `en`, `eng` or `english` routes to the English checkpoint, and every other code that names a language routes to the multilingual one. `"en_US"` and `"en_US.UTF-8"` are read as English, so `$LANG` can be passed straight through. Returning `None`, or a code that names no language, makes it abstain and the built-in detector decides as before — so a LID model that is unsure does not force a checkpoint. `C`, `POSIX` and `C.UTF-8` abstain, which matters because `C.UTF-8` is the default `$LANG` in the official Python image: passing it through no longer pins every request to the multilingual checkpoint, which is what it used to do. The ISO 639-2 special codes `und`, `zxx` and `mul` abstain for the same reason. An explicit `model=`, `task=` or `lang=` still wins, and the default path is unchanged.
---
diff --git a/laya/router.py b/laya/router.py
index 6becc646..d446c66a 100644
--- a/laya/router.py
+++ b/laya/router.py
@@ -137,16 +137,26 @@ def _question_schema(questions: Dict[str, Any]) -> str:
# Subtags that mean "the English checkpoint can read this". Routing needs one bit -- is this
# English Latin text, or something the English checkpoint cannot read -- not a language id, so
-# every other code resolves to the multilingual checkpoint.
+# every other code that names a language resolves to the multilingual checkpoint.
_ENGLISH_SUBTAGS = ("en", "eng", "english")
+# Codes that are valid `$LANG` values but name no language, so they answer nothing about the
+# state. `C`, `POSIX` and `C.UTF-8` are what minimal images ship -- `C.UTF-8` is the default
+# `LANG` in the official Python image, which is where `laya-serve` runs -- and the ISO 639-2
+# special codes say the same thing in the standard's own vocabulary: `und` undetermined,
+# `zxx` no linguistic content, `mul` multiple languages. They abstain, which is what the blank
+# case below already does, rather than forcing the multilingual checkpoint on English text.
+_LANGUAGE_AGNOSTIC_CODES = ("c", "posix", "und", "zxx", "mul")
+
def _english_from_code(value: Any) -> Optional[bool]:
"""True/False for a language code, or None when the code identifies nothing.
Accepts the forms a caller is likely to have to hand: `"en"`, `"EN"`, `"en-US"`, the
POSIX `"en_US"` (which `$LANG` holds), and `"en_US.UTF-8"`. `None` here means "no usable
- hint", which is what lets a language-identification model abstain.
+ hint", which is what lets a language-identification model abstain -- and it is also what a
+ code that names no language returns, so `LANG=C` falls through to detection instead of
+ pinning every request to one checkpoint.
"""
if value is None:
return None
@@ -155,7 +165,7 @@ def _english_from_code(value: Any) -> Optional[bool]:
return None
code = code.split(".", 1)[0] # en_US.UTF-8 -> en_US
primary = code.replace("_", "-").split("-", 1)[0] # en_US -> en
- if not primary:
+ if not primary or primary in _LANGUAGE_AGNOSTIC_CODES:
return None
return primary in _ENGLISH_SUBTAGS
diff --git a/tests/test_router.py b/tests/test_router.py
index 24d65d90..81f6fb6a 100644
--- a/tests/test_router.py
+++ b/tests/test_router.py
@@ -13,6 +13,7 @@ from laya.router import ( # noqa: E402
BUNDLE_REPO,
DEFAULT_MODELS,
STANDALONE_MODELS,
+ _english_from_code,
_repo_str,
Router,
match_typed_decisions_workflow,
@@ -820,6 +821,45 @@ for label, text, script in (
):
check("unlisted/regression " + label + " script", detect_script(text), script)
+# ------------------------------------------------- lang codes that name no language (#359)
+# `C`, `POSIX` and `C.UTF-8` are valid `$LANG` values that identify nothing, and `C.UTF-8` is
+# the default in the official Python image -- which is where `laya-serve` runs. Passing one to
+# `Router(lang=...)` used to resolve to "not English" and pin every request to the multilingual
+# checkpoint, before detection ever ran. The blank-string case below already abstained; these
+# codes are the same kind of non-answer, so they abstain too. The ISO 639-2 special codes say
+# the same thing in the standard's vocabulary.
+_ENGLISH_STATE = "Please refund the duplicate charge on invoice 4411"
+for code in ("C", "POSIX", "C.UTF-8", "c.utf8", "c", "posix",
+ "und", "zxx", "mul", "UND", "Zxx", " und "):
+ check("lang-code/%r abstains at the code level" % code, _english_from_code(code), None)
+ decision = Router().route(_ENGLISH_STATE, lang=code)
+ check("lang-code/%r lets detection name the checkpoint" % code, decision.model, "english")
+ check("lang-code/%r says the hint was not used" % code,
+ "explicit" in str(decision.reason), False)
+
+# An empty hint is the case this mirrors, so it must still behave the same way.
+check("lang-code/empty string still abstains", _english_from_code(""), None)
+check("lang-code/None still abstains", _english_from_code(None), None)
+check("lang-code/whitespace still abstains", _english_from_code(" "), None)
+
+# The change must not touch codes that do name a language: English still routes now, and a
+# non-English code still forces the multilingual checkpoint rather than being second-guessed.
+for code in ("en", "eng", "english", "EN", "en-US", "en_US.UTF-8"):
+ check("lang-code/%r is still decisive English" % code, _english_from_code(code), True)
+ check("lang-code/%r routes without detection" % code,
+ Router().route(_ENGLISH_STATE, lang=code).reason.count("explicit"), 1)
+for code in ("de", "fr", "zh", "ja", "pt-BR", "de_DE.UTF-8"):
+ check("lang-code/%r is still decisive non-English" % code, _english_from_code(code), False)
+ check("lang-code/%r routes to the multilingual checkpoint" % code,
+ Router().route(_ENGLISH_STATE, lang=code).model, "multilingual")
+
+# A subtag after an agnostic primary is not itself agnostic: `C` names nothing, but the primary
+# subtag is what is compared, so a hypothetical `C-something` also abstains and is not treated
+# as a language by accident.
+check("lang-code/agnostic primary wins over its subtag", _english_from_code("C.UTF-8"), None)
+check("lang-code/posix with a modifier abstains", _english_from_code("POSIX-1"), None)
+
+
# --------------------------------------------------------------------- report
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL: