Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(router): a lang code that names no language must not pick a checkpoint
PerryLinkc15b97dbb2b3
diff --git a/README.md b/README.mdindex 507e230e..754591e3 100644--- a/README.md+++ b/README.md@@ -415,7 +415,7 @@ router.predict(state, questions, lang_guess=lambda s: my_lid(s))router = Router(preload=True, lang_guess=my_lid)```-The hint only decides *English or not*: a code whose primary subtag is `en`, `eng` or `english` routes to the English checkpoint and everything else routes to the multilingual one. `"en_US"` and `"en_US.UTF-8"` are read as English, so `$LANG` can be passed straight through. Returning `None`, or an empty code, makes it abstain and the built-in detector decides as before — so a LID model that is unsure does not force a checkpoint. An explicit `model=`, `task=` or `lang=` still wins, and the default path is unchanged.+The hint only decides *English or not*: a code whose primary subtag is `en`, `eng` or `english` routes to the English checkpoint, and every other code that names a language routes to the multilingual one. `"en_US"` and `"en_US.UTF-8"` are read as English, so `$LANG` can be passed straight through. Returning `None`, or a code that names no language, makes it abstain and the built-in detector decides as before — so a LID model that is unsure does not force a checkpoint. `C`, `POSIX` and `C.UTF-8` abstain, which matters because `C.UTF-8` is the default `$LANG` in the official Python image: passing it through no longer pins every request to the multilingual checkpoint, which is what it used to do. The ISO 639-2 special codes `und`, `zxx` and `mul` abstain for the same reason. An explicit `model=`, `task=` or `lang=` still wins, and the default path is unchanged.---diff --git a/laya/router.py b/laya/router.pyindex 6becc646..d446c66a 100644--- a/laya/router.py+++ b/laya/router.py@@ -137,16 +137,26 @@ def _question_schema(questions: Dict[str, Any]) -> str:# Subtags that mean "the English checkpoint can read this". Routing needs one bit -- is this# English Latin text, or something the English checkpoint cannot read -- not a language id, so-# every other code resolves to the multilingual checkpoint.+# every other code that names a language resolves to the multilingual checkpoint._ENGLISH_SUBTAGS = ("en", "eng", "english")+# Codes that are valid `$LANG` values but name no language, so they answer nothing about the+# state. `C`, `POSIX` and `C.UTF-8` are what minimal images ship -- `C.UTF-8` is the default+# `LANG` in the official Python image, which is where `laya-serve` runs -- and the ISO 639-2+# special codes say the same thing in the standard's own vocabulary: `und` undetermined,+# `zxx` no linguistic content, `mul` multiple languages. They abstain, which is what the blank+# case below already does, rather than forcing the multilingual checkpoint on English text.+_LANGUAGE_AGNOSTIC_CODES = ("c", "posix", "und", "zxx", "mul")+def _english_from_code(value: Any) -> Optional[bool]:"""True/False for a language code, or None when the code identifies nothing.Accepts the forms a caller is likely to have to hand: `"en"`, `"EN"`, `"en-US"`, thePOSIX `"en_US"` (which `$LANG` holds), and `"en_US.UTF-8"`. `None` here means "no usable- hint", which is what lets a language-identification model abstain.+ hint", which is what lets a language-identification model abstain -- and it is also what a+ code that names no language returns, so `LANG=C` falls through to detection instead of+ pinning every request to one checkpoint."""if value is None:return None@@ -155,7 +165,7 @@ def _english_from_code(value: Any) -> Optional[bool]:return Nonecode = code.split(".", 1)[0] # en_US.UTF-8 -> en_USprimary = code.replace("_", "-").split("-", 1)[0] # en_US -> en- if not primary:+ if not primary or primary in _LANGUAGE_AGNOSTIC_CODES:return Nonereturn primary in _ENGLISH_SUBTAGSdiff --git a/tests/test_router.py b/tests/test_router.pyindex 24d65d90..81f6fb6a 100644--- a/tests/test_router.py+++ b/tests/test_router.py@@ -13,6 +13,7 @@ from laya.router import ( # noqa: E402BUNDLE_REPO,DEFAULT_MODELS,STANDALONE_MODELS,+ _english_from_code,_repo_str,Router,match_typed_decisions_workflow,@@ -820,6 +821,45 @@ for label, text, script in ():check("unlisted/regression " + label + " script", detect_script(text), script)+# ------------------------------------------------- lang codes that name no language (#359)+# `C`, `POSIX` and `C.UTF-8` are valid `$LANG` values that identify nothing, and `C.UTF-8` is+# the default in the official Python image -- which is where `laya-serve` runs. Passing one to+# `Router(lang=...)` used to resolve to "not English" and pin every request to the multilingual+# checkpoint, before detection ever ran. The blank-string case below already abstained; these+# codes are the same kind of non-answer, so they abstain too. The ISO 639-2 special codes say+# the same thing in the standard's vocabulary.+_ENGLISH_STATE = "Please refund the duplicate charge on invoice 4411"+for code in ("C", "POSIX", "C.UTF-8", "c.utf8", "c", "posix",+ "und", "zxx", "mul", "UND", "Zxx", " und "):+ check("lang-code/%r abstains at the code level" % code, _english_from_code(code), None)+ decision = Router().route(_ENGLISH_STATE, lang=code)+ check("lang-code/%r lets detection name the checkpoint" % code, decision.model, "english")+ check("lang-code/%r says the hint was not used" % code,+ "explicit" in str(decision.reason), False)++# An empty hint is the case this mirrors, so it must still behave the same way.+check("lang-code/empty string still abstains", _english_from_code(""), None)+check("lang-code/None still abstains", _english_from_code(None), None)+check("lang-code/whitespace still abstains", _english_from_code(" "), None)++# The change must not touch codes that do name a language: English still routes now, and a+# non-English code still forces the multilingual checkpoint rather than being second-guessed.+for code in ("en", "eng", "english", "EN", "en-US", "en_US.UTF-8"):+ check("lang-code/%r is still decisive English" % code, _english_from_code(code), True)+ check("lang-code/%r routes without detection" % code,+ Router().route(_ENGLISH_STATE, lang=code).reason.count("explicit"), 1)+for code in ("de", "fr", "zh", "ja", "pt-BR", "de_DE.UTF-8"):+ check("lang-code/%r is still decisive non-English" % code, _english_from_code(code), False)+ check("lang-code/%r routes to the multilingual checkpoint" % code,+ Router().route(_ENGLISH_STATE, lang=code).model, "multilingual")++# A subtag after an agnostic primary is not itself agnostic: `C` names nothing, but the primary+# subtag is what is compared, so a hypothetical `C-something` also abstains and is not treated+# as a language by accident.+check("lang-code/agnostic primary wins over its subtag", _english_from_code("C.UTF-8"), None)+check("lang-code/posix with a modifier abstains", _english_from_code("POSIX-1"), None)++# --------------------------------------------------------------------- reportprint("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL: