Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(lang): add opt-in detection reasons and source diagnostics
xpeb494474ff69
diff --git a/docs/routing.md b/docs/routing.mdindex a9a452f9..ff63e8fd 100644--- a/docs/routing.md+++ b/docs/routing.md@@ -70,6 +70,99 @@ print(decision.reason) # why that checkpoint was selected`decision["model"]` and `decision["reason"]`. `Router.predict()` includes the same decision underthe result's `"routing"` key.+## Explain a detection result++Use `explain=True` to inspect why built-in detection reached its result, without loading a+checkpoint. The existing detection fields and routing rules stay unchanged; only the optional+`diagnostic` field is added.++```python+from laya import Router+from laya.lang import analyse++detection = analyse("Turn lights off", explain=True)+print(detection["diagnostic"])+# {"stage": "whole_state", "reason": "insufficient_words", "source": None}++router = Router(default="english")+decision = router.route("Turn lights off", explain=True)+assert decision["model"] == "english"+assert decision["detection"]["language_undecided"] is True+```++The detector's reason is separate from the router's policy: an unidentified language follows+the configured `default`, which need not be multilingual. Reasons describe heuristic branches,+not calibrated language confidence. Explicit model/task/language overrides, an accepted+`lang_guess`, and automatic workflow selection still skip detection and return+`detection=None`. As before, an `on_route` hook can replace the whole decision.++For structured input, a later value can change the result even when a long earlier value has+filled the joined-text window:++```python+state = {+ "note": "This is an English note about the order. " * 140,+ "payload": {"customer": "Ich kann mich nicht anmelden und brauche Hilfe"},+}+detection = analyse(state, explain=True)+assert detection["language"] == "de"+assert detection["mixed_segment"] is None # its existing meaning is unchanged+assert detection["diagnostic"] == {+ "stage": "value",+ "reason": "named_value",+ "source": {"leaf_index": 1, "path": ["payload", "customer"], "line": 1},+}+```++`stage` is `whole_state` for the initial joined sample, `segment` for the first qualifying+line in the shared scan window, or `value` for the subsequent per-value fallback. Only the+selected result is reported; this is not a trace of every candidate.+An override changes only the fields it changed before: for example, a late non-Latin value can+make `is_english=False` while `script_profile` still describes the initial Latin sample. The+diagnostic identifies the override's source; it does not relabel those older measurements.++| Reason | Meaning |+|---|---|+| `no_letters` | The initial sample contains no alphabetic characters. |+| `non_latin_script` | The script check selected a non-Latin script, including mixed-script promotion. |+| `insufficient_words` | The Latin language guess returned before general scoring because it had fewer than four words. |+| `no_language_evidence` | General word scoring did not identify a language. |+| `language_evidence` | Word evidence identified a language; see the existing `language` field. |+| `short_swedish` | The existing short Swedish phrase rule identified the language. |+| `caps_veto` | The all-capitals evidence check rejected the language name. |+| `caps_filtered` | The language name was reconsidered with acronym-shaped capitals removed. |+| `non_english_markers` | No language was named, but diacritics or a shared Nordic marker still made the text non-English. This takes precedence over an inconclusive word/caps reason. |+| `named_segment` | A named non-English prose segment overrode the joined result. |+| `named_value` | The selected fallback line passed the named-language prose check. |+| `non_latin_value` | The selected fallback line passed the non-Latin word/letter check. |+| `diacritic_value` | The selected fallback line passed the diacritic and word-count check. |++For `whole_state`, `source` is `None`: the joined sample need not correspond to a single field.+Other stages report a zero-based `leaf_index` in the existing string-leaf traversal and a+one-based `line` within that decoded leaf. `path` is an array of mapping keys and sequence+indices, so keys containing dots or slashes remain unambiguous; a root string has path `[]`.+The detector's existing depth and character limits still apply. Invalid UTF-8 byte leaves and+non-text values do not contribute a leaf index.++A path retains string keys of at most 128 characters and integer keys with at most 63 bits of+magnitude, including negative integers. If a key anywhere on that path is unsupported or exceeds+those bounds, the entire path is `None`; the leaf index and line remain available. No arbitrary+key is stringified, and the diagnostic adds no input text. The existing `mixed_segment` can+still contain text from the segment scan, as it did before.++`Router.route_batch(requests, explain=True)` applies the same option to the whole routing batch,+preserving order. The CLI exposes it in route-only mode:++```bash+laya "Turn lights off" --explain --json+laya --batch tickets.txt --explain --json+```++The flag also works in interactive mode and ordinary text output. Combining it with+`--predict`, `--preset` or `--questions` is rejected: diagnostics belong to the routing call,+not a separate detector pass attached to an inference result.++## Override language routingUse `lang=` when the application already knows the request language. Language tags such asdiff --git a/laya/cli.py b/laya/cli.pyindex 4b31d5c7..24121be6 100644--- a/laya/cli.py+++ b/laya/cli.py@@ -130,6 +130,8 @@ def build_parser():" rather than a refusal. Needs --predict, --preset or --questions.")parser.add_argument("--device", help="torch device, e.g. cpu or cuda")parser.add_argument("--json", action="store_true", help="print the raw result as JSON")+ parser.add_argument("--explain", action="store_true",+ help="include language-detection reasons and sources in routing mode (no model loading)")parser.add_argument("--batch", metavar="FILE",help="score a file of requests, one per line (use '-' for stdin), instead ""of a single text; implies neither --predict nor --preset, but the "@@ -284,7 +286,8 @@ def run(text, args, router=None):else:state = {"text": text}decision = router.route(state, model=args.model, task=args.task, lang=args.lang,- lang_guess=args.lang_guess)+ lang_guess=args.lang_guess,+ **({"explain": True} if args.explain else {}))if args.json:print(json.dumps(dict(decision), ensure_ascii=False, indent=2, default=str))else:@@ -407,10 +410,13 @@ def run_batch(lines, args, router=None):print()else:decisions = router.route_batch([{"state": {"text": line}, "questions": {},- **overrides} for line in lines])+ **overrides} for line in lines],+ **({"explain": True} if args.explain else {}))for line, decision in zip(lines, decisions):if args.json:print(json.dumps(dict(decision), ensure_ascii=False, default=str))+ elif args.explain:+ show_decision(decision)else:print("%-14s %s <- %s" % (decision["model"], decision["reason"],line if len(line) <= 48 else line[:45] + "..."))@@ -460,6 +466,10 @@ def main(argv=None):if hasattr(sys.stdout, "reconfigure"):sys.stdout.reconfigure(encoding="utf-8")text = " ".join(args.text).strip()+ if args.explain and (args.predict or args.preset or args.questions):+ print("laya: --explain is for routing only; omit --predict, --preset and --questions.",+ file=sys.stderr)+ return 2# Routing answers nothing, so there is no confidence to gate: `route` does not take the# threshold at all. Accepting the flag there would let `laya --min-confidence 0.9 "..."` exit 0# having ignored it, which is the thing this flag exists to avoid.diff --git a/laya/lang.py b/laya/lang.pyindex 58ea55ea..28a76aa3 100644--- a/laya/lang.py+++ b/laya/lang.py@@ -239,7 +239,8 @@ _WORD = re.compile(r"[^\W\d_]+", re.UNICODE)_IDENTIFIER = re.compile(r"(?<![\w-])[\w-]*(?:[.@][\w-]+)+", re.UNICODE)-def _iter_text(state: Union[str, bytes, Mapping, list, None], _depth: int = 0) -> List[str]:+def _iter_text(state: Union[str, bytes, Mapping, list, None], _depth: int = 0,+ _paths=None, _path=()) -> List[str]:"""Collect the string leaves of a state, so detection sees real content.Keys are ignored: they are usually English field names. Any mapping counts, not only@@ -249,21 +250,35 @@ def _iter_text(state: Union[str, bytes, Mapping, list, None], _depth: int = 0) -if _depth > 6 or state is None:return []if isinstance(state, str):+ if _paths is not None:+ _paths.append(_path)return [state]if isinstance(state, (bytes, bytearray)):try:- return [bytes(state).decode("utf-8")]+ text = bytes(state).decode("utf-8")except UnicodeDecodeError:return []+ if _paths is not None:+ _paths.append(_path)+ return [text]if isinstance(state, Mapping):out = []- for v in state.values():- out.extend(_iter_text(v, _depth + 1))+ if _paths is None:+ for v in state.values():+ out.extend(_iter_text(v, _depth + 1))+ else:+ for key, v in state.items():+ # Do not stringify arbitrary keys or copy an unbounded field name into a report.+ bounded = ((type(key) is str and len(key) <= 128)+ or (type(key) is int and key.bit_length() <= 63))+ path = _path + (key,) if _path is not None and bounded else None+ out.extend(_iter_text(v, _depth + 1, _paths, path))return outif isinstance(state, (list, tuple)):out = []- for v in state:- out.extend(_iter_text(v, _depth + 1))+ for i, v in enumerate(state):+ path = _path + (i,) if _paths is not None and _path is not None else _path+ out.extend(_iter_text(v, _depth + 1, _paths, path))return outreturn []@@ -442,6 +457,10 @@ def latin_profile(text: str) -> Dict[str, object]:A non-English language is only named when it matched at least one word that no other listclaims: shared function words alone (`la`, `e`, `o`) identify no particular language."""+ return _latin_profile(text)+++def _latin_profile(text: str, diagnostic=None) -> Dict[str, object]:# 'İ'.lower() is 'i' + a combining dot, which matches no word listwords = _WORD.findall(_IDENTIFIER.sub(" ", text).replace("İ", "i").lower())lowered = text.lower()@@ -450,9 +469,13 @@ def latin_profile(text: str) -> Dict[str, object]:non_english = diac_rate >= NON_EN_DIACRITIC_RATEnordic_overlap = bool(set(words) & _NORDIC_OVERLAP_WORDS) and not bool(set(words) & _EN_ONLY_WORDS)if 1 < len(words) < 4 and set(words) & _SHORT_SWEDISH_WORDS:+ if diagnostic is not None:+ diagnostic["reason"] = "short_swedish"return {"language": "sv", "english_hits": 0, "diacritic_rate": diac_rate,"looks_non_english": non_english}if len(words) < 4:+ if diagnostic is not None:+ diagnostic["reason"] = "insufficient_words"return {"language": None, "english_hits": 0, "diacritic_rate": diac_rate,"looks_non_english": non_english or nordic_overlap}@@ -486,6 +509,8 @@ def latin_profile(text: str) -> Dict[str, object]:lang = best_lgelif en and (not non_english or _english_rescued_by_words(words, diac_rate)):lang = "en"+ if diagnostic is not None:+ diagnostic["reason"] = "language_evidence" if lang else "no_language_evidence"return {"language": lang, "english_hits": en, "diacritic_rate": diac_rate,"looks_non_english": non_english or (lang is None and nordic_overlap)}@@ -639,7 +664,8 @@ def _named_prose_language(segment: str):return lang-def _non_english_segment(state: Union[str, bytes, Mapping, list, None], max_chars: int = 4000):+def _non_english_segment(state: Union[str, bytes, Mapping, list, None], max_chars: int = 4000,+ diagnostic=None):"""First line or field that, read on its own, is named a non-English language, else None.Returns (language, segment). A segment needs the evidence a whole state needs -- at least four@@ -650,19 +676,28 @@ def _non_english_segment(state: Union[str, bytes, Mapping, list, None], max_charlonger English part from outvoting a foreign one. Reads at most `max_chars` characters in all."""seen = 0- for leaf in _iter_text(state):- for seg in leaf.split("\n"):+ paths = [] if diagnostic is not None else None+ for index, leaf in enumerate(_iter_text(state, _paths=paths)):+ for line, seg in enumerate(leaf.split("\n"), 1):if seen >= max_chars:return Noneseg = seg[:max_chars - seen]seen += len(seg)lang = _named_prose_language(seg)if lang:+ if diagnostic is not None:+ diagnostic.update(stage="segment", reason="named_segment",+ source=_diagnostic_source(paths, index, line))return lang, seg.strip()return None-def _analyse_text(text: str) -> Dict[str, object]:+def _diagnostic_source(paths, index, line):+ path = paths[index]+ return {"leaf_index": index, "path": list(path) if path is not None else None, "line": line}+++def _analyse_text(text: str, diagnostic=None) -> Dict[str, object]:"""Detection result for one already-flattened string.Does not look for a foreign line inside mostly-English text. `analyse` does that, because it@@ -677,14 +712,18 @@ def _analyse_text(text: str) -> Dict[str, object]:and _non_latin_carries(non_latin, n_non_latin)):script = max((s for s in prof if s != "latin"), key=prof.get)if script == "unknown":+ if diagnostic is not None:+ diagnostic["reason"] = "no_letters"return {"script": "unknown", "script_profile": prof, "language": None,"is_english": True, "language_undecided": True, "diacritic_rate": 0.0,"non_latin_fraction": 0.0, "mixed_segment": None}if script != "latin":+ if diagnostic is not None:+ diagnostic["reason"] = "non_latin_script"return {"script": script, "script_profile": prof, "language": None,"is_english": False, "language_undecided": True, "diacritic_rate": 0.0,"non_latin_fraction": non_latin, "mixed_segment": None}- prof_lat = latin_profile(text)+ prof_lat = _latin_profile(text, diagnostic)lang = prof_lat["language"]# The acronym bar belongs here too, not only in the segment scan. The scan runs only once a# state already reads English overall, so a state that is *nothing but* an acronym line --@@ -707,6 +746,8 @@ def _analyse_text(text: str) -> Dict[str, object]:if not _shouted_evidence(_WORD.findall(text), lang,float(prof_lat["diacritic_rate"])):lang = None+ if diagnostic is not None:+ diagnostic["reason"] = "caps_veto"else:# Mixed-case text: the segment scan has always held that an acronym is not a word,# but the whole-state verdict never applied that rule, so the acronyms voted in it.@@ -734,19 +775,23 @@ def _analyse_text(text: str) -> Dict[str, object]:lambda m: " " if m.group().isupper() else m.group(), text)if blanked != text:lang = latin_profile(blanked)["language"]+ if diagnostic is not None:+ diagnostic["reason"] = "caps_filtered"# Undecided is not English. Treating it as English sent every Latin-script language we hold no# stopwords for to the checkpoint that cannot read it, silently. When nothing identifies the# language, non-English letters or a shared Swedish-Danish marker can still prefer the# multilingual checkpoint; text with neither signal still goes to the English one.undecided = lang is Noneenglish = lang == "en" or (undecided and not prof_lat["looks_non_english"])+ if diagnostic is not None and undecided and not english:+ diagnostic["reason"] = "non_english_markers"return {"script": "latin", "script_profile": prof, "language": lang,"is_english": english, "language_undecided": undecided,"diacritic_rate": round(float(prof_lat["diacritic_rate"]), 4),"non_latin_fraction": non_latin, "mixed_segment": None}-def _leaf_non_english(leaf: str) -> Optional[Dict[str, object]]:+def _leaf_non_english(leaf: str, diagnostic=None) -> Optional[Dict[str, object]]:"""A string value that is itself not safe for the English checkpoint, else None.A one-word name ("José") and a capitalised non-Latin name stay out: the same rules@@ -758,7 +803,7 @@ def _leaf_non_english(leaf: str) -> Optional[Dict[str, object]]:"""best_n = -1best: Optional[Dict[str, object]] = None- for line in leaf.split("\n"):+ for line_number, line in enumerate(leaf.split("\n"), 1):# A line this short cannot be selected, so skip it before the slice, `strip`,# `_CODE_LINE` and a full `_analyse_text` pass. Each branch below needs four `_WORD`# tokens or NON_LATIN_MIN_LETTERS letters. `_WORD` matches maximal runs of letters and@@ -802,10 +847,13 @@ def _leaf_non_english(leaf: str) -> Optional[Dict[str, object]]:if n_alpha > best_n:best_n = n_alphabest = det+ if diagnostic is not None:+ diagnostic.update(line=line_number, reason=("named_value" if named else+ "non_latin_value" if det["script"] not in ("latin", "unknown") else "diacritic_value"))return best-def analyse(state: Union[str, bytes, Mapping, list, None]) -> Dict[str, object]:+def analyse(state: Union[str, bytes, Mapping, list, None], *, explain: bool = False) -> Dict[str, object]:"""Full detection result for a state.Returns `script`, `script_profile`, `language` (best effort, may be None),@@ -819,8 +867,18 @@ def analyse(state: Union[str, bytes, Mapping, list, None]) -> Dict[str, object]:or outvote a short German message, and that message was then sent to the English checkpoint(#384). The segment scan still stops at 4000 characters, which is what keeps a huge fieldcheap; a value it did not reach is read on its own afterwards.++ With `explain=True`, adds `diagnostic`: the selected analysis stage, its reason and, for a+ segment or value override, the source leaf index, path and 1-based line. Paths retain string+ keys (up to 128 characters), integers (up to 63 bits) and sequence indices. Other keys make+ the path None; the leaf index remains available. No input text is added to the diagnostic.+ The original fields and detection rules are unchanged. These are heuristic reasons, not+ calibrated confidence or the router's final model choice."""- result = _analyse_text(state_text(state))+ diagnostic = {"stage": "whole_state", "reason": None, "source": None} if explain else None+ result = _analyse_text(state_text(state), diagnostic)+ if diagnostic is not None:+ result["diagnostic"] = diagnosticif result["script"] == "latin" and result["is_english"]:# A Portuguese ticket with an English stack trace, error payload or form template reads as# English as a whole, because the English part is longer -- yet the part a question is about@@ -831,7 +889,7 @@ def analyse(state: Union[str, bytes, Mapping, list, None]) -> Dict[str, object]:leaves = _iter_text(state)# a single line has no other part to be outvoted by, and was just read wholeif len(leaves) > 1 or any("\n" in leaf for leaf in leaves):- found = _non_english_segment(state)+ found = _non_english_segment(state, diagnostic=diagnostic)if found:lang, mixed = foundresult = dict(result)@@ -845,20 +903,29 @@ def analyse(state: Union[str, bytes, Mapping, list, None]) -> Dict[str, object]:return resultbest_n = -1best: Optional[Dict[str, object]] = None- for leaf in _iter_text(state):- det = _leaf_non_english(leaf)+ paths = [] if explain else None+ best_source = None+ best_reason = None+ for index, leaf in enumerate(_iter_text(state, _paths=paths)):+ leaf_diagnostic = {} if explain else None+ det = _leaf_non_english(leaf, leaf_diagnostic)if det is None:continuen_alpha = sum(ch.isalpha() for ch in leaf[:4000])if n_alpha > best_n:best_n = n_alphabest = det+ if explain:+ best_source = _diagnostic_source(paths, index, leaf_diagnostic["line"])+ best_reason = leaf_diagnostic["reason"]if best is None:return resultupdated = dict(result)updated["language"] = best["language"]updated["is_english"] = Falseupdated["language_undecided"] = best["language_undecided"]+ if diagnostic is not None:+ diagnostic.update(stage="value", reason=best_reason, source=best_source)return updateddiff --git a/laya/router.py b/laya/router.pyindex 4113ba3f..f538d589 100644--- a/laya/router.py+++ b/laya/router.py@@ -1183,16 +1183,22 @@ class Router(HookRegistry):hooks=None,hooks_raise: Optional[bool] = None,hooks_timeout: Optional[float] = None,+ *,+ explain: bool = False,) -> RouteDecision:"""Decide which checkpoint to use, then let `on_route` hooks observe or replace it.`ctx.decision` is the `RouteDecision`; a hook may replace it (for example to pin acheckpoint) and the replacement is what gets returned and used. `hooks` are per-callhooks, appended after any installed on the Router.++ `explain=True` adds language diagnostics when built-in detection runs. Explicit routing+ overrides still skip it. A hook can replace the decision, including its diagnostics."""if questions is not None and not isinstance(questions, dict):raise TypeError("questions must be a dict of question id -> definition, got %s" % type(questions).__name__)- decision = self._route(state, questions, model=model, task=task, lang=lang, lang_guess=lang_guess)+ decision = self._route(state, questions, model=model, task=task, lang=lang, lang_guess=lang_guess,+ **({"explain": True} if explain else {}))raise_errors = self.hooks_raise if hooks_raise is None else bool(hooks_raise)active = compose_hooks(self.hooks, hooks)timeout = self.hooks_timeout if hooks_timeout is None else validate_timeout(hooks_timeout)@@ -1208,6 +1214,8 @@ class Router(HookRegistry):task: Optional[str] = None,lang: Optional[str] = None,lang_guess: Optional[Any] = None,+ *,+ explain: bool = False,) -> RouteDecision:"""Decide which checkpoint to use, without loading or running anything.@@ -1261,7 +1269,7 @@ class Router(HookRegistry):source, "English" if resolved else "non-English"),detection=None, workflow=workflow)- det = analyse(state)+ det = analyse(state, explain=True) if explain else analyse(state)if det["script"] == "unknown":key = self.defaultreason = "no letters detected in state; using default (%s)" % key@@ -1534,6 +1542,7 @@ class Router(HookRegistry):*,hooks=None,hooks_raise: Optional[bool] = None,+ explain: bool = False,) -> List[RouteDecision]:"""Route a heterogeneous request batch without loading any checkpoints.@@ -1551,6 +1560,7 @@ class Router(HookRegistry):every request's ``on_route`` dispatch, as on :meth:`route`.hooks (HookArg): Per-call hook or sequence of hooks for this call.hooks_raise: Override the Router's ``hooks_raise`` policy for this call.+ explain: Include built-in language diagnostics for each request that uses detection."""if not isinstance(requests, SequenceABC) or isinstance(requests, (str, bytes)):raise TypeError("requests must be a sequence of request dictionaries")@@ -1584,6 +1594,7 @@ class Router(HookRegistry):hooks=hooks,hooks_raise=hooks_raise,hooks_timeout=hooks_timeout,+ **({"explain": True} if explain else {}),))diff --git a/scripts/test_suites.py b/scripts/test_suites.pyindex a5a6859a..a720e352 100644--- a/scripts/test_suites.py+++ b/scripts/test_suites.py@@ -114,6 +114,7 @@ SCRIPT_SUITES = []PYTEST_SUITES = [+ "tests/test_lang_diagnostics.py","tests/test_evals_agreement.py","tests/test_evals_tournament.py","tests/test_serve.py",diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.pyindex 1a5840d9..3f03f282 100644--- a/tests/test_hooks_api.py+++ b/tests/test_hooks_api.py@@ -243,6 +243,19 @@ check("predict_tournament/positional prefix",check_param("predict_tournament", laya.predict_tournament, "predict_kwargs", inspect.Parameter.empty,inspect.Parameter.VAR_KEYWORD)+# Detection diagnostics are opt-in and do not replace existing analysis fields.+from laya.lang import analyse, latin_profile # noqa: E402++for label, fn in (("analyse", analyse), ("Router.route", Router.route),+ ("Router.route_batch", Router.route_batch)):+ check_param(label, fn, "explain", False, inspect.Parameter.KEYWORD_ONLY)+check("latin_profile keeps its signature", list(sig(latin_profile)), ["text"])+_plain_analysis = analyse("Turn lights off")+_explained_analysis = analyse("Turn lights off", explain=True)+check("analyse opts into one extra key", set(_explained_analysis) - set(_plain_analysis), {"diagnostic"})+check("analyse diagnostic fields", set(_explained_analysis.pop("diagnostic")), {"stage", "reason", "source"})+check("analyse retains original fields", _explained_analysis, _plain_analysis)+# route() and route_batch() take per-call hooks so a hook can pin a checkpoint for one callfor label, fn in (("Router.route", Router.route),("Router.route_batch", Router.route_batch)):diff --git a/tests/test_lang_diagnostics.py b/tests/test_lang_diagnostics.pynew file mode 100644index 00000000..d7a9b4b3--- /dev/null+++ b/tests/test_lang_diagnostics.py@@ -0,0 +1,258 @@+"""Opt-in diagnostics explain the executed detector, without changing its decisions."""+import copy+import json+import os+from pathlib import Path+import subprocess+import sys+from collections import UserDict+from concurrent.futures import ThreadPoolExecutor+from types import MappingProxyType+from unittest.mock import patch++import pytest++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))++from laya import Router, BaseHook # noqa: E402+from laya import lang # noqa: E402++EN = "This is an English note about the order. "+DE = "Ich kann mich nicht anmelden und brauche Hilfe"+PT = "Quero cancelar a minha conta porque não consigo entrar"+++def explained(state):+ plain = lang.analyse(state)+ result = lang.analyse(state, explain=True)+ diagnostic = result.pop("diagnostic")+ assert result == plain+ assert set(diagnostic) == {"stage", "reason", "source"}+ json.dumps(diagnostic, ensure_ascii=False, allow_nan=False)+ return diagnostic++[email protected](("state", "reason"), [+ ("Turn lights off", "insufficient_words"),+ ("xyzzy frobnitz quux blorb", "no_language_evidence"),+ ("I would like to book a flight to Berlin tomorrow", "language_evidence"),+ (DE, "language_evidence"),+ ("glömt lösenord", "short_swedish"),+ ("MON DES EST LA", "caps_veto"),+ ("This is my ticket. MON DES EST LA", "caps_filtered"),+ ("åå øø äa", "non_english_markers"),+ ("请帮我取消订单", "non_latin_script"),+ ("1234", "no_letters"),+ ("", "no_letters"),+ (None, "no_letters"),+ (b"\xff", "no_letters"),+])+def test_whole_state_reason(state, reason):+ assert explained(state) == {"stage": "whole_state", "reason": reason, "source": None}+++def test_original_fields_stay_fixed():+ assert lang.analyse("Turn lights off") == {+ "script": "latin", "script_profile": {"latin": 1.0}, "language": None,+ "is_english": True, "language_undecided": True, "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0, "mixed_segment": None,+ }+ assert set(lang.latin_profile("Turn lights off")) == {+ "language", "english_hits", "diacritic_rate", "looks_non_english",+ }+++def test_segment_keeps_its_original_text_and_identifies_the_first_matching_line():+ state = {"note": EN * 20, "messages": [{"customer": "status\n" + DE + "\n" + PT}]}+ d = explained(state)+ assert d == {"stage": "segment", "reason": "named_segment",+ "source": {"leaf_index": 1, "path": ["messages", 0, "customer"], "line": 2}}+ assert lang.analyse(state)["mixed_segment"] == DE+ # Equal text in another field must not replace the source by searching the state afterwards.+ state["duplicate"] = DE+ assert explained(state)["source"] == d["source"]+++def test_root_string_source_is_a_line_not_a_fabricated_field():+ d = explained(EN * 20 + "\n" + DE)+ assert d["stage"] == "segment"+ assert d["source"] == {"leaf_index": 0, "path": [], "line": 2}++[email protected]("wrap", [dict, UserDict, MappingProxyType])+def test_late_value_records_nested_path_without_changing_mixed_segment(wrap):+ state = wrap({"note": EN * 140, "messages": ({"customer": "status\n" + DE},)})+ d = explained(state)+ assert d == {"stage": "value", "reason": "named_value",+ "source": {"leaf_index": 1, "path": ["messages", 0, "customer"], "line": 2}}+ assert lang.analyse(state)["mixed_segment"] is None+ assert DE not in json.dumps(d)++[email protected](("message", "reason"), [+ ("请帮我取消这个订单并且退款给我", "non_latin_value"),+ ("WIE SPÄT IST ES IN KÖLN", "diacritic_value"),+])+def test_value_eligibility_reason(message, reason):+ d = explained({"note": EN * 140, "customer": message})+ assert d["stage"] == "value"+ assert d["reason"] == reason+ assert d["source"]["path"] == ["customer"]+ assert lang.analyse({"note": EN * 140, "customer": message})["script_profile"] == {"latin": 1.0}+++def test_value_selection_keeps_both_existing_winner_rules():+ # Inner scan chooses the longest qualifying line; outer scan weighs the entire leaf window.+ state = {"note": EN * 140, "short": DE,+ "long": "status\n" + DE + "\n" + PT * 3}+ d = explained(state)+ assert d["source"] == {"leaf_index": 2, "path": ["long"], "line": 3}+ tie = {"note": EN * 140, "first": DE, "second": DE}+ assert explained(tie)["source"]["path"] == ["first"]+ lines_tie = {"note": EN * 140, "first": DE + "\n" + DE}+ assert explained(lines_tie)["source"]["line"] == 1+++class UnprintableKey:+ def __repr__(self):+ raise AssertionError("diagnostics must not stringify arbitrary keys")++[email protected]("key", [UnprintableKey(), ("tuple",), "k" * 129, 2 ** 80, True])+def test_unrepresentable_path_retains_leaf_index(key):+ d = explained({"note": EN * 140, key: {"customer": DE}})+ assert d["source"] == {"leaf_index": 1, "path": None, "line": 1}++[email protected]("key", ["k" * 128, -10, "a/b~c\nfield"])+def test_bounded_keys_are_preserved_as_path_components(key):+ d = explained({"note": EN * 140, key: DE})+ assert d["source"]["path"] == [key]+++def test_decode_depth_and_window_limits_are_unchanged():+ d = explained({"note": EN * 140, "bad": b"\xff", "good": DE.encode()})+ assert d["source"] == {"leaf_index": 1, "path": ["good"], "line": 1}+ at_limit = DE+ for _ in range(5):+ at_limit = [at_limit]+ d = explained({"note": EN * 140, "deep": at_limit})+ assert d["source"]["path"] == ["deep", 0, 0, 0, 0, 0]+ beyond = {"note": EN * 140, "deep": [at_limit]}+ assert explained(beyond)["source"] is None+ # The fallback reads each line's first 4000 characters, not an unbounded field.+ assert explained({"note": EN * 140, "hidden": EN * 140 + DE})["source"] is None+++def test_diagnostics_are_per_call_and_do_not_mutate_state():+ states = [{"note": EN * 140, "customer": DE}, "Turn lights off", "MON DES EST LA"]+ original = copy.deepcopy(states)+ expected = [lang.analyse(s, explain=True) for s in states]+ with ThreadPoolExecutor(max_workers=4) as executor:+ actual = list(executor.map(lambda s: lang.analyse(s, explain=True), states * 8))+ assert actual == expected * 8+ assert states == original+ actual[0]["diagnostic"]["source"]["path"].append("changed")+ assert lang.analyse(states[0], explain=True) == expected[0]+++def test_explain_does_not_run_a_second_detection_pass():+ state = {"note": EN * 140, "customer": DE}+ with patch.object(lang, "_analyse_text", wraps=lang._analyse_text) as detector:+ lang.analyse(state)+ plain_calls = detector.call_args_list+ with patch.object(lang, "_analyse_text", wraps=lang._analyse_text) as detector:+ lang.analyse(state, explain=True)+ assert [c.args[0] for c in detector.call_args_list] == [c.args[0] for c in plain_calls]++[email protected]("default", ["english", "multilingual"])+def test_router_keeps_fallback_and_never_loads_models(default):+ router = Router(default=default)+ with patch.object(router, "load", side_effect=AssertionError("must not load")):+ plain = router.route("Turn lights off")+ result = router.route("Turn lights off", explain=True)+ assert result["model"] == default+ assert result["detection"].pop("diagnostic")["reason"] == "insufficient_words"+ assert result == plain++[email protected]("override", [+ {"model": "english"}, {"task": "typed-decisions"}, {"lang": "de"}, {"lang_guess": "en"},+])+def test_explicit_routes_do_not_invent_detector_evidence(override):+ router = Router()+ with patch("laya.router.analyse", side_effect=AssertionError("detector must be skipped")):+ result = router.route(DE, explain=True, **override)+ assert result["detection"] is None+++def test_abstaining_hint_runs_detection_once():+ calls = []+ def hint(state):+ calls.append(state)+ return None+ result = Router(lang_guess=hint).route("Turn lights off", explain=True)+ assert calls == ["Turn lights off"]+ assert result["detection"]["diagnostic"]["reason"] == "insufficient_words"+++def test_automatic_workflow_selection_skips_detection():+ questions = {key: {"type": "bool"} for key in+ ("action", "category", "churn_risk", "needs_human", "urgency")}+ with patch("laya.router.analyse", side_effect=AssertionError("detector must be skipped")):+ result = Router(auto_task_detection=True).route(DE, questions, explain=True)+ assert result["workflow"] == "customer_service"+ assert result["detection"] is None+++def test_hooks_observe_and_may_replace_the_actual_decision():+ seen = []+ class Hook(BaseHook):+ def on_route(self, ctx):+ seen.append(ctx.decision["detection"]["diagnostic"]["reason"])+ ctx.decision = {"model": "custom", "reason": "hook"}+ assert Router().route("Turn lights off", explain=True, hooks=Hook()) == {+ "model": "custom", "reason": "hook"}+ assert seen == ["insufficient_words"]+++def test_batch_uses_same_explanation_and_order():+ states = ["Turn lights off", {"note": EN * 140, "customer": DE}]+ router = Router()+ batch = router.route_batch([{"state": s, "questions": {}} for s in states], explain=True)+ assert batch == [router.route(s, explain=True) for s in states]+ assert all("diagnostic" not in r["detection"] for r in router.route_batch(+ [{"state": s, "questions": {}} for s in states]))+++def cli(*args, stdin=None):+ return subprocess.run([sys.executable, "-m", "laya.cli", *args],+ cwd=Path(__file__).resolve().parents[1], input=stdin, text=True, encoding="utf-8",+ capture_output=True, timeout=45,+ env={**os.environ, "PYTHONUTF8": "1", "HF_HUB_OFFLINE": "1", "TRANSFORMERS_OFFLINE": "1"})+++def test_cli_real_process_json_text_batch_and_interactive():+ result = cli("Turn lights off", "--explain", "--json")+ assert result.returncode == 0, result.stderr+ assert json.loads(result.stdout)["detection"]["diagnostic"]["reason"] == "insufficient_words"+ result = cli("Turn lights off", "--explain")+ assert result.returncode == 0, result.stderr+ assert "Detected" in result.stdout and "insufficient_words" in result.stdout+ result = cli("--batch", "-", "--explain", "--json", stdin="Turn lights off\n" + DE + "\n")+ assert result.returncode == 0, result.stderr+ assert len([json.loads(line)["detection"]["diagnostic"] for line in result.stdout.splitlines()]) == 2+ result = cli("--batch", "-", "--explain", stdin="Turn lights off\n")+ assert result.returncode == 0 and "insufficient_words" in result.stdout+ result = cli("--explain", stdin="Turn lights off\nquit\n")+ assert result.returncode == 0 and "insufficient_words" in result.stdout++[email protected]("args", [("--predict",), ("--preset", "email"), ("--questions", "absent.json")])+def test_cli_rejects_prediction_modes_before_any_loading(args):+ result = cli("Turn lights off", "--explain", *args)+ assert result.returncode == 2+ assert "--explain is for routing only" in result.stderr+ assert result.stdout == ""+