Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(serve): make the routing fallback reachable from the environment
Aashishd47443d3b91c
diff --git a/README.md b/README.mdindex 328a6194..edf177d3 100644--- a/README.md+++ b/README.md@@ -414,6 +414,11 @@ router.route({"body": "Esqueci minha senha"}).model # -> multilirouter.route({"body": "Please refund the duplicate charge"}).model # -> english```+Running one of the shipped servers rather than your own `Router`, the same setting is+`LAYA_DEFAULT_MODEL=multilingual` — in the environment of `laya-serve`, the MCP server or the+`laya-serve` container ([docs/docker.md](docs/docker.md)). It is the routing fallback only: text+the detector can place is routed on what it detects, whatever this is set to.+### Heterogeneous routed batchesIf a lazy router receives an interleaved workload whose requests route to different checkpoints, calling `predict()` in a loop can still cause unnecessary checkpoint churn when the required checkpoints exceed the resident cache, for example with `max_loaded=1` or when `typed-decisions` is also used.@@ -561,7 +566,9 @@ The endpoint shares forward passes via `Router.predict_batch` and returns an arrConfiguration is by environment variable: `LAYA_HOST`, `LAYA_PORT`,`LAYA_DEVICE`, `LAYA_PRELOAD`, `LAYA_MODELS` (comma list to preload),`LAYA_THREADS` (cap torch intra-op threads for CPU inference — keep at or below-physical cores), `LAYA_AUTO_TASK`, `LAYA_MAX_LOADED` (checkpoints resident at+physical cores), `LAYA_AUTO_TASK`, `LAYA_DEFAULT_MODEL` (the checkpoint a state with no+language evidence falls back to, `english` by default; set it to `multilingual` when most of+your traffic is not English), `LAYA_MAX_LOADED` (checkpoints resident atonce, 2 by default; raise it to 3 when `LAYA_AUTO_TASK` makes a third onereachable on demand, or the server rebuilds one every time routing switches),and `LAYA_API_KEY` (when set, clients mustdiff --git a/compose.http.yaml b/compose.http.yamlindex 0c7eeb39..e1fb2d11 100644--- a/compose.http.yaml+++ b/compose.http.yaml@@ -47,6 +47,10 @@ services:LAYA_MODELS: "${LAYA_MODELS:-}"LAYA_THREADS: "${LAYA_THREADS:-${OMP_NUM_THREADS:-4}}"LAYA_AUTO_TASK: "${LAYA_AUTO_TASK:-0}"+ # The checkpoint a state with no language evidence falls back to. Empty keeps Router's own+ # default (english); set `multilingual` for mostly-non-English traffic. An unresolvable+ # name stops the container at startup rather than serving a configuration nobody asked for.+ LAYA_DEFAULT_MODEL: "${LAYA_DEFAULT_MODEL:-}"# Empty keeps Router's own default (2). Set 3 when LAYA_AUTO_TASK is on: that adds a# third checkpoint routing can choose on demand, and a cap below it reloads one per switch.LAYA_MAX_LOADED: "${LAYA_MAX_LOADED:-}"diff --git a/docs/cli-mcp.md b/docs/cli-mcp.mdindex c880bcf3..547f9b30 100644--- a/docs/cli-mcp.md+++ b/docs/cli-mcp.md@@ -216,6 +216,7 @@ the server is ready.| `LAYA_MODELS` | `english,multilingual` | Comma-separated checkpoints to preload. An empty value keeps the MCP default rather than preloading every checkpoint. || `LAYA_THREADS` | PyTorch default | Caps Torch intra-op threads for CPU inference; keep it at or below the physical core count. || `LAYA_AUTO_TASK` | `0` | Set to `1` to let a request auto-route to the `typed-decisions` checkpoint. Same meaning as in `laya.serve`; it does not preload that checkpoint, so `LAYA_MODELS` still decides what is built at startup. |+| `LAYA_DEFAULT_MODEL` | `english` | The checkpoint a state with no language evidence falls back to, same meaning as in `laya.serve`. Unlike `laya.serve`, an unresolvable name does not stop the server: it comes back as a `router construction failed` tool error on the next call, because a stdio server has no startup to refuse. |The stock `laya-mcp-server` launcher creates its Router without installing hooks. If you needprediction hooks, use a custom launcher that installs them, for example withdiff --git a/docs/docker.md b/docs/docker.mdindex 41eec446..eab8fb76 100644--- a/docs/docker.md+++ b/docs/docker.md@@ -237,6 +237,7 @@ These apply to the `laya-serve` service only.| `LAYA_MODELS` | (all) | comma list to preload: `english,multilingual,typed-decisions` || `LAYA_THREADS` | `OMP_NUM_THREADS` | caps torch intra-op threads; keep at or below physical cores || `LAYA_AUTO_TASK` | `0` | `1` lets the router reach `typed-decisions` automatically |+| `LAYA_DEFAULT_MODEL` | `english` | Checkpoint a state with no language evidence falls back to (no letters, or Latin text too short to identify). Set `multilingual` for mostly non-English traffic; an unresolvable name stops the container at startup instead of serving a configuration nobody asked for || `LAYA_MAX_LOADED` | `2` | Checkpoints kept resident; `LAYA_AUTO_TASK` makes a third reachable on demand, and a cap below what routing chooses rebuilds one per switch || `LAYA_MAX_CONCURRENT` | `16` | requests admitted at once; later ones get `503` (a value that does not parse, or is not positive, falls back to `16`) || `LAYA_LOG_LEVEL` | `info` | uvicorn log level |diff --git a/docs/http-api.md b/docs/http-api.mdindex 8b517dc3..da5a16a3 100644--- a/docs/http-api.md+++ b/docs/http-api.md@@ -28,6 +28,7 @@ Everything is environment variables, so one image serves a laptop dev run and a| `LAYA_MODELS` | comma list to preload (`english,multilingual,typed-decisions`); empty = all | all || `LAYA_THREADS` | cap torch intra-op threads on CPU; keep it <= physical cores -- oversubscribing logical cores is a large regression | torch default || `LAYA_AUTO_TASK` | auto-route to the typed-decisions checkpoint | `0` |+| `LAYA_DEFAULT_MODEL` | checkpoint a state with no language evidence falls back to; aliases such as `ml` resolve the way core resolves them, and an unresolvable name stops the server at startup | `english` || `LAYA_API_KEY` | if set, require `Authorization: Bearer <key>` | none || `LAYA_LOG_LEVEL` | uvicorn log level | `info` || `LAYA_MAX_CONCURRENT` | requests admitted past auth at once; excess gets `503` | `16` |diff --git a/laya/mcp/server.py b/laya/mcp/server.pyindex e7d362e6..418478e0 100644--- a/laya/mcp/server.py+++ b/laya/mcp/server.py@@ -15,6 +15,9 @@ Environment (same meaning as laya.serve where it exists):LAYA_AUTO_TASK "1" lets a request auto-route to the typed-decisions checkpoint(same as laya.serve). It does not preload it: LAYA_MODELS stilldecides what is built at startup.+ LAYA_DEFAULT_MODEL the checkpoint a state with no language evidence falls back to+ (same as laya.serve, including its aliases); an unset value leaves+ it to Router, an unresolvable one is a tool error."""from __future__ import annotations@@ -36,8 +39,8 @@ except ImportError as exc: # mcp extra not installed) from exc# laya.serve only imports os/typing at module level, so reusing its helpers-# keeps one meaning for LAYA_PRELOAD / LAYA_THREADS across the package.-from laya.serve import _apply_thread_limit, _env_bool+# keeps one meaning for LAYA_PRELOAD / LAYA_THREADS / LAYA_DEFAULT_MODEL across the package.+from laya.serve import _apply_thread_limit, _default_model_option, _env_boolfrom .device import env_devicefrom .tools import (@@ -123,12 +126,14 @@ def _models_from_env() -> list[str]:def _ensure_router() -> Any:"""Build the Router from the environment, following the laya.serve contract.- LAYA_DEVICE / LAYA_PRELOAD / LAYA_THREADS / LAYA_AUTO_TASK keep the same meaning as- in laya.serve (the helpers are reused, not duplicated). LAYA_MODELS follows the- serve comma-list but defaults to english+multilingual here, so typed-decisions stays- lazy: LAYA_AUTO_TASK=1 only lets a matching question schema route to it, and it is- then loaded on demand. The global is only set once the router is fully built, so a- failed preload stays retriable on the next tool call, and construction errors+ LAYA_DEVICE / LAYA_PRELOAD / LAYA_THREADS / LAYA_AUTO_TASK / LAYA_DEFAULT_MODEL keep the+ same meaning as in laya.serve (the helpers are reused, not duplicated). The last one+ differs only in what an unresolvable value costs: serve refuses to start, while a stdio+ server has no startup to refuse, so the ValueError rides on into the ToolError below.+ LAYA_MODELS follows the serve comma-list but defaults to english+multilingual here, so+ typed-decisions stays lazy: LAYA_AUTO_TASK=1 only lets a matching question schema route+ to it, and it is then loaded on demand. The global is only set once the router is fully+ built, so a failed preload stays retriable on the next tool call, and construction errorssurface as ToolError payloads instead of being swallowed."""global _ROUTER@@ -144,7 +149,8 @@ def _ensure_router() -> Any:try:_apply_thread_limit()router = Router(device=env_device(),- auto_task_detection=_env_bool("LAYA_AUTO_TASK", False))+ auto_task_detection=_env_bool("LAYA_AUTO_TASK", False),+ **_default_model_option())if _env_bool("LAYA_PRELOAD", True):router.preload(_models_from_env())except Exception as exc:diff --git a/laya/serve.py b/laya/serve.pyindex 59a30753..a3f00997 100644--- a/laya/serve.py+++ b/laya/serve.py@@ -24,6 +24,8 @@ env var meaning defaulKeep <= physical cores; oversubscribing the default)logical/hyperthread count is a large regression.``LAYA_AUTO_TASK`` auto-route to the typed-decisions checkpoint 0+``LAYA_DEFAULT_MODEL`` fallback checkpoint when a state carries no (english)+ language evidence; aliases like ml work``LAYA_MAX_LOADED`` checkpoints kept resident at once. Below what 2routing can choose, one reloads per switch.``LAYA_API_KEY`` if set, require ``Authorization: Bearer <it>`` (none)@@ -264,6 +266,35 @@ def _resolve_max_loaded() -> Optional[int]:return n if n > 0 else None+def _default_model_option() -> Dict[str, str]:+ """Routing fallback from ``LAYA_DEFAULT_MODEL``, as a ``Router`` keyword; unset sends nothing.++ ``Router.default`` answers the two states that carry no language evidence at all: no letters,+ and Latin script too short to identify ("Quero cancelar", "Esqueci minha senha"). The README+ tells a deployment whose traffic is mostly non-English to set ``Router(default="multilingual")``,+ and this is the only way such a deployment can say so without writing its own server. Left out+ of the constructor when unset, so the value cannot drift from ``Router``'s own default -- the+ same reasoning as ``_resolve_max_loaded`` above.++ The name goes through ``normalise_name``, so the accepted set and its aliases are core's and+ not a list restated here. Unlike the numeric knobs, a typo here has no harmless fallback: a+ silently-ignored value would keep routing the ambiguous states to the checkpoint the operator+ just said cannot read them, so this raises and the caller refuses to start rather than serve a+ configuration nobody asked for. ``laya.mcp.server`` turns the same error into a ``ToolError``,+ because a stdio server has no startup to refuse.+ """+ raw = os.environ.get("LAYA_DEFAULT_MODEL")+ if raw is None or not raw.strip():+ return {}+ from .router import normalise_name++ try:+ name = normalise_name(raw)+ except ValueError as error:+ raise ValueError("invalid LAYA_DEFAULT_MODEL %r: %s" % (raw.strip(), error)) from None+ return {"default": name}++def _resolve_port() -> int:"""Port from LAYA_PORT, validated. Exits with a message instead of a traceback."""raw = os.environ.get("LAYA_PORT", "8000")@@ -566,6 +597,12 @@ def build_router():max_loaded = _resolve_max_loaded()if max_loaded is not None:options["max_loaded"] = max_loaded+ # Resolved before the Router is built: a name `Router` would reject is a configuration error,+ # and `_resolve_port`'s idiom applies -- exit with the message, not a traceback.+ try:+ options.update(_default_model_option())+ except ValueError as error:+ raise SystemExit(str(error)) from Nonerouter = Router(**options)if _env_bool("LAYA_PRELOAD", True):router.preload(preload_names)diff --git a/nix/laya-serve.nix b/nix/laya-serve.nixindex 2f5a0e28..911bba3d 100644--- a/nix/laya-serve.nix+++ b/nix/laya-serve.nix@@ -244,6 +244,23 @@ indescription = "Let the router auto-select the typed-decisions checkpoint when question ids match its workflows.";};+ defaultModel = lib.mkOption {+ type = lib.types.nullOr (lib.types.strMatching "[A-Za-z0-9._/-]+");+ default = null;+ example = "multilingual";+ description = ''+ Checkpoint a state with no language evidence falls back to (sets+ `LAYA_DEFAULT_MODEL`): text with no letters at all, or Latin script too+ short to identify a language. Set it to `multilingual` when most traffic+ on this host is not English. Text the detector can place is routed on+ what it detects, so this is a fallback and not a pin. The spellings+ accepted, and the aliases among them, are laya's own -- the server+ checks the value against them at startup and refuses to start on a name+ it does not know, so this module deliberately lists none. null leaves+ laya's default, which is `english`.+ '';+ };+apiKeyFile = lib.mkOption {type = lib.types.nullOr lib.types.path;default = null;@@ -303,6 +320,8 @@ inLAYA_MAX_TOKEN_BUDGET = toString cfg.maxTokenBudget;} // lib.optionalAttrs (cfg.revision != null) {LAYA_REVISION = cfg.revision;+ } // lib.optionalAttrs (cfg.defaultModel != null) {+ LAYA_DEFAULT_MODEL = cfg.defaultModel;} // {HF_HOME = "/var/lib/${cfg.stateDirectory}/huggingface";# torch-bin bundles its own CUDA runtime but still needs the hostdiff --git a/tests/test_mcp.py b/tests/test_mcp.pyindex d878d8c6..d202226f 100644--- a/tests/test_mcp.py+++ b/tests/test_mcp.py@@ -7,7 +7,8 @@ Skips cleanly (exit 0) when the mcp package is not installed, so the coreinstall keeps working.Device and preload-list tests follow the laya.serve environment contract-(LAYA_DEVICE / LAYA_PRELOAD / LAYA_MODELS / LAYA_THREADS / LAYA_AUTO_TASK).+(LAYA_DEVICE / LAYA_PRELOAD / LAYA_MODELS / LAYA_THREADS / LAYA_AUTO_TASK /+LAYA_DEFAULT_MODEL)."""import asyncioimport os@@ -2065,6 +2066,119 @@ def test_auto_task_env():os.environ[key] = value+def test_default_model_env():+ """LAYA_DEFAULT_MODEL has to reach both servers' Routers, with one meaning.++ README's MCP section says these variables follow the contract at the top of laya.serve, so+ both halves are built through the real builders -- `laya.mcp.server._ensure_router()` and+ `laya.serve.build_router()` -- and read off the Router each one actually returns. The+ variable itself is the routing fallback README prescribes for a mostly-non-English deployment+ (`Router(default="multilingual")`), which until now only examples/server.py could be told.+ The one place the two surfaces part is what an unresolvable name costs: serve refuses to+ start, a stdio server has no startup to refuse, so MCP has to carry the same words in a+ tool error.+ """+ import laya.mcp.server as mcp_mod # the module, not the MCPServer instance+ from laya.serve import build_router++ saved = {k: os.environ.get(k)+ for k in ("LAYA_DEFAULT_MODEL", "LAYA_PRELOAD", "LAYA_AUTO_TASK")}+ saved_router = mcp_mod._ROUTER+ try:+ os.environ["LAYA_PRELOAD"] = "0" # neither surface may build a checkpoint here+ os.environ["LAYA_AUTO_TASK"] = "0" # leaves the fallback the only thing moving++ def build(value):+ if value is None:+ os.environ.pop("LAYA_DEFAULT_MODEL", None)+ else:+ os.environ["LAYA_DEFAULT_MODEL"] = value+ mcp_mod._ROUTER = None # the server caches the Router it built+ return mcp_mod._ensure_router(), build_router()++ for label, value, want in (+ ("unset", None, "english"),+ ("empty", "", "english"),+ ("blank", " ", "english"),+ ("canonical", "multilingual", "multilingual"),+ ("alias", "ml", "multilingual"),+ ("padded_upper", " MULTI ", "multilingual"),+ ("typed", "typed-decisions", "typed-decisions")):+ try:+ mcp_router, serve_router = build(value)+ except (Exception, SystemExit) as exc: # noqa: BLE001 -- every value here is valid+ # Named rather than fatal: a surface that raises for a name core accepts is the+ # bug, and a suite that dies on the way to reporting it leaves the cause unsaid.+ ok("default_model/%s_builds" % label, False,+ "raised %r: %s" % (type(exc).__name__, exc))+ continue+ ok("default_model/%s_mcp" % label, mcp_router.default == want,+ repr(mcp_router.default))+ # The parity the README claims: one variable, one meaning on both surfaces.+ ok("default_model/%s_matches_serve" % label,+ mcp_router.default == serve_router.default,+ "mcp=%r serve=%r" % (mcp_router.default, serve_router.default))++ # What the setting decides, at `route()` rather than at the attribute. `route` is+ # documented as deciding "without loading or running anything", so nothing is downloaded.+ stock_mcp, stock_serve = build(None)+ nonenglish_mcp, nonenglish_serve = build("multilingual")+ for index, state in enumerate(("12345 !!!", "Quero cancelar")):+ ok("default_model/stock_english_%d" % index,+ stock_mcp.route(state).model == "english" == stock_serve.route(state).model,+ "mcp=%r serve=%r" % (stock_mcp.route(state).model, stock_serve.route(state).model))+ ok("default_model/fallback_multilingual_%d" % index,+ nonenglish_mcp.route(state).model == "multilingual"+ == nonenglish_serve.route(state).model,+ "mcp=%r serve=%r" % (nonenglish_mcp.route(state).model,+ nonenglish_serve.route(state).model))+ # A fallback, not a pin, on both surfaces.+ ok("default_model/placed_text_unaffected",+ nonenglish_mcp.route({"body": "Please refund the duplicate charge"}).model == "english"+ == nonenglish_serve.route({"body": "Please refund the duplicate charge"}).model)++ serve_message = mcp_message = ""+ os.environ["LAYA_DEFAULT_MODEL"] = "mutli-lingual"+ try:+ build_router()+ ok("default_model/serve_refuses_to_start", False, "build_router() returned a Router")+ except SystemExit as exc:+ serve_message = str(exc)+ ok("default_model/serve_refuses_to_start", True)+ except Exception as exc: # noqa: BLE001 -- a traceback is not "exits with a message"+ ok("default_model/serve_refuses_to_start", False,+ "raised %r instead of exiting: %s" % (type(exc).__name__, exc))+ mcp_mod._ROUTER = None+ try:+ mcp_mod._ensure_router()+ ok("default_model/mcp_refuses", False, "_ensure_router() returned a Router")+ except ToolError as exc:+ mcp_message = exc.message+ ok("default_model/mcp_refuses_with_a_tool_error",+ exc.code == "internal_error", repr(exc.code))+ except Exception as exc: # noqa: BLE001 -- anything else escapes the tool wrapper+ ok("default_model/mcp_refuses_with_a_tool_error", False,+ "raised %r instead of a ToolError, so the client gets a server crash" % type(exc).__name__)+ # One message, both surfaces: the operator fixes the typo from what they were told,+ # whether or not the server came up.+ ok("default_model/both_surfaces_name_the_variable_and_the_value",+ all("invalid LAYA_DEFAULT_MODEL 'mutli-lingual'" in m for m in (serve_message, mcp_message)),+ "serve=%r mcp=%r" % (serve_message, mcp_message))+ ok("default_model/both_surfaces_carry_core_s_words",+ all("unknown model" in m for m in (serve_message, mcp_message)),+ "serve=%r mcp=%r" % (serve_message, mcp_message))+ # The docstring promises a failed build is retriable rather than cached.+ ok("default_model/failed_build_is_not_cached", mcp_mod._ROUTER is None,+ repr(mcp_mod._ROUTER))+ finally:+ mcp_mod._ROUTER = saved_router+ for key, value in saved.items():+ if value is None:+ os.environ.pop(key, None)+ else:+ os.environ[key] = value++def test_server_registration():tools = asyncio.run(mcp_server.list_tools())names = sorted(t.name for t in tools)@@ -2146,6 +2260,7 @@ test_timeout_removed()test_models_from_env()test_batch_item_shape_as_documented()test_auto_task_env()+test_default_model_env()test_server_registration()test_server_shortlist_k_default()diff --git a/tests/test_packaging.py b/tests/test_packaging.pyindex 656a2cfa..6e32ae32 100644--- a/tests/test_packaging.py+++ b/tests/test_packaging.py@@ -473,7 +473,7 @@ def option_type(opt):# Every new knob is opt-in: unset means the unit exports nothing and the runtime's own default# applies, so a host that ignores them gets today's behaviour byte for byte.for opt in ("rootPath", "logLevel", "maxConcurrent", "cudaAmp", "cpuAmp", "mpsAmpMinRows",- "maxLoaded", "maxTokenBudget", "revision"):+ "maxLoaded", "maxTokenBudget", "revision", "defaultModel"):_t = option_text(opt)check_true("nix/module declares %s" % opt, _t != "", "option not found")check_true("nix/%s is opt-in (nullOr, default null)" % opt,diff --git a/tests/test_serve.py b/tests/test_serve.pyindex e41dc9f8..37a6e602 100644--- a/tests/test_serve.py+++ b/tests/test_serve.py@@ -488,6 +488,7 @@ def _server_router(monkeypatch, **env):monkeypatch.setenv("LAYA_PRELOAD", "0") # nothing may downloadmonkeypatch.setenv("LAYA_AUTO_TASK", "1") # the config that puts three checkpoints in playmonkeypatch.delenv("LAYA_MAX_LOADED", raising=False)+ monkeypatch.delenv("LAYA_DEFAULT_MODEL", raising=False)for name, value in env.items():monkeypatch.setenv(name, value)router = build_router()@@ -574,6 +575,83 @@ def test_raising_the_cap_stops_the_server_rebuilding_a_checkpoint(monkeypatch):assert sorted(roomy.loaded) == sorted(DEFAULT_MODELS)+# README's other answer written as a constructor argument is `Router(default="multilingual")`,+# given where it explains that very short Latin-script text ("Quero cancelar", "Esqueci minha+# senha") carries nothing identifying its language and so goes to `default`. Both packaged+# servers build their Router from the environment and neither could pass it: the only reader of+# LAYA_DEFAULT_MODEL was examples/server.py, so the demo server honoured the variable while+# laya-serve routed the ambiguous states to the English checkpoint it was told not to use.+#+# Unlike the numeric knobs above, an unresolvable name is fatal here. `LAYA_MAX_LOADED=abc`+# falling back to 2 costs a rebuild; `LAYA_DEFAULT_MODEL=mutli` falling back to english costs the+# wrong checkpoint answering every ambiguous state, which is the failure the operator was trying+# to configure away. So the message is core's and the server refuses to start.+def test_default_model_reaches_the_router_the_server_builds(monkeypatch):+ from laya.router import DEFAULT_MODELS, Router, _ALIASES, normalise_name+ from laya.serve import _default_model_option++ # Unset is "not asked for", not a copy of Router's default -- the same reason+ # `_resolve_max_loaded` returns None, and the same drift it prevents.+ monkeypatch.delenv("LAYA_DEFAULT_MODEL", raising=False)+ assert _default_model_option() == {}+ for blank in ("", " ", "\n"):+ monkeypatch.setenv("LAYA_DEFAULT_MODEL", blank)+ assert _default_model_option() == {}, repr(blank)++ # The accepted spellings are core's tables read out of core, so a name added to _ALIASES+ # arrives here on its own -- and a value this module resolves differently from the way+ # `Router` itself resolves it fails the same line.+ for name in sorted(set(DEFAULT_MODELS) | set(_ALIASES)):+ monkeypatch.setenv("LAYA_DEFAULT_MODEL", name)+ assert _default_model_option() == {"default": normalise_name(name)}, name++ # And it reaches the Router the server builds. The literal is the one the README and+ # docs/docker.md quote, so moving Router's default has to move those too.+ router, _ = _server_router(monkeypatch)+ assert router.default == Router().default+ assert router.default == "english"+ for raw, want in (("multilingual", "multilingual"), ("ml", "multilingual"),+ (" MULTI ", "multilingual"), ("typed-decisions", "typed-decisions")):+ router, _ = _server_router(monkeypatch, LAYA_DEFAULT_MODEL=raw)+ assert router.default == want, raw++ # The decision the whole knob exists to change, measured at `route()` rather than at the+ # attribute -- `route` documents that it decides "without loading or running anything", so+ # this costs no weights.+ ambiguous = ("12345 !!!", "Quero cancelar", "Esqueci minha senha")+ stock, _ = _server_router(monkeypatch)+ portuguese, _ = _server_router(monkeypatch, LAYA_DEFAULT_MODEL="multilingual")+ for state in ambiguous:+ assert stock.route(state).model == "english", state+ assert "using default (english)" in stock.route(state).reason, state+ assert portuguese.route(state).model == "multilingual", state+ assert "using default (multilingual)" in portuguese.route(state).reason, state+ # A fallback, not a pin: text the detector can place routes on what it detects.+ assert portuguese.route({"body": "Please refund the duplicate charge"}).model == "english"+++def test_an_unresolvable_default_stops_the_server_with_core_s_words(monkeypatch):+ from laya.serve import build_router++ monkeypatch.setenv("LAYA_PRELOAD", "0")+ monkeypatch.delenv("LAYA_DEFAULT_MODEL", raising=False)+ monkeypatch.setenv("LAYA_DEFAULT_MODEL", "mutli-lingual")+ with pytest.raises(SystemExit) as raised:+ build_router()+ message = str(raised.value)+ # Names the variable, quotes the value the operator typed, and then says in core's own words+ # what the accepted set is -- so the typo is fixed from the message, not from the source.+ assert message.startswith("invalid LAYA_DEFAULT_MODEL 'mutli-lingual': unknown model"), message+ for name in ("english", "multilingual", "typed-decisions", "ml"):+ assert name in message, message+ # The contrast with the numeric knobs has to stay the contrast: with the name fixed, the bad+ # number below still falls back instead of stopping the server.+ monkeypatch.setenv("LAYA_DEFAULT_MODEL", "english")+ monkeypatch.setenv("LAYA_MAX_LOADED", "abc")+ assert build_router().default == "english"+ assert build_router().max_loaded == 2++# The endpoint is `async def` and inference is synchronous torch, which on CPU takes# hundreds of milliseconds to seconds. Calling it from the coroutine puts that work on