Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(serve): make the routing fallback reachable from the environment

Aashishd47443d3b91c
diff --git a/README.md b/README.md
index 328a6194..edf177d3 100644
--- a/README.md
+++ b/README.md
@@ -414,6 +414,11 @@ router.route({"body": "Esqueci minha senha"}).model # -> multili
router.route({"body": "Please refund the duplicate charge"}).model # -> english
```
+Running one of the shipped servers rather than your own `Router`, the same setting is
+`LAYA_DEFAULT_MODEL=multilingual` — in the environment of `laya-serve`, the MCP server or the
+`laya-serve` container ([docs/docker.md](docs/docker.md)). It is the routing fallback only: text
+the detector can place is routed on what it detects, whatever this is set to.
+
### Heterogeneous routed batches
If a lazy router receives an interleaved workload whose requests route to different checkpoints, calling `predict()` in a loop can still cause unnecessary checkpoint churn when the required checkpoints exceed the resident cache, for example with `max_loaded=1` or when `typed-decisions` is also used.
@@ -561,7 +566,9 @@ The endpoint shares forward passes via `Router.predict_batch` and returns an arr
Configuration is by environment variable: `LAYA_HOST`, `LAYA_PORT`,
`LAYA_DEVICE`, `LAYA_PRELOAD`, `LAYA_MODELS` (comma list to preload),
`LAYA_THREADS` (cap torch intra-op threads for CPU inference — keep at or below
-physical cores), `LAYA_AUTO_TASK`, `LAYA_MAX_LOADED` (checkpoints resident at
+physical cores), `LAYA_AUTO_TASK`, `LAYA_DEFAULT_MODEL` (the checkpoint a state with no
+language evidence falls back to, `english` by default; set it to `multilingual` when most of
+your traffic is not English), `LAYA_MAX_LOADED` (checkpoints resident at
once, 2 by default; raise it to 3 when `LAYA_AUTO_TASK` makes a third one
reachable on demand, or the server rebuilds one every time routing switches),
and `LAYA_API_KEY` (when set, clients must
diff --git a/compose.http.yaml b/compose.http.yaml
index 0c7eeb39..e1fb2d11 100644
--- a/compose.http.yaml
+++ b/compose.http.yaml
@@ -47,6 +47,10 @@ services:
LAYA_MODELS: "${LAYA_MODELS:-}"
LAYA_THREADS: "${LAYA_THREADS:-${OMP_NUM_THREADS:-4}}"
LAYA_AUTO_TASK: "${LAYA_AUTO_TASK:-0}"
+ # The checkpoint a state with no language evidence falls back to. Empty keeps Router's own
+ # default (english); set `multilingual` for mostly-non-English traffic. An unresolvable
+ # name stops the container at startup rather than serving a configuration nobody asked for.
+ LAYA_DEFAULT_MODEL: "${LAYA_DEFAULT_MODEL:-}"
# Empty keeps Router's own default (2). Set 3 when LAYA_AUTO_TASK is on: that adds a
# third checkpoint routing can choose on demand, and a cap below it reloads one per switch.
LAYA_MAX_LOADED: "${LAYA_MAX_LOADED:-}"
diff --git a/docs/cli-mcp.md b/docs/cli-mcp.md
index c880bcf3..547f9b30 100644
--- a/docs/cli-mcp.md
+++ b/docs/cli-mcp.md
@@ -216,6 +216,7 @@ the server is ready.
| `LAYA_MODELS` | `english,multilingual` | Comma-separated checkpoints to preload. An empty value keeps the MCP default rather than preloading every checkpoint. |
| `LAYA_THREADS` | PyTorch default | Caps Torch intra-op threads for CPU inference; keep it at or below the physical core count. |
| `LAYA_AUTO_TASK` | `0` | Set to `1` to let a request auto-route to the `typed-decisions` checkpoint. Same meaning as in `laya.serve`; it does not preload that checkpoint, so `LAYA_MODELS` still decides what is built at startup. |
+| `LAYA_DEFAULT_MODEL` | `english` | The checkpoint a state with no language evidence falls back to, same meaning as in `laya.serve`. Unlike `laya.serve`, an unresolvable name does not stop the server: it comes back as a `router construction failed` tool error on the next call, because a stdio server has no startup to refuse. |
The stock `laya-mcp-server` launcher creates its Router without installing hooks. If you need
prediction hooks, use a custom launcher that installs them, for example with
diff --git a/docs/docker.md b/docs/docker.md
index 41eec446..eab8fb76 100644
--- a/docs/docker.md
+++ b/docs/docker.md
@@ -237,6 +237,7 @@ These apply to the `laya-serve` service only.
| `LAYA_MODELS` | (all) | comma list to preload: `english,multilingual,typed-decisions` |
| `LAYA_THREADS` | `OMP_NUM_THREADS` | caps torch intra-op threads; keep at or below physical cores |
| `LAYA_AUTO_TASK` | `0` | `1` lets the router reach `typed-decisions` automatically |
+| `LAYA_DEFAULT_MODEL` | `english` | Checkpoint a state with no language evidence falls back to (no letters, or Latin text too short to identify). Set `multilingual` for mostly non-English traffic; an unresolvable name stops the container at startup instead of serving a configuration nobody asked for |
| `LAYA_MAX_LOADED` | `2` | Checkpoints kept resident; `LAYA_AUTO_TASK` makes a third reachable on demand, and a cap below what routing chooses rebuilds one per switch |
| `LAYA_MAX_CONCURRENT` | `16` | requests admitted at once; later ones get `503` (a value that does not parse, or is not positive, falls back to `16`) |
| `LAYA_LOG_LEVEL` | `info` | uvicorn log level |
diff --git a/docs/http-api.md b/docs/http-api.md
index 8b517dc3..da5a16a3 100644
--- a/docs/http-api.md
+++ b/docs/http-api.md
@@ -28,6 +28,7 @@ Everything is environment variables, so one image serves a laptop dev run and a
| `LAYA_MODELS` | comma list to preload (`english,multilingual,typed-decisions`); empty = all | all |
| `LAYA_THREADS` | cap torch intra-op threads on CPU; keep it <= physical cores -- oversubscribing logical cores is a large regression | torch default |
| `LAYA_AUTO_TASK` | auto-route to the typed-decisions checkpoint | `0` |
+| `LAYA_DEFAULT_MODEL` | checkpoint a state with no language evidence falls back to; aliases such as `ml` resolve the way core resolves them, and an unresolvable name stops the server at startup | `english` |
| `LAYA_API_KEY` | if set, require `Authorization: Bearer <key>` | none |
| `LAYA_LOG_LEVEL` | uvicorn log level | `info` |
| `LAYA_MAX_CONCURRENT` | requests admitted past auth at once; excess gets `503` | `16` |
diff --git a/laya/mcp/server.py b/laya/mcp/server.py
index e7d362e6..418478e0 100644
--- a/laya/mcp/server.py
+++ b/laya/mcp/server.py
@@ -15,6 +15,9 @@ Environment (same meaning as laya.serve where it exists):
LAYA_AUTO_TASK "1" lets a request auto-route to the typed-decisions checkpoint
(same as laya.serve). It does not preload it: LAYA_MODELS still
decides what is built at startup.
+ LAYA_DEFAULT_MODEL the checkpoint a state with no language evidence falls back to
+ (same as laya.serve, including its aliases); an unset value leaves
+ it to Router, an unresolvable one is a tool error.
"""
from __future__ import annotations
@@ -36,8 +39,8 @@ except ImportError as exc: # mcp extra not installed
) from exc
# laya.serve only imports os/typing at module level, so reusing its helpers
-# keeps one meaning for LAYA_PRELOAD / LAYA_THREADS across the package.
-from laya.serve import _apply_thread_limit, _env_bool
+# keeps one meaning for LAYA_PRELOAD / LAYA_THREADS / LAYA_DEFAULT_MODEL across the package.
+from laya.serve import _apply_thread_limit, _default_model_option, _env_bool
from .device import env_device
from .tools import (
@@ -123,12 +126,14 @@ def _models_from_env() -> list[str]:
def _ensure_router() -> Any:
"""Build the Router from the environment, following the laya.serve contract.
- LAYA_DEVICE / LAYA_PRELOAD / LAYA_THREADS / LAYA_AUTO_TASK keep the same meaning as
- in laya.serve (the helpers are reused, not duplicated). LAYA_MODELS follows the
- serve comma-list but defaults to english+multilingual here, so typed-decisions stays
- lazy: LAYA_AUTO_TASK=1 only lets a matching question schema route to it, and it is
- then loaded on demand. The global is only set once the router is fully built, so a
- failed preload stays retriable on the next tool call, and construction errors
+ LAYA_DEVICE / LAYA_PRELOAD / LAYA_THREADS / LAYA_AUTO_TASK / LAYA_DEFAULT_MODEL keep the
+ same meaning as in laya.serve (the helpers are reused, not duplicated). The last one
+ differs only in what an unresolvable value costs: serve refuses to start, while a stdio
+ server has no startup to refuse, so the ValueError rides on into the ToolError below.
+ LAYA_MODELS follows the serve comma-list but defaults to english+multilingual here, so
+ typed-decisions stays lazy: LAYA_AUTO_TASK=1 only lets a matching question schema route
+ to it, and it is then loaded on demand. The global is only set once the router is fully
+ built, so a failed preload stays retriable on the next tool call, and construction errors
surface as ToolError payloads instead of being swallowed.
"""
global _ROUTER
@@ -144,7 +149,8 @@ def _ensure_router() -> Any:
try:
_apply_thread_limit()
router = Router(device=env_device(),
- auto_task_detection=_env_bool("LAYA_AUTO_TASK", False))
+ auto_task_detection=_env_bool("LAYA_AUTO_TASK", False),
+ **_default_model_option())
if _env_bool("LAYA_PRELOAD", True):
router.preload(_models_from_env())
except Exception as exc:
diff --git a/laya/serve.py b/laya/serve.py
index 59a30753..a3f00997 100644
--- a/laya/serve.py
+++ b/laya/serve.py
@@ -24,6 +24,8 @@ env var meaning defaul
Keep <= physical cores; oversubscribing the default)
logical/hyperthread count is a large regression.
``LAYA_AUTO_TASK`` auto-route to the typed-decisions checkpoint 0
+``LAYA_DEFAULT_MODEL`` fallback checkpoint when a state carries no (english)
+ language evidence; aliases like ml work
``LAYA_MAX_LOADED`` checkpoints kept resident at once. Below what 2
routing can choose, one reloads per switch.
``LAYA_API_KEY`` if set, require ``Authorization: Bearer <it>`` (none)
@@ -264,6 +266,35 @@ def _resolve_max_loaded() -> Optional[int]:
return n if n > 0 else None
+def _default_model_option() -> Dict[str, str]:
+ """Routing fallback from ``LAYA_DEFAULT_MODEL``, as a ``Router`` keyword; unset sends nothing.
+
+ ``Router.default`` answers the two states that carry no language evidence at all: no letters,
+ and Latin script too short to identify ("Quero cancelar", "Esqueci minha senha"). The README
+ tells a deployment whose traffic is mostly non-English to set ``Router(default="multilingual")``,
+ and this is the only way such a deployment can say so without writing its own server. Left out
+ of the constructor when unset, so the value cannot drift from ``Router``'s own default -- the
+ same reasoning as ``_resolve_max_loaded`` above.
+
+ The name goes through ``normalise_name``, so the accepted set and its aliases are core's and
+ not a list restated here. Unlike the numeric knobs, a typo here has no harmless fallback: a
+ silently-ignored value would keep routing the ambiguous states to the checkpoint the operator
+ just said cannot read them, so this raises and the caller refuses to start rather than serve a
+ configuration nobody asked for. ``laya.mcp.server`` turns the same error into a ``ToolError``,
+ because a stdio server has no startup to refuse.
+ """
+ raw = os.environ.get("LAYA_DEFAULT_MODEL")
+ if raw is None or not raw.strip():
+ return {}
+ from .router import normalise_name
+
+ try:
+ name = normalise_name(raw)
+ except ValueError as error:
+ raise ValueError("invalid LAYA_DEFAULT_MODEL %r: %s" % (raw.strip(), error)) from None
+ return {"default": name}
+
+
def _resolve_port() -> int:
"""Port from LAYA_PORT, validated. Exits with a message instead of a traceback."""
raw = os.environ.get("LAYA_PORT", "8000")
@@ -566,6 +597,12 @@ def build_router():
max_loaded = _resolve_max_loaded()
if max_loaded is not None:
options["max_loaded"] = max_loaded
+ # Resolved before the Router is built: a name `Router` would reject is a configuration error,
+ # and `_resolve_port`'s idiom applies -- exit with the message, not a traceback.
+ try:
+ options.update(_default_model_option())
+ except ValueError as error:
+ raise SystemExit(str(error)) from None
router = Router(**options)
if _env_bool("LAYA_PRELOAD", True):
router.preload(preload_names)
diff --git a/nix/laya-serve.nix b/nix/laya-serve.nix
index 2f5a0e28..911bba3d 100644
--- a/nix/laya-serve.nix
+++ b/nix/laya-serve.nix
@@ -244,6 +244,23 @@ in
description = "Let the router auto-select the typed-decisions checkpoint when question ids match its workflows.";
};
+ defaultModel = lib.mkOption {
+ type = lib.types.nullOr (lib.types.strMatching "[A-Za-z0-9._/-]+");
+ default = null;
+ example = "multilingual";
+ description = ''
+ Checkpoint a state with no language evidence falls back to (sets
+ `LAYA_DEFAULT_MODEL`): text with no letters at all, or Latin script too
+ short to identify a language. Set it to `multilingual` when most traffic
+ on this host is not English. Text the detector can place is routed on
+ what it detects, so this is a fallback and not a pin. The spellings
+ accepted, and the aliases among them, are laya's own -- the server
+ checks the value against them at startup and refuses to start on a name
+ it does not know, so this module deliberately lists none. null leaves
+ laya's default, which is `english`.
+ '';
+ };
+
apiKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.path;
default = null;
@@ -303,6 +320,8 @@ in
LAYA_MAX_TOKEN_BUDGET = toString cfg.maxTokenBudget;
} // lib.optionalAttrs (cfg.revision != null) {
LAYA_REVISION = cfg.revision;
+ } // lib.optionalAttrs (cfg.defaultModel != null) {
+ LAYA_DEFAULT_MODEL = cfg.defaultModel;
} // {
HF_HOME = "/var/lib/${cfg.stateDirectory}/huggingface";
# torch-bin bundles its own CUDA runtime but still needs the host
diff --git a/tests/test_mcp.py b/tests/test_mcp.py
index d878d8c6..d202226f 100644
--- a/tests/test_mcp.py
+++ b/tests/test_mcp.py
@@ -7,7 +7,8 @@ Skips cleanly (exit 0) when the mcp package is not installed, so the core
install keeps working.
Device and preload-list tests follow the laya.serve environment contract
-(LAYA_DEVICE / LAYA_PRELOAD / LAYA_MODELS / LAYA_THREADS / LAYA_AUTO_TASK).
+(LAYA_DEVICE / LAYA_PRELOAD / LAYA_MODELS / LAYA_THREADS / LAYA_AUTO_TASK /
+LAYA_DEFAULT_MODEL).
"""
import asyncio
import os
@@ -2065,6 +2066,119 @@ def test_auto_task_env():
os.environ[key] = value
+def test_default_model_env():
+ """LAYA_DEFAULT_MODEL has to reach both servers' Routers, with one meaning.
+
+ README's MCP section says these variables follow the contract at the top of laya.serve, so
+ both halves are built through the real builders -- `laya.mcp.server._ensure_router()` and
+ `laya.serve.build_router()` -- and read off the Router each one actually returns. The
+ variable itself is the routing fallback README prescribes for a mostly-non-English deployment
+ (`Router(default="multilingual")`), which until now only examples/server.py could be told.
+ The one place the two surfaces part is what an unresolvable name costs: serve refuses to
+ start, a stdio server has no startup to refuse, so MCP has to carry the same words in a
+ tool error.
+ """
+ import laya.mcp.server as mcp_mod # the module, not the MCPServer instance
+ from laya.serve import build_router
+
+ saved = {k: os.environ.get(k)
+ for k in ("LAYA_DEFAULT_MODEL", "LAYA_PRELOAD", "LAYA_AUTO_TASK")}
+ saved_router = mcp_mod._ROUTER
+ try:
+ os.environ["LAYA_PRELOAD"] = "0" # neither surface may build a checkpoint here
+ os.environ["LAYA_AUTO_TASK"] = "0" # leaves the fallback the only thing moving
+
+ def build(value):
+ if value is None:
+ os.environ.pop("LAYA_DEFAULT_MODEL", None)
+ else:
+ os.environ["LAYA_DEFAULT_MODEL"] = value
+ mcp_mod._ROUTER = None # the server caches the Router it built
+ return mcp_mod._ensure_router(), build_router()
+
+ for label, value, want in (
+ ("unset", None, "english"),
+ ("empty", "", "english"),
+ ("blank", " ", "english"),
+ ("canonical", "multilingual", "multilingual"),
+ ("alias", "ml", "multilingual"),
+ ("padded_upper", " MULTI ", "multilingual"),
+ ("typed", "typed-decisions", "typed-decisions")):
+ try:
+ mcp_router, serve_router = build(value)
+ except (Exception, SystemExit) as exc: # noqa: BLE001 -- every value here is valid
+ # Named rather than fatal: a surface that raises for a name core accepts is the
+ # bug, and a suite that dies on the way to reporting it leaves the cause unsaid.
+ ok("default_model/%s_builds" % label, False,
+ "raised %r: %s" % (type(exc).__name__, exc))
+ continue
+ ok("default_model/%s_mcp" % label, mcp_router.default == want,
+ repr(mcp_router.default))
+ # The parity the README claims: one variable, one meaning on both surfaces.
+ ok("default_model/%s_matches_serve" % label,
+ mcp_router.default == serve_router.default,
+ "mcp=%r serve=%r" % (mcp_router.default, serve_router.default))
+
+ # What the setting decides, at `route()` rather than at the attribute. `route` is
+ # documented as deciding "without loading or running anything", so nothing is downloaded.
+ stock_mcp, stock_serve = build(None)
+ nonenglish_mcp, nonenglish_serve = build("multilingual")
+ for index, state in enumerate(("12345 !!!", "Quero cancelar")):
+ ok("default_model/stock_english_%d" % index,
+ stock_mcp.route(state).model == "english" == stock_serve.route(state).model,
+ "mcp=%r serve=%r" % (stock_mcp.route(state).model, stock_serve.route(state).model))
+ ok("default_model/fallback_multilingual_%d" % index,
+ nonenglish_mcp.route(state).model == "multilingual"
+ == nonenglish_serve.route(state).model,
+ "mcp=%r serve=%r" % (nonenglish_mcp.route(state).model,
+ nonenglish_serve.route(state).model))
+ # A fallback, not a pin, on both surfaces.
+ ok("default_model/placed_text_unaffected",
+ nonenglish_mcp.route({"body": "Please refund the duplicate charge"}).model == "english"
+ == nonenglish_serve.route({"body": "Please refund the duplicate charge"}).model)
+
+ serve_message = mcp_message = ""
+ os.environ["LAYA_DEFAULT_MODEL"] = "mutli-lingual"
+ try:
+ build_router()
+ ok("default_model/serve_refuses_to_start", False, "build_router() returned a Router")
+ except SystemExit as exc:
+ serve_message = str(exc)
+ ok("default_model/serve_refuses_to_start", True)
+ except Exception as exc: # noqa: BLE001 -- a traceback is not "exits with a message"
+ ok("default_model/serve_refuses_to_start", False,
+ "raised %r instead of exiting: %s" % (type(exc).__name__, exc))
+ mcp_mod._ROUTER = None
+ try:
+ mcp_mod._ensure_router()
+ ok("default_model/mcp_refuses", False, "_ensure_router() returned a Router")
+ except ToolError as exc:
+ mcp_message = exc.message
+ ok("default_model/mcp_refuses_with_a_tool_error",
+ exc.code == "internal_error", repr(exc.code))
+ except Exception as exc: # noqa: BLE001 -- anything else escapes the tool wrapper
+ ok("default_model/mcp_refuses_with_a_tool_error", False,
+ "raised %r instead of a ToolError, so the client gets a server crash" % type(exc).__name__)
+ # One message, both surfaces: the operator fixes the typo from what they were told,
+ # whether or not the server came up.
+ ok("default_model/both_surfaces_name_the_variable_and_the_value",
+ all("invalid LAYA_DEFAULT_MODEL 'mutli-lingual'" in m for m in (serve_message, mcp_message)),
+ "serve=%r mcp=%r" % (serve_message, mcp_message))
+ ok("default_model/both_surfaces_carry_core_s_words",
+ all("unknown model" in m for m in (serve_message, mcp_message)),
+ "serve=%r mcp=%r" % (serve_message, mcp_message))
+ # The docstring promises a failed build is retriable rather than cached.
+ ok("default_model/failed_build_is_not_cached", mcp_mod._ROUTER is None,
+ repr(mcp_mod._ROUTER))
+ finally:
+ mcp_mod._ROUTER = saved_router
+ for key, value in saved.items():
+ if value is None:
+ os.environ.pop(key, None)
+ else:
+ os.environ[key] = value
+
+
def test_server_registration():
tools = asyncio.run(mcp_server.list_tools())
names = sorted(t.name for t in tools)
@@ -2146,6 +2260,7 @@ test_timeout_removed()
test_models_from_env()
test_batch_item_shape_as_documented()
test_auto_task_env()
+test_default_model_env()
test_server_registration()
test_server_shortlist_k_default()
diff --git a/tests/test_packaging.py b/tests/test_packaging.py
index 656a2cfa..6e32ae32 100644
--- a/tests/test_packaging.py
+++ b/tests/test_packaging.py
@@ -473,7 +473,7 @@ def option_type(opt):
# Every new knob is opt-in: unset means the unit exports nothing and the runtime's own default
# applies, so a host that ignores them gets today's behaviour byte for byte.
for opt in ("rootPath", "logLevel", "maxConcurrent", "cudaAmp", "cpuAmp", "mpsAmpMinRows",
- "maxLoaded", "maxTokenBudget", "revision"):
+ "maxLoaded", "maxTokenBudget", "revision", "defaultModel"):
_t = option_text(opt)
check_true("nix/module declares %s" % opt, _t != "", "option not found")
check_true("nix/%s is opt-in (nullOr, default null)" % opt,
diff --git a/tests/test_serve.py b/tests/test_serve.py
index e41dc9f8..37a6e602 100644
--- a/tests/test_serve.py
+++ b/tests/test_serve.py
@@ -488,6 +488,7 @@ def _server_router(monkeypatch, **env):
monkeypatch.setenv("LAYA_PRELOAD", "0") # nothing may download
monkeypatch.setenv("LAYA_AUTO_TASK", "1") # the config that puts three checkpoints in play
monkeypatch.delenv("LAYA_MAX_LOADED", raising=False)
+ monkeypatch.delenv("LAYA_DEFAULT_MODEL", raising=False)
for name, value in env.items():
monkeypatch.setenv(name, value)
router = build_router()
@@ -574,6 +575,83 @@ def test_raising_the_cap_stops_the_server_rebuilding_a_checkpoint(monkeypatch):
assert sorted(roomy.loaded) == sorted(DEFAULT_MODELS)
+# README's other answer written as a constructor argument is `Router(default="multilingual")`,
+# given where it explains that very short Latin-script text ("Quero cancelar", "Esqueci minha
+# senha") carries nothing identifying its language and so goes to `default`. Both packaged
+# servers build their Router from the environment and neither could pass it: the only reader of
+# LAYA_DEFAULT_MODEL was examples/server.py, so the demo server honoured the variable while
+# laya-serve routed the ambiguous states to the English checkpoint it was told not to use.
+#
+# Unlike the numeric knobs above, an unresolvable name is fatal here. `LAYA_MAX_LOADED=abc`
+# falling back to 2 costs a rebuild; `LAYA_DEFAULT_MODEL=mutli` falling back to english costs the
+# wrong checkpoint answering every ambiguous state, which is the failure the operator was trying
+# to configure away. So the message is core's and the server refuses to start.
+def test_default_model_reaches_the_router_the_server_builds(monkeypatch):
+ from laya.router import DEFAULT_MODELS, Router, _ALIASES, normalise_name
+ from laya.serve import _default_model_option
+
+ # Unset is "not asked for", not a copy of Router's default -- the same reason
+ # `_resolve_max_loaded` returns None, and the same drift it prevents.
+ monkeypatch.delenv("LAYA_DEFAULT_MODEL", raising=False)
+ assert _default_model_option() == {}
+ for blank in ("", " ", "\n"):
+ monkeypatch.setenv("LAYA_DEFAULT_MODEL", blank)
+ assert _default_model_option() == {}, repr(blank)
+
+ # The accepted spellings are core's tables read out of core, so a name added to _ALIASES
+ # arrives here on its own -- and a value this module resolves differently from the way
+ # `Router` itself resolves it fails the same line.
+ for name in sorted(set(DEFAULT_MODELS) | set(_ALIASES)):
+ monkeypatch.setenv("LAYA_DEFAULT_MODEL", name)
+ assert _default_model_option() == {"default": normalise_name(name)}, name
+
+ # And it reaches the Router the server builds. The literal is the one the README and
+ # docs/docker.md quote, so moving Router's default has to move those too.
+ router, _ = _server_router(monkeypatch)
+ assert router.default == Router().default
+ assert router.default == "english"
+ for raw, want in (("multilingual", "multilingual"), ("ml", "multilingual"),
+ (" MULTI ", "multilingual"), ("typed-decisions", "typed-decisions")):
+ router, _ = _server_router(monkeypatch, LAYA_DEFAULT_MODEL=raw)
+ assert router.default == want, raw
+
+ # The decision the whole knob exists to change, measured at `route()` rather than at the
+ # attribute -- `route` documents that it decides "without loading or running anything", so
+ # this costs no weights.
+ ambiguous = ("12345 !!!", "Quero cancelar", "Esqueci minha senha")
+ stock, _ = _server_router(monkeypatch)
+ portuguese, _ = _server_router(monkeypatch, LAYA_DEFAULT_MODEL="multilingual")
+ for state in ambiguous:
+ assert stock.route(state).model == "english", state
+ assert "using default (english)" in stock.route(state).reason, state
+ assert portuguese.route(state).model == "multilingual", state
+ assert "using default (multilingual)" in portuguese.route(state).reason, state
+ # A fallback, not a pin: text the detector can place routes on what it detects.
+ assert portuguese.route({"body": "Please refund the duplicate charge"}).model == "english"
+
+
+def test_an_unresolvable_default_stops_the_server_with_core_s_words(monkeypatch):
+ from laya.serve import build_router
+
+ monkeypatch.setenv("LAYA_PRELOAD", "0")
+ monkeypatch.delenv("LAYA_DEFAULT_MODEL", raising=False)
+ monkeypatch.setenv("LAYA_DEFAULT_MODEL", "mutli-lingual")
+ with pytest.raises(SystemExit) as raised:
+ build_router()
+ message = str(raised.value)
+ # Names the variable, quotes the value the operator typed, and then says in core's own words
+ # what the accepted set is -- so the typo is fixed from the message, not from the source.
+ assert message.startswith("invalid LAYA_DEFAULT_MODEL 'mutli-lingual': unknown model"), message
+ for name in ("english", "multilingual", "typed-decisions", "ml"):
+ assert name in message, message
+ # The contrast with the numeric knobs has to stay the contrast: with the name fixed, the bad
+ # number below still falls back instead of stopping the server.
+ monkeypatch.setenv("LAYA_DEFAULT_MODEL", "english")
+ monkeypatch.setenv("LAYA_MAX_LOADED", "abc")
+ assert build_router().default == "english"
+ assert build_router().max_loaded == 2
+
+
# The endpoint is `async def` and inference is synchronous torch, which on CPU takes
# hundreds of milliseconds to seconds. Calling it from the coroutine puts that work on