Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(serve): reach Router's resident-checkpoint cap from LAYA_MAX_LOADED
Aashish6c2f03ffadfe
diff --git a/README.md b/README.mdindex 54c20cf9..47242bb5 100644--- a/README.md+++ b/README.md@@ -457,7 +457,10 @@ curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{Configuration is by environment variable: `LAYA_HOST`, `LAYA_PORT`,`LAYA_DEVICE`, `LAYA_PRELOAD`, `LAYA_MODELS` (comma list to preload),`LAYA_THREADS` (cap torch intra-op threads for CPU inference — keep at or below-physical cores), `LAYA_AUTO_TASK`, and `LAYA_API_KEY` (when set, clients must+physical cores), `LAYA_AUTO_TASK`, `LAYA_MAX_LOADED` (checkpoints resident at+once, 2 by default; raise it to 3 when `LAYA_AUTO_TASK` makes a third one+reachable on demand, or the server rebuilds one every time routing switches),+and `LAYA_API_KEY` (when set, clients mustsend `Authorization: Bearer <key>`). A client's `model` field is honoured when itnames a Laya checkpoint (`english`/`multilingual`/`typed-decisions`), otherwisethe router auto-selects by script/language.diff --git a/compose.http.yaml b/compose.http.yamlindex 2c5c82aa..d5f0ffd3 100644--- a/compose.http.yaml+++ b/compose.http.yaml@@ -41,6 +41,9 @@ services:LAYA_MODELS: "${LAYA_MODELS:-}"LAYA_THREADS: "${LAYA_THREADS:-${OMP_NUM_THREADS:-4}}"LAYA_AUTO_TASK: "${LAYA_AUTO_TASK:-0}"+ # Empty keeps Router's own default (2). Set 3 when LAYA_AUTO_TASK is on: that adds a+ # third checkpoint routing can choose on demand, and a cap below it reloads one per switch.+ LAYA_MAX_LOADED: "${LAYA_MAX_LOADED:-}"LAYA_LOG_LEVEL: "${LAYA_LOG_LEVEL:-info}"LAYA_API_KEY: "${LAYA_API_KEY:-}"LAYA_API_KEY_FILE: "${LAYA_API_KEY_FILE:-}"diff --git a/docs/docker.md b/docs/docker.mdindex 27487236..ae6c704f 100644--- a/docs/docker.md+++ b/docs/docker.md@@ -225,6 +225,7 @@ These apply to the `laya-serve` service only.| `LAYA_MODELS` | (all) | comma list to preload: `english,multilingual,typed-decisions` || `LAYA_THREADS` | `OMP_NUM_THREADS` | caps torch intra-op threads; keep at or below physical cores || `LAYA_AUTO_TASK` | `0` | `1` lets the router reach `typed-decisions` automatically |+| `LAYA_MAX_LOADED` | `2` | Checkpoints kept resident; `LAYA_AUTO_TASK` makes a third reachable on demand, and a cap below what routing chooses rebuilds one per switch || `LAYA_LOG_LEVEL` | `info` | uvicorn log level || `LAYA_API_KEY` | (none) | when set, requires `Authorization: Bearer <key>` |diff --git a/laya/serve.py b/laya/serve.pyindex 2cd30922..012ed2e2 100644--- a/laya/serve.py+++ b/laya/serve.py@@ -24,6 +24,8 @@ env var meaning defaultKeep <= physical cores; oversubscribing the default)logical/hyperthread count is a large regression.``LAYA_AUTO_TASK`` auto-route to the typed-decisions checkpoint 0+``LAYA_MAX_LOADED`` checkpoints kept resident at once. Below what 2+ routing can choose, one reloads per switch.``LAYA_API_KEY`` if set, require ``Authorization: Bearer <it>`` (none)``LAYA_LOG_LEVEL`` uvicorn log level info``LAYA_MAX_CONCURRENT`` cap on requests past auth at once; excess 16@@ -115,6 +117,31 @@ def _resolve_max_concurrent() -> int:return n if n > 0 else DEFAULT_MAX_CONCURRENT+def _resolve_max_loaded() -> Optional[int]:+ """Resident-checkpoint cap from ``LAYA_MAX_LOADED``; ``None`` leaves it to ``Router``.++ ``Router`` keeps two checkpoints resident, which is exactly the number automatic routing+ picks between. ``LAYA_AUTO_TASK`` adds `typed-decisions` as a third possibility, and a cap+ below the number of checkpoints in play unloads the least recently used one on every+ switch -- #172 measured 20-23 s per request rebuilding a checkpoint on CPU against+ 49-136 ms with it resident. The value is left out of the constructor when unset rather+ than defaulted to a copy of ``Router``'s own default here, so the two cannot drift.+ Unparseable or non-positive input falls back the way ``_resolve_max_concurrent`` does: a+ typo in a deployment file should not stop the server from starting.++ Only checkpoints reached on demand are bounded by it: ``Router.preload()`` raises the cap+ to hold whatever it builds, so a lower value never evicts a preloaded checkpoint.+ """+ raw = os.environ.get("LAYA_MAX_LOADED")+ if not raw:+ return None+ try:+ n = int(raw)+ except ValueError:+ return None+ return n if n > 0 else None++def _resolve_port() -> int:"""Port from LAYA_PORT, validated. Exits with a message instead of a traceback."""raw = os.environ.get("LAYA_PORT", "8000")@@ -235,7 +262,11 @@ def build_router():device = os.environ.get("LAYA_DEVICE") or Nonemodels_env = os.environ.get("LAYA_MODELS", "").strip()preload_names = [m.strip() for m in models_env.split(",") if m.strip()] or None- router = Router(device=device, auto_task_detection=_env_bool("LAYA_AUTO_TASK", False))+ options = {"device": device, "auto_task_detection": _env_bool("LAYA_AUTO_TASK", False)}+ max_loaded = _resolve_max_loaded()+ if max_loaded is not None:+ options["max_loaded"] = max_loaded+ router = Router(**options)if _env_bool("LAYA_PRELOAD", True):router.preload(preload_names)return router