Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(serve): reach Router's resident-checkpoint cap from LAYA_MAX_LOADED

Aashish6c2f03ffadfe
diff --git a/README.md b/README.md
index 54c20cf9..47242bb5 100644
--- a/README.md
+++ b/README.md
@@ -457,7 +457,10 @@ curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
Configuration is by environment variable: `LAYA_HOST`, `LAYA_PORT`,
`LAYA_DEVICE`, `LAYA_PRELOAD`, `LAYA_MODELS` (comma list to preload),
`LAYA_THREADS` (cap torch intra-op threads for CPU inference — keep at or below
-physical cores), `LAYA_AUTO_TASK`, and `LAYA_API_KEY` (when set, clients must
+physical cores), `LAYA_AUTO_TASK`, `LAYA_MAX_LOADED` (checkpoints resident at
+once, 2 by default; raise it to 3 when `LAYA_AUTO_TASK` makes a third one
+reachable on demand, or the server rebuilds one every time routing switches),
+and `LAYA_API_KEY` (when set, clients must
send `Authorization: Bearer <key>`). A client's `model` field is honoured when it
names a Laya checkpoint (`english`/`multilingual`/`typed-decisions`), otherwise
the router auto-selects by script/language.
diff --git a/compose.http.yaml b/compose.http.yaml
index 2c5c82aa..d5f0ffd3 100644
--- a/compose.http.yaml
+++ b/compose.http.yaml
@@ -41,6 +41,9 @@ services:
LAYA_MODELS: "${LAYA_MODELS:-}"
LAYA_THREADS: "${LAYA_THREADS:-${OMP_NUM_THREADS:-4}}"
LAYA_AUTO_TASK: "${LAYA_AUTO_TASK:-0}"
+ # Empty keeps Router's own default (2). Set 3 when LAYA_AUTO_TASK is on: that adds a
+ # third checkpoint routing can choose on demand, and a cap below it reloads one per switch.
+ LAYA_MAX_LOADED: "${LAYA_MAX_LOADED:-}"
LAYA_LOG_LEVEL: "${LAYA_LOG_LEVEL:-info}"
LAYA_API_KEY: "${LAYA_API_KEY:-}"
LAYA_API_KEY_FILE: "${LAYA_API_KEY_FILE:-}"
diff --git a/docs/docker.md b/docs/docker.md
index 27487236..ae6c704f 100644
--- a/docs/docker.md
+++ b/docs/docker.md
@@ -225,6 +225,7 @@ These apply to the `laya-serve` service only.
| `LAYA_MODELS` | (all) | comma list to preload: `english,multilingual,typed-decisions` |
| `LAYA_THREADS` | `OMP_NUM_THREADS` | caps torch intra-op threads; keep at or below physical cores |
| `LAYA_AUTO_TASK` | `0` | `1` lets the router reach `typed-decisions` automatically |
+| `LAYA_MAX_LOADED` | `2` | Checkpoints kept resident; `LAYA_AUTO_TASK` makes a third reachable on demand, and a cap below what routing chooses rebuilds one per switch |
| `LAYA_LOG_LEVEL` | `info` | uvicorn log level |
| `LAYA_API_KEY` | (none) | when set, requires `Authorization: Bearer <key>` |
diff --git a/laya/serve.py b/laya/serve.py
index 2cd30922..012ed2e2 100644
--- a/laya/serve.py
+++ b/laya/serve.py
@@ -24,6 +24,8 @@ env var meaning default
Keep <= physical cores; oversubscribing the default)
logical/hyperthread count is a large regression.
``LAYA_AUTO_TASK`` auto-route to the typed-decisions checkpoint 0
+``LAYA_MAX_LOADED`` checkpoints kept resident at once. Below what 2
+ routing can choose, one reloads per switch.
``LAYA_API_KEY`` if set, require ``Authorization: Bearer <it>`` (none)
``LAYA_LOG_LEVEL`` uvicorn log level info
``LAYA_MAX_CONCURRENT`` cap on requests past auth at once; excess 16
@@ -115,6 +117,31 @@ def _resolve_max_concurrent() -> int:
return n if n > 0 else DEFAULT_MAX_CONCURRENT
+def _resolve_max_loaded() -> Optional[int]:
+ """Resident-checkpoint cap from ``LAYA_MAX_LOADED``; ``None`` leaves it to ``Router``.
+
+ ``Router`` keeps two checkpoints resident, which is exactly the number automatic routing
+ picks between. ``LAYA_AUTO_TASK`` adds `typed-decisions` as a third possibility, and a cap
+ below the number of checkpoints in play unloads the least recently used one on every
+ switch -- #172 measured 20-23 s per request rebuilding a checkpoint on CPU against
+ 49-136 ms with it resident. The value is left out of the constructor when unset rather
+ than defaulted to a copy of ``Router``'s own default here, so the two cannot drift.
+ Unparseable or non-positive input falls back the way ``_resolve_max_concurrent`` does: a
+ typo in a deployment file should not stop the server from starting.
+
+ Only checkpoints reached on demand are bounded by it: ``Router.preload()`` raises the cap
+ to hold whatever it builds, so a lower value never evicts a preloaded checkpoint.
+ """
+ raw = os.environ.get("LAYA_MAX_LOADED")
+ if not raw:
+ return None
+ try:
+ n = int(raw)
+ except ValueError:
+ return None
+ return n if n > 0 else None
+
+
def _resolve_port() -> int:
"""Port from LAYA_PORT, validated. Exits with a message instead of a traceback."""
raw = os.environ.get("LAYA_PORT", "8000")
@@ -235,7 +262,11 @@ def build_router():
device = os.environ.get("LAYA_DEVICE") or None
models_env = os.environ.get("LAYA_MODELS", "").strip()
preload_names = [m.strip() for m in models_env.split(",") if m.strip()] or None
- router = Router(device=device, auto_task_detection=_env_bool("LAYA_AUTO_TASK", False))
+ options = {"device": device, "auto_task_detection": _env_bool("LAYA_AUTO_TASK", False)}
+ max_loaded = _resolve_max_loaded()
+ if max_loaded is not None:
+ options["max_loaded"] = max_loaded
+ router = Router(**options)
if _env_bool("LAYA_PRELOAD", True):
router.preload(preload_names)
return router