Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(router): keep two checkpoints by default, not one (#180)

devloper961-maker270e1bc1cb26
diff --git a/README.md b/README.md
index 16744f96..ee355309 100644
--- a/README.md
+++ b/README.md
@@ -141,7 +141,7 @@ The English checkpoint collapses on non-Latin scripts (Khmer scores **0.000 accu
### Production Preload & Memory
-A cold checkpoint build costs seconds; language detection costs microseconds. At the default `max_loaded=1`, traffic that alternates languages rebuilds a model on *every* request (measured at a 7.4 s median reload on CPU and 10.3 s on T4).
+A cold checkpoint build costs seconds; language detection costs microseconds. The lazy default keeps **two** checkpoints resident — `english` and `multilingual`, the only two automatic routing chooses between — so a language flip costs detection only once each has been built. `max_loaded=1` rebuilds the checkpoint it just evicted on *every* switch (measured at a 7.4 s median reload on CPU and 10.3 s on T4), and traffic that only ever sees one language never builds the second, so the default costs a single-language deployment nothing.
For a server or production app, preload:
@@ -156,14 +156,16 @@ router.preload(["english", "multilingual"])
# If your app already built an agent, attach it to avoid duplicate VRAM:
router.attach("english", existing_agent)
-# Manage resident memory (default keeps 1 hot, LRU eviction)
-router = Router(max_loaded=2) # keep two hot
+# Manage resident memory (default keeps two hot: english + multilingual, LRU eviction)
+router = Router(max_loaded=3) # keep all three hot, e.g. with auto_task_detection
+router = Router(max_loaded=1) # memory-constrained host, reloads on every switch
router.unload() # free memory
```
| Deployment Mode | Per-Request Latency | Model Reloads |
|---|---|---|
-| `Router()` (lazy, `max_loaded=1`) | 7 to 10 s on every language switch | 1 per switch |
+| `Router()` (lazy, `max_loaded=2`) | detection only (<1 ms) on a switch, after each language's first load | 1 the first time a language appears |
+| `Router(max_loaded=1)` | 7 to 10 s on every language switch | 1 per switch |
| `Router(preload=True)` | **32.8 ms (GPU) / 193–464 ms (CPU)** | **none** |
---
diff --git a/laya/router.py b/laya/router.py
index 6577bec9..3d79d7cc 100644
--- a/laya/router.py
+++ b/laya/router.py
@@ -133,9 +133,16 @@ class Router:
Models are downloaded and built on first use. `max_loaded` caps how many stay resident
(least-recently-used is evicted), because all three together are ~1.16B parameters.
+ The default is 2, because automatic routing only ever chooses between `english` and
+ `multilingual`: a cap of one rebuilds the checkpoint it just evicted on every script switch,
+ which is seconds per request on exactly the traffic the Router exists for. Traffic that only
+ ever sees one language never builds the second checkpoint, so the default costs it nothing.
+ Lower it to 1 for a memory-constrained host, and raise it to 3 (or preload) when
+ `auto_task_detection`, an explicit `model=` or an explicit `task=` can reach
+ `typed-decisions` as well.
+
For a server or a demo, preload instead: a cold load costs seconds, while detection costs
- microseconds, so anything that alternates languages at `max_loaded=1` reloads on every
- request.
+ microseconds, so even the default still pays a load the first time a language appears.
r = Router(preload=True) # all three resident, routing is free
r = Router(preload=True, device="cuda")
@@ -147,7 +154,7 @@ class Router:
models: Optional[Dict[str, str]] = None,
device: Optional[str] = None,
token: Optional[str] = None,
- max_loaded: int = 1,
+ max_loaded: int = 2,
default: str = "english",
auto_task_detection: bool = False,
standalone_repos: bool = False,
diff --git a/tests/test_router.py b/tests/test_router.py
index 6ca0be7c..9e9ffff6 100644
--- a/tests/test_router.py
+++ b/tests/test_router.py
@@ -351,6 +351,47 @@ rr.unload()
check("lru/unload all", rr.loaded, [])
+# --------------------------------------------------------------------- default cap (#172)
+# #172 measured a workload that alternates languages at 20-23 s per request on CPU (reloading a
+# checkpoint every request) against 49-136 ms with both resident. Automatic routing only ever
+# chooses between `english` and `multilingual`, so the default holds both, and a deployment that
+# never alternates never builds the second.
+def counting_router(cap=None):
+ """Router whose loader records which checkpoints it had to build."""
+ rr = Router() if cap is None else Router(max_loaded=cap)
+ built = []
+
+ def load(name, _rr=rr, _built=built):
+ key = normalise_name(name)
+ if key in _rr._agents:
+ _rr._touch(key)
+ return _rr._agents[key]
+ _built.append(key)
+ _rr._agents[key] = _Stub(key)
+ _rr._order.append(key)
+ _rr._evict()
+ return _rr._agents[key]
+
+ rr.load = load
+ return rr, built
+
+
+check("lru/default is two", Router().max_loaded, 2)
+_en = {"body": "I was charged twice for invoice 4411, please refund."}
+_ml = {"body": "Der Kunde wurde zweimal belastet und moechte eine Rueckerstattung"}
+for cap, want_built in ((1, 20), (2, 2)):
+ cr, built = counting_router(cap)
+ for _ in range(10): # the reported alternating workload
+ cr.predict(_en, Q_GENERIC)
+ cr.predict(_ml, Q_GENERIC)
+ check("lru/alternating traffic, cap=%d builds" % cap, len(built), want_built)
+# and the default costs a single-language deployment nothing at all
+cr, built = counting_router()
+for _ in range(5):
+ cr.predict(_en, Q_GENERIC)
+check("lru/single-language traffic builds one checkpoint", built, ["english"])
+
+
# --------------------------------------------------------------------- bundle vs standalone
check("bundle/english is repo root", DEFAULT_MODELS["english"], (BUNDLE_REPO, None))
check("bundle/multilingual subfolder", DEFAULT_MODELS["multilingual"], (BUNDLE_REPO, "multilingual"))