Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(router): keep two checkpoints by default, not one (#180)
devloper961-maker270e1bc1cb26
diff --git a/README.md b/README.mdindex 16744f96..ee355309 100644--- a/README.md+++ b/README.md@@ -141,7 +141,7 @@ The English checkpoint collapses on non-Latin scripts (Khmer scores **0.000 accu### Production Preload & Memory-A cold checkpoint build costs seconds; language detection costs microseconds. At the default `max_loaded=1`, traffic that alternates languages rebuilds a model on *every* request (measured at a 7.4 s median reload on CPU and 10.3 s on T4).+A cold checkpoint build costs seconds; language detection costs microseconds. The lazy default keeps **two** checkpoints resident — `english` and `multilingual`, the only two automatic routing chooses between — so a language flip costs detection only once each has been built. `max_loaded=1` rebuilds the checkpoint it just evicted on *every* switch (measured at a 7.4 s median reload on CPU and 10.3 s on T4), and traffic that only ever sees one language never builds the second, so the default costs a single-language deployment nothing.For a server or production app, preload:@@ -156,14 +156,16 @@ router.preload(["english", "multilingual"])# If your app already built an agent, attach it to avoid duplicate VRAM:router.attach("english", existing_agent)-# Manage resident memory (default keeps 1 hot, LRU eviction)-router = Router(max_loaded=2) # keep two hot+# Manage resident memory (default keeps two hot: english + multilingual, LRU eviction)+router = Router(max_loaded=3) # keep all three hot, e.g. with auto_task_detection+router = Router(max_loaded=1) # memory-constrained host, reloads on every switchrouter.unload() # free memory```| Deployment Mode | Per-Request Latency | Model Reloads ||---|---|---|-| `Router()` (lazy, `max_loaded=1`) | 7 to 10 s on every language switch | 1 per switch |+| `Router()` (lazy, `max_loaded=2`) | detection only (<1 ms) on a switch, after each language's first load | 1 the first time a language appears |+| `Router(max_loaded=1)` | 7 to 10 s on every language switch | 1 per switch || `Router(preload=True)` | **32.8 ms (GPU) / 193–464 ms (CPU)** | **none** |---diff --git a/laya/router.py b/laya/router.pyindex 6577bec9..3d79d7cc 100644--- a/laya/router.py+++ b/laya/router.py@@ -133,9 +133,16 @@ class Router:Models are downloaded and built on first use. `max_loaded` caps how many stay resident(least-recently-used is evicted), because all three together are ~1.16B parameters.+ The default is 2, because automatic routing only ever chooses between `english` and+ `multilingual`: a cap of one rebuilds the checkpoint it just evicted on every script switch,+ which is seconds per request on exactly the traffic the Router exists for. Traffic that only+ ever sees one language never builds the second checkpoint, so the default costs it nothing.+ Lower it to 1 for a memory-constrained host, and raise it to 3 (or preload) when+ `auto_task_detection`, an explicit `model=` or an explicit `task=` can reach+ `typed-decisions` as well.+For a server or a demo, preload instead: a cold load costs seconds, while detection costs- microseconds, so anything that alternates languages at `max_loaded=1` reloads on every- request.+ microseconds, so even the default still pays a load the first time a language appears.r = Router(preload=True) # all three resident, routing is freer = Router(preload=True, device="cuda")@@ -147,7 +154,7 @@ class Router:models: Optional[Dict[str, str]] = None,device: Optional[str] = None,token: Optional[str] = None,- max_loaded: int = 1,+ max_loaded: int = 2,default: str = "english",auto_task_detection: bool = False,standalone_repos: bool = False,diff --git a/tests/test_router.py b/tests/test_router.pyindex 6ca0be7c..9e9ffff6 100644--- a/tests/test_router.py+++ b/tests/test_router.py@@ -351,6 +351,47 @@ rr.unload()check("lru/unload all", rr.loaded, [])+# --------------------------------------------------------------------- default cap (#172)+# #172 measured a workload that alternates languages at 20-23 s per request on CPU (reloading a+# checkpoint every request) against 49-136 ms with both resident. Automatic routing only ever+# chooses between `english` and `multilingual`, so the default holds both, and a deployment that+# never alternates never builds the second.+def counting_router(cap=None):+ """Router whose loader records which checkpoints it had to build."""+ rr = Router() if cap is None else Router(max_loaded=cap)+ built = []++ def load(name, _rr=rr, _built=built):+ key = normalise_name(name)+ if key in _rr._agents:+ _rr._touch(key)+ return _rr._agents[key]+ _built.append(key)+ _rr._agents[key] = _Stub(key)+ _rr._order.append(key)+ _rr._evict()+ return _rr._agents[key]++ rr.load = load+ return rr, built+++check("lru/default is two", Router().max_loaded, 2)+_en = {"body": "I was charged twice for invoice 4411, please refund."}+_ml = {"body": "Der Kunde wurde zweimal belastet und moechte eine Rueckerstattung"}+for cap, want_built in ((1, 20), (2, 2)):+ cr, built = counting_router(cap)+ for _ in range(10): # the reported alternating workload+ cr.predict(_en, Q_GENERIC)+ cr.predict(_ml, Q_GENERIC)+ check("lru/alternating traffic, cap=%d builds" % cap, len(built), want_built)+# and the default costs a single-language deployment nothing at all+cr, built = counting_router()+for _ in range(5):+ cr.predict(_en, Q_GENERIC)+check("lru/single-language traffic builds one checkpoint", built, ["english"])++# --------------------------------------------------------------------- bundle vs standalonecheck("bundle/english is repo root", DEFAULT_MODELS["english"], (BUNDLE_REPO, None))check("bundle/multilingual subfolder", DEFAULT_MODELS["multilingual"], (BUNDLE_REPO, "multilingual"))