Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(examples): example 24 must state the resident cap Router actually defaults to
Aashish83f8dd8cec9b
diff --git a/examples/24_preload_and_memory.py b/examples/24_preload_and_memory.pyindex 189728f9..4e527ffb 100644--- a/examples/24_preload_and_memory.py+++ b/examples/24_preload_and_memory.py@@ -3,20 +3,30 @@Shows what stays hot in a `Router`: lazy loading with `max_loaded=1` evicting on a languageswitch, the reload cost that `preload` removes, `max_loaded=2` LRU behaviour, `attach()`adopting an already-built agent, and `unload()`.++`Router`'s own default cap is read from the library below rather than typed in here. It used to+be 1 and was raised to 2, so a page that copies the old number teaches readers to configure a cap+the library no longer has."""import time-from _common import STATE_EN, STATE_HI, banner, heading, router+from _common import STATE_EN, STATE_HI, banner, heading, laya, router, timed++# The library's default, taken from a `Router` instance. Constructing one loads nothing: agents are+# built lazily by `load()`, so this line costs no checkpoint and cannot drift from `laya/router.py`.+DEFAULT_MAX_LOADED = laya.Router().max_loadedbanner("24", "Preload and memory", """- A cold checkpoint build costs seconds; language detection costs microseconds. At the- default `max_loaded=1`, traffic that alternates languages rebuilds a model on every- switch. `router.loaded` is the residency view, least-recently-used first, so you can- watch models enter and leave as traffic moves between languages.+ A cold checkpoint build costs seconds; language detection costs microseconds. `Router` keeps+ `max_loaded=%d` checkpoints resident by default -- that number is read from an instance above+ rather than typed in here, so it tracks the library. To make eviction visible this demo pins+ one resident checkpoint (`router(max_loaded=1)`); under that setting traffic that alternates+ languages rebuilds a model on every switch. `router.loaded` is the residency view,+ least-recently-used first, so you can watch models enter and leave as traffic moves.A server preloads instead. Here we also show `attach()`, which registers an Agent theprocess already built rather than paying for -- and holding -- a duplicate copy.- """)+ """ % DEFAULT_MAX_LOADED)QUESTION = {"department": {@@ -39,7 +49,8 @@ heading("1. lazy load: one checkpoint hot")en_agent = r.load("english") # cold build #1out = r.predict(STATE_EN, QUESTION) # first forward also pays MPS kernel compilationprint(" routed to %r via %r" % (out["routing"]["model"], out["routing"]["reason"]))-print(" loaded: %r max_loaded=%d" % (r.loaded, r.max_loaded))+print(" loaded: %r max_loaded=%d (pinned here; laya's default is %d)"+ % (r.loaded, r.max_loaded, DEFAULT_MAX_LOADED))heading("2. a language switch evicts")ml_agent = r.load("multilingual") # cold build #2 -> evicts english (LRU)@@ -55,6 +66,11 @@ reload_s = time.perf_counter() - t0print(" reloading the 421M-parameter english checkpoint after eviction")print(" measured cold build: %.1f s" % reload_s)print(" loaded: %r" % r.loaded)+# `route()` decides which checkpoint *would* serve a state and loads nothing on the way (see+# `Router._route`), so this is exactly the work a cold build replaces. `timed` warms up first and+# returns the median, so the ratio at the end of this page compares two measured numbers.+_, route_ms = timed(lambda: r.route(STATE_HI, QUESTION))+print(" measured route for that same switch: %.3f ms" % route_ms)print(" With max_loaded=1 and alternating languages, a server pays that on every switch.")heading("4. max_loaded=2: least-recently-used is dropped")@@ -91,7 +107,9 @@ print(" hot.unload() -> %r" % hot.loaded)lru.unload()print(" lru.unload() -> %r" % lru.loaded)print("""- Rule of thumb: `Router(max_loaded=1)` is fine for a single-language service; anything- that alternates languages should `preload` (or raise `max_loaded`), because the reload- measured above is thousands of times more expensive than the routing itself.- """)+ Rule of thumb: a service that only ever sees one language can pin `Router(max_loaded=1)`.+ Anything that alternates languages should `preload`, or raise `max_loaded` -- laya's own+ default, which this page reads from an instance above, is %d. The cold reload measured in+ section 3 is not quoted from a document: it is %.0f times the route it replaces (%.1f s+ against %.3f ms), both timed here, so the ratio moves with the hardware.+ """ % (DEFAULT_MAX_LOADED, reload_s * 1000.0 / route_ms, reload_s, route_ms))