Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
Merge pull request #604 from aashish254/fix-demo-max-loaded
Nandakishord28aef54f6a4
diff --cc tests/test_example_server_limits.pyindex f926e596,44750662..8287584a--- a/tests/test_example_server_limits.py+++ b/tests/test_example_server_limits.py@@@ -8,17 -8,40 +8,44 @@@ examples/server.py bounded `states` toa 5 MB state were both accepted where the shipped server answers 413. The bounds areread from laya.serve rather than restated, so the two cannot drift.-Scope: the question count and the state size, answered 413 as laya.serve answers them, and-the resident-checkpoint cap the app's Router is built with. `Question.instructions` and-`criteria` still carry unbounded text that no per-field bound can see; capping the request-body is the backstop for those and is left out deliberately -- see the PR description.+Scope: the question count, the state size, and the per-question and total option counts,+answered 413 as laya.serve answers them. The option budgets matter for the same reason the+other two do -- a choice or score question encodes one sequence per option, and those+sequences share the head budget -- so a request laya.serve refuses with 413 must not be+answered here. The bounds are read from laya.serve, never restated.+`Question.instructions` and `criteria` still carry unbounded *text* that no per-field+bound can see; capping the request body is the backstop for those and is left out+deliberately -- see the PR description.Driven over HTTP through TestClient. No weights are loaded; the router stays unbuilt, so- a request that passes validation answers 503, which is the assertion for "accepted".+ a request that passes validation answers 503, which is the assertion for "accepted". The+ one arm that needs a Router enters the lifespan, which is also the arm that checks the cap.++ On `main` this file is 19 checks, all of them about request size. It is now 38: the other+ nineteen follow the cap from the environment to the constructor, to the running Router, to+ `/health` in both JSON and HTML, and back out through the `--reload` push. The first+ nineteen are untouched.++ FAILS_ON_MAIN -- this file, against `main`'s `examples/server.py` at 4066d5d:++ PASS the page and health endpoints are unaffected+ FAIL an unset LAYA_MAX_LOADED means 'not asked for', not a number of this file's own 1+ AttributeError: module 'server' has no attribute '_router_kwargs'++ One check names the number, then the run aborts because the helper it drives does not exist+ there. So the same claims are checked on `main` through nothing but the public surface -- its+ own lifespan, the Router that builds, `/health` in both representations -- which prints:++ Router()'s own default = 2 built Router's cap = 1 /health JSON = 1+ /health HTML says = 'Checkpoints are loaded on demand; up to 1 kept in memory.'+ preload() lifts the cap to -> 3, while the page prints 1++ and `False` for "the cap came from the environment", "the Router holds laya's default", and+ "that default is more than one"; on this branch all three read `True` and the cap reads 2.+ That last line is the half that is not about the default at all: `Router.preload()` raises the+ cap to fit what it preloads (`laya/router.py:275` -> `:372`), so the demo's own default mode ran+ with three checkpoints resident while `/health` reported one. Nothing in the file ever asked the+ running Router what it was holding.Run: python tests/test_example_server_limits.py"""@@@ -195,45 -164,111 +227,150 @@@ def main()ok("the page and health endpoints are unaffected",client.get("/").status_code == 200 and client.get("/health").status_code == 200)+ # --- the two surfaces must reach the same verdict on the same numbers ---------+ # The demo reads `Question` instances where serve reads plain dicts, so the counting+ # lives behind a shape branch that could drift from serve's. Pin the verdicts against+ # each other rather than restating either one: serve's validator is called with the+ # plain dicts it actually receives, the demo through HTTP with the models it actually+ # receives, and the two have to agree.+ import laya.serve as _serve_mod # noqa: E402++ def serve_verdict(state, questions):+ try:+ _serve_mod._check_request_limits(state, questions)+ except Exception: # noqa: BLE001 -- HTTPException carries the 413+ return "refused"+ return "accepted"++ def demo_verdict(state, questions):+ return "refused" if client.post("/predict", json={"state": state,+ "questions": questions}).status_code == 413 else "accepted"++ noul_crit = {"type": "noul", "instructions": "True?",+ "criteria": {"false": "no", "true": "yes"}}+ parity_cases = [+ ("101 choice options", "hi", choice(MAX_CHOICE_OPTIONS + 1)),+ ("33 score levels", "hi", score(MAX_SCORE_LEVELS + 1)),+ ("600 options total", "hi",+ dict(("q%d" % i, choice(20, "q%d" % i)["q%d" % i]) for i in range(30))),+ ("exactly 100 choice options", "hi", choice(MAX_CHOICE_OPTIONS)),+ ("exactly 32 score levels", "hi", score(MAX_SCORE_LEVELS)),+ ("noul criteria only, near the total", "hi",+ dict([("c%d" % i, choice(MAX_CHOICE_OPTIONS, "c%d" % i)["c%d" % i]) for i in range(5)]+ + [("last", choice(12, "last")["last"])]+ + [("n%d" % i, noul_crit) for i in range(30)])),+ ("noul only", "hi", {"n%d" % i: noul_crit for i in range(10)}),+ ("an ordinary request", {"body": "billed twice"}, one),+ ]+ for label, st, qs in parity_cases:+ s, d = serve_verdict(st, qs), demo_verdict(st, qs)+ ok("parity with laya.serve: %s" % label, s == d, "serve=%s demo=%s" % (s, d))++ # --- the resident-checkpoint cap: derived, not copied -------------------+ # examples/server.py used to build its Router with `max_loaded=1`, a copy of a default+ # laya/router.py retired in #180. A cap of one cannot hold both english and+ # multilingual, so the demo ran the churn #172 measured and fixed -- one checkpoint+ # rebuilt per alternating-language request -- while the library and laya.serve did not.+ # What is asserted here is that the demo asks for nothing unless asked to, and that+ # what it then reports is the number the running Router holds.+ from laya.router import Router++ ok("an unset LAYA_MAX_LOADED means 'not asked for', not a number of this file's own",+ demo._CFG["max_loaded"] is None, repr(demo._CFG["max_loaded"]))+ ok("so the key never reaches the constructor",+ "max_loaded" not in demo._router_kwargs(demo._CFG),+ repr(demo._router_kwargs(demo._CFG)))+ # Before a Router exists the page has no resident count to state. This is the check+ # that fails if the page goes back to guessing one (`cfg.get('max_loaded', 1)`).+ ok("and while loading, the page states no resident count",+ "kept in memory" not in client.get("/health", headers={"accept": "text/html"}).text,+ client.get("/health", headers={"accept": "text/html"}).text[:200])++ with client: # the lifespan builds the Router; preload is off+ # The witness that this arm needed no weights: LAYA_PRELOAD=0 above means the+ # Router the app built holds no checkpoint yet, so the cap is the only thing here+ # that could have been loaded.+ ok("building it downloaded nothing",+ not demo.ROUTER._agents, repr(sorted(demo.ROUTER._agents)))+ cap = demo.ROUTER.max_loaded+ ok("the running Router holds laya's own default",+ cap == Router().max_loaded, "demo %r vs laya %r" % (cap, Router().max_loaded))+ ok("and laya's own default is more than one checkpoint",+ cap > 1, repr(cap))+ ok("/health reports the cap the Router really holds",+ client.get("/health").json()["config"]["max_loaded"] == cap, repr(cap))+ ok("and the HTML page prints that same number",+ ("up to %d kept in memory" % cap)+ in client.get("/health", headers={"accept": "text/html"}).text, repr(cap))+ # The number cannot come from the requested config, because the config and the+ # Router are allowed to disagree: `Router(preload=True)` raises its own cap to fit+ # what it preloads (laya/router.py:275 -> :372), so the demo's default mode has run+ # with three resident while its env said one. Move the Router and the page must move.+ demo.ROUTER.max_loaded = cap + 1+ moved = client.get("/health")+ ok("and both follow the Router when the Router raises its own cap",+ moved.json()["config"]["max_loaded"] == cap + 1+ and ("up to %d kept in memory" % (cap + 1))+ in client.get("/health", headers={"accept": "text/html"}).text,+ repr(moved.json()["config"]))++ def with_cap(value):+ """Re-import the demo the way uvicorn starts it, with LAYA_MAX_LOADED set to `value`."""+ if value is None:+ os.environ.pop("LAYA_MAX_LOADED", None)+ else:+ os.environ["LAYA_MAX_LOADED"] = value+ return importlib.reload(demo)++ # The default moving must not take the operator's own number away with it.+ for raw, want in (("3", 3), ("1", 1), (" 4 ", 4)):+ fresh = with_cap(raw)+ with TestClient(fresh.app) as c:+ ok("LAYA_MAX_LOADED=%r still reaches the Router" % raw,+ fresh.ROUTER.max_loaded == want, repr(fresh.ROUTER.max_loaded))+ ok("LAYA_MAX_LOADED=%r still shows up in /health" % raw,+ c.get("/health").json()["config"]["max_loaded"] == want, raw)+ fresh = with_cap(None)+ from laya.serve import build_router++ ok("the demo and laya.serve leave the cap to the same place",+ fresh._router_kwargs(fresh._CFG).get("max_loaded", Router().max_loaded)+ == build_router().max_loaded,+ "%r vs %r" % (fresh._router_kwargs(fresh._CFG), build_router().max_loaded))++ # --- the --reload env push has to survive the round trip ----------------+ # With reload=True uvicorn re-imports `server:app` in a child process and only the+ # environment crosses over. Writing str(None) for "not asked for" would land on the+ # int() that reads LAYA_MAX_LOADED in that child and stop the server at import, so the+ # unset case must push nothing at all. uvicorn is replaced so main() never binds a port.+ real_uvicorn = sys.modules.get("uvicorn")+ started = {}+ fake = types.ModuleType("uvicorn")+ fake.run = lambda target, **kw: started.update(target=target)+ sys.modules["uvicorn"] = fake+ argv = sys.argv++ def start(args):+ with_cap(None)+ sys.argv = ["server.py"] + args+ demo.main()+ return os.environ.get("LAYA_MAX_LOADED", "(absent)")++ try:+ ok("--reload with no flag pushes no cap, so the reimport can read it",+ start(["--reload"]) == "(absent)", repr(started))+ ok("--reload --max-loaded 3 pushes 3",+ start(["--reload", "--max-loaded", "3"]) == "3", repr(started))+ ok("without --reload nothing is pushed at all",+ start([]) == "(absent)", repr(started))+ finally:+ sys.argv = argv+ if real_uvicorn is not None:+ sys.modules["uvicorn"] = real_uvicorn+ else:+ sys.modules.pop("uvicorn", None)+ with_cap(None) # leave the module as the rest of the file found it+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL:print(" FAIL " + f)