Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
"""Regression: examples/server.py must bound a request the way laya.serve does.
`laya/serve.py` refuses a request carrying more than MAX_QUESTIONS questions, a state
over MAX_STATE_CHARS, or a body over its own cap, because Laya encodes the state once
per question -- cost is questions x state size, collated into one tensor.
examples/server.py bounded `states` to 64 and left the rest open: 20 000 questions and
a 5 MB state were both accepted where the shipped server answers 413. The bounds are
read from laya.serve rather than restated, so the two cannot drift.
Scope: the question count, the state size, and the per-question and total option counts,
answered 413 as laya.serve answers them. The option budgets matter for the same reason the
other two do -- a choice or score question encodes one sequence per option, and those
sequences share the head budget -- so a request laya.serve refuses with 413 must not be
answered here. The bounds are read from laya.serve, never restated.
`Question.instructions` and `criteria` still carry unbounded *text* that no per-field
bound can see; capping the request body is the backstop for those and is left out
deliberately -- see the PR description.
Driven over HTTP through TestClient. No weights are loaded; the router stays unbuilt, so
a request that passes validation answers 503, which is the assertion for "accepted". The
one arm that needs a Router enters the lifespan, which is also the arm that checks the cap.
On `main` this file is 19 checks, all of them about request size. It is now 38: the other
nineteen follow the cap from the environment to the constructor, to the running Router, to
`/health` in both JSON and HTML, and back out through the `--reload` push. The first
nineteen are untouched.
FAILS_ON_MAIN -- this file, against `main`'s `examples/server.py` at 4066d5d:
PASS the page and health endpoints are unaffected
FAIL an unset LAYA_MAX_LOADED means 'not asked for', not a number of this file's own 1
AttributeError: module 'server' has no attribute '_router_kwargs'
One check names the number, then the run aborts because the helper it drives does not exist
there. So the same claims are checked on `main` through nothing but the public surface -- its
own lifespan, the Router that builds, `/health` in both representations -- which prints:
Router()'s own default = 2 built Router's cap = 1 /health JSON = 1
/health HTML says = 'Checkpoints are loaded on demand; up to 1 kept in memory.'
preload() lifts the cap to -> 3, while the page prints 1
and `False` for "the cap came from the environment", "the Router holds laya's default", and
"that default is more than one"; on this branch all three read `True` and the cap reads 2.
That last line is the half that is not about the default at all: `laya/router.py::Router.preload`
raises the cap to fit what it preloads, so the demo's own default mode ran with three checkpoints
resident while `/health` reported one. Nothing in the file ever asked the running Router what it
was holding.
`main` is at 66 checks now, and this file is at 81. The fifteen new ones are the request `model`
field. `laya.router.normalise_name` is core's one resolver -- trim, lower-case, alias table,
registry -- and `laya --model` and `laya/serve.py` both hand it the caller's string, but this demo
compared it against a tuple of the three checkpoint names. So on `main` the aliases, the checkpoint
ids with the `laya-` prefix, `typed_decisions`, and the upper-cased and space-padded form of every
one of them answer 422 here: 60 of the 67 spellings core resolves, which
`laya-bench/model_alias_witness_main.log` counts against `main` at 9d95567 through
`_check_model` and through `POST /predict` on a TestClient -- `{"model": "english"}` answers 503,
`{"model": "laya"}` answers 422, while `laya --model laya` pins the checkpoint. The new checks take
the probe from `laya.router`'s own tables rather than listing the aliases here, require the
canonical name to reach the Router, keep blank meaning "do not pin" as the playground sends it,
read the alias list back out of `/models`, its HTML page, the published request schema and the
README section for this server, and then add a fourth checkpoint to core at runtime, reload the
demo, and require it to be pinnable by name and by alias and to appear in that text with no edit to
this file.
Run: python tests/test_example_server_limits.py
"""
import ast
import importlib
import json
import os
import re as _re # `main()` already binds a local `re` further down; F811 off both
import sys
import types
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
os.environ.setdefault("LAYA_PRELOAD", "0")
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)
sys.path.insert(0, os.path.join(ROOT, "examples"))
PASS, FAIL = [], []
def ok(name, cond, detail=""):
(PASS if cond else FAIL).append("%s%s" % (name, (" -- " + detail) if detail and not cond else ""))
print(" %s %s%s" % ("PASS" if cond else "FAIL", name, (" " + detail) if detail and not cond else ""), flush=True)
def main():
try:
from fastapi.testclient import TestClient
except ImportError as exc:
# Only the optional serving stack may be missing. An ImportError naming anything
# else -- in particular `cannot import name MAX_QUESTIONS from laya.serve`, the
# drift this test exists to catch -- must fail rather than skip.
if (getattr(exc, "name", None) or "").split(".")[0] not in ("fastapi", "httpx", "starlette"):
raise
print("SKIP: fastapi/httpx not installed -- pip install laya[serve] httpx")
return 0
try:
# examples/server.py reads the environment at import, so the cap under test has to
# be absent/present *before* the import, never patched after it.
os.environ.pop("LAYA_MAX_LOADED", None)
import server as demo
except ImportError as exc:
if (getattr(exc, "name", None) or "").split(".")[0] not in ("fastapi", "httpx", "starlette", "multipart"):
raise
print("SKIP: examples/server.py needs the serve extra -- pip install laya[serve]")
return 0
from laya.serve import (
MAX_CHOICE_OPTIONS,
MAX_QUESTIONS,
MAX_SCORE_LEVELS,
MAX_STATE_CHARS,
MAX_TOTAL_OPTIONS,
)
client = TestClient(demo.app, raise_server_exceptions=False)
one = {"a": {"type": "noul", "instructions": "x"}}
def questions(n):
return {"q%d" % i: {"type": "noul", "instructions": "x"} for i in range(n)}
def choice(n_opts, qid="a"):
return {qid: {"type": "choice", "instructions": "Which?",
"criteria": {"opt%d" % i: "desc %d" % i for i in range(n_opts)}}}
def score(n_levels, qid="a"):
return {qid: {"type": "score", "instructions": "How bad?",
"criteria": ["level %d" % i for i in range(n_levels)]}}
def code(**kw):
return client.post("/predict", **kw).status_code
# The bounds must come from laya.serve, not a local copy, or the two drift.
demo_q = getattr(demo, "MAX_QUESTIONS", None)
demo_s = getattr(demo, "MAX_STATE_CHARS", None)
ok("the per-request bounds are laya.serve's",
demo_q == MAX_QUESTIONS and demo_s == MAX_STATE_CHARS,
"demo %r/%r vs laya.serve %r/%r" % (demo_q, demo_s, MAX_QUESTIONS, MAX_STATE_CHARS))
demo_opts = (getattr(demo, "MAX_CHOICE_OPTIONS", None),
getattr(demo, "MAX_SCORE_LEVELS", None),
getattr(demo, "MAX_TOTAL_OPTIONS", None))
ok("the option budgets are laya.serve's too",
demo_opts == (MAX_CHOICE_OPTIONS, MAX_SCORE_LEVELS, MAX_TOTAL_OPTIONS),
"demo %r vs laya.serve %r" % (demo_opts, (MAX_CHOICE_OPTIONS, MAX_SCORE_LEVELS,
MAX_TOTAL_OPTIONS)))
# --- too many questions, too large a state: 413, as laya.serve answers ---
ok("more than MAX_QUESTIONS questions is 413",
code(json={"state": "hi", "questions": questions(MAX_QUESTIONS + 1)}) == 413)
# A noul question carries no answer options, so an options-based budget cannot see
# it; the count is what has to be bounded.
ok("a noul-only flood is 413 (it carries no answer options)",
code(json={"state": "hi", "questions": questions(20_000)}) == 413)
ok("a state over MAX_STATE_CHARS is 413",
code(json={"state": "A" * (MAX_STATE_CHARS + 1), "questions": one}) == 413)
ok("an oversized dict state is 413",
code(json={"state": {"body": "A" * (MAX_STATE_CHARS + 1)}, "questions": one}) == 413)
ok("an oversized state inside a batch is 413",
client.post("/predict/batch",
json={"states": ["hi", "A" * (MAX_STATE_CHARS + 1)], "questions": one}
).status_code == 413)
# --- the option budgets, which laya.serve also answers 413 -----------------
# A choice or score question encodes one sequence per option against a shared head
# budget, so the option counts are size limits for the same reason the state is.
ok("more than MAX_CHOICE_OPTIONS options in one question is 413",
code(json={"state": "hi", "questions": choice(MAX_CHOICE_OPTIONS + 1)}) == 413)
ok("more than MAX_SCORE_LEVELS levels in one question is 413",
code(json={"state": "hi", "questions": score(MAX_SCORE_LEVELS + 1)}) == 413)
# Under the per-question caps but over the shared total: the case a per-question check
# alone would still accept.
per_q = MAX_CHOICE_OPTIONS
n_questions = MAX_TOTAL_OPTIONS // per_q + 1
ok("over MAX_TOTAL_OPTIONS across questions is 413",
code(json={"state": "hi",
"questions": {"q%d" % i: choice(per_q, "q%d" % i)[ "q%d" % i]
for i in range(n_questions)}}) == 413,
"%d questions x %d options" % (n_questions, per_q))
# A noul question contributes no options, so it cannot move the total either way.
ok("a noul flood is still bounded by the question count, not the option total",
code(json={"state": "hi", "questions": questions(MAX_QUESTIONS)}) == 503)
# ...including when it carries the ordinary false/true criteria, which are option
# *texts* but not answer options. laya.serve adds them nowhere; a total that counted
# them would refuse a request serve accepts, which is the drift this file exists to
# catch -- in the direction of refusing something legal.
counted = 5 * MAX_CHOICE_OPTIONS + 12 # exactly MAX_TOTAL_OPTIONS by laya.serve's count
mixed = {"q%d" % i: choice(MAX_CHOICE_OPTIONS, "q%d" % i)["q%d" % i] for i in range(5)}
mixed["last"] = choice(12, "last")["last"]
for i in range(30): # 30 noul questions, 2 criteria each, 60 if wrongly counted
mixed["n%d" % i] = {"type": "noul", "instructions": "True?",
"criteria": {"false": "no", "true": "yes"}}
ok("noul criteria do not count toward the shared option total",
len(mixed) <= MAX_QUESTIONS and counted == MAX_TOTAL_OPTIONS
and code(json={"state": "hi", "questions": mixed}) == 503,
"%d questions, %d options by laya.serve's count" % (len(mixed), counted))
# --- the refusal must not echo the rejected payload back ----------------
# Declaring these as Field(max_length=...) instead would report 422 *and* include
# the offending `input` in FastAPI's validation-error body, so refusing a 5 MB
# state would write 5 MB back to the caller -- a size limit that amplifies.
big = {"state": "A" * 5_000_000, "questions": one}
sent = len(json.dumps(big).encode())
resp = client.post("/predict", json=big)
ok("a rejected 5 MB state answers 413 without echoing it",
resp.status_code == 413 and len(resp.content) < 1_000,
"%s, %d bytes returned for %d sent" % (resp.status_code, len(resp.content), sent))
# --- a genuine schema error is still 422, not 413 -----------------------
ok("an unknown question type is 422",
code(json={"state": "hi", "questions": {"a": {"type": "bogus", "instructions": "x"}}}) == 422)
ok("a choice question with no criteria is 422",
code(json={"state": "hi", "questions": {"a": {"type": "choice", "instructions": "x"}}}) == 422)
ok("an empty state is 422", code(json={"state": "", "questions": one}) == 422)
ok("zero questions is 422", code(json={"state": "hi", "questions": {}}) == 422)
# --- the limits are limits, not walls -----------------------------------
# No router is built, so anything that passes validation answers 503.
ok("an ordinary request passes validation",
code(json={"state": {"body": "billed twice, please refund"}, "questions": one}) == 503)
ok("exactly MAX_QUESTIONS passes",
code(json={"state": "hi", "questions": questions(MAX_QUESTIONS)}) == 503)
ok("a state of exactly MAX_STATE_CHARS passes",
code(json={"state": "A" * MAX_STATE_CHARS, "questions": one}) == 503)
ok("exactly MAX_CHOICE_OPTIONS options passes",
code(json={"state": "hi", "questions": choice(MAX_CHOICE_OPTIONS)}) == 503)
ok("exactly MAX_SCORE_LEVELS levels passes",
code(json={"state": "hi", "questions": score(MAX_SCORE_LEVELS)}) == 503)
ok("a list state passes", code(json={"state": ["a", "b"], "questions": one}) == 503)
ok("a small chunked body passes",
code(content=(lambda: (yield json.dumps({"state": "hi", "questions": one}).encode()))(),
headers={"content-type": "application/json"}) == 503)
# /predict/batch reports per-item failures inside a 200 envelope, so an accepted
# batch is a 200 here; over the state bound it is refused outright, and 413 rather
# than 422 because "too many states" is a size violation like the others.
ok("a 64-state batch is accepted",
client.post("/predict/batch", json={"states": ["hi"] * 64, "questions": one}).status_code == 200)
ok("a 65-state batch is still refused by the existing bound",
client.post("/predict/batch", json={"states": ["hi"] * 65, "questions": one}).status_code == 422)
ok("the page and health endpoints are unaffected",
client.get("/").status_code == 200 and client.get("/health").status_code == 200)
# --- the two surfaces must reach the same verdict on the same numbers ---------
# The demo reads `Question` instances where serve reads plain dicts, so the counting
# lives behind a shape branch that could drift from serve's. Pin the verdicts against
# each other rather than restating either one: serve's validator is called with the
# plain dicts it actually receives, the demo through HTTP with the models it actually
# receives, and the two have to agree.
import laya.serve as _serve_mod # noqa: E402
def serve_verdict(state, questions):
try:
_serve_mod._check_request_limits(state, questions)
except Exception: # noqa: BLE001 -- HTTPException carries the 413
return "refused"
return "accepted"
def demo_verdict(state, questions):
return "refused" if client.post("/predict", json={"state": state,
"questions": questions}).status_code == 413 else "accepted"
noul_crit = {"type": "noul", "instructions": "True?",
"criteria": {"false": "no", "true": "yes"}}
parity_cases = [
("101 choice options", "hi", choice(MAX_CHOICE_OPTIONS + 1)),
("33 score levels", "hi", score(MAX_SCORE_LEVELS + 1)),
("600 options total", "hi",
dict(("q%d" % i, choice(20, "q%d" % i)["q%d" % i]) for i in range(30))),
("exactly 100 choice options", "hi", choice(MAX_CHOICE_OPTIONS)),
("exactly 32 score levels", "hi", score(MAX_SCORE_LEVELS)),
("noul criteria only, near the total", "hi",
dict([("c%d" % i, choice(MAX_CHOICE_OPTIONS, "c%d" % i)["c%d" % i]) for i in range(5)]
+ [("last", choice(12, "last")["last"])]
+ [("n%d" % i, noul_crit) for i in range(30)])),
("noul only", "hi", {"n%d" % i: noul_crit for i in range(10)}),
("an ordinary request", {"body": "billed twice"}, one),
# A state whose `str()` and whose JSON serialization are different lengths (50000 vs 99988).
# This case catches the two surfaces DIVERGING -- one measuring the serialized text and the
# other `str(state)`. It cannot catch them being wrong together: `examples/server.py` binds
# `_state_length` from `laya.serve` by `getattr`, so a regression inside that helper moves
# both and parity stays green. Measured: regressing `_state_length` fails 0 of these parity
# checks and both of the absolute ones below. Those are the real coverage -- do not prune
# them as redundant.
("a state whose repr is half its JSON", {"body": '"' * 49988}, one),
]
for label, st, qs in parity_cases:
s, d = serve_verdict(st, qs), demo_verdict(st, qs)
ok("parity with laya.serve: %s" % label, s == d, "serve=%s demo=%s" % (s, d))
# The BATCH shape, which these parity cases could not reach: every case above drives
# `_check_request_limits` with one state, so nothing compared how the two surfaces SIZE a batch.
# `laya.serve` splits a batch so one forward pass stays inside its token budget rather than
# refusing it, and the demo must split it identically -- it binds the planner from `laya.serve` by
# getattr, so this checks the binding took AND that the plan reaches the call.
serve_plan = _serve_mod._batch_chunk_size
demo_plan = demo._batch_chunk_size
tiny = {"q": {"type": "noul", "instructions": "True?"}}
four = dict(("q%03d" % i, tiny["q"]) for i in range(4))
thirty_two = dict(("q%03d" % i, tiny["q"]) for i in range(32))
many = dict(("q%03d" % i, tiny["q"]) for i in range(MAX_QUESTIONS))
wide = _serve_mod.DEFAULT_MAX_TOKEN_BUDGET
for label, ns, nq, mlen in [
("an ordinary batch fits, no split", 2, len(four), None),
("64 states x 4 questions fits", 64, len(four), None),
("8 states x 32 questions fits", 8, len(thirty_two), None),
("9 states x 32 questions splits", 9, len(thirty_two), None),
("64 states x 64 questions splits", 64, len(many), None),
("a wide max_len splits harder", 8, len(thirty_two), wide),
("a wide max_len with few rows fits", 2, 1, wide)]:
a, b = serve_plan(ns, nq, mlen), demo_plan(ns, nq, mlen)
ok("batch plan parity with laya.serve: %s" % label, a == b, "serve=%r demo=%r" % (a, b))
# Equal on both sides is not enough -- they bind the same function, so both could be wrong
# together. The plan itself has to be the right shape.
ok("batch plan/a fitting batch is not split", serve_plan(8, len(thirty_two)) is None,
repr(serve_plan(8, len(thirty_two))))
split = serve_plan(64, len(many))
ok("batch plan/an oversized batch is split", split is not None and split >= 1, repr(split))
ok("batch plan/one pass fits the budget",
split * len(many) * _serve_mod._BATCH_ROW_TOKENS_ASSUMED
<= _serve_mod.DEFAULT_MAX_BATCH_TOKENS,
"%r states x %d questions" % (split, len(many)))
# And the demo's route sends it, rather than computing it and dropping it.
sent = {}
real_router = demo._router
class _Recorder:
def predict_batch(self, requests, **kwargs):
sent.update(kwargs)
sent["n"] = len(requests)
return [{"model": "m", "answers": {}, "usage": {"input_tokens": 1}} for _ in requests]
demo._router = lambda: _Recorder()
try:
code = client.post("/predict/batch",
json={"states": ["hi"] * 64, "questions": many}).status_code
ok("batch plan/the demo answers an oversized batch instead of refusing it", code == 200, code)
ok("batch plan/the demo forwards the planned batch_size",
sent.get("batch_size") == serve_plan(64, len(many)),
"sent=%r planned=%r" % (sent.get("batch_size"), serve_plan(64, len(many))))
ok("batch plan/every state is still sent", sent.get("n") == 64, sent.get("n"))
sent.clear()
client.post("/predict/batch", json={"states": ["hi"] * 2, "questions": four})
ok("batch plan/a fitting batch is passed no batch_size", "batch_size" not in sent, sent)
finally:
demo._router = real_router
# Parity alone cannot see a bug both surfaces share, and `getattr` guarantees they share one:
# with `_state_length` measuring `str(state)` again, both agree on accepting a state that
# serializes to 99 988 characters and every parity check above stays green (measured: 0 of them
# fail, both of these do). So the verdict itself is asserted, not just the agreement.
quote_heavy = {"body": '"' * 49988}
# Recorded through `ok` like everything else in this file: a bare `assert` here would raise out
# of `main()` and abandon the ~30 checks that follow instead of recording one failure.
ok("the quote-heavy fixture passes a str()-based gate",
len(str(quote_heavy)) <= MAX_STATE_CHARS, len(str(quote_heavy)))
ok("the quote-heavy fixture fails a serialization-based gate",
len(json.dumps(quote_heavy, ensure_ascii=False)) > MAX_STATE_CHARS,
len(json.dumps(quote_heavy, ensure_ascii=False)))
for who, verdict in (("laya.serve", serve_verdict(quote_heavy, one)),
("the demo server", demo_verdict(quote_heavy, one))):
ok("%s refuses a state whose JSON is twice its repr" % who, verdict == "refused", verdict)
# --- the resident-checkpoint cap: derived, not copied -------------------
# examples/server.py used to build its Router with `max_loaded=1`, a copy of a default
# laya/router.py retired in #180. A cap of one cannot hold both english and
# multilingual, so the demo ran the churn #172 measured and fixed -- one checkpoint
# rebuilt per alternating-language request -- while the library and laya.serve did not.
# What is asserted here is that the demo asks for nothing unless asked to, and that
# what it then reports is the number the running Router holds.
from laya.router import Router
ok("an unset LAYA_MAX_LOADED means 'not asked for', not a number of this file's own",
demo._CFG["max_loaded"] is None, repr(demo._CFG["max_loaded"]))
ok("so the key never reaches the constructor",
"max_loaded" not in demo._router_kwargs(demo._CFG),
repr(demo._router_kwargs(demo._CFG)))
# Before a Router exists the page has no resident count to state. This is the check
# that fails if the page goes back to guessing one (`cfg.get('max_loaded', 1)`).
ok("and while loading, the page states no resident count",
"kept in memory" not in client.get("/health", headers={"accept": "text/html"}).text,
client.get("/health", headers={"accept": "text/html"}).text[:200])
with client: # the lifespan builds the Router; preload is off
# The witness that this arm needed no weights: LAYA_PRELOAD=0 above means the
# Router the app built holds no checkpoint yet, so the cap is the only thing here
# that could have been loaded.
ok("building it downloaded nothing",
not demo.ROUTER._agents, repr(sorted(demo.ROUTER._agents)))
cap = demo.ROUTER.max_loaded
ok("the running Router holds laya's own default",
cap == Router().max_loaded, "demo %r vs laya %r" % (cap, Router().max_loaded))
ok("and laya's own default is more than one checkpoint",
cap > 1, repr(cap))
ok("/health reports the cap the Router really holds",
client.get("/health").json()["config"]["max_loaded"] == cap, repr(cap))
ok("and the HTML page prints that same number",
("up to %d kept in memory" % cap)
in client.get("/health", headers={"accept": "text/html"}).text, repr(cap))
# The number cannot come from the requested config, because the config and the
# Router are allowed to disagree: `Router(preload=True)` raises its own cap to fit what it
# preloads (`laya/router.py::Router.preload`), so the demo's default mode has run with
# three resident while its env said one. Move the Router and the page must move.
demo.ROUTER.max_loaded = cap + 1
moved = client.get("/health")
ok("and both follow the Router when the Router raises its own cap",
moved.json()["config"]["max_loaded"] == cap + 1
and ("up to %d kept in memory" % (cap + 1))
in client.get("/health", headers={"accept": "text/html"}).text,
repr(moved.json()["config"]))
def with_cap(value):
"""Re-import the demo the way uvicorn starts it, with LAYA_MAX_LOADED set to `value`."""
if value is None:
os.environ.pop("LAYA_MAX_LOADED", None)
else:
os.environ["LAYA_MAX_LOADED"] = value
return importlib.reload(demo)
# The default moving must not take the operator's own number away with it.
for raw, want in (("3", 3), ("1", 1), (" 4 ", 4)):
fresh = with_cap(raw)
with TestClient(fresh.app) as c:
ok("LAYA_MAX_LOADED=%r still reaches the Router" % raw,
fresh.ROUTER.max_loaded == want, repr(fresh.ROUTER.max_loaded))
ok("LAYA_MAX_LOADED=%r still shows up in /health" % raw,
c.get("/health").json()["config"]["max_loaded"] == want, raw)
fresh = with_cap(None)
from laya.serve import build_router
ok("the demo and laya.serve leave the cap to the same place",
fresh._router_kwargs(fresh._CFG).get("max_loaded", Router().max_loaded)
== build_router().max_loaded,
"%r vs %r" % (fresh._router_kwargs(fresh._CFG), build_router().max_loaded))
# A bad or non-positive LAYA_MAX_LOADED must fall back to "not asked for" the way
# laya.serve's `_resolve_max_loaded` documents ("a typo in a deployment file should not stop
# the server from starting"), not reach `int()` as an import crash and not forward a cap
# Router cannot honour. FAILS_ON_MAIN: the bare int() at module level aborts the reload.
for raw in ("abc", "2.5", "0", "-3", ""):
fresh = with_cap(raw)
ok("LAYA_MAX_LOADED=%r falls back to 'not asked for'" % raw,
"max_loaded" not in fresh._router_kwargs(fresh._CFG),
repr(fresh._router_kwargs(fresh._CFG)))
with_cap(None) # leave the ambient env clean for the checks below
# --- the routing fallback: derived, not copied --------------------------
# The copy retired above said a number. The one beside it named a checkpoint: 3cf26cb moved
# `Router(default=...)` from `english` to `multilingual` in 0.4.0, and `laya.serve` followed
# by sending no `default` keyword at all when LAYA_DEFAULT_MODEL is unset
# (`_default_model_option`), which `laya/mcp/server.py::_ensure_router` reuses rather than
# duplicates. This file went on constructing `Router(default="english")`, so one environment
# answered three ways: a bare `laya-serve` and a bare `laya mcp` took a state with no language
# evidence to the multilingual checkpoint, a bare `python examples/server.py` took it to the
# English arm. 3cf26cb's own migration note is that the old fallback is now something you ask
# for by name, so the pin here was not a demo of the old default -- it was the demo ignoring
# the change. `Router.route` decides without loading or running anything, so none of this
# needs weights.
def with_default(value):
"""Re-import the demo the way uvicorn starts it, with LAYA_DEFAULT_MODEL set to `value`."""
if value is None:
os.environ.pop("LAYA_DEFAULT_MODEL", None)
else:
os.environ["LAYA_DEFAULT_MODEL"] = value
return importlib.reload(demo)
fresh = with_default(None)
ok("an unset LAYA_DEFAULT_MODEL means 'not asked for', not a checkpoint of this file's own",
fresh._CFG["default"] is None, repr(fresh._CFG["default"]))
ok("so the key never reaches the constructor",
"default" not in fresh._router_kwargs(fresh._CFG),
repr(fresh._router_kwargs(fresh._CFG)))
# Blank has to mean the same thing at both surfaces. laya.serve treats "", " " and "\n" as
# unset (tests/test_serve.py pins all three against `_default_model_option`); a demo that read
# them as a name would raise at `Router(default="")` where the packaged server starts.
from laya.serve import _default_model_option
for blank in ("", " ", "\n"):
os.environ["LAYA_DEFAULT_MODEL"] = blank
ok("LAYA_DEFAULT_MODEL=%r is unset for laya.serve" % blank,
_default_model_option() == {}, repr(_default_model_option()))
fresh = with_default(blank)
ok("LAYA_DEFAULT_MODEL=%r is unset for the demo too" % blank,
"default" not in fresh._router_kwargs(fresh._CFG),
repr(fresh._router_kwargs(fresh._CFG)))
fresh = with_default(None)
with TestClient(fresh.app) as c:
ok("the Router the demo builds holds laya's own fallback",
fresh.ROUTER.default == Router().default,
"demo %r vs laya %r" % (fresh.ROUTER.default, Router().default))
ok("and laya's own fallback is the multilingual checkpoint",
fresh.ROUTER.default == "multilingual", repr(fresh.ROUTER.default))
# The decision, not the attribute: this is the branch 3cf26cb moved and #54 measured.
for state in ("12345 !!!", "Quero cancelar", "Esqueci minha senha"):
d = fresh.ROUTER.route(state)
ok("the demo routes %r off the fallback, from multilingual" % state,
d.model == "multilingual" and "using default (multilingual)" in d.reason,
"%r %r" % (d.model, d.reason))
ok("/models reports the fallback the running Router holds",
c.get("/models").json()["default"] == fresh.ROUTER.default,
repr(c.get("/models").json()["default"]))
# One badge, on the card the Router actually falls back to. The badge is rendered from
# `payload["default"]`, so this is the HTML half of the line above.
badged = [chunk.split("</h2>")[0] for chunk in
c.get("/models", headers={"accept": "text/html"}).text
.split("<section class='panel'>") if "data-s='ok'>default</span>" in chunk]
ok("and the page badges exactly one checkpoint, the multilingual one",
len(badged) == 1 and "<h2>multilingual" in badged[0], repr(badged)[:160])
# Before the lifespan has built anything, the page has no Router to ask, and must not guess.
fresh = with_default(None)
loading = TestClient(fresh.app) # entered without a lifespan, so ROUTER stays None
ok("while the app is loading, /models claims no fallback",
fresh.ROUTER is None and loading.get("/models").json()["default"] is None,
repr(fresh.ROUTER))
# The default moving must not take the operator's own answer away with it -- and an alias or
# a padded spelling resolves to the same checkpoint it resolves to in laya.serve.
for raw, want in (("english", "english"), ("ml", "multilingual"),
(" MULTI ", "multilingual"), ("typed-decisions", "typed-decisions")):
fresh = with_default(raw)
with TestClient(fresh.app) as cc:
ok("LAYA_DEFAULT_MODEL=%r still reaches the Router" % raw,
fresh.ROUTER.default == want, repr(fresh.ROUTER.default))
ok("LAYA_DEFAULT_MODEL=%r still shows up in /models" % raw,
cc.get("/models").json()["default"] == want,
repr(cc.get("/models").json()["default"]))
d = fresh.ROUTER.route("Quero cancelar")
ok("and %r decides that state from that checkpoint" % raw,
d.model == want and "using default (%s)" % want in d.reason,
"%r %r" % (d.model, d.reason))
fresh = with_default(None)
ok("the demo and laya.serve leave the fallback to the same place",
fresh._router_kwargs(fresh._CFG).get("default", Router().default) == build_router().default,
"%r vs %r" % (fresh._router_kwargs(fresh._CFG), build_router().default))
# --- the --reload env push has to survive the round trip ----------------
# With reload=True uvicorn re-imports `server:app` in a child process and only the
# environment crosses over. Writing str(None) for "not asked for" would land on the
# int() that reads LAYA_MAX_LOADED in that child and stop the server at import, so the
# unset case must push nothing at all. uvicorn is replaced so main() never binds a port.
# The same round trip now carries the fallback, where str(None) is worse than a bad number:
# the child reads it as a checkpoint literally named "None" and refuses to start over a
# configuration nobody asked for.
real_uvicorn = sys.modules.get("uvicorn")
started = {}
fake = types.ModuleType("uvicorn")
fake.run = lambda target, **kw: started.update(target=target)
sys.modules["uvicorn"] = fake
argv = sys.argv
def start(args):
with_cap(None)
sys.argv = ["server.py"] + args
demo.main()
return os.environ.get("LAYA_MAX_LOADED", "(absent)")
def start_default(args):
"""The same run, reporting what crossed over as the fallback."""
with_default(None)
sys.argv = ["server.py"] + args
demo.main()
return os.environ.get("LAYA_DEFAULT_MODEL", "(absent)")
try:
ok("--reload with no flag pushes no cap, so the reimport can read it",
start(["--reload"]) == "(absent)", repr(started))
ok("--reload --max-loaded 3 pushes 3",
start(["--reload", "--max-loaded", "3"]) == "3", repr(started))
ok("without --reload nothing is pushed at all",
start([]) == "(absent)", repr(started))
ok("--reload with no flag pushes no fallback either, so the child inherits laya's",
start_default(["--reload"]) == "(absent)", repr(started))
ok("--reload --default-model english pushes english",
start_default(["--reload", "--default-model", "english"]) == "english",
repr(started))
ok("and without --reload the flag is not pushed as a name",
start_default(["--default-model", "english"]) == "(absent)", repr(started))
ok("--default-model '' is the same as not passing it",
start_default(["--reload", "--default-model", ""]) == "(absent)", repr(started))
finally:
sys.argv = argv
if real_uvicorn is not None:
sys.modules["uvicorn"] = real_uvicorn
else:
sys.modules.pop("uvicorn", None)
with_cap(None) # leave the module as the rest of the file found it
with_default(None) # ... including the fallback the arms above pushed
# --- /predict/batch must make ONE Router call, not one per state ---------
#
# README teaches `Router.predict_batch` as the way to answer many states with shared forward
# passes ("routes the full workload first, groups requests by checkpoint ... results are
# restored to the original request order"), and this endpoint is the batch-shaped surface the
# README points at for trying Laya without writing code. The handler used to call
# `Router.predict` once per state anyway, so 64 states were 64 forwards. These drive the real
# app -- validation, `_questions()`, the handler body, the JSON envelope -- over a recording
# stand-in at `demo.ROUTER`, so no weights are needed to count the calls.
import inspect
from laya.router import Router as CoreRouter
class RecordingRouter:
"""Answers like the Router does, and remembers how it was asked."""
def __init__(self, fail_on=None):
self.predict_calls = []
self.batch_calls = []
self.fail_on = fail_on
def predict(self, state, questions, **kw):
self.predict_calls.append((state, dict(kw)))
return self._answer(state, questions)
def predict_batch(self, requests, **kw):
self.batch_calls.append((list(requests), dict(kw)))
return [self._answer(r["state"], r["questions"]) for r in requests]
def _answer(self, state, questions):
if self.fail_on and self.fail_on in str(state):
raise ValueError("simulated failure for " + self.fail_on)
return {"answers": {"a": {"type": "noul", "choice": False}}, "state": state,
"questions": questions}
states = ["ticket %d" % i for i in range(8)]
class LegacyRouter(RecordingRouter):
"""A Router-like object whose `predict_batch` predates the batch path entirely."""
predict_batch = None
# A Router that predates `predict_batch` must still be usable through the fallback.
older = LegacyRouter()
demo.ROUTER = older
legacy = TestClient(demo.app, raise_server_exceptions=False).post(
"/predict/batch", json={"states": states, "questions": one})
ok("a router without predict_batch still answers every state",
legacy.status_code == 200 and len(legacy.json()["results"]) == 8
and not [r for r in legacy.json()["results"] if "error" in r],
"%s / %s" % (legacy.status_code, json.dumps(legacy.json())[:200]))
router = RecordingRouter()
demo.ROUTER = router
batched = TestClient(demo.app, raise_server_exceptions=False).post(
"/predict/batch", json={"states": states, "questions": one})
body = batched.json()
ok("an 8-state batch is ONE predict_batch call and zero predict calls",
len(router.batch_calls) == 1 and not router.predict_calls,
"predict_batch=%d predict=%d" % (len(router.batch_calls), len(router.predict_calls)))
ok("the batch envelope is unchanged: count and one result per state, in order",
body.get("count") == 8 and [r.get("state") for r in body.get("results", [])] == states)
# Read defensively: an endpoint that never batches has nothing to inspect, and the checks below
# say so by name instead of letting this script die at the unpack.
requests_sent, call_kwargs = (router.batch_calls[0] if router.batch_calls else ([], {}))
ok("each request carries state + questions, and the questions map is the same one",
len(requests_sent) == 8
and all(sorted(r) == ["questions", "state"] for r in requests_sent)
and all(r["questions"] == requests_sent[0]["questions"] for r in requests_sent))
controls_sent = [k for r in requests_sent for k in r if k in ("model", "task", "lang")]
ok("unset controls are absent, not sent as null",
len(requests_sent) == 8 and not call_kwargs and not controls_sent,
"call kwargs=%r controls=%r" % (call_kwargs, controls_sent))
# Every key the endpoint puts in a request must be one core actually reads out of it, and the
# set of those keys is derived from `Router.route_batch`'s own source, so a rename or a new
# override in core shows up here rather than silently stopping reaching the router.
import re
route_src = inspect.getsource(CoreRouter.route_batch)
read = ({"state", "questions"}
| set(re.findall(r'request\["(\w+)"\]', route_src))
| set(re.findall(r'request\.get\("(\w+)"', route_src)))
sent_keys = {k for r in requests_sent for k in r}
ok("the request keys the endpoint sends are ones core reads",
len(requests_sent) == 8 and read >= sent_keys,
"core reads %r, endpoint sends %r" % (sorted(read), sorted(sent_keys)))
ok("predict_batch is still a one-positional-list call",
list(inspect.signature(CoreRouter.predict_batch).parameters)[1] == "requests")
pinned = RecordingRouter()
demo.ROUTER = pinned
TestClient(demo.app, raise_server_exceptions=False).post(
"/predict/batch", json={"states": states, "questions": one,
"model": "multilingual", "lang": "de"})
sent = pinned.batch_calls[0][0] if pinned.batch_calls else []
ok("a pinned model/lang travels with every request",
len(sent) == 8
and all(r["model"] == "multilingual" and r["lang"] == "de" for r in sent)
and not any("task" in r for r in sent),
json.dumps(sent[:1])[:200])
# --- how the batch is packed is askable ---------------------------------------------
#
# `Router.predict_batch` has taken `batch_size` and `sort_by_length` since #294, and the README
# teaches `router.predict_batch(requests, batch_size=8, sort_by_length=True)`. The endpoint that
# answers up to 64 states in one call forwarded neither, so every batch was one forward pass
# padded to its longest state: a caller who wanted smaller passes had no way to ask, in code or
# over HTTP. These drive the real request model, so a body that named the keys and had them
# dropped by validation fails here rather than returning a silently slower 200.
shape_params = set(inspect.signature(CoreRouter.predict_batch).parameters) - {"self", "requests"}
ok("core still takes the shape keys this endpoint forwards",
{"batch_size", "sort_by_length"} <= shape_params, str(sorted(shape_params)))
shaped = RecordingRouter()
demo.ROUTER = shaped
body = TestClient(demo.app, raise_server_exceptions=False).post(
"/predict/batch", json={"states": states, "questions": one,
"batch_size": 4, "sort_by_length": True})
ok("a shape request is still one batch call, in order",
len(shaped.batch_calls) == 1 and not shaped.predict_calls
and body.json().get("count") == 8
and [r.get("state") for r in body.json()["results"]] == states,
"predict_batch=%d predict=%d" % (len(shaped.batch_calls), len(shaped.predict_calls)))
ok("batch_size and sort_by_length reach predict_batch",
shaped.batch_calls[0][1] == {"batch_size": 4, "sort_by_length": True},
str(shaped.batch_calls[0][1]))
sized = RecordingRouter()
demo.ROUTER = sized
TestClient(demo.app, raise_server_exceptions=False).post(
"/predict/batch", json={"states": states, "questions": one, "batch_size": 2})
ok("batch_size alone does not invent a sort request",
sized.batch_calls[0][1] == {"batch_size": 2}, str(sized.batch_calls[0][1]))
# Core documents `sort_by_length` without a split batch as a no-op, not an error; the endpoint
# must not be the surface that turns a valid call into a 422.
unsized = RecordingRouter()
demo.ROUTER = unsized
no_split = TestClient(demo.app, raise_server_exceptions=False).post(
"/predict/batch", json={"states": states, "questions": one, "sort_by_length": True})
ok("sort_by_length alone is accepted and forwarded, as core takes it",
no_split.status_code == 200 and unsized.batch_calls[0][1] == {"sort_by_length": True},
"%s %s" % (no_split.status_code, str(unsized.batch_calls[0][1])))
junk = TestClient(demo.app, raise_server_exceptions=False).post(
"/predict/batch", json={"states": states, "questions": one, "batch_size": 0})
ok("a zero-size forward pass is a 422 before any router call",
junk.status_code == 422, "%s %s" % (junk.status_code, str(junk.json())[:120]))
# The shape belongs to the batch call only. When a state fails, the endpoint retries per state
# through `Router.predict`, which takes neither key: leaking them there would turn one bad
# state into 8 TypeErrors and an empty batch.
leaked = RecordingRouter(fail_on="ticket 3")
demo.ROUTER = leaked
retried = TestClient(demo.app, raise_server_exceptions=False).post(
"/predict/batch", json={"states": states, "questions": one,
"batch_size": 4, "sort_by_length": True})
ok("the per-state fallback carries no batch shape",
leaked.batch_calls[0][1] == {"batch_size": 4, "sort_by_length": True}
and len(leaked.predict_calls) == 8
and not any("batch_size" in kw or "sort_by_length" in kw
for _, kw in leaked.predict_calls)
and [r for r in retried.json()["results"] if "error" in r][0]["index"] == 3,
str([kw for _, kw in leaked.predict_calls][:1])[:200])
legacy_shaped = LegacyRouter()
demo.ROUTER = legacy_shaped
old = TestClient(demo.app, raise_server_exceptions=False).post(
"/predict/batch", json={"states": states, "questions": one,
"batch_size": 4, "sort_by_length": True})
ok("a router without predict_batch still answers a shaped request",
old.status_code == 200 and len(old.json()["results"]) == 8
and not [r for r in old.json()["results"] if "error" in r],
"%s %s" % (old.status_code, json.dumps(old.json())[:200]))
# One state failing must not cost its neighbours their answer: the endpoint's published
# contract is per-item errors inside a 200.
partial = RecordingRouter(fail_on="ticket 3")
demo.ROUTER = partial
poison = TestClient(demo.app, raise_server_exceptions=False).post(
"/predict/batch", json={"states": states, "questions": one}).json()
errors = [r for r in poison["results"] if "error" in r]
ok("one failing state yields one error entry, not an empty batch",
len(poison["results"]) == 8 and [r["index"] for r in errors] == [3]
# #625: the item names the failure without its exception text, which stays in the log
and errors[0]["error"] == "prediction failed",
json.dumps(poison)[:240])
ok("the failing batch retried per state, so its neighbours still answered",
len(partial.batch_calls) == 1 and len(partial.predict_calls) == 8)
# The single-state surface must keep going through predict(), and `/predict/batch` with one
# state must still batch -- otherwise the two endpoints diverge on where hooks fire.
single = RecordingRouter()
demo.ROUTER = single
one_state = TestClient(demo.app, raise_server_exceptions=False)
one_state.post("/predict", json={"state": "ticket 0", "questions": one})
ok("/predict still calls Router.predict once",
len(single.predict_calls) == 1 and not single.batch_calls)
demo.ROUTER = None
not_ready = TestClient(demo.app, raise_server_exceptions=False).post(
"/predict/batch", json={"states": states, "questions": one})
ok("an unready router keeps answering 200 with per-item 503s",
not_ready.status_code == 200
and len(not_ready.json()["results"]) == 8
and all("503" in r.get("error", "") for r in not_ready.json()["results"]),
json.dumps(not_ready.json())[:200])
demo.ROUTER = None
# ---- the `model` field resolves the way core resolves one ----------------
# `laya.router.normalise_name` trims, lower-cases and maps its alias table before it checks
# the checkpoint registry, and `laya --model laya` and `POST /v1/systemone {"model":"laya"}`
# both go through it. This demo compared the string against a tuple of the three names, so the
# alias worked on the CLI and answered 422 here. The probe below is built from laya.router's
# own tables, so it covers spellings this file has never heard of and cannot fall behind.
from laya import router as _router_mod
names = sorted(_router_mod.DEFAULT_MODELS)
aliases = sorted(a for a, target in _router_mod._ALIASES.items()
if target in _router_mod.DEFAULT_MODELS)
spellings = [spelling for value in names + aliases
for spelling in (value, value.upper(), " %s " % value)]
refused, not_canonical = [], []
for value in spellings:
want = _router_mod.normalise_name(value)
try:
got = demo._check_model(value)
except Exception as exc:
refused.append("%r -> %s" % (value, exc))
continue
if got != want:
not_canonical.append("%r -> %r, core says %r" % (value, got, want))
ok("every one of the %d spellings core resolves, the demo resolves too" % len(spellings),
not refused, "; ".join(refused[:3]))
ok("and what reaches the Router is the checkpoint, not the spelling that named it",
not not_canonical, "; ".join(not_canonical[:3]))
# The negatives, derived from the same tables so one of them cannot accidentally be real.
unknown = [v + "-nope" for v in names + aliases]
def _refuses(fn, value):
try:
fn(value)
except ValueError:
return True
return False
ok("a name neither core nor the demo knows still refuses on both surfaces",
all(_refuses(_router_mod.normalise_name, v) and _refuses(demo._check_model, v)
for v in unknown),
repr([v for v in unknown if not _refuses(demo._check_model, v)][:3]))
# Blank is the demo's own convenience, not core's: the playground posts an empty field, and
# core's resolver raises on one. Parity must not take that away.
def _resolves(v):
"""The value the field validator accepts, or the refusal it gave -- never a raised error."""
try:
return demo._check_model(v)
except Exception as exc:
return "refused: %s" % exc
ok("a blank or whitespace-only model still means 'do not pin'",
[_resolves(v) for v in (None, "", " ")] == [None, None, None],
repr([(v, _resolves(v)) for v in (None, "", " ")])[:180])
regressed = demo.ROUTER
demo.ROUTER = RecordingRouter()
http = TestClient(demo.app, raise_server_exceptions=False)
single = http.post("/predict", json={"state": "ticket 0", "questions": one, "model": "ml"})
ok("an alias is accepted by /predict and the Router is asked for its checkpoint",
single.status_code == 200 and demo.ROUTER.predict_calls
and demo.ROUTER.predict_calls[-1][1].get("model") == "multilingual",
"%s %r" % (single.status_code, demo.ROUTER.predict_calls[-1:]))
batch = http.post("/predict/batch", json={"states": states, "questions": one, "model": "LAYA"})
ok("the same alias on /predict/batch pins english for every state in the batch",
batch.status_code == 200 and demo.ROUTER.batch_calls
and [r.get("model") for r in demo.ROUTER.batch_calls[-1][0]] == ["english"] * len(states),
"%s %r" % (batch.status_code, [r.get("model")
for r in (demo.ROUTER.batch_calls[-1][0]
if demo.ROUTER.batch_calls else [])][:3]))
bad = http.post("/predict", json={"state": "ticket 0", "questions": one,
"model": names[0] + "-nope"})
bad_text = json.dumps(bad.json())
ok("an unknown name is still a 422, in core's words plus this server's own hint",
bad.status_code == 422 and "unknown model" in bad_text and "choose one of" in bad_text
and "(or omit it)" in bad_text, bad_text[:220])
listing = http.get("/models").json()
ok("/models publishes the aliases a caller may use, and every one it publishes validates",
listing["allowed"] == names and listing.get("aliases") == {a: _router_mod._ALIASES[a]
for a in aliases}
and all(_resolves(a) == c for a, c in (listing.get("aliases") or {}).items()),
json.dumps(listing)[:220])
page = http.get("/models", headers={"accept": "text/html"}).text
ok("the page that lists the checkpoints names the count and the spellings it accepts",
("%d checkpoints, one router" % len(names)) in page
and all("<code>%s</code>" % a in page for a in aliases), page[:160])
demo.ROUTER = regressed
def _rendered(desc):
"""The checkpoint list and the alias list, read back out of the field's own description.
That sentence is generated from `MODELS`/`MODEL_ALIASES`, so the honest gate is to parse
the lists out of it and compare them with what `laya.router` accepts -- a substring check
would also pass a sentence somebody typed beside the tables. A render this cannot read
comes back as two empty lists, which fails by name instead of raising and aborting the run.
"""
try:
names_part, alias_part = desc.split("one of ", 1)[1].split(
", or an alias core resolves (", 1)
alias_part = alias_part.split("), in any casing", 1)[0]
except (IndexError, ValueError):
return [], []
return ([p.strip() for p in names_part.split(" | ")] if names_part.strip() else [],
[p.strip() for p in alias_part.split(", ")] if alias_part.strip() else [])
schema = http.get("/openapi.json").json()["components"]["schemas"]
description = schema["PredictRequest"]["properties"]["model"].get("description", "")
ok("the published request schema lists every name and alias the endpoint takes, in neither "
"more nor less", _rendered(description) == (names, aliases),
"rendered %r %r from tables %r %r" % (_rendered(description)[0], _rendered(description)[1],
names, aliases))
ok("and both request models describe the one field identically",
description and description
== schema["BatchRequest"]["properties"]["model"].get("description", ""),
schema["BatchRequest"]["properties"]["model"].get("description", ""))
# The section of the README that teaches this server has to say the field exists and where its
# spellings come from, or the only way to learn it is to get a 422.
with open(os.path.join(ROOT, "README.md"), encoding="utf-8") as handle:
readme = handle.read()
demo_page = readme.split("## Try it locally: web GUI + JSON API", 1)[1].split("\n---", 1)[0]
ok("the README section for this server teaches `model` and where its names come from",
"`model`" in demo_page and "laya.router" in demo_page and "/models" in demo_page,
demo_page[-200:])
# The registry is read, not copied. A checkpoint added to laya.router has to become pinnable
# here -- by name, by alias, in the schema text and on /models -- with no edit to this file.
# `laya.DEFAULT_MODELS` is the same dict object, so one insert reaches both reads.
_router_mod.DEFAULT_MODELS["spanish"] = (_router_mod.BUNDLE_REPO, "spanish")
_router_mod._ALIASES["es"] = "spanish"
try:
grown = importlib.reload(demo)
grown_names = sorted(_router_mod.DEFAULT_MODELS)
ok("a checkpoint core gains is pinnable here without editing this file",
"spanish" in grown.MODELS and _resolves("spanish") == "spanish"
and _resolves(" ES ") == "spanish"
and grown.MODEL_ALIASES.get("es") == "spanish",
"%r %r" % (sorted(grown.MODELS), _resolves(" es ")))
grown_schema = TestClient(grown.app, raise_server_exceptions=False).get(
"/openapi.json").json()["components"]["schemas"]
grown_description = grown_schema["PredictRequest"]["properties"]["model"].get("description", "")
ok("and the text that teaches the list is rendered from it, not typed beside it",
_rendered(grown_description) == (grown_names, sorted(grown.MODEL_ALIASES))
and grown_description
== grown_schema["BatchRequest"]["properties"]["model"].get("description", ""),
"rendered %r from %r -- %s" % (_rendered(grown_description), grown_names,
grown_description[:120]))
grown_listing = TestClient(grown.app, raise_server_exceptions=False).get("/models").json()
ok("/models reports the new checkpoint and its new alias",
grown_listing["allowed"] == grown_names
and (grown_listing.get("aliases") or {}).get("es") == "spanish",
json.dumps(grown_listing)[:220])
finally:
_router_mod.DEFAULT_MODELS.pop("spanish", None)
_router_mod._ALIASES.pop("es", None)
demo = importlib.reload(demo)
# --- the certainty chip: it names a formula, so it has to name that type's formula ---
# `confidence` is per-type in the decoders (normalized entropy on `choice`/`score`, the reported
# side's probability on `noul`), but the demo's chip said "entropy" for all three and its tooltip
# defined the field as one formula. The three have to agree: what the agents write, what the chip
# says per type, and what the tooltip attributes to which type. Weight-free -- no Router is built.
with open(os.path.join(ROOT, "examples", "server.py"), encoding="utf-8") as handle:
srv = handle.read()
def _decoded_kinds():
"""The question types this page draws distribution rows for."""
tree = ast.parse(srv)
fn = next(n for n in ast.walk(tree)
if isinstance(n, ast.FunctionDef) and n.name == "_answer_dist")
return sorted({c.value for n in ast.walk(fn) if isinstance(n, ast.Compare)
and isinstance(n.left, ast.Name) and n.left.id == "kind"
for c in n.comparators
if isinstance(c, ast.Constant) and isinstance(c.value, str)})
def _decoder_formulas(relpath):
"""{question type: "entropy" | "maxp"}, read off the `confidence` each builder writes."""
with open(os.path.join(ROOT, relpath), encoding="utf-8") as handle:
tree = ast.parse(handle.read())
fn = next(n for n in ast.walk(tree)
if isinstance(n, ast.FunctionDef) and n.name == "_decode_answers")
bound = {}
for node in ast.walk(fn):
if isinstance(node, ast.Assign) and len(node.targets) == 1 and isinstance(node.targets[0], ast.Name):
bound[node.targets[0].id] = node.value
got = {}
for node in ast.walk(fn):
if not isinstance(node, ast.Dict):
continue
fields = {k.value: v for k, v in zip(node.keys, node.values)
if isinstance(k, ast.Constant) and isinstance(k.value, str)}
if "type" not in fields or "confidence" not in fields:
continue
kind = fields["type"].value
expr = fields["confidence"]
while isinstance(expr, ast.Name) and expr.id in bound: # `conf_score = round(...)`
expr = bound[expr.id]
called = {c.func.id for c in ast.walk(expr)
if isinstance(c, ast.Call) and isinstance(c.func, ast.Name)}
formula = "entropy" if "confidence_from_probs" in called else "maxp" if "max" in called else None
if formula is None:
raise SystemExit("%s: %s's `confidence` is neither confidence_from_probs nor a max(): %s"
% (relpath, kind, ast.dump(expr)))
if got.get(kind) not in (None, formula):
raise SystemExit("%s: %s writes `confidence` two ways" % (relpath, kind))
got[kind] = formula
return got
def _sentence_types(prose, kinds):
"""{formula: the question types named, in backticks, in the same sentence as it}."""
got = {}
for sentence in _re.split(r"(?<=[.!?])\s+", prose):
named = {t for t in _re.findall(r"`([a-z]+)`", sentence) if t in kinds}
if not named:
continue
for formula, words in (("entropy", "entropy"), ("maxp", "max(p")):
if words in sentence:
got.setdefault(formula, set()).update(named)
return {key: sorted(value) for key, value in sorted(got.items())}
def _chip(row):
hit = _re.search(r"<code>confidence</code> <b>([0-9.]+)</b> (.*?) <svg", row)
return (hit.group(1), hit.group(2)) if hit else (None, None)
kinds = _decoded_kinds()
label = demo._CERTAINTY_LABEL
tip = demo._CERTAINTY_TIP
ok("the chip has a label for every type the page draws rows for, and no other",
sorted(label) == kinds, "%r vs the page's %r" % (sorted(label), kinds))
page_formula = {kind: ("entropy" if "entropy" in text else "maxp") for kind, text in sorted(label.items())}
for relpath in ("laya/agent.py", "laya/onnx_agent.py"):
decoders = _decoder_formulas(relpath)
ok("%s writes `confidence` the way this page labels it" % relpath,
decoders == page_formula, "%r vs the chip's %r" % (decoders, page_formula))
said = _sentence_types(tip, set(kinds))
ok("the tooltip attributes entropy to exactly the types whose chip says entropy",
said.get("entropy") == sorted(k for k, f in page_formula.items() if f == "entropy"),
"tooltip %r vs chip %r" % (said, page_formula))
ok("and the reported side's max(p_true, 1 - p_true) to exactly the types whose chip does not",
said.get("maxp") == sorted(k for k, f in page_formula.items() if f == "maxp"),
"tooltip %r vs chip %r" % (said, page_formula))
ok("so the tooltip names both formulas rather than defining the field as one",
len(said) == 2 and set(said) == {"entropy", "maxp"}, repr(said))
OLD_BLANKET = "The `confidence` field is 1 minus the normalized entropy of the whole distribution:"
ok("the blanket sentence this replaced is gone", OLD_BLANKET not in srv, OLD_BLANKET)
ok("and the page no longer promises an accuracy rate for a confidence value",
"are right about 90% of the time" not in srv
and "gate on a threshold you have measured" in tip, tip)
OLD_JS_CHIP = 'toFixed(4)}), "entropy, not calibrated",\n icon("info")'
ok("the browser's chip reads the table by the row's type, in the block the page ships",
OLD_JS_CHIP not in srv and "CERTAINTY_LABEL[kind]" in demo._PLAYGROUND_JS)
page = client.get("/").text
ok("the browser gets the label table from this dict, not a copy of it",
"const CERTAINTY_LABEL = " + json.dumps(label) in page, json.dumps(label))
ok("and the tooltip it shows is this same text",
"is not one formula, and it is not calibrated, so do not gate on it" in page)
import numpy as np
from laya.common import confidence_from_probs
spread = [0.62, 0.25, 0.13]
ent = round(confidence_from_probs(np.asarray(spread), 3), 4)
row = demo._answer_row("q", {"type": "choice", "choice": "billing", "answer_confidence": 0.62,
"confidence": ent, "probabilities": {"billing": 0.62, "support": 0.25,
"sales": 0.13}},
{"type": "choice", "instructions": "?"}, 1)
ok("a choice row reads as the entropy it is", _chip(row) == ("%.4f" % ent, label["choice"]),
"%r" % (_chip(row),))
row = demo._answer_row("q", {"type": "score", "score": 1.2, "answer_confidence": 0.5,
"confidence": round(confidence_from_probs(np.asarray([0.3, 0.5, 0.2]), 3), 4),
"legend": {"0": "low", "1": "mid", "2": "high"},
"probabilities": {"0": 0.3, "1": 0.5, "2": 0.2}},
{"type": "score", "instructions": "?"}, 1)
ok("so does a score row", _chip(row) == ("%.4f" % round(
confidence_from_probs(np.asarray([0.3, 0.5, 0.2]), 3), 4), label["score"]), "%r" % (_chip(row),))
p_true = 0.8727
side = round(max(p_true, 1 - p_true), 4)
two_option_entropy = round(confidence_from_probs(np.asarray([p_true, 1 - p_true]), 2), 4)
row = demo._answer_row("q", {"type": "noul", "noul": p_true, "answer_confidence": 0.79,
"confidence": side}, {"type": "noul", "instructions": "?"}, 1)
ok("a noul row's confidence is the side being reported, not an entropy",
side == p_true and abs(side - two_option_entropy) > 0.4
and _chip(row) == ("%.4f" % side, label["noul"])
and "entropy" not in _chip(row)[1], "%r %r" % (_chip(row), two_option_entropy))
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL " + f)
return 1 if FAIL else 0
if __name__ == "__main__":
sys.exit(main())