Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

63.0 KB
Raw Blame
"""Regression: examples/server.py must bound a request the way laya.serve does.

`laya/serve.py` refuses a request carrying more than MAX_QUESTIONS questions, a state
over MAX_STATE_CHARS, or a body over its own cap, because Laya encodes the state once
per question -- cost is questions x state size, collated into one tensor.

examples/server.py bounded `states` to 64 and left the rest open: 20 000 questions and
a 5 MB state were both accepted where the shipped server answers 413. The bounds are
read from laya.serve rather than restated, so the two cannot drift.

Scope: the question count, the state size, and the per-question and total option counts,
answered 413 as laya.serve answers them. The option budgets matter for the same reason the
other two do -- a choice or score question encodes one sequence per option, and those
sequences share the head budget -- so a request laya.serve refuses with 413 must not be
answered here. The bounds are read from laya.serve, never restated.
`Question.instructions` and `criteria` still carry unbounded *text* that no per-field
bound can see; capping the request body is the backstop for those and is left out
deliberately -- see the PR description.

Driven over HTTP through TestClient. No weights are loaded; the router stays unbuilt, so
a request that passes validation answers 503, which is the assertion for "accepted". The
one arm that needs a Router enters the lifespan, which is also the arm that checks the cap.

On `main` this file is 19 checks, all of them about request size. It is now 38: the other
nineteen follow the cap from the environment to the constructor, to the running Router, to
`/health` in both JSON and HTML, and back out through the `--reload` push. The first
nineteen are untouched.

FAILS_ON_MAIN -- this file, against `main`'s `examples/server.py` at 4066d5d:

    PASS the page and health endpoints are unaffected
    FAIL an unset LAYA_MAX_LOADED means 'not asked for', not a number of this file's own  1
    AttributeError: module 'server' has no attribute '_router_kwargs'

One check names the number, then the run aborts because the helper it drives does not exist
there. So the same claims are checked on `main` through nothing but the public surface -- its
own lifespan, the Router that builds, `/health` in both representations -- which prints:

    Router()'s own default = 2      built Router's cap = 1      /health JSON = 1
    /health HTML says      = 'Checkpoints are loaded on demand; up to 1 kept in memory.'
    preload() lifts the cap to -> 3, while the page prints 1

and `False` for "the cap came from the environment", "the Router holds laya's default", and
"that default is more than one"; on this branch all three read `True` and the cap reads 2.
That last line is the half that is not about the default at all: `laya/router.py::Router.preload`
raises the cap to fit what it preloads, so the demo's own default mode ran with three checkpoints
resident while `/health` reported one. Nothing in the file ever asked the running Router what it
was holding.

`main` is at 66 checks now, and this file is at 81. The fifteen new ones are the request `model`
field. `laya.router.normalise_name` is core's one resolver -- trim, lower-case, alias table,
registry -- and `laya --model` and `laya/serve.py` both hand it the caller's string, but this demo
compared it against a tuple of the three checkpoint names. So on `main` the aliases, the checkpoint
ids with the `laya-` prefix, `typed_decisions`, and the upper-cased and space-padded form of every
one of them answer 422 here: 60 of the 67 spellings core resolves, which
`laya-bench/model_alias_witness_main.log` counts against `main` at 9d95567 through
`_check_model` and through `POST /predict` on a TestClient -- `{"model": "english"}` answers 503,
`{"model": "laya"}` answers 422, while `laya --model laya` pins the checkpoint. The new checks take
the probe from `laya.router`'s own tables rather than listing the aliases here, require the
canonical name to reach the Router, keep blank meaning "do not pin" as the playground sends it,
read the alias list back out of `/models`, its HTML page, the published request schema and the
README section for this server, and then add a fourth checkpoint to core at runtime, reload the
demo, and require it to be pinnable by name and by alias and to appear in that text with no edit to
this file.

Run: python tests/test_example_server_limits.py
"""
import ast
import importlib
import json
import os
import re as _re          # `main()` already binds a local `re` further down; F811 off both
import sys
import types

os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
os.environ.setdefault("LAYA_PRELOAD", "0")

ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)
sys.path.insert(0, os.path.join(ROOT, "examples"))

PASS, FAIL = [], []


def ok(name, cond, detail=""):
    (PASS if cond else FAIL).append("%s%s" % (name, ("  -- " + detail) if detail and not cond else ""))
    print("   %s %s%s" % ("PASS" if cond else "FAIL", name, ("  " + detail) if detail and not cond else ""), flush=True)


def main():
    try:
        from fastapi.testclient import TestClient
    except ImportError as exc:
        # Only the optional serving stack may be missing. An ImportError naming anything
        # else -- in particular `cannot import name MAX_QUESTIONS from laya.serve`, the
        # drift this test exists to catch -- must fail rather than skip.
        if (getattr(exc, "name", None) or "").split(".")[0] not in ("fastapi", "httpx", "starlette"):
            raise
        print("SKIP: fastapi/httpx not installed -- pip install laya[serve] httpx")
        return 0
    try:
        # examples/server.py reads the environment at import, so the cap under test has to
        # be absent/present *before* the import, never patched after it.
        os.environ.pop("LAYA_MAX_LOADED", None)
        import server as demo
    except ImportError as exc:
        if (getattr(exc, "name", None) or "").split(".")[0] not in ("fastapi", "httpx", "starlette", "multipart"):
            raise
        print("SKIP: examples/server.py needs the serve extra -- pip install laya[serve]")
        return 0

    from laya.serve import (
        MAX_CHOICE_OPTIONS,
        MAX_QUESTIONS,
        MAX_SCORE_LEVELS,
        MAX_STATE_CHARS,
        MAX_TOTAL_OPTIONS,
    )

    client = TestClient(demo.app, raise_server_exceptions=False)
    one = {"a": {"type": "noul", "instructions": "x"}}

    def questions(n):
        return {"q%d" % i: {"type": "noul", "instructions": "x"} for i in range(n)}

    def choice(n_opts, qid="a"):
        return {qid: {"type": "choice", "instructions": "Which?",
                      "criteria": {"opt%d" % i: "desc %d" % i for i in range(n_opts)}}}

    def score(n_levels, qid="a"):
        return {qid: {"type": "score", "instructions": "How bad?",
                      "criteria": ["level %d" % i for i in range(n_levels)]}}

    def code(**kw):
        return client.post("/predict", **kw).status_code

    # The bounds must come from laya.serve, not a local copy, or the two drift.
    demo_q = getattr(demo, "MAX_QUESTIONS", None)
    demo_s = getattr(demo, "MAX_STATE_CHARS", None)
    ok("the per-request bounds are laya.serve's",
       demo_q == MAX_QUESTIONS and demo_s == MAX_STATE_CHARS,
       "demo %r/%r vs laya.serve %r/%r" % (demo_q, demo_s, MAX_QUESTIONS, MAX_STATE_CHARS))
    demo_opts = (getattr(demo, "MAX_CHOICE_OPTIONS", None),
                 getattr(demo, "MAX_SCORE_LEVELS", None),
                 getattr(demo, "MAX_TOTAL_OPTIONS", None))
    ok("the option budgets are laya.serve's too",
       demo_opts == (MAX_CHOICE_OPTIONS, MAX_SCORE_LEVELS, MAX_TOTAL_OPTIONS),
       "demo %r vs laya.serve %r" % (demo_opts, (MAX_CHOICE_OPTIONS, MAX_SCORE_LEVELS,
                                                  MAX_TOTAL_OPTIONS)))

    # --- too many questions, too large a state: 413, as laya.serve answers ---
    ok("more than MAX_QUESTIONS questions is 413",
       code(json={"state": "hi", "questions": questions(MAX_QUESTIONS + 1)}) == 413)
    # A noul question carries no answer options, so an options-based budget cannot see
    # it; the count is what has to be bounded.
    ok("a noul-only flood is 413 (it carries no answer options)",
       code(json={"state": "hi", "questions": questions(20_000)}) == 413)
    ok("a state over MAX_STATE_CHARS is 413",
       code(json={"state": "A" * (MAX_STATE_CHARS + 1), "questions": one}) == 413)
    ok("an oversized dict state is 413",
       code(json={"state": {"body": "A" * (MAX_STATE_CHARS + 1)}, "questions": one}) == 413)
    ok("an oversized state inside a batch is 413",
       client.post("/predict/batch",
                   json={"states": ["hi", "A" * (MAX_STATE_CHARS + 1)], "questions": one}
                   ).status_code == 413)

    # --- the option budgets, which laya.serve also answers 413 -----------------
    # A choice or score question encodes one sequence per option against a shared head
    # budget, so the option counts are size limits for the same reason the state is.
    ok("more than MAX_CHOICE_OPTIONS options in one question is 413",
       code(json={"state": "hi", "questions": choice(MAX_CHOICE_OPTIONS + 1)}) == 413)
    ok("more than MAX_SCORE_LEVELS levels in one question is 413",
       code(json={"state": "hi", "questions": score(MAX_SCORE_LEVELS + 1)}) == 413)
    # Under the per-question caps but over the shared total: the case a per-question check
    # alone would still accept.
    per_q = MAX_CHOICE_OPTIONS
    n_questions = MAX_TOTAL_OPTIONS // per_q + 1
    ok("over MAX_TOTAL_OPTIONS across questions is 413",
       code(json={"state": "hi",
                  "questions": {"q%d" % i: choice(per_q, "q%d" % i)[ "q%d" % i]
                                for i in range(n_questions)}}) == 413,
       "%d questions x %d options" % (n_questions, per_q))
    # A noul question contributes no options, so it cannot move the total either way.
    ok("a noul flood is still bounded by the question count, not the option total",
       code(json={"state": "hi", "questions": questions(MAX_QUESTIONS)}) == 503)
    # ...including when it carries the ordinary false/true criteria, which are option
    # *texts* but not answer options. laya.serve adds them nowhere; a total that counted
    # them would refuse a request serve accepts, which is the drift this file exists to
    # catch -- in the direction of refusing something legal.
    counted = 5 * MAX_CHOICE_OPTIONS + 12  # exactly MAX_TOTAL_OPTIONS by laya.serve's count
    mixed = {"q%d" % i: choice(MAX_CHOICE_OPTIONS, "q%d" % i)["q%d" % i] for i in range(5)}
    mixed["last"] = choice(12, "last")["last"]
    for i in range(30):  # 30 noul questions, 2 criteria each, 60 if wrongly counted
        mixed["n%d" % i] = {"type": "noul", "instructions": "True?",
                            "criteria": {"false": "no", "true": "yes"}}
    ok("noul criteria do not count toward the shared option total",
       len(mixed) <= MAX_QUESTIONS and counted == MAX_TOTAL_OPTIONS
       and code(json={"state": "hi", "questions": mixed}) == 503,
       "%d questions, %d options by laya.serve's count" % (len(mixed), counted))


    # --- the refusal must not echo the rejected payload back ----------------
    # Declaring these as Field(max_length=...) instead would report 422 *and* include
    # the offending `input` in FastAPI's validation-error body, so refusing a 5 MB
    # state would write 5 MB back to the caller -- a size limit that amplifies.
    big = {"state": "A" * 5_000_000, "questions": one}
    sent = len(json.dumps(big).encode())
    resp = client.post("/predict", json=big)
    ok("a rejected 5 MB state answers 413 without echoing it",
       resp.status_code == 413 and len(resp.content) < 1_000,
       "%s, %d bytes returned for %d sent" % (resp.status_code, len(resp.content), sent))

    # --- a genuine schema error is still 422, not 413 -----------------------
    ok("an unknown question type is 422",
       code(json={"state": "hi", "questions": {"a": {"type": "bogus", "instructions": "x"}}}) == 422)
    ok("a choice question with no criteria is 422",
       code(json={"state": "hi", "questions": {"a": {"type": "choice", "instructions": "x"}}}) == 422)
    ok("an empty state is 422", code(json={"state": "", "questions": one}) == 422)
    ok("zero questions is 422", code(json={"state": "hi", "questions": {}}) == 422)

    # --- the limits are limits, not walls -----------------------------------
    # No router is built, so anything that passes validation answers 503.
    ok("an ordinary request passes validation",
       code(json={"state": {"body": "billed twice, please refund"}, "questions": one}) == 503)
    ok("exactly MAX_QUESTIONS passes",
       code(json={"state": "hi", "questions": questions(MAX_QUESTIONS)}) == 503)
    ok("a state of exactly MAX_STATE_CHARS passes",
       code(json={"state": "A" * MAX_STATE_CHARS, "questions": one}) == 503)
    ok("exactly MAX_CHOICE_OPTIONS options passes",
       code(json={"state": "hi", "questions": choice(MAX_CHOICE_OPTIONS)}) == 503)
    ok("exactly MAX_SCORE_LEVELS levels passes",
       code(json={"state": "hi", "questions": score(MAX_SCORE_LEVELS)}) == 503)
    ok("a list state passes", code(json={"state": ["a", "b"], "questions": one}) == 503)
    ok("a small chunked body passes",
       code(content=(lambda: (yield json.dumps({"state": "hi", "questions": one}).encode()))(),
            headers={"content-type": "application/json"}) == 503)
    # /predict/batch reports per-item failures inside a 200 envelope, so an accepted
    # batch is a 200 here; over the state bound it is refused outright, and 413 rather
    # than 422 because "too many states" is a size violation like the others.
    ok("a 64-state batch is accepted",
       client.post("/predict/batch", json={"states": ["hi"] * 64, "questions": one}).status_code == 200)
    ok("a 65-state batch is still refused by the existing bound",
       client.post("/predict/batch", json={"states": ["hi"] * 65, "questions": one}).status_code == 422)
    ok("the page and health endpoints are unaffected",
       client.get("/").status_code == 200 and client.get("/health").status_code == 200)

    # --- the two surfaces must reach the same verdict on the same numbers ---------
    # The demo reads `Question` instances where serve reads plain dicts, so the counting
    # lives behind a shape branch that could drift from serve's. Pin the verdicts against
    # each other rather than restating either one: serve's validator is called with the
    # plain dicts it actually receives, the demo through HTTP with the models it actually
    # receives, and the two have to agree.
    import laya.serve as _serve_mod  # noqa: E402

    def serve_verdict(state, questions):
        try:
            _serve_mod._check_request_limits(state, questions)
        except Exception:  # noqa: BLE001 -- HTTPException carries the 413
            return "refused"
        return "accepted"

    def demo_verdict(state, questions):
        return "refused" if client.post("/predict", json={"state": state,
                                                          "questions": questions}).status_code == 413 else "accepted"

    noul_crit = {"type": "noul", "instructions": "True?",
                 "criteria": {"false": "no", "true": "yes"}}
    parity_cases = [
        ("101 choice options", "hi", choice(MAX_CHOICE_OPTIONS + 1)),
        ("33 score levels", "hi", score(MAX_SCORE_LEVELS + 1)),
        ("600 options total", "hi",
         dict(("q%d" % i, choice(20, "q%d" % i)["q%d" % i]) for i in range(30))),
        ("exactly 100 choice options", "hi", choice(MAX_CHOICE_OPTIONS)),
        ("exactly 32 score levels", "hi", score(MAX_SCORE_LEVELS)),
        ("noul criteria only, near the total", "hi",
         dict([("c%d" % i, choice(MAX_CHOICE_OPTIONS, "c%d" % i)["c%d" % i]) for i in range(5)]
               + [("last", choice(12, "last")["last"])]
               + [("n%d" % i, noul_crit) for i in range(30)])),
        ("noul only", "hi", {"n%d" % i: noul_crit for i in range(10)}),
        ("an ordinary request", {"body": "billed twice"}, one),
        # A state whose `str()` and whose JSON serialization are different lengths (50000 vs 99988).
        # This case catches the two surfaces DIVERGING -- one measuring the serialized text and the
        # other `str(state)`. It cannot catch them being wrong together: `examples/server.py` binds
        # `_state_length` from `laya.serve` by `getattr`, so a regression inside that helper moves
        # both and parity stays green. Measured: regressing `_state_length` fails 0 of these parity
        # checks and both of the absolute ones below. Those are the real coverage -- do not prune
        # them as redundant.
        ("a state whose repr is half its JSON", {"body": '"' * 49988}, one),
    ]
    for label, st, qs in parity_cases:
        s, d = serve_verdict(st, qs), demo_verdict(st, qs)
        ok("parity with laya.serve: %s" % label, s == d, "serve=%s demo=%s" % (s, d))

    # The BATCH shape, which these parity cases could not reach: every case above drives
    # `_check_request_limits` with one state, so nothing compared how the two surfaces SIZE a batch.
    # `laya.serve` splits a batch so one forward pass stays inside its token budget rather than
    # refusing it, and the demo must split it identically -- it binds the planner from `laya.serve` by
    # getattr, so this checks the binding took AND that the plan reaches the call.
    serve_plan = _serve_mod._batch_chunk_size
    demo_plan = demo._batch_chunk_size

    tiny = {"q": {"type": "noul", "instructions": "True?"}}
    four = dict(("q%03d" % i, tiny["q"]) for i in range(4))
    thirty_two = dict(("q%03d" % i, tiny["q"]) for i in range(32))
    many = dict(("q%03d" % i, tiny["q"]) for i in range(MAX_QUESTIONS))
    wide = _serve_mod.DEFAULT_MAX_TOKEN_BUDGET
    for label, ns, nq, mlen in [
            ("an ordinary batch fits, no split", 2, len(four), None),
            ("64 states x 4 questions fits", 64, len(four), None),
            ("8 states x 32 questions fits", 8, len(thirty_two), None),
            ("9 states x 32 questions splits", 9, len(thirty_two), None),
            ("64 states x 64 questions splits", 64, len(many), None),
            ("a wide max_len splits harder", 8, len(thirty_two), wide),
            ("a wide max_len with few rows fits", 2, 1, wide)]:
        a, b = serve_plan(ns, nq, mlen), demo_plan(ns, nq, mlen)
        ok("batch plan parity with laya.serve: %s" % label, a == b, "serve=%r demo=%r" % (a, b))

    # Equal on both sides is not enough -- they bind the same function, so both could be wrong
    # together. The plan itself has to be the right shape.
    ok("batch plan/a fitting batch is not split", serve_plan(8, len(thirty_two)) is None,
       repr(serve_plan(8, len(thirty_two))))
    split = serve_plan(64, len(many))
    ok("batch plan/an oversized batch is split", split is not None and split >= 1, repr(split))
    ok("batch plan/one pass fits the budget",
       split * len(many) * _serve_mod._BATCH_ROW_TOKENS_ASSUMED
       <= _serve_mod.DEFAULT_MAX_BATCH_TOKENS,
       "%r states x %d questions" % (split, len(many)))

    # And the demo's route sends it, rather than computing it and dropping it.
    sent = {}
    real_router = demo._router

    class _Recorder:
        def predict_batch(self, requests, **kwargs):
            sent.update(kwargs)
            sent["n"] = len(requests)
            return [{"model": "m", "answers": {}, "usage": {"input_tokens": 1}} for _ in requests]

    demo._router = lambda: _Recorder()
    try:
        code = client.post("/predict/batch",
                           json={"states": ["hi"] * 64, "questions": many}).status_code
        ok("batch plan/the demo answers an oversized batch instead of refusing it", code == 200, code)
        ok("batch plan/the demo forwards the planned batch_size",
           sent.get("batch_size") == serve_plan(64, len(many)),
           "sent=%r planned=%r" % (sent.get("batch_size"), serve_plan(64, len(many))))
        ok("batch plan/every state is still sent", sent.get("n") == 64, sent.get("n"))
        sent.clear()
        client.post("/predict/batch", json={"states": ["hi"] * 2, "questions": four})
        ok("batch plan/a fitting batch is passed no batch_size", "batch_size" not in sent, sent)
    finally:
        demo._router = real_router

    # Parity alone cannot see a bug both surfaces share, and `getattr` guarantees they share one:
    # with `_state_length` measuring `str(state)` again, both agree on accepting a state that
    # serializes to 99 988 characters and every parity check above stays green (measured: 0 of them
    # fail, both of these do). So the verdict itself is asserted, not just the agreement.
    quote_heavy = {"body": '"' * 49988}
    # Recorded through `ok` like everything else in this file: a bare `assert` here would raise out
    # of `main()` and abandon the ~30 checks that follow instead of recording one failure.
    ok("the quote-heavy fixture passes a str()-based gate",
       len(str(quote_heavy)) <= MAX_STATE_CHARS, len(str(quote_heavy)))
    ok("the quote-heavy fixture fails a serialization-based gate",
       len(json.dumps(quote_heavy, ensure_ascii=False)) > MAX_STATE_CHARS,
       len(json.dumps(quote_heavy, ensure_ascii=False)))
    for who, verdict in (("laya.serve", serve_verdict(quote_heavy, one)),
                         ("the demo server", demo_verdict(quote_heavy, one))):
        ok("%s refuses a state whose JSON is twice its repr" % who, verdict == "refused", verdict)

    # --- the resident-checkpoint cap: derived, not copied -------------------
    # examples/server.py used to build its Router with `max_loaded=1`, a copy of a default
    # laya/router.py retired in #180. A cap of one cannot hold both english and
    # multilingual, so the demo ran the churn #172 measured and fixed -- one checkpoint
    # rebuilt per alternating-language request -- while the library and laya.serve did not.
    # What is asserted here is that the demo asks for nothing unless asked to, and that
    # what it then reports is the number the running Router holds.
    from laya.router import Router

    ok("an unset LAYA_MAX_LOADED means 'not asked for', not a number of this file's own",
       demo._CFG["max_loaded"] is None, repr(demo._CFG["max_loaded"]))
    ok("so the key never reaches the constructor",
       "max_loaded" not in demo._router_kwargs(demo._CFG),
       repr(demo._router_kwargs(demo._CFG)))
    # Before a Router exists the page has no resident count to state. This is the check
    # that fails if the page goes back to guessing one (`cfg.get('max_loaded', 1)`).
    ok("and while loading, the page states no resident count",
       "kept in memory" not in client.get("/health", headers={"accept": "text/html"}).text,
       client.get("/health", headers={"accept": "text/html"}).text[:200])

    with client:                       # the lifespan builds the Router; preload is off
        # The witness that this arm needed no weights: LAYA_PRELOAD=0 above means the
        # Router the app built holds no checkpoint yet, so the cap is the only thing here
        # that could have been loaded.
        ok("building it downloaded nothing",
           not demo.ROUTER._agents, repr(sorted(demo.ROUTER._agents)))
        cap = demo.ROUTER.max_loaded
        ok("the running Router holds laya's own default",
           cap == Router().max_loaded, "demo %r vs laya %r" % (cap, Router().max_loaded))
        ok("and laya's own default is more than one checkpoint",
           cap > 1, repr(cap))
        ok("/health reports the cap the Router really holds",
           client.get("/health").json()["config"]["max_loaded"] == cap, repr(cap))
        ok("and the HTML page prints that same number",
           ("up to %d kept in memory" % cap)
           in client.get("/health", headers={"accept": "text/html"}).text, repr(cap))
        # The number cannot come from the requested config, because the config and the
        # Router are allowed to disagree: `Router(preload=True)` raises its own cap to fit what it
        # preloads (`laya/router.py::Router.preload`), so the demo's default mode has run with
        # three resident while its env said one. Move the Router and the page must move.
        demo.ROUTER.max_loaded = cap + 1
        moved = client.get("/health")
        ok("and both follow the Router when the Router raises its own cap",
           moved.json()["config"]["max_loaded"] == cap + 1
           and ("up to %d kept in memory" % (cap + 1))
           in client.get("/health", headers={"accept": "text/html"}).text,
           repr(moved.json()["config"]))

    def with_cap(value):
        """Re-import the demo the way uvicorn starts it, with LAYA_MAX_LOADED set to `value`."""
        if value is None:
            os.environ.pop("LAYA_MAX_LOADED", None)
        else:
            os.environ["LAYA_MAX_LOADED"] = value
        return importlib.reload(demo)

    # The default moving must not take the operator's own number away with it.
    for raw, want in (("3", 3), ("1", 1), (" 4 ", 4)):
        fresh = with_cap(raw)
        with TestClient(fresh.app) as c:
            ok("LAYA_MAX_LOADED=%r still reaches the Router" % raw,
               fresh.ROUTER.max_loaded == want, repr(fresh.ROUTER.max_loaded))
            ok("LAYA_MAX_LOADED=%r still shows up in /health" % raw,
               c.get("/health").json()["config"]["max_loaded"] == want, raw)
    fresh = with_cap(None)
    from laya.serve import build_router

    ok("the demo and laya.serve leave the cap to the same place",
       fresh._router_kwargs(fresh._CFG).get("max_loaded", Router().max_loaded)
       == build_router().max_loaded,
       "%r vs %r" % (fresh._router_kwargs(fresh._CFG), build_router().max_loaded))

    # A bad or non-positive LAYA_MAX_LOADED must fall back to "not asked for" the way
    # laya.serve's `_resolve_max_loaded` documents ("a typo in a deployment file should not stop
    # the server from starting"), not reach `int()` as an import crash and not forward a cap
    # Router cannot honour. FAILS_ON_MAIN: the bare int() at module level aborts the reload.
    for raw in ("abc", "2.5", "0", "-3", ""):
        fresh = with_cap(raw)
        ok("LAYA_MAX_LOADED=%r falls back to 'not asked for'" % raw,
           "max_loaded" not in fresh._router_kwargs(fresh._CFG),
           repr(fresh._router_kwargs(fresh._CFG)))
    with_cap(None)   # leave the ambient env clean for the checks below

    # --- the routing fallback: derived, not copied --------------------------
    # The copy retired above said a number. The one beside it named a checkpoint: 3cf26cb moved
    # `Router(default=...)` from `english` to `multilingual` in 0.4.0, and `laya.serve` followed
    # by sending no `default` keyword at all when LAYA_DEFAULT_MODEL is unset
    # (`_default_model_option`), which `laya/mcp/server.py::_ensure_router` reuses rather than
    # duplicates. This file went on constructing `Router(default="english")`, so one environment
    # answered three ways: a bare `laya-serve` and a bare `laya mcp` took a state with no language
    # evidence to the multilingual checkpoint, a bare `python examples/server.py` took it to the
    # English arm. 3cf26cb's own migration note is that the old fallback is now something you ask
    # for by name, so the pin here was not a demo of the old default -- it was the demo ignoring
    # the change. `Router.route` decides without loading or running anything, so none of this
    # needs weights.
    def with_default(value):
        """Re-import the demo the way uvicorn starts it, with LAYA_DEFAULT_MODEL set to `value`."""
        if value is None:
            os.environ.pop("LAYA_DEFAULT_MODEL", None)
        else:
            os.environ["LAYA_DEFAULT_MODEL"] = value
        return importlib.reload(demo)

    fresh = with_default(None)
    ok("an unset LAYA_DEFAULT_MODEL means 'not asked for', not a checkpoint of this file's own",
       fresh._CFG["default"] is None, repr(fresh._CFG["default"]))
    ok("so the key never reaches the constructor",
       "default" not in fresh._router_kwargs(fresh._CFG),
       repr(fresh._router_kwargs(fresh._CFG)))

    # Blank has to mean the same thing at both surfaces. laya.serve treats "", "   " and "\n" as
    # unset (tests/test_serve.py pins all three against `_default_model_option`); a demo that read
    # them as a name would raise at `Router(default="")` where the packaged server starts.
    from laya.serve import _default_model_option

    for blank in ("", "   ", "\n"):
        os.environ["LAYA_DEFAULT_MODEL"] = blank
        ok("LAYA_DEFAULT_MODEL=%r is unset for laya.serve" % blank,
           _default_model_option() == {}, repr(_default_model_option()))
        fresh = with_default(blank)
        ok("LAYA_DEFAULT_MODEL=%r is unset for the demo too" % blank,
           "default" not in fresh._router_kwargs(fresh._CFG),
           repr(fresh._router_kwargs(fresh._CFG)))

    fresh = with_default(None)
    with TestClient(fresh.app) as c:
        ok("the Router the demo builds holds laya's own fallback",
           fresh.ROUTER.default == Router().default,
           "demo %r vs laya %r" % (fresh.ROUTER.default, Router().default))
        ok("and laya's own fallback is the multilingual checkpoint",
           fresh.ROUTER.default == "multilingual", repr(fresh.ROUTER.default))
        # The decision, not the attribute: this is the branch 3cf26cb moved and #54 measured.
        for state in ("12345 !!!", "Quero cancelar", "Esqueci minha senha"):
            d = fresh.ROUTER.route(state)
            ok("the demo routes %r off the fallback, from multilingual" % state,
               d.model == "multilingual" and "using default (multilingual)" in d.reason,
               "%r %r" % (d.model, d.reason))
        ok("/models reports the fallback the running Router holds",
           c.get("/models").json()["default"] == fresh.ROUTER.default,
           repr(c.get("/models").json()["default"]))
        # One badge, on the card the Router actually falls back to. The badge is rendered from
        # `payload["default"]`, so this is the HTML half of the line above.
        badged = [chunk.split("</h2>")[0] for chunk in
                  c.get("/models", headers={"accept": "text/html"}).text
                  .split("<section class='panel'>") if "data-s='ok'>default</span>" in chunk]
        ok("and the page badges exactly one checkpoint, the multilingual one",
           len(badged) == 1 and "<h2>multilingual" in badged[0], repr(badged)[:160])

    # Before the lifespan has built anything, the page has no Router to ask, and must not guess.
    fresh = with_default(None)
    loading = TestClient(fresh.app)        # entered without a lifespan, so ROUTER stays None
    ok("while the app is loading, /models claims no fallback",
       fresh.ROUTER is None and loading.get("/models").json()["default"] is None,
       repr(fresh.ROUTER))

    # The default moving must not take the operator's own answer away with it -- and an alias or
    # a padded spelling resolves to the same checkpoint it resolves to in laya.serve.
    for raw, want in (("english", "english"), ("ml", "multilingual"),
                      (" MULTI ", "multilingual"), ("typed-decisions", "typed-decisions")):
        fresh = with_default(raw)
        with TestClient(fresh.app) as cc:
            ok("LAYA_DEFAULT_MODEL=%r still reaches the Router" % raw,
               fresh.ROUTER.default == want, repr(fresh.ROUTER.default))
            ok("LAYA_DEFAULT_MODEL=%r still shows up in /models" % raw,
               cc.get("/models").json()["default"] == want,
               repr(cc.get("/models").json()["default"]))
            d = fresh.ROUTER.route("Quero cancelar")
            ok("and %r decides that state from that checkpoint" % raw,
               d.model == want and "using default (%s)" % want in d.reason,
               "%r %r" % (d.model, d.reason))

    fresh = with_default(None)
    ok("the demo and laya.serve leave the fallback to the same place",
       fresh._router_kwargs(fresh._CFG).get("default", Router().default) == build_router().default,
       "%r vs %r" % (fresh._router_kwargs(fresh._CFG), build_router().default))

    # --- the --reload env push has to survive the round trip ----------------
    # With reload=True uvicorn re-imports `server:app` in a child process and only the
    # environment crosses over. Writing str(None) for "not asked for" would land on the
    # int() that reads LAYA_MAX_LOADED in that child and stop the server at import, so the
    # unset case must push nothing at all. uvicorn is replaced so main() never binds a port.
    # The same round trip now carries the fallback, where str(None) is worse than a bad number:
    # the child reads it as a checkpoint literally named "None" and refuses to start over a
    # configuration nobody asked for.
    real_uvicorn = sys.modules.get("uvicorn")
    started = {}
    fake = types.ModuleType("uvicorn")
    fake.run = lambda target, **kw: started.update(target=target)
    sys.modules["uvicorn"] = fake
    argv = sys.argv

    def start(args):
        with_cap(None)
        sys.argv = ["server.py"] + args
        demo.main()
        return os.environ.get("LAYA_MAX_LOADED", "(absent)")

    def start_default(args):
        """The same run, reporting what crossed over as the fallback."""
        with_default(None)
        sys.argv = ["server.py"] + args
        demo.main()
        return os.environ.get("LAYA_DEFAULT_MODEL", "(absent)")

    try:
        ok("--reload with no flag pushes no cap, so the reimport can read it",
           start(["--reload"]) == "(absent)", repr(started))
        ok("--reload --max-loaded 3 pushes 3",
           start(["--reload", "--max-loaded", "3"]) == "3", repr(started))
        ok("without --reload nothing is pushed at all",
           start([]) == "(absent)", repr(started))
        ok("--reload with no flag pushes no fallback either, so the child inherits laya's",
           start_default(["--reload"]) == "(absent)", repr(started))
        ok("--reload --default-model english pushes english",
           start_default(["--reload", "--default-model", "english"]) == "english",
           repr(started))
        ok("and without --reload the flag is not pushed as a name",
           start_default(["--default-model", "english"]) == "(absent)", repr(started))
        ok("--default-model '' is the same as not passing it",
           start_default(["--reload", "--default-model", ""]) == "(absent)", repr(started))
    finally:
        sys.argv = argv
        if real_uvicorn is not None:
            sys.modules["uvicorn"] = real_uvicorn
        else:
            sys.modules.pop("uvicorn", None)
    with_cap(None)                     # leave the module as the rest of the file found it
    with_default(None)                 # ... including the fallback the arms above pushed

    # --- /predict/batch must make ONE Router call, not one per state ---------
    #
    # README teaches `Router.predict_batch` as the way to answer many states with shared forward
    # passes ("routes the full workload first, groups requests by checkpoint ... results are
    # restored to the original request order"), and this endpoint is the batch-shaped surface the
    # README points at for trying Laya without writing code. The handler used to call
    # `Router.predict` once per state anyway, so 64 states were 64 forwards. These drive the real
    # app -- validation, `_questions()`, the handler body, the JSON envelope -- over a recording
    # stand-in at `demo.ROUTER`, so no weights are needed to count the calls.

    import inspect

    from laya.router import Router as CoreRouter

    class RecordingRouter:
        """Answers like the Router does, and remembers how it was asked."""

        def __init__(self, fail_on=None):
            self.predict_calls = []
            self.batch_calls = []
            self.fail_on = fail_on

        def predict(self, state, questions, **kw):
            self.predict_calls.append((state, dict(kw)))
            return self._answer(state, questions)

        def predict_batch(self, requests, **kw):
            self.batch_calls.append((list(requests), dict(kw)))
            return [self._answer(r["state"], r["questions"]) for r in requests]

        def _answer(self, state, questions):
            if self.fail_on and self.fail_on in str(state):
                raise ValueError("simulated failure for " + self.fail_on)
            return {"answers": {"a": {"type": "noul", "choice": False}}, "state": state,
                    "questions": questions}

    states = ["ticket %d" % i for i in range(8)]

    class LegacyRouter(RecordingRouter):
        """A Router-like object whose `predict_batch` predates the batch path entirely."""

        predict_batch = None

    # A Router that predates `predict_batch` must still be usable through the fallback.
    older = LegacyRouter()
    demo.ROUTER = older
    legacy = TestClient(demo.app, raise_server_exceptions=False).post(
        "/predict/batch", json={"states": states, "questions": one})
    ok("a router without predict_batch still answers every state",
       legacy.status_code == 200 and len(legacy.json()["results"]) == 8
       and not [r for r in legacy.json()["results"] if "error" in r],
       "%s / %s" % (legacy.status_code, json.dumps(legacy.json())[:200]))

    router = RecordingRouter()
    demo.ROUTER = router
    batched = TestClient(demo.app, raise_server_exceptions=False).post(
        "/predict/batch", json={"states": states, "questions": one})
    body = batched.json()
    ok("an 8-state batch is ONE predict_batch call and zero predict calls",
       len(router.batch_calls) == 1 and not router.predict_calls,
       "predict_batch=%d predict=%d" % (len(router.batch_calls), len(router.predict_calls)))
    ok("the batch envelope is unchanged: count and one result per state, in order",
       body.get("count") == 8 and [r.get("state") for r in body.get("results", [])] == states)
    # Read defensively: an endpoint that never batches has nothing to inspect, and the checks below
    # say so by name instead of letting this script die at the unpack.
    requests_sent, call_kwargs = (router.batch_calls[0] if router.batch_calls else ([], {}))
    ok("each request carries state + questions, and the questions map is the same one",
       len(requests_sent) == 8
       and all(sorted(r) == ["questions", "state"] for r in requests_sent)
       and all(r["questions"] == requests_sent[0]["questions"] for r in requests_sent))
    controls_sent = [k for r in requests_sent for k in r if k in ("model", "task", "lang")]
    ok("unset controls are absent, not sent as null",
       len(requests_sent) == 8 and not call_kwargs and not controls_sent,
       "call kwargs=%r controls=%r" % (call_kwargs, controls_sent))
    # Every key the endpoint puts in a request must be one core actually reads out of it, and the
    # set of those keys is derived from `Router.route_batch`'s own source, so a rename or a new
    # override in core shows up here rather than silently stopping reaching the router.
    import re

    route_src = inspect.getsource(CoreRouter.route_batch)
    read = ({"state", "questions"}
            | set(re.findall(r'request\["(\w+)"\]', route_src))
            | set(re.findall(r'request\.get\("(\w+)"', route_src)))
    sent_keys = {k for r in requests_sent for k in r}
    ok("the request keys the endpoint sends are ones core reads",
       len(requests_sent) == 8 and read >= sent_keys,
       "core reads %r, endpoint sends %r" % (sorted(read), sorted(sent_keys)))
    ok("predict_batch is still a one-positional-list call",
       list(inspect.signature(CoreRouter.predict_batch).parameters)[1] == "requests")

    pinned = RecordingRouter()
    demo.ROUTER = pinned
    TestClient(demo.app, raise_server_exceptions=False).post(
        "/predict/batch", json={"states": states, "questions": one,
                                "model": "multilingual", "lang": "de"})
    sent = pinned.batch_calls[0][0] if pinned.batch_calls else []
    ok("a pinned model/lang travels with every request",
       len(sent) == 8
       and all(r["model"] == "multilingual" and r["lang"] == "de" for r in sent)
       and not any("task" in r for r in sent),
       json.dumps(sent[:1])[:200])

    # --- how the batch is packed is askable ---------------------------------------------
    #
    # `Router.predict_batch` has taken `batch_size` and `sort_by_length` since #294, and the README
    # teaches `router.predict_batch(requests, batch_size=8, sort_by_length=True)`. The endpoint that
    # answers up to 64 states in one call forwarded neither, so every batch was one forward pass
    # padded to its longest state: a caller who wanted smaller passes had no way to ask, in code or
    # over HTTP. These drive the real request model, so a body that named the keys and had them
    # dropped by validation fails here rather than returning a silently slower 200.
    shape_params = set(inspect.signature(CoreRouter.predict_batch).parameters) - {"self", "requests"}
    ok("core still takes the shape keys this endpoint forwards",
       {"batch_size", "sort_by_length"} <= shape_params, str(sorted(shape_params)))

    shaped = RecordingRouter()
    demo.ROUTER = shaped
    body = TestClient(demo.app, raise_server_exceptions=False).post(
        "/predict/batch", json={"states": states, "questions": one,
                                "batch_size": 4, "sort_by_length": True})
    ok("a shape request is still one batch call, in order",
       len(shaped.batch_calls) == 1 and not shaped.predict_calls
       and body.json().get("count") == 8
       and [r.get("state") for r in body.json()["results"]] == states,
       "predict_batch=%d predict=%d" % (len(shaped.batch_calls), len(shaped.predict_calls)))
    ok("batch_size and sort_by_length reach predict_batch",
       shaped.batch_calls[0][1] == {"batch_size": 4, "sort_by_length": True},
       str(shaped.batch_calls[0][1]))

    sized = RecordingRouter()
    demo.ROUTER = sized
    TestClient(demo.app, raise_server_exceptions=False).post(
        "/predict/batch", json={"states": states, "questions": one, "batch_size": 2})
    ok("batch_size alone does not invent a sort request",
       sized.batch_calls[0][1] == {"batch_size": 2}, str(sized.batch_calls[0][1]))

    # Core documents `sort_by_length` without a split batch as a no-op, not an error; the endpoint
    # must not be the surface that turns a valid call into a 422.
    unsized = RecordingRouter()
    demo.ROUTER = unsized
    no_split = TestClient(demo.app, raise_server_exceptions=False).post(
        "/predict/batch", json={"states": states, "questions": one, "sort_by_length": True})
    ok("sort_by_length alone is accepted and forwarded, as core takes it",
       no_split.status_code == 200 and unsized.batch_calls[0][1] == {"sort_by_length": True},
       "%s %s" % (no_split.status_code, str(unsized.batch_calls[0][1])))

    junk = TestClient(demo.app, raise_server_exceptions=False).post(
        "/predict/batch", json={"states": states, "questions": one, "batch_size": 0})
    ok("a zero-size forward pass is a 422 before any router call",
       junk.status_code == 422, "%s %s" % (junk.status_code, str(junk.json())[:120]))

    # The shape belongs to the batch call only. When a state fails, the endpoint retries per state
    # through `Router.predict`, which takes neither key: leaking them there would turn one bad
    # state into 8 TypeErrors and an empty batch.
    leaked = RecordingRouter(fail_on="ticket 3")
    demo.ROUTER = leaked
    retried = TestClient(demo.app, raise_server_exceptions=False).post(
        "/predict/batch", json={"states": states, "questions": one,
                                "batch_size": 4, "sort_by_length": True})
    ok("the per-state fallback carries no batch shape",
       leaked.batch_calls[0][1] == {"batch_size": 4, "sort_by_length": True}
       and len(leaked.predict_calls) == 8
       and not any("batch_size" in kw or "sort_by_length" in kw
                   for _, kw in leaked.predict_calls)
       and [r for r in retried.json()["results"] if "error" in r][0]["index"] == 3,
       str([kw for _, kw in leaked.predict_calls][:1])[:200])

    legacy_shaped = LegacyRouter()
    demo.ROUTER = legacy_shaped
    old = TestClient(demo.app, raise_server_exceptions=False).post(
        "/predict/batch", json={"states": states, "questions": one,
                                "batch_size": 4, "sort_by_length": True})
    ok("a router without predict_batch still answers a shaped request",
       old.status_code == 200 and len(old.json()["results"]) == 8
       and not [r for r in old.json()["results"] if "error" in r],
       "%s %s" % (old.status_code, json.dumps(old.json())[:200]))

    # One state failing must not cost its neighbours their answer: the endpoint's published
    # contract is per-item errors inside a 200.
    partial = RecordingRouter(fail_on="ticket 3")
    demo.ROUTER = partial
    poison = TestClient(demo.app, raise_server_exceptions=False).post(
        "/predict/batch", json={"states": states, "questions": one}).json()
    errors = [r for r in poison["results"] if "error" in r]
    ok("one failing state yields one error entry, not an empty batch",
       len(poison["results"]) == 8 and [r["index"] for r in errors] == [3]
       # #625: the item names the failure without its exception text, which stays in the log
       and errors[0]["error"] == "prediction failed",
       json.dumps(poison)[:240])
    ok("the failing batch retried per state, so its neighbours still answered",
       len(partial.batch_calls) == 1 and len(partial.predict_calls) == 8)

    # The single-state surface must keep going through predict(), and `/predict/batch` with one
    # state must still batch -- otherwise the two endpoints diverge on where hooks fire.
    single = RecordingRouter()
    demo.ROUTER = single
    one_state = TestClient(demo.app, raise_server_exceptions=False)
    one_state.post("/predict", json={"state": "ticket 0", "questions": one})
    ok("/predict still calls Router.predict once",
       len(single.predict_calls) == 1 and not single.batch_calls)

    demo.ROUTER = None
    not_ready = TestClient(demo.app, raise_server_exceptions=False).post(
        "/predict/batch", json={"states": states, "questions": one})
    ok("an unready router keeps answering 200 with per-item 503s",
       not_ready.status_code == 200
       and len(not_ready.json()["results"]) == 8
       and all("503" in r.get("error", "") for r in not_ready.json()["results"]),
       json.dumps(not_ready.json())[:200])
    demo.ROUTER = None

    # ---- the `model` field resolves the way core resolves one ----------------
    # `laya.router.normalise_name` trims, lower-cases and maps its alias table before it checks
    # the checkpoint registry, and `laya --model laya` and `POST /v1/systemone {"model":"laya"}`
    # both go through it. This demo compared the string against a tuple of the three names, so the
    # alias worked on the CLI and answered 422 here. The probe below is built from laya.router's
    # own tables, so it covers spellings this file has never heard of and cannot fall behind.
    from laya import router as _router_mod

    names = sorted(_router_mod.DEFAULT_MODELS)
    aliases = sorted(a for a, target in _router_mod._ALIASES.items()
                     if target in _router_mod.DEFAULT_MODELS)
    spellings = [spelling for value in names + aliases
                 for spelling in (value, value.upper(), " %s " % value)]

    refused, not_canonical = [], []
    for value in spellings:
        want = _router_mod.normalise_name(value)
        try:
            got = demo._check_model(value)
        except Exception as exc:
            refused.append("%r -> %s" % (value, exc))
            continue
        if got != want:
            not_canonical.append("%r -> %r, core says %r" % (value, got, want))
    ok("every one of the %d spellings core resolves, the demo resolves too" % len(spellings),
       not refused, "; ".join(refused[:3]))
    ok("and what reaches the Router is the checkpoint, not the spelling that named it",
       not not_canonical, "; ".join(not_canonical[:3]))

    # The negatives, derived from the same tables so one of them cannot accidentally be real.
    unknown = [v + "-nope" for v in names + aliases]

    def _refuses(fn, value):
        try:
            fn(value)
        except ValueError:
            return True
        return False

    ok("a name neither core nor the demo knows still refuses on both surfaces",
       all(_refuses(_router_mod.normalise_name, v) and _refuses(demo._check_model, v)
           for v in unknown),
       repr([v for v in unknown if not _refuses(demo._check_model, v)][:3]))

    # Blank is the demo's own convenience, not core's: the playground posts an empty field, and
    # core's resolver raises on one. Parity must not take that away.
    def _resolves(v):
        """The value the field validator accepts, or the refusal it gave -- never a raised error."""
        try:
            return demo._check_model(v)
        except Exception as exc:
            return "refused: %s" % exc

    ok("a blank or whitespace-only model still means 'do not pin'",
       [_resolves(v) for v in (None, "", "   ")] == [None, None, None],
       repr([(v, _resolves(v)) for v in (None, "", "   ")])[:180])

    regressed = demo.ROUTER
    demo.ROUTER = RecordingRouter()
    http = TestClient(demo.app, raise_server_exceptions=False)
    single = http.post("/predict", json={"state": "ticket 0", "questions": one, "model": "ml"})
    ok("an alias is accepted by /predict and the Router is asked for its checkpoint",
       single.status_code == 200 and demo.ROUTER.predict_calls
       and demo.ROUTER.predict_calls[-1][1].get("model") == "multilingual",
       "%s %r" % (single.status_code, demo.ROUTER.predict_calls[-1:]))
    batch = http.post("/predict/batch", json={"states": states, "questions": one, "model": "LAYA"})
    ok("the same alias on /predict/batch pins english for every state in the batch",
       batch.status_code == 200 and demo.ROUTER.batch_calls
       and [r.get("model") for r in demo.ROUTER.batch_calls[-1][0]] == ["english"] * len(states),
       "%s %r" % (batch.status_code, [r.get("model")
                                      for r in (demo.ROUTER.batch_calls[-1][0]
                                                if demo.ROUTER.batch_calls else [])][:3]))
    bad = http.post("/predict", json={"state": "ticket 0", "questions": one,
                                      "model": names[0] + "-nope"})
    bad_text = json.dumps(bad.json())
    ok("an unknown name is still a 422, in core's words plus this server's own hint",
       bad.status_code == 422 and "unknown model" in bad_text and "choose one of" in bad_text
       and "(or omit it)" in bad_text, bad_text[:220])

    listing = http.get("/models").json()
    ok("/models publishes the aliases a caller may use, and every one it publishes validates",
       listing["allowed"] == names and listing.get("aliases") == {a: _router_mod._ALIASES[a]
                                                                  for a in aliases}
       and all(_resolves(a) == c for a, c in (listing.get("aliases") or {}).items()),
       json.dumps(listing)[:220])
    page = http.get("/models", headers={"accept": "text/html"}).text
    ok("the page that lists the checkpoints names the count and the spellings it accepts",
       ("%d checkpoints, one router" % len(names)) in page
       and all("<code>%s</code>" % a in page for a in aliases), page[:160])
    demo.ROUTER = regressed

    def _rendered(desc):
        """The checkpoint list and the alias list, read back out of the field's own description.

        That sentence is generated from `MODELS`/`MODEL_ALIASES`, so the honest gate is to parse
        the lists out of it and compare them with what `laya.router` accepts -- a substring check
        would also pass a sentence somebody typed beside the tables. A render this cannot read
        comes back as two empty lists, which fails by name instead of raising and aborting the run.
        """
        try:
            names_part, alias_part = desc.split("one of ", 1)[1].split(
                ", or an alias core resolves (", 1)
            alias_part = alias_part.split("), in any casing", 1)[0]
        except (IndexError, ValueError):
            return [], []
        return ([p.strip() for p in names_part.split(" | ")] if names_part.strip() else [],
                [p.strip() for p in alias_part.split(", ")] if alias_part.strip() else [])

    schema = http.get("/openapi.json").json()["components"]["schemas"]
    description = schema["PredictRequest"]["properties"]["model"].get("description", "")
    ok("the published request schema lists every name and alias the endpoint takes, in neither "
       "more nor less", _rendered(description) == (names, aliases),
       "rendered %r %r from tables %r %r" % (_rendered(description)[0], _rendered(description)[1],
                                             names, aliases))
    ok("and both request models describe the one field identically",
       description and description
       == schema["BatchRequest"]["properties"]["model"].get("description", ""),
       schema["BatchRequest"]["properties"]["model"].get("description", ""))

    # The section of the README that teaches this server has to say the field exists and where its
    # spellings come from, or the only way to learn it is to get a 422.
    with open(os.path.join(ROOT, "README.md"), encoding="utf-8") as handle:
        readme = handle.read()
    demo_page = readme.split("## Try it locally: web GUI + JSON API", 1)[1].split("\n---", 1)[0]
    ok("the README section for this server teaches `model` and where its names come from",
       "`model`" in demo_page and "laya.router" in demo_page and "/models" in demo_page,
       demo_page[-200:])

    # The registry is read, not copied. A checkpoint added to laya.router has to become pinnable
    # here -- by name, by alias, in the schema text and on /models -- with no edit to this file.
    # `laya.DEFAULT_MODELS` is the same dict object, so one insert reaches both reads.
    _router_mod.DEFAULT_MODELS["spanish"] = (_router_mod.BUNDLE_REPO, "spanish")
    _router_mod._ALIASES["es"] = "spanish"
    try:
        grown = importlib.reload(demo)
        grown_names = sorted(_router_mod.DEFAULT_MODELS)
        ok("a checkpoint core gains is pinnable here without editing this file",
           "spanish" in grown.MODELS and _resolves("spanish") == "spanish"
           and _resolves(" ES ") == "spanish"
           and grown.MODEL_ALIASES.get("es") == "spanish",
           "%r %r" % (sorted(grown.MODELS), _resolves(" es ")))
        grown_schema = TestClient(grown.app, raise_server_exceptions=False).get(
            "/openapi.json").json()["components"]["schemas"]
        grown_description = grown_schema["PredictRequest"]["properties"]["model"].get("description", "")
        ok("and the text that teaches the list is rendered from it, not typed beside it",
           _rendered(grown_description) == (grown_names, sorted(grown.MODEL_ALIASES))
           and grown_description
           == grown_schema["BatchRequest"]["properties"]["model"].get("description", ""),
           "rendered %r from %r -- %s" % (_rendered(grown_description), grown_names,
                                          grown_description[:120]))
        grown_listing = TestClient(grown.app, raise_server_exceptions=False).get("/models").json()
        ok("/models reports the new checkpoint and its new alias",
           grown_listing["allowed"] == grown_names
           and (grown_listing.get("aliases") or {}).get("es") == "spanish",
           json.dumps(grown_listing)[:220])
    finally:
        _router_mod.DEFAULT_MODELS.pop("spanish", None)
        _router_mod._ALIASES.pop("es", None)
        demo = importlib.reload(demo)

    # --- the certainty chip: it names a formula, so it has to name that type's formula ---
    # `confidence` is per-type in the decoders (normalized entropy on `choice`/`score`, the reported
    # side's probability on `noul`), but the demo's chip said "entropy" for all three and its tooltip
    # defined the field as one formula. The three have to agree: what the agents write, what the chip
    # says per type, and what the tooltip attributes to which type. Weight-free -- no Router is built.
    with open(os.path.join(ROOT, "examples", "server.py"), encoding="utf-8") as handle:
        srv = handle.read()

    def _decoded_kinds():
        """The question types this page draws distribution rows for."""
        tree = ast.parse(srv)
        fn = next(n for n in ast.walk(tree)
                  if isinstance(n, ast.FunctionDef) and n.name == "_answer_dist")
        return sorted({c.value for n in ast.walk(fn) if isinstance(n, ast.Compare)
                       and isinstance(n.left, ast.Name) and n.left.id == "kind"
                       for c in n.comparators
                       if isinstance(c, ast.Constant) and isinstance(c.value, str)})

    def _decoder_formulas(relpath):
        """{question type: "entropy" | "maxp"}, read off the `confidence` each builder writes."""
        with open(os.path.join(ROOT, relpath), encoding="utf-8") as handle:
            tree = ast.parse(handle.read())
        fn = next(n for n in ast.walk(tree)
                  if isinstance(n, ast.FunctionDef) and n.name == "_decode_answers")
        bound = {}
        for node in ast.walk(fn):
            if isinstance(node, ast.Assign) and len(node.targets) == 1 and isinstance(node.targets[0], ast.Name):
                bound[node.targets[0].id] = node.value
        got = {}
        for node in ast.walk(fn):
            if not isinstance(node, ast.Dict):
                continue
            fields = {k.value: v for k, v in zip(node.keys, node.values)
                      if isinstance(k, ast.Constant) and isinstance(k.value, str)}
            if "type" not in fields or "confidence" not in fields:
                continue
            kind = fields["type"].value
            expr = fields["confidence"]
            while isinstance(expr, ast.Name) and expr.id in bound:   # `conf_score = round(...)`
                expr = bound[expr.id]
            called = {c.func.id for c in ast.walk(expr)
                      if isinstance(c, ast.Call) and isinstance(c.func, ast.Name)}
            formula = "entropy" if "confidence_from_probs" in called else "maxp" if "max" in called else None
            if formula is None:
                raise SystemExit("%s: %s's `confidence` is neither confidence_from_probs nor a max(): %s"
                                 % (relpath, kind, ast.dump(expr)))
            if got.get(kind) not in (None, formula):
                raise SystemExit("%s: %s writes `confidence` two ways" % (relpath, kind))
            got[kind] = formula
        return got

    def _sentence_types(prose, kinds):
        """{formula: the question types named, in backticks, in the same sentence as it}."""
        got = {}
        for sentence in _re.split(r"(?<=[.!?])\s+", prose):
            named = {t for t in _re.findall(r"`([a-z]+)`", sentence) if t in kinds}
            if not named:
                continue
            for formula, words in (("entropy", "entropy"), ("maxp", "max(p")):
                if words in sentence:
                    got.setdefault(formula, set()).update(named)
        return {key: sorted(value) for key, value in sorted(got.items())}

    def _chip(row):
        hit = _re.search(r"<code>confidence</code> <b>([0-9.]+)</b> (.*?) <svg", row)
        return (hit.group(1), hit.group(2)) if hit else (None, None)

    kinds = _decoded_kinds()
    label = demo._CERTAINTY_LABEL
    tip = demo._CERTAINTY_TIP
    ok("the chip has a label for every type the page draws rows for, and no other",
       sorted(label) == kinds, "%r vs the page's %r" % (sorted(label), kinds))

    page_formula = {kind: ("entropy" if "entropy" in text else "maxp") for kind, text in sorted(label.items())}
    for relpath in ("laya/agent.py", "laya/onnx_agent.py"):
        decoders = _decoder_formulas(relpath)
        ok("%s writes `confidence` the way this page labels it" % relpath,
           decoders == page_formula, "%r vs the chip's %r" % (decoders, page_formula))

    said = _sentence_types(tip, set(kinds))
    ok("the tooltip attributes entropy to exactly the types whose chip says entropy",
       said.get("entropy") == sorted(k for k, f in page_formula.items() if f == "entropy"),
       "tooltip %r vs chip %r" % (said, page_formula))
    ok("and the reported side's max(p_true, 1 - p_true) to exactly the types whose chip does not",
       said.get("maxp") == sorted(k for k, f in page_formula.items() if f == "maxp"),
       "tooltip %r vs chip %r" % (said, page_formula))
    ok("so the tooltip names both formulas rather than defining the field as one",
       len(said) == 2 and set(said) == {"entropy", "maxp"}, repr(said))

    OLD_BLANKET = "The `confidence` field is 1 minus the normalized entropy of the whole distribution:"
    ok("the blanket sentence this replaced is gone", OLD_BLANKET not in srv, OLD_BLANKET)
    ok("and the page no longer promises an accuracy rate for a confidence value",
       "are right about 90% of the time" not in srv
       and "gate on a threshold you have measured" in tip, tip)

    OLD_JS_CHIP = 'toFixed(4)}), "entropy, not calibrated",\n        icon("info")'
    ok("the browser's chip reads the table by the row's type, in the block the page ships",
       OLD_JS_CHIP not in srv and "CERTAINTY_LABEL[kind]" in demo._PLAYGROUND_JS)
    page = client.get("/").text
    ok("the browser gets the label table from this dict, not a copy of it",
       "const CERTAINTY_LABEL = " + json.dumps(label) in page, json.dumps(label))
    ok("and the tooltip it shows is this same text",
       "is not one formula, and it is not calibrated, so do not gate on it" in page)

    import numpy as np
    from laya.common import confidence_from_probs

    spread = [0.62, 0.25, 0.13]
    ent = round(confidence_from_probs(np.asarray(spread), 3), 4)
    row = demo._answer_row("q", {"type": "choice", "choice": "billing", "answer_confidence": 0.62,
                                 "confidence": ent, "probabilities": {"billing": 0.62, "support": 0.25,
                                                                     "sales": 0.13}},
                           {"type": "choice", "instructions": "?"}, 1)
    ok("a choice row reads as the entropy it is", _chip(row) == ("%.4f" % ent, label["choice"]),
       "%r" % (_chip(row),))
    row = demo._answer_row("q", {"type": "score", "score": 1.2, "answer_confidence": 0.5,
                                 "confidence": round(confidence_from_probs(np.asarray([0.3, 0.5, 0.2]), 3), 4),
                                 "legend": {"0": "low", "1": "mid", "2": "high"},
                                 "probabilities": {"0": 0.3, "1": 0.5, "2": 0.2}},
                           {"type": "score", "instructions": "?"}, 1)
    ok("so does a score row", _chip(row) == ("%.4f" % round(
        confidence_from_probs(np.asarray([0.3, 0.5, 0.2]), 3), 4), label["score"]), "%r" % (_chip(row),))

    p_true = 0.8727
    side = round(max(p_true, 1 - p_true), 4)
    two_option_entropy = round(confidence_from_probs(np.asarray([p_true, 1 - p_true]), 2), 4)
    row = demo._answer_row("q", {"type": "noul", "noul": p_true, "answer_confidence": 0.79,
                                 "confidence": side}, {"type": "noul", "instructions": "?"}, 1)
    ok("a noul row's confidence is the side being reported, not an entropy",
       side == p_true and abs(side - two_option_entropy) > 0.4
       and _chip(row) == ("%.4f" % side, label["noul"])
       and "entropy" not in _chip(row)[1], "%r %r" % (_chip(row), two_option_entropy))

    print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
    for f in FAIL:
        print("  FAIL " + f)
    return 1 if FAIL else 0


if __name__ == "__main__":
    sys.exit(main())