Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(serve): forward every /v1/systemone control a JSON body can state, refuse the hook five
Aashish2f6f4bc73da8
diff --git a/README.md b/README.mdindex 4aadb0c4..fdd4e731 100644--- a/README.md+++ b/README.md@@ -498,7 +498,11 @@ reachable on demand, or the server rebuilds one every time routing switches),and `LAYA_API_KEY` (when set, clients mustsend `Authorization: Bearer <key>`). A client's `model` field is honoured when itnames a Laya checkpoint (`english`/`multilingual`/`typed-decisions`), otherwise-the router auto-selects by script/language.+the router auto-selects by script/language. The same body can also carry `task`,+`lang`, `lang_guess`, `min_confidence`, `max_len` and `head_max_len` — the controls+`Router.predict` takes that a JSON body can state — while the five hook arguments+(`hooks`, `on_predict_start`, `on_predict_end`, `hooks_raise`, `hooks_timeout`) are refused+with 422, because a hook is a callable that has to live where the server runs.Three things differ from Jev when you port a client:diff --git a/docs/http-api.md b/docs/http-api.mdindex a1bb894a..5b376896 100644--- a/docs/http-api.md+++ b/docs/http-api.md@@ -69,6 +69,22 @@ curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{| `state` | yes | text, email, ticket or JSON document to decide on; a missing or `null` state is a `400` || `questions` | yes | object keyed by question id; each question is `choice` / `score` / `noul` with `instructions` and `criteria` || `model` | no | names a checkpoint; anything else is ignored (see below) |+| `task` | no | forces a checkpoint by workflow name instead of letting routing decide; an unknown name is a `422` naming it |+| `lang` | no | a language code (`de`, `en-US`) that skips detection when it names a language; a blank or unrecognised code falls through to detection |+| `lang_guess` | no | a language code from the client's own identifier, consulted after `lang` and before detection; any non-English code routes to the multilingual checkpoint |+| `max_len` | no | total token window for this request, capped by `LAYA_MAX_TOKEN_BUDGET` |+| `head_max_len` | no | token window the option prompt shares, same cap; see [Widening the Token Budget](langchain.md) for when a question needs it |+| `min_confidence` | no | abstention threshold in `[0.0, 1.0]`; an answer whose `answer_confidence` falls below it comes back marked `low_confidence`, and the answer itself is kept |++`model`, `task`, `lang`, `lang_guess`, `max_len`, `head_max_len` and `min_confidence` are the+arguments `Router.predict` takes that a JSON body can state; each is forwarded only when the request+sends it, so an absent one leaves the deployment's own `Router(...)` setting in charge. The five+hook arguments `predict` also takes -- `hooks`, `on_predict_start`, `on_predict_end`,+`hooks_raise`, `hooks_timeout` -- are refused with a `422` rather than dropped: a hook is a callable+that runs inside the server process, and the last two say how the hooks a deployment installed+execute, so no value a caller sends has a meaning here. The same five are refused client-side by a+LangChain node with a `base_url` (`laya.integrations.langchain`), so a chain and a raw HTTP client+now get the same answer.`model` is accepted so a Jev client can keep sending one. The public Hugging Face ids(`convaiinnovations/laya-multilingual`, `convaiinnovations/laya-typed-decisions`), the checkpoint@@ -151,7 +167,7 @@ requests in-process without the HTTP layer.| `400` | body is not valid JSON, not an object, has no `questions`, `state` is missing or `null`, or `questions` is not an object | what is wrong || `401` | `LAYA_API_KEY` is set and the bearer token is missing or wrong | `invalid or missing bearer token` || `413` | any limit above | which limit and by how much |-| `422` | the question is well-formed JSON but invalid to Laya (unknown type, options over the head budget) | names the question and what to fix |+| `422` | the question is well-formed JSON but invalid to Laya (unknown type, options over the head budget), or a request control (`lang`, `min_confidence`, a hook argument) is not in the form this endpoint accepts | names the question or the field and what to fix || `500` | inference failed for any other reason | `inference failed` -- always this string, so paths, weights and memory state never leak; the cause is in the server log || `503` | `LAYA_MAX_CONCURRENT` requests are already in flight | `server busy, try again later` |diff --git a/laya/serve.py b/laya/serve.pyindex ed08d39a..6704d1f9 100644--- a/laya/serve.py+++ b/laya/serve.py@@ -86,6 +86,17 @@ _PUBLISHED_MODEL_IDS = {"convaiinnovations/laya-typed-decisions": "typed-decisions",}+# Every control ``Router.predict`` takes beyond ``state`` and ``questions``, split by whether a+# JSON body can mean it. ``tests/test_serve.py`` pins the two lists against+# ``inspect.signature(Router.predict)``, so a control added to the core call has to be placed on+# one side of that line -- forwarded or refused -- before the suite goes green again. The line is+# not drawn by value type: a hook is a callable that runs in the server process, and+# ``hooks_raise`` / ``hooks_timeout`` govern how the hooks the *operator* installed execute, so+# none of those five has a meaning a request could give it. Which checkpoint answers, which+# language it reads, how many tokens it gets and where it abstains are all plain data.+BODY_CONTROLS = ("model", "max_len", "head_max_len", "task", "lang", "lang_guess", "min_confidence")+BODY_REFUSALS = ("hooks", "on_predict_start", "on_predict_end", "hooks_raise", "hooks_timeout")+def _env_bool(name: str, default: bool) -> bool:v = os.environ.get(name)@@ -162,6 +173,65 @@ def _validate_budget_param(body: Dict[str, Any], key: str, max_cap: int) -> Optireturn val+def _refuse_body_refusals(body: Dict[str, Any]) -> None:+ """Reject a request that sends one of ``BODY_REFUSALS`` (422) instead of ignoring it.++ A hook is a callable that runs inside ``predict`` -- it can cache a decision, gate one or+ rewrite its state -- and ``laya-serve`` has no way to receive or run one, which is why+ ``laya.integrations.langchain`` already refuses them on a node with a ``base_url``+ (``_reject_remote_hooks``). ``hooks_raise`` and ``hooks_timeout`` belong with them: they say+ how the hooks the deployment installed execute, so honouring a caller's value would let a+ request change server-side behaviour. Today all five are read into the body and dropped, so a+ client that asks for one is told nothing.+ """+ from fastapi import HTTPException++ given = sorted(key for key in BODY_REFUSALS if key in body and body[key] is not None)+ if given:+ raise HTTPException(+ status_code=422,+ detail="%s run inside the server process and cannot be sent to this endpoint; "+ "install them where laya-serve runs, or drop them" % ", ".join(given))+++def _validate_language_param(body: Dict[str, Any], key: str) -> Optional[str]:+ """Return a ``lang`` / ``lang_guess`` value, requiring the code-string form (422).++ ``Router`` reads both through ``_english_from_code``, which stringifies its input, so a JSON+ ``true`` would become the code ``"true"`` -- a real, non-English one -- and decide the+ checkpoint. A callable hint, the other form core accepts, cannot cross an HTTP body. Explicit+ ``null`` stays "no hint", which is what lets a deployment's ``Router(lang_guess=...)`` answer.+ """+ from fastapi import HTTPException++ val = body.get(key)+ if val is None:+ return None+ if not isinstance(val, str):+ raise HTTPException(status_code=422,+ detail='%s must be a language code string such as "de", or null' % key)+ return val+++def _validate_min_confidence(body: Dict[str, Any]) -> Optional[float]:+ """Round-trip the body's ``min_confidence`` through core's validator, reporting its 422.++ Validation happens here rather than inside ``predict`` so a bad threshold is refused before the+ request takes an inference slot, and through ``laya.confidence`` rather than a copy of its+ bounds so the accepted range cannot drift from what the abstention gate itself enforces.+ """+ from fastapi import HTTPException++ if body.get("min_confidence") is None:+ return None+ from .confidence import check_min_confidence++ try:+ return check_min_confidence(body["min_confidence"])+ except ValueError as error:+ raise HTTPException(status_code=422, detail=str(error)) from None++def _resolve_max_loaded() -> Optional[int]:"""Resident-checkpoint cap from ``LAYA_MAX_LOADED``; ``None`` leaves it to ``Router``.@@ -466,15 +536,31 @@ def create_app(router: Optional[Any] = None):state = body.get("state")questions = body["questions"]_check_request_limits(state, questions)+ _refuse_body_refusals(body)model = _resolve_model(body.get("model"))max_budget_cap = _resolve_max_token_budget()max_len = _validate_budget_param(body, "max_len", max_budget_cap)head_max_len = _validate_budget_param(body, "head_max_len", max_budget_cap)+ # Each control is sent only when the client sent it: core reads an absent argument as+ # "inherit what the Router was built with", so passing None would override a deployment's+ # own `Router(lang_guess=...)` or abstention threshold with the server's default.predict_kwargs = {}if max_len is not None:predict_kwargs["max_len"] = max_lenif head_max_len is not None:predict_kwargs["head_max_len"] = head_max_len+ # An unknown task is left to `route`, which normalises it through `normalise_name` and+ # raises; the `except ValueError` below turns that into a 422 naming the task, so the+ # accepted set is core's and not a list restated here.+ if body.get("task") is not None:+ predict_kwargs["task"] = body["task"]+ for key in ("lang", "lang_guess"):+ value = _validate_language_param(body, key)+ if value is not None:+ predict_kwargs[key] = value+ min_confidence = _validate_min_confidence(body)+ if min_confidence is not None:+ predict_kwargs["min_confidence"] = min_confidenceif gate is None:gate = asyncio.Lock()try:diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.pyindex 6ca39ec8..dd6273bb 100644--- a/tests/test_hooks_api.py+++ b/tests/test_hooks_api.py@@ -816,6 +816,24 @@ for path, name, anchor in BUDGET[1:]:(None, _widest.head_max_len, _widest.max_len, None, _raised.head_max_len))+# --------------------------------------------------------------- the HTTP boundary+# A hook is a callable that runs inside `predict`, so it cannot cross `/v1/systemone`. The+# LangChain node refuses the five client-side (`_reject_remote_hooks`) and the packaged server+# refuses them server-side. Those must stay the same five: if the server's list gained an argument+# or dropped one, a chain step and a raw HTTP client would get different answers for one request+# body -- and a caller that was told "no" on one path would be ignored on the other.+from laya.integrations.langchain import _hook_kwargs as _lc_hook_kwargs # noqa: E402+from laya.serve import BODY_CONTROLS as _http_controls, BODY_REFUSALS as _http_refusals # noqa: E402++check("serve/BODY_REFUSALS is the hook argument set LangChain refuses", sorted(_http_refusals),+ sorted(_lc_hook_kwargs(hooks=[object()], on_predict_start=object(),+ on_predict_end=object(), hooks_raise=False, hooks_timeout=1.0)))+# A list that grew past hooks would refuse something a body can legitimately state.+check("serve/BODY_REFUSALS names nothing but hooks",+ [key for key in _http_refusals if "hook" not in key and "predict" not in key], [])+check("serve forwards and refuses disjoint sets", sorted(set(_http_controls) & set(_http_refusals)), [])++print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL:print(" FAIL", f)diff --git a/tests/test_serve.py b/tests/test_serve.pyindex dab066ab..6c9c6d77 100644--- a/tests/test_serve.py+++ b/tests/test_serve.py@@ -3,6 +3,7 @@A fake Router is injected so nothing loads a checkpoint; we only assert that theHTTP layer maps requests/responses and enforces auth as hs-jev expects."""+import inspectimport jsonimport loggingimport os@@ -18,6 +19,8 @@ from fastapi.testclient import TestClient # noqa: E402ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))from laya.serve import ( # noqa: E402+ BODY_CONTROLS,+ BODY_REFUSALS,DEFAULT_MAX_TOKEN_BUDGET,MAX_BODY_BYTES,_apply_thread_limit,@@ -941,3 +944,209 @@ def test_resolve_max_token_budget_fallback(monkeypatch, caplog):assert "invalid LAYA_MAX_TOKEN_BUDGET" in caplog.textassert "LAYA_MAX_TOKEN_BUDGET must be positive" in caplog.text++# --------------------------------------------------------------------- per-call controls+#+# `Router.predict` takes nine arguments beyond `state` and `questions` that a JSON body could+# state. `laya/serve.py` splits them into the two lists the tests below read, and the first test+# pins those lists against core's own signature: a control added to `predict` cannot be silently+# ignored by the HTTP surface, because it has to be placed on one side of that line first.++NEW_CONTROLS = ("task", "lang", "lang_guess", "min_confidence")+NEW_CONTROL_VALUES = {"task": "typed-decisions", "lang": "de", "lang_guess": "de",+ "min_confidence": 0.9}+++def test_every_predict_control_is_forwarded_or_refused():+ """`BODY_CONTROLS` and `BODY_REFUSALS` must cover `Router.predict` exactly, both directions."""+ from laya.router import Router++ taken = set(inspect.signature(Router.predict).parameters) - {"self", "state", "questions"}+ declared = set(BODY_CONTROLS) | set(BODY_REFUSALS)+ assert not set(BODY_CONTROLS) & set(BODY_REFUSALS)+ assert taken == declared, "predict() takes %s; serve declares %s" % (+ sorted(taken), sorted(declared))++[email protected]("key", NEW_CONTROLS)+def test_each_control_reaches_predict(monkeypatch, key):+ client, fake = _budget_client(monkeypatch)+ r = client.post("/v1/systemone", json={**REQ, key: NEW_CONTROL_VALUES[key]})+ assert r.status_code == 200, r.text+ assert fake.calls[0][key] == NEW_CONTROL_VALUES[key]+++def test_unset_controls_are_not_sent_as_none(monkeypatch):+ """An absent control must stay absent rather than be forwarded as `None`.++ Core reads `None` as "inherit what the Router was built with", so sending `lang_guess=None`+ would override a deployment's `Router(lang_guess=...)`. It would also 500 on any router whose+ `predict` does not take the keyword -- the `FakeRouter` this suite has always used.+ """+ client, fake = _budget_client(monkeypatch)+ assert client.post("/v1/systemone", json=REQ).status_code == 200+ assert not [key for key in NEW_CONTROLS if key in fake.calls[0]]++[email protected]("key", NEW_CONTROLS)+def test_explicit_null_is_no_control(monkeypatch, key):+ client, fake = _budget_client(monkeypatch)+ assert client.post("/v1/systemone", json={**REQ, key: None}).status_code == 200+ assert not [name for name in NEW_CONTROLS if name in fake.calls[0]]+++def test_min_confidence_zero_is_still_a_threshold(monkeypatch):+ """`0.0` is falsy but is a value the caller chose; `if min_confidence:` would drop it."""+ client, fake = _budget_client(monkeypatch)+ assert client.post("/v1/systemone", json={**REQ, "min_confidence": 0.0}).status_code == 200+ assert fake.calls[0]["min_confidence"] == 0.0++[email protected]("key", ["lang", "lang_guess"])[email protected]("value", [True, 5, ["de"], {"code": "de"}])+def test_a_language_control_must_be_a_code_string(monkeypatch, key, value):+ """`Router` stringifies a language hint, so `true` becomes the code `"true"`.++ That is a real, non-English code, so the JSON boolean would decide the checkpoint. A callable+ hint -- core's other accepted form -- cannot cross an HTTP body either, which leaves the code+ string as the only form to accept here.+ """+ client, fake = _budget_client(monkeypatch)+ r = client.post("/v1/systemone", json={**REQ, key: value})+ assert r.status_code == 422, r.text+ assert "must be a language code string" in r.json()["detail"]+ assert not fake.calls, "a refused request must not run inference"++[email protected]("value", [1.5, -0.1, True, "0.9"])+def test_min_confidence_is_validated_by_core(monkeypatch, value):+ """The bounds belong to `laya.confidence`, so serve must report core's own message.++ Comparing against the `ValueError` core raises for the same value is what keeps the accepted+ range from drifting: a restated `[0.0, 1.0]` check here would pass CI on the day core widened+ it and start rejecting requests the abstention gate would have honoured.+ """+ from laya.confidence import check_min_confidence++ try:+ check_min_confidence(value)+ except ValueError as error:+ expected = str(error)+ else:+ raise AssertionError("core accepted %r; this probe needs a rejecting value" % (value,))++ client, fake = _budget_client(monkeypatch)+ r = client.post("/v1/systemone", json={**REQ, "min_confidence": value})+ assert r.status_code == 422, r.text+ assert r.json()["detail"] == expected+ assert not fake.calls+++def test_an_infinite_threshold_is_refused_by_core(monkeypatch):+ """JSON writes no NaN, but a number literal can parse to infinity.++ `check_min_confidence` tests `math.isfinite` for exactly this. A range check written here would+ happen to reject it too, and then keep passing on the day core's rule changed -- so the request+ is sent as raw bytes: Python cannot serialise an infinity back to JSON, and `1e999` is what a+ client that never round-tripped through Python would actually put on the wire.+ """+ client, fake = _budget_client(monkeypatch)+ raw = (json.dumps(REQ)[:-1] + ', "min_confidence": 1e999}').encode("utf-8")+ r = client.post("/v1/systemone", content=raw,+ headers={"Content-Type": "application/json"})+ assert r.status_code == 422, r.text+ assert "min_confidence must be a float in [0.0, 1.0]" in r.json()["detail"]+ assert not fake.calls+++def test_an_unknown_task_gets_cores_answer_not_a_500(monkeypatch):+ """`task` is forwarded verbatim, as the CLI forwards `--task`, and core names the valid set.++ `route()` normalises a task through `normalise_name`, whose `ValueError` the existing mapping+ reports as a 422 carrying its message. Listing the accepted tasks in `serve` instead would be+ a second registry to keep in step -- the failure mode #544 and #638 removed elsewhere.+ """+ from laya.router import normalise_name++ class RoutingRouter(BudgetRouter):+ def predict(self, state, questions, model=None, **kwargs):+ if "task" in kwargs:+ normalise_name(kwargs["task"]) # core's guard, called exactly as route() calls it+ return super().predict(state, questions, model=model, **kwargs)++ monkeypatch.delenv("LAYA_API_KEY", raising=False)+ client = TestClient(create_app(router=RoutingRouter()), raise_server_exceptions=False)++ r = client.post("/v1/systemone", json={**REQ, "task": "not-a-task"})+ assert r.status_code == 422, r.text+ assert "unknown model" in r.json()["detail"]++ ok = client.post("/v1/systemone", json={**REQ, "task": "typed-decisions"})+ assert ok.status_code == 200, ok.text++[email protected]("key", BODY_REFUSALS)+def test_a_hook_control_is_refused_not_dropped(monkeypatch, key):+ """All five used to be read into the body and ignored, so a client got a silent no.++ `laya.integrations.langchain::_reject_remote_hooks` already refuses the same five on a node+ with a `base_url`, on the grounds that a hook runs in the server's process. The endpoint now+ says that itself instead of answering as though the control had been honoured.+ """+ value = {"hooks": [], "on_predict_start": "cache", "on_predict_end": "audit",+ "hooks_raise": False, "hooks_timeout": 5.0}[key]+ client, fake = _budget_client(monkeypatch)+ r = client.post("/v1/systemone", json={**REQ, key: value})+ assert r.status_code == 422, r.text+ detail = r.json()["detail"]+ assert key in detail+ assert "cannot be sent to this endpoint" in detail+ assert not fake.calls, "the refusal must come before inference"+++def test_a_refusal_names_every_hook_control_sent(monkeypatch):+ client, fake = _budget_client(monkeypatch)+ r = client.post("/v1/systemone", json={**REQ, "hooks_raise": True, "hooks_timeout": 5.0})+ assert r.status_code == 422, r.text+ detail = r.json()["detail"]+ assert "hooks_raise" in detail and "hooks_timeout" in detail+ assert not fake.calls++[email protected]("key", BODY_REFUSALS)+def test_a_null_hook_control_is_not_a_refusal(monkeypatch, key):+ """`{"hooks": null}` means "no per-call hooks", which is what core reads as inherit.++ A Jev client that serialises its absent fields must keep working; only a value says the caller+ asked for something this endpoint cannot do.+ """+ client, fake = _budget_client(monkeypatch)+ assert client.post("/v1/systemone", json={**REQ, key: None}).status_code == 200+++def test_http_api_page_documents_exactly_the_forwarded_controls():+ """The request-body table and the code must not drift apart in either direction.++ Read back out of the markdown rather than substring-matched: a row this parse cannot see is a+ field the documentation stopped describing, and a documented field with no code behind it is+ the same lie in the other direction. The five refusals have to be named where the page says+ why they are refused, so the reason travels with the field list.+ """+ path = os.path.join(ROOT, "docs", "http-api.md")+ with open(path, encoding="utf-8") as handle:+ lines = handle.read().splitlines()++ header = lines.index("| field | required | meaning |")+ rows = []+ for line in lines[header + 2:]:+ if not line.strip():+ break+ rows.append(line.split("|")[1].strip().strip("`"))+ documented = set(rows) - {"state", "questions"}+ assert documented == set(BODY_CONTROLS), "table says %s, serve forwards %s" % (+ sorted(documented), sorted(BODY_CONTROLS))++ page = "\n".join(lines)+ prose = page[page.index("`model`, `task`, `lang`, `lang_guess`"):]+ for key in BODY_REFUSALS:+ assert "`%s`" % key in prose, "%s is not named where the refusal is explained" % key