Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(serve): forward every /v1/systemone control a JSON body can state, refuse the hook five

Aashish2f6f4bc73da8
diff --git a/README.md b/README.md
index 4aadb0c4..fdd4e731 100644
--- a/README.md
+++ b/README.md
@@ -498,7 +498,11 @@ reachable on demand, or the server rebuilds one every time routing switches),
and `LAYA_API_KEY` (when set, clients must
send `Authorization: Bearer <key>`). A client's `model` field is honoured when it
names a Laya checkpoint (`english`/`multilingual`/`typed-decisions`), otherwise
-the router auto-selects by script/language.
+the router auto-selects by script/language. The same body can also carry `task`,
+`lang`, `lang_guess`, `min_confidence`, `max_len` and `head_max_len` — the controls
+`Router.predict` takes that a JSON body can state — while the five hook arguments
+(`hooks`, `on_predict_start`, `on_predict_end`, `hooks_raise`, `hooks_timeout`) are refused
+with 422, because a hook is a callable that has to live where the server runs.
Three things differ from Jev when you port a client:
diff --git a/docs/http-api.md b/docs/http-api.md
index a1bb894a..5b376896 100644
--- a/docs/http-api.md
+++ b/docs/http-api.md
@@ -69,6 +69,22 @@ curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
| `state` | yes | text, email, ticket or JSON document to decide on; a missing or `null` state is a `400` |
| `questions` | yes | object keyed by question id; each question is `choice` / `score` / `noul` with `instructions` and `criteria` |
| `model` | no | names a checkpoint; anything else is ignored (see below) |
+| `task` | no | forces a checkpoint by workflow name instead of letting routing decide; an unknown name is a `422` naming it |
+| `lang` | no | a language code (`de`, `en-US`) that skips detection when it names a language; a blank or unrecognised code falls through to detection |
+| `lang_guess` | no | a language code from the client's own identifier, consulted after `lang` and before detection; any non-English code routes to the multilingual checkpoint |
+| `max_len` | no | total token window for this request, capped by `LAYA_MAX_TOKEN_BUDGET` |
+| `head_max_len` | no | token window the option prompt shares, same cap; see [Widening the Token Budget](langchain.md) for when a question needs it |
+| `min_confidence` | no | abstention threshold in `[0.0, 1.0]`; an answer whose `answer_confidence` falls below it comes back marked `low_confidence`, and the answer itself is kept |
+
+`model`, `task`, `lang`, `lang_guess`, `max_len`, `head_max_len` and `min_confidence` are the
+arguments `Router.predict` takes that a JSON body can state; each is forwarded only when the request
+sends it, so an absent one leaves the deployment's own `Router(...)` setting in charge. The five
+hook arguments `predict` also takes -- `hooks`, `on_predict_start`, `on_predict_end`,
+`hooks_raise`, `hooks_timeout` -- are refused with a `422` rather than dropped: a hook is a callable
+that runs inside the server process, and the last two say how the hooks a deployment installed
+execute, so no value a caller sends has a meaning here. The same five are refused client-side by a
+LangChain node with a `base_url` (`laya.integrations.langchain`), so a chain and a raw HTTP client
+now get the same answer.
`model` is accepted so a Jev client can keep sending one. The public Hugging Face ids
(`convaiinnovations/laya-multilingual`, `convaiinnovations/laya-typed-decisions`), the checkpoint
@@ -151,7 +167,7 @@ requests in-process without the HTTP layer.
| `400` | body is not valid JSON, not an object, has no `questions`, `state` is missing or `null`, or `questions` is not an object | what is wrong |
| `401` | `LAYA_API_KEY` is set and the bearer token is missing or wrong | `invalid or missing bearer token` |
| `413` | any limit above | which limit and by how much |
-| `422` | the question is well-formed JSON but invalid to Laya (unknown type, options over the head budget) | names the question and what to fix |
+| `422` | the question is well-formed JSON but invalid to Laya (unknown type, options over the head budget), or a request control (`lang`, `min_confidence`, a hook argument) is not in the form this endpoint accepts | names the question or the field and what to fix |
| `500` | inference failed for any other reason | `inference failed` -- always this string, so paths, weights and memory state never leak; the cause is in the server log |
| `503` | `LAYA_MAX_CONCURRENT` requests are already in flight | `server busy, try again later` |
diff --git a/laya/serve.py b/laya/serve.py
index ed08d39a..6704d1f9 100644
--- a/laya/serve.py
+++ b/laya/serve.py
@@ -86,6 +86,17 @@ _PUBLISHED_MODEL_IDS = {
"convaiinnovations/laya-typed-decisions": "typed-decisions",
}
+# Every control ``Router.predict`` takes beyond ``state`` and ``questions``, split by whether a
+# JSON body can mean it. ``tests/test_serve.py`` pins the two lists against
+# ``inspect.signature(Router.predict)``, so a control added to the core call has to be placed on
+# one side of that line -- forwarded or refused -- before the suite goes green again. The line is
+# not drawn by value type: a hook is a callable that runs in the server process, and
+# ``hooks_raise`` / ``hooks_timeout`` govern how the hooks the *operator* installed execute, so
+# none of those five has a meaning a request could give it. Which checkpoint answers, which
+# language it reads, how many tokens it gets and where it abstains are all plain data.
+BODY_CONTROLS = ("model", "max_len", "head_max_len", "task", "lang", "lang_guess", "min_confidence")
+BODY_REFUSALS = ("hooks", "on_predict_start", "on_predict_end", "hooks_raise", "hooks_timeout")
+
def _env_bool(name: str, default: bool) -> bool:
v = os.environ.get(name)
@@ -162,6 +173,65 @@ def _validate_budget_param(body: Dict[str, Any], key: str, max_cap: int) -> Opti
return val
+def _refuse_body_refusals(body: Dict[str, Any]) -> None:
+ """Reject a request that sends one of ``BODY_REFUSALS`` (422) instead of ignoring it.
+
+ A hook is a callable that runs inside ``predict`` -- it can cache a decision, gate one or
+ rewrite its state -- and ``laya-serve`` has no way to receive or run one, which is why
+ ``laya.integrations.langchain`` already refuses them on a node with a ``base_url``
+ (``_reject_remote_hooks``). ``hooks_raise`` and ``hooks_timeout`` belong with them: they say
+ how the hooks the deployment installed execute, so honouring a caller's value would let a
+ request change server-side behaviour. Today all five are read into the body and dropped, so a
+ client that asks for one is told nothing.
+ """
+ from fastapi import HTTPException
+
+ given = sorted(key for key in BODY_REFUSALS if key in body and body[key] is not None)
+ if given:
+ raise HTTPException(
+ status_code=422,
+ detail="%s run inside the server process and cannot be sent to this endpoint; "
+ "install them where laya-serve runs, or drop them" % ", ".join(given))
+
+
+def _validate_language_param(body: Dict[str, Any], key: str) -> Optional[str]:
+ """Return a ``lang`` / ``lang_guess`` value, requiring the code-string form (422).
+
+ ``Router`` reads both through ``_english_from_code``, which stringifies its input, so a JSON
+ ``true`` would become the code ``"true"`` -- a real, non-English one -- and decide the
+ checkpoint. A callable hint, the other form core accepts, cannot cross an HTTP body. Explicit
+ ``null`` stays "no hint", which is what lets a deployment's ``Router(lang_guess=...)`` answer.
+ """
+ from fastapi import HTTPException
+
+ val = body.get(key)
+ if val is None:
+ return None
+ if not isinstance(val, str):
+ raise HTTPException(status_code=422,
+ detail='%s must be a language code string such as "de", or null' % key)
+ return val
+
+
+def _validate_min_confidence(body: Dict[str, Any]) -> Optional[float]:
+ """Round-trip the body's ``min_confidence`` through core's validator, reporting its 422.
+
+ Validation happens here rather than inside ``predict`` so a bad threshold is refused before the
+ request takes an inference slot, and through ``laya.confidence`` rather than a copy of its
+ bounds so the accepted range cannot drift from what the abstention gate itself enforces.
+ """
+ from fastapi import HTTPException
+
+ if body.get("min_confidence") is None:
+ return None
+ from .confidence import check_min_confidence
+
+ try:
+ return check_min_confidence(body["min_confidence"])
+ except ValueError as error:
+ raise HTTPException(status_code=422, detail=str(error)) from None
+
+
def _resolve_max_loaded() -> Optional[int]:
"""Resident-checkpoint cap from ``LAYA_MAX_LOADED``; ``None`` leaves it to ``Router``.
@@ -466,15 +536,31 @@ def create_app(router: Optional[Any] = None):
state = body.get("state")
questions = body["questions"]
_check_request_limits(state, questions)
+ _refuse_body_refusals(body)
model = _resolve_model(body.get("model"))
max_budget_cap = _resolve_max_token_budget()
max_len = _validate_budget_param(body, "max_len", max_budget_cap)
head_max_len = _validate_budget_param(body, "head_max_len", max_budget_cap)
+ # Each control is sent only when the client sent it: core reads an absent argument as
+ # "inherit what the Router was built with", so passing None would override a deployment's
+ # own `Router(lang_guess=...)` or abstention threshold with the server's default.
predict_kwargs = {}
if max_len is not None:
predict_kwargs["max_len"] = max_len
if head_max_len is not None:
predict_kwargs["head_max_len"] = head_max_len
+ # An unknown task is left to `route`, which normalises it through `normalise_name` and
+ # raises; the `except ValueError` below turns that into a 422 naming the task, so the
+ # accepted set is core's and not a list restated here.
+ if body.get("task") is not None:
+ predict_kwargs["task"] = body["task"]
+ for key in ("lang", "lang_guess"):
+ value = _validate_language_param(body, key)
+ if value is not None:
+ predict_kwargs[key] = value
+ min_confidence = _validate_min_confidence(body)
+ if min_confidence is not None:
+ predict_kwargs["min_confidence"] = min_confidence
if gate is None:
gate = asyncio.Lock()
try:
diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.py
index 6ca39ec8..dd6273bb 100644
--- a/tests/test_hooks_api.py
+++ b/tests/test_hooks_api.py
@@ -816,6 +816,24 @@ for path, name, anchor in BUDGET[1:]:
(None, _widest.head_max_len, _widest.max_len, None, _raised.head_max_len))
+# --------------------------------------------------------------- the HTTP boundary
+# A hook is a callable that runs inside `predict`, so it cannot cross `/v1/systemone`. The
+# LangChain node refuses the five client-side (`_reject_remote_hooks`) and the packaged server
+# refuses them server-side. Those must stay the same five: if the server's list gained an argument
+# or dropped one, a chain step and a raw HTTP client would get different answers for one request
+# body -- and a caller that was told "no" on one path would be ignored on the other.
+from laya.integrations.langchain import _hook_kwargs as _lc_hook_kwargs # noqa: E402
+from laya.serve import BODY_CONTROLS as _http_controls, BODY_REFUSALS as _http_refusals # noqa: E402
+
+check("serve/BODY_REFUSALS is the hook argument set LangChain refuses", sorted(_http_refusals),
+ sorted(_lc_hook_kwargs(hooks=[object()], on_predict_start=object(),
+ on_predict_end=object(), hooks_raise=False, hooks_timeout=1.0)))
+# A list that grew past hooks would refuse something a body can legitimately state.
+check("serve/BODY_REFUSALS names nothing but hooks",
+ [key for key in _http_refusals if "hook" not in key and "predict" not in key], [])
+check("serve forwards and refuses disjoint sets", sorted(set(_http_controls) & set(_http_refusals)), [])
+
+
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL", f)
diff --git a/tests/test_serve.py b/tests/test_serve.py
index dab066ab..6c9c6d77 100644
--- a/tests/test_serve.py
+++ b/tests/test_serve.py
@@ -3,6 +3,7 @@
A fake Router is injected so nothing loads a checkpoint; we only assert that the
HTTP layer maps requests/responses and enforces auth as hs-jev expects.
"""
+import inspect
import json
import logging
import os
@@ -18,6 +19,8 @@ from fastapi.testclient import TestClient # noqa: E402
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
from laya.serve import ( # noqa: E402
+ BODY_CONTROLS,
+ BODY_REFUSALS,
DEFAULT_MAX_TOKEN_BUDGET,
MAX_BODY_BYTES,
_apply_thread_limit,
@@ -941,3 +944,209 @@ def test_resolve_max_token_budget_fallback(monkeypatch, caplog):
assert "invalid LAYA_MAX_TOKEN_BUDGET" in caplog.text
assert "LAYA_MAX_TOKEN_BUDGET must be positive" in caplog.text
+
+# --------------------------------------------------------------------- per-call controls
+#
+# `Router.predict` takes nine arguments beyond `state` and `questions` that a JSON body could
+# state. `laya/serve.py` splits them into the two lists the tests below read, and the first test
+# pins those lists against core's own signature: a control added to `predict` cannot be silently
+# ignored by the HTTP surface, because it has to be placed on one side of that line first.
+
+NEW_CONTROLS = ("task", "lang", "lang_guess", "min_confidence")
+NEW_CONTROL_VALUES = {"task": "typed-decisions", "lang": "de", "lang_guess": "de",
+ "min_confidence": 0.9}
+
+
+def test_every_predict_control_is_forwarded_or_refused():
+ """`BODY_CONTROLS` and `BODY_REFUSALS` must cover `Router.predict` exactly, both directions."""
+ from laya.router import Router
+
+ taken = set(inspect.signature(Router.predict).parameters) - {"self", "state", "questions"}
+ declared = set(BODY_CONTROLS) | set(BODY_REFUSALS)
+ assert not set(BODY_CONTROLS) & set(BODY_REFUSALS)
+ assert taken == declared, "predict() takes %s; serve declares %s" % (
+ sorted(taken), sorted(declared))
+
+
[email protected]("key", NEW_CONTROLS)
+def test_each_control_reaches_predict(monkeypatch, key):
+ client, fake = _budget_client(monkeypatch)
+ r = client.post("/v1/systemone", json={**REQ, key: NEW_CONTROL_VALUES[key]})
+ assert r.status_code == 200, r.text
+ assert fake.calls[0][key] == NEW_CONTROL_VALUES[key]
+
+
+def test_unset_controls_are_not_sent_as_none(monkeypatch):
+ """An absent control must stay absent rather than be forwarded as `None`.
+
+ Core reads `None` as "inherit what the Router was built with", so sending `lang_guess=None`
+ would override a deployment's `Router(lang_guess=...)`. It would also 500 on any router whose
+ `predict` does not take the keyword -- the `FakeRouter` this suite has always used.
+ """
+ client, fake = _budget_client(monkeypatch)
+ assert client.post("/v1/systemone", json=REQ).status_code == 200
+ assert not [key for key in NEW_CONTROLS if key in fake.calls[0]]
+
+
[email protected]("key", NEW_CONTROLS)
+def test_explicit_null_is_no_control(monkeypatch, key):
+ client, fake = _budget_client(monkeypatch)
+ assert client.post("/v1/systemone", json={**REQ, key: None}).status_code == 200
+ assert not [name for name in NEW_CONTROLS if name in fake.calls[0]]
+
+
+def test_min_confidence_zero_is_still_a_threshold(monkeypatch):
+ """`0.0` is falsy but is a value the caller chose; `if min_confidence:` would drop it."""
+ client, fake = _budget_client(monkeypatch)
+ assert client.post("/v1/systemone", json={**REQ, "min_confidence": 0.0}).status_code == 200
+ assert fake.calls[0]["min_confidence"] == 0.0
+
+
[email protected]("key", ["lang", "lang_guess"])
[email protected]("value", [True, 5, ["de"], {"code": "de"}])
+def test_a_language_control_must_be_a_code_string(monkeypatch, key, value):
+ """`Router` stringifies a language hint, so `true` becomes the code `"true"`.
+
+ That is a real, non-English code, so the JSON boolean would decide the checkpoint. A callable
+ hint -- core's other accepted form -- cannot cross an HTTP body either, which leaves the code
+ string as the only form to accept here.
+ """
+ client, fake = _budget_client(monkeypatch)
+ r = client.post("/v1/systemone", json={**REQ, key: value})
+ assert r.status_code == 422, r.text
+ assert "must be a language code string" in r.json()["detail"]
+ assert not fake.calls, "a refused request must not run inference"
+
+
[email protected]("value", [1.5, -0.1, True, "0.9"])
+def test_min_confidence_is_validated_by_core(monkeypatch, value):
+ """The bounds belong to `laya.confidence`, so serve must report core's own message.
+
+ Comparing against the `ValueError` core raises for the same value is what keeps the accepted
+ range from drifting: a restated `[0.0, 1.0]` check here would pass CI on the day core widened
+ it and start rejecting requests the abstention gate would have honoured.
+ """
+ from laya.confidence import check_min_confidence
+
+ try:
+ check_min_confidence(value)
+ except ValueError as error:
+ expected = str(error)
+ else:
+ raise AssertionError("core accepted %r; this probe needs a rejecting value" % (value,))
+
+ client, fake = _budget_client(monkeypatch)
+ r = client.post("/v1/systemone", json={**REQ, "min_confidence": value})
+ assert r.status_code == 422, r.text
+ assert r.json()["detail"] == expected
+ assert not fake.calls
+
+
+def test_an_infinite_threshold_is_refused_by_core(monkeypatch):
+ """JSON writes no NaN, but a number literal can parse to infinity.
+
+ `check_min_confidence` tests `math.isfinite` for exactly this. A range check written here would
+ happen to reject it too, and then keep passing on the day core's rule changed -- so the request
+ is sent as raw bytes: Python cannot serialise an infinity back to JSON, and `1e999` is what a
+ client that never round-tripped through Python would actually put on the wire.
+ """
+ client, fake = _budget_client(monkeypatch)
+ raw = (json.dumps(REQ)[:-1] + ', "min_confidence": 1e999}').encode("utf-8")
+ r = client.post("/v1/systemone", content=raw,
+ headers={"Content-Type": "application/json"})
+ assert r.status_code == 422, r.text
+ assert "min_confidence must be a float in [0.0, 1.0]" in r.json()["detail"]
+ assert not fake.calls
+
+
+def test_an_unknown_task_gets_cores_answer_not_a_500(monkeypatch):
+ """`task` is forwarded verbatim, as the CLI forwards `--task`, and core names the valid set.
+
+ `route()` normalises a task through `normalise_name`, whose `ValueError` the existing mapping
+ reports as a 422 carrying its message. Listing the accepted tasks in `serve` instead would be
+ a second registry to keep in step -- the failure mode #544 and #638 removed elsewhere.
+ """
+ from laya.router import normalise_name
+
+ class RoutingRouter(BudgetRouter):
+ def predict(self, state, questions, model=None, **kwargs):
+ if "task" in kwargs:
+ normalise_name(kwargs["task"]) # core's guard, called exactly as route() calls it
+ return super().predict(state, questions, model=model, **kwargs)
+
+ monkeypatch.delenv("LAYA_API_KEY", raising=False)
+ client = TestClient(create_app(router=RoutingRouter()), raise_server_exceptions=False)
+
+ r = client.post("/v1/systemone", json={**REQ, "task": "not-a-task"})
+ assert r.status_code == 422, r.text
+ assert "unknown model" in r.json()["detail"]
+
+ ok = client.post("/v1/systemone", json={**REQ, "task": "typed-decisions"})
+ assert ok.status_code == 200, ok.text
+
+
[email protected]("key", BODY_REFUSALS)
+def test_a_hook_control_is_refused_not_dropped(monkeypatch, key):
+ """All five used to be read into the body and ignored, so a client got a silent no.
+
+ `laya.integrations.langchain::_reject_remote_hooks` already refuses the same five on a node
+ with a `base_url`, on the grounds that a hook runs in the server's process. The endpoint now
+ says that itself instead of answering as though the control had been honoured.
+ """
+ value = {"hooks": [], "on_predict_start": "cache", "on_predict_end": "audit",
+ "hooks_raise": False, "hooks_timeout": 5.0}[key]
+ client, fake = _budget_client(monkeypatch)
+ r = client.post("/v1/systemone", json={**REQ, key: value})
+ assert r.status_code == 422, r.text
+ detail = r.json()["detail"]
+ assert key in detail
+ assert "cannot be sent to this endpoint" in detail
+ assert not fake.calls, "the refusal must come before inference"
+
+
+def test_a_refusal_names_every_hook_control_sent(monkeypatch):
+ client, fake = _budget_client(monkeypatch)
+ r = client.post("/v1/systemone", json={**REQ, "hooks_raise": True, "hooks_timeout": 5.0})
+ assert r.status_code == 422, r.text
+ detail = r.json()["detail"]
+ assert "hooks_raise" in detail and "hooks_timeout" in detail
+ assert not fake.calls
+
+
[email protected]("key", BODY_REFUSALS)
+def test_a_null_hook_control_is_not_a_refusal(monkeypatch, key):
+ """`{"hooks": null}` means "no per-call hooks", which is what core reads as inherit.
+
+ A Jev client that serialises its absent fields must keep working; only a value says the caller
+ asked for something this endpoint cannot do.
+ """
+ client, fake = _budget_client(monkeypatch)
+ assert client.post("/v1/systemone", json={**REQ, key: None}).status_code == 200
+
+
+def test_http_api_page_documents_exactly_the_forwarded_controls():
+ """The request-body table and the code must not drift apart in either direction.
+
+ Read back out of the markdown rather than substring-matched: a row this parse cannot see is a
+ field the documentation stopped describing, and a documented field with no code behind it is
+ the same lie in the other direction. The five refusals have to be named where the page says
+ why they are refused, so the reason travels with the field list.
+ """
+ path = os.path.join(ROOT, "docs", "http-api.md")
+ with open(path, encoding="utf-8") as handle:
+ lines = handle.read().splitlines()
+
+ header = lines.index("| field | required | meaning |")
+ rows = []
+ for line in lines[header + 2:]:
+ if not line.strip():
+ break
+ rows.append(line.split("|")[1].strip().strip("`"))
+ documented = set(rows) - {"state", "questions"}
+ assert documented == set(BODY_CONTROLS), "table says %s, serve forwards %s" % (
+ sorted(documented), sorted(BODY_CONTROLS))
+
+ page = "\n".join(lines)
+ prose = page[page.index("`model`, `task`, `lang`, `lang_guess`"):]
+ for key in BODY_REFUSALS:
+ assert "`%s`" % key in prose, "%s is not named where the refusal is explained" % key