Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(serve): answer large choices by elimination behind LAYA_TOURNAMENT_GROUP (#302)
kmitulc3e631bcd936
diff --git a/docs/http-api.md b/docs/http-api.mdindex a93644ab..18a8a88e 100644--- a/docs/http-api.md+++ b/docs/http-api.md@@ -43,6 +43,7 @@ Everything is environment variables, so one image serves a laptop dev run and a| `LAYA_MAX_CONCURRENT` | requests admitted past auth at once; excess gets `503` | `16` || `LAYA_MAX_BATCH_TOKENS` | tokens one `/v1/systemone/batch` FORWARD PASS may collate (`states` x questions x row width); a larger batch is split into several passes, not refused | `131072` || `LAYA_JEV_STRICT` | serve the strict Jev wire contract: no root `routing`, no per-answer `action` / `answer_confidence`, no `confidence` on noul answers, and `usage` reduced to `input_tokens` + `output_tokens`. For clients that validate the response against the Jev contract with no extra fields | `0` |+| `LAYA_TOURNAMENT_GROUP` | answer a `choice` with more options than this by elimination, in groups of this size, on `/v1/systemone`, and raise its option cap from 100 to 255. See [Large choices](#large-choices-laya_tournament_group). Unset, or outside 2 to 255, is off | `0` |For a deployment published under a prefix such as `/laya`, set `LAYA_ROOT_PATH=/laya`.FastAPI uses it when generating OpenAPI and Swagger UI URLs. Configure the reverse proxy to@@ -285,6 +286,37 @@ stringifies its own criteria may not match byte for byte.Successful responses also carry `Server-Timing: inference;dur=<ms>` and `X-Inference-Time-Ms`.+### Large choices: `LAYA_TOURNAMENT_GROUP`++One pass reads every option of a `choice` inside one option budget, so past a few dozen options+they stop getting a token span each (`usage.options` reports when that happens), and past about 100+the request is refused. Jev accepts up to 255. With `LAYA_TOURNAMENT_GROUP=16`, `/v1/systemone`+answers a choice with more than 16 options the way `laya.predict_tournament` does: the options are+cut into groups of at most 16, each group is answered in one shared call, and the group winners go+to a final call. Its option cap becomes 255. A choice of 16 or fewer, and every other question,+takes the usual path unchanged.++The answer keeps the Jev shape, so the official SDK parses it:++- `probabilities` names every option, in criteria order. The finalists carry the final round's+ probabilities and an option eliminated in an earlier round carries `0.0`, so `choice` is still+ the argmax and the values still sum to 1;+- `x_tournament` lists the `finalists` and the number of `rounds`, so a caller can tell an+ eliminated option from one the final round scored. Under `LAYA_JEV_STRICT` it is not sent;+- `confidence` and `answer_confidence` are the final round's, over the finalists;+- `usage.input_tokens` is summed over every call; the other `usage` keys are the final call's.++A choice that sets `option_order`, or uses a non-string label, keeps the one-pass path, since an order+over the full label set cannot be carried through a narrowing (#1062). `/v1/systemone/batch` is not+covered and keeps the 100-option cap.++Measured on the `english` checkpoint (CPU) with one test query per intent from CLINC150 (150+intents, `random_state=0`), the question listing every intent and the instruction "Which intent does+this request express?": with the flag off the request is refused (`150 > 100`); with+`LAYA_TOURNAMENT_GROUP=16` it is answered at 86.7% accuracy (130/150), median 0.77 s per request. On+50-option subsets of the same queries, the tournament scored 94.0% against 82.7% for one pass. That+is one dataset and one sample; measure on your own labels before relying on it.+## LimitsRequest guardrails are checked before tokenization, so an oversized request costs the server@@ -298,7 +330,7 @@ it answers `503` with `Retry-After: 1`.| `state` | 50,000 characters of the text the model is given -- the string itself for a string state, `json.dumps(state, ensure_ascii=False)` for an object or array || questions per request | 64 || `states` per batch request | 64 |-| options per `choice` question | 100 |+| options per `choice` question | 100; 255 on `/v1/systemone` with `LAYA_TOURNAMENT_GROUP` set || levels per `score` question | 32 || options across all questions | 512 || concurrent admitted requests (`503`, not `413`) | `LAYA_MAX_CONCURRENT` (16) |diff --git a/laya/serve.py b/laya/serve.pyindex fd3da64d..549689a8 100644--- a/laya/serve.py+++ b/laya/serve.py@@ -53,6 +53,10 @@ env var meaning defaul`answer_confidence`, no `confidence` on noulanswers, and `usage` reduced to`input_tokens` + `output_tokens`+``LAYA_TOURNAMENT_GROUP`` answer a choice with more options than this 0 (off)+ by elimination in groups of this size, on+ ``/v1/systemone``; raises its option cap from+ 100 to 255 (#302)========================= ============================================ =========``LAYA_DEVICE`` is a preference, not a guarantee: an ``Agent`` that asks for a@@ -120,6 +124,10 @@ DEFAULT_MAX_BATCH_TOKENS = 131072_BATCH_ROW_TOKENS_ASSUMED = 512# HTTP-only amplification guard; the library keeps its head_max_len-aware budget.MAX_CHOICE_OPTIONS = 100+# With `LAYA_TOURNAMENT_GROUP` set, a choice is narrowed in groups rather than read in one pass, so+# its cap is Jev's documented one instead (#302). The tournament reads every label inside a group's+# option budget, so the one-pass budget that sets MAX_CHOICE_OPTIONS no longer binds.+MAX_TOURNAMENT_CHOICE_OPTIONS = 255MAX_SCORE_LEVELS = 32MAX_TOTAL_OPTIONS = 512@@ -226,6 +234,61 @@ def _add_jev_confidence(result: Dict[str, Any]) -> Dict[str, Any]:return {**result, "answers": answers}+def _predict_with_tournament(router: Any, state: Any, questions: Dict[str, Any], group: int,+ **predict_kwargs: Any) -> Dict[str, Any]:+ """Answer `questions`, narrowing every choice over `group` options by elimination (#302).++ One pass reads every option inside a single option budget, and past a few dozen options they+ no longer each get a token span of their own. `predict_tournament` reads them in groups of+ `group` instead. When no choice is over `group`, this is exactly `router.predict`.++ The answer keeps the Jev shape: `probabilities` names every option in criteria order, the+ finalists carry the final round's probabilities, and an option eliminated in an earlier round+ carries 0.0, so `choice` is still the argmax and the values still sum to 1. `x_tournament`+ records the finalists and the round count, so a caller can tell an eliminated option from one+ the final round scored. `confidence` and `answer_confidence` are the final round's.+ `usage.input_tokens` is summed over every call; the other usage keys are the final call's.++ A choice with `option_order`, or with a label that is not a string, keeps the one-pass path:+ narrowing cannot carry an order over the full label set (#1062).+ """+ from .shortlist import _criteria_items, predict_tournament+ large = {}+ for qid, question in questions.items():+ if not (isinstance(question, dict) and question.get("type") == "choice"):+ continue+ criteria = question.get("criteria")+ if isinstance(criteria, (dict, list)) and len(criteria) > group:+ large[qid] = [label for label, _ in _criteria_items(criteria)]+ if not large or any(questions[qid].get("option_order") is not None+ or not all(isinstance(label, str) for label in labels)+ for qid, labels in large.items()):+ return router.predict(state, questions, **predict_kwargs)++ calls = []++ class _Recorded:+ def predict(self, *args, **kwargs):+ result = router.predict(*args, **kwargs)+ calls.append(result)+ return result++ result = dict(predict_tournament(_Recorded(), state, questions, group_size=group, **predict_kwargs))+ rounds = result.pop("tournament", {})+ answers = dict(result.get("answers") or {})+ for qid, labels in large.items():+ answer, meta = answers.get(qid), rounds.get(qid) or {}+ if not isinstance(answer, dict) or not meta.get("rounds"):+ continue+ final = answer.get("probabilities") or {}+ answers[qid] = {**answer,+ "probabilities": {label: final.get(label, 0.0) for label in labels},+ "x_tournament": {"finalists": list(meta["labels"]), "rounds": meta["rounds"]}}+ usage = dict(result.get("usage") or {})+ usage["input_tokens"] = sum((call.get("usage") or {}).get("input_tokens", 0) for call in calls)+ return {**result, "answers": answers, "usage": usage}++def _published_model_ids() -> Dict[str, str]:"""Public Hugging Face ids accepted so a client can name a checkpoint.@@ -312,6 +375,23 @@ def _resolve_max_concurrent() -> int:return n if n > 0 else DEFAULT_MAX_CONCURRENT+def _resolve_tournament_group() -> int:+ """Group size for choices answered by elimination, from LAYA_TOURNAMENT_GROUP; 0 is off (#302).++ A choice with more options than this goes through `predict_tournament` in groups of this size.+ Anything unset, unparseable, below 2 or above MAX_TOURNAMENT_CHOICE_OPTIONS leaves it off, so+ the one-pass path is the default: no choice the cap admits has more options than such a group.+ """+ raw = os.environ.get("LAYA_TOURNAMENT_GROUP")+ if not raw:+ return 0+ try:+ n = int(raw)+ except ValueError:+ return 0+ return n if 2 <= n <= MAX_TOURNAMENT_CHOICE_OPTIONS else 0++def _resolve_idle_unload_seconds() -> float:"""Idle window from LAYA_IDLE_UNLOAD_SECONDS; zero disables unloading."""raw = os.environ.get("LAYA_IDLE_UNLOAD_SECONDS", "").strip()@@ -715,7 +795,7 @@ def _state_length(state: Any) -> int:raise HTTPException(status_code=400, detail="'state' must be JSON-serializable")-def _check_request_limits(state: Any, questions: Any) -> None:+def _check_request_limits(state: Any, questions: Any, max_choice_options: int = MAX_CHOICE_OPTIONS) -> None:"""Reject absent or oversized inference requests before tokenization (400/413)."""from fastapi import HTTPException@@ -742,10 +822,10 @@ def _check_request_limits(state: Any, questions: Any) -> None:if qtype == "choice" and isinstance(crit, (dict, list)):count = len(crit)total_options += count- if count > MAX_CHOICE_OPTIONS:+ if count > max_choice_options:raise HTTPException(status_code=413,- detail="too many choice options for %r (%d > %d)" % (qid, count, MAX_CHOICE_OPTIONS),+ detail="too many choice options for %r (%d > %d)" % (qid, count, max_choice_options),)elif qtype == "score" and isinstance(crit, list):count = len(crit)@@ -981,6 +1061,7 @@ def create_app(router: Optional[Any] = None):admission: Optional[asyncio.Semaphore] = Noneidle_unload_seconds = _resolve_idle_unload_seconds()+ tournament_group = _resolve_tournament_group()last_request = time.monotonic()def _mark_request():@@ -1148,7 +1229,8 @@ def create_app(router: Optional[Any] = None):raise HTTPException(status_code=400, detail="request body must be an object with a 'questions' field")state = body.get("state")questions = body["questions"]- _check_request_limits(state, questions)+ _check_request_limits(state, questions, MAX_TOURNAMENT_CHOICE_OPTIONS if tournament_group+ else MAX_CHOICE_OPTIONS)_refuse_body_refusals(body)# After the size checks, so an oversized body is refused before anything walks it, and# `MAX_STATE_CHARS`/`MAX_QUESTIONS` bound what the walk can reach. A `\udXXX` escape with@@ -1194,8 +1276,12 @@ def create_app(router: Optional[Any] = None):async with gate:loop = asyncio.get_running_loop()t0 = time.perf_counter()- result = await loop.run_in_executor(- pool, _run_inference, partial(router.predict, state, questions, model=model, **predict_kwargs))+ if tournament_group:+ run = partial(_predict_with_tournament, router, state, questions, tournament_group,+ model=model, **predict_kwargs)+ else:+ run = partial(router.predict, state, questions, model=model, **predict_kwargs)+ result = await loop.run_in_executor(pool, _run_inference, run)infer_ms = (time.perf_counter() - t0) * 1000.0if _env_bool("LAYA_JEV_STRICT", False):result = _project_jev_strict(result)diff --git a/nix/laya-serve.nix b/nix/laya-serve.nixindex 3ae5b63b..1efc4ad8 100644--- a/nix/laya-serve.nix+++ b/nix/laya-serve.nix@@ -251,6 +251,19 @@ in'';};+ tournamentGroup = lib.mkOption {+ type = lib.types.nullOr (lib.types.ints.between 2 255);+ default = null;+ example = 16;+ description = ''+ Answer a `choice` with more options than this by elimination, in groups+ of this size, on /v1/systemone (sets `LAYA_TOURNAMENT_GROUP`), and raise+ its option cap from 100 to 255. Every option still comes back in+ `probabilities`; one eliminated in an earlier round carries 0.0. null+ leaves it off, so every choice is read in one pass as before.+ '';+ };+revision = lib.mkOption {type = lib.types.nullOr (lib.types.strMatching "[A-Za-z0-9._/-]+");default = null;@@ -379,6 +392,8 @@ inLAYA_MAX_TOKEN_BUDGET = toString cfg.maxTokenBudget;} // lib.optionalAttrs (cfg.maxBatchTokens != null) {LAYA_MAX_BATCH_TOKENS = toString cfg.maxBatchTokens;+ } // lib.optionalAttrs (cfg.tournamentGroup != null) {+ LAYA_TOURNAMENT_GROUP = toString cfg.tournamentGroup;} // lib.optionalAttrs (cfg.revision != null) {LAYA_REVISION = cfg.revision;} // lib.optionalAttrs (cfg.extraModels != null) {diff --git a/tests/test_packaging.py b/tests/test_packaging.pyindex 7199fca4..a9cba7e2 100644--- a/tests/test_packaging.py+++ b/tests/test_packaging.py@@ -594,7 +594,8 @@ def option_type(opt):# Every new knob is opt-in: unset means the unit exports nothing and the runtime's own default# applies, so a host that ignores them gets today's behaviour byte for byte.for opt in ("rootPath", "logLevel", "maxConcurrent", "cudaAmp", "cpuAmp", "mpsAmpMinRows",- "maxLoaded", "maxTokenBudget", "revision", "defaultModel", "idleUnloadSeconds"):+ "maxLoaded", "maxTokenBudget", "revision", "defaultModel", "idleUnloadSeconds",+ "tournamentGroup"):_t = option_text(opt)check_true("nix/module declares %s" % opt, _t != "", "option not found")check_true("nix/%s is opt-in (nullOr, default null)" % opt,diff --git a/tests/test_serve.py b/tests/test_serve.pyindex 0850cd66..4b2aec90 100644--- a/tests/test_serve.py+++ b/tests/test_serve.py@@ -38,6 +38,7 @@ from laya.serve import ( # noqa: E402_resolve_idle_unload_seconds,_resolve_max_loaded,_resolve_model,+ _resolve_tournament_group,create_app,)@@ -288,6 +289,141 @@ def test_jev_strict_leaves_out_jev_confidence(monkeypatch):assert all("x_jev_confidence" not in answer for answer in answers.values())+# --- LAYA_TOURNAMENT_GROUP: large choices answered by elimination (#302) ----------------------++def _queues(n):+ return {"Queue %03d" % i: "Tickets routed to queue %d" % i for i in range(n)}+++def _large_req(n, **question):+ return {"state": "I was charged twice, please refund one.",+ "questions": {"queue": {"type": "choice", "instructions": "Which queue?",+ "criteria": _queues(n), **question}}}+++class EliminationRouter:+ """Answers every choice it is asked: `Queue 077` wins when offered, else the first label.++ Records each call's question set and reports 10 input tokens per question, so a test can count+ the calls a tournament made and check that their usage was summed.+ """++ loaded = ["english"]++ def __init__(self):+ self.calls = []++ def predict(self, state, questions, model=None, **kwargs):+ self.calls.append(questions)+ answers = {}+ for qid, q in questions.items():+ labels = list(q["criteria"])+ winner = "Queue 077" if "Queue 077" in labels else labels[0]+ rest = (0.3 / (len(labels) - 1)) if len(labels) > 1 else 0.0+ probabilities = {label: (0.7 if label == winner else rest) for label in labels}+ if len(labels) == 1:+ probabilities[winner] = 1.0+ answers[qid] = {"type": "choice", "choice": winner, "probabilities": probabilities,+ "confidence": 0.5, "answer_confidence": probabilities[winner]}+ return {"model": "laya-rl-agent", "answers": answers,+ "usage": {"input_tokens": 10 * len(questions), "output_tokens": 0},+ "routing": {"model": "english", "reason": "English Latin text"}}+++def _tournament_client(monkeypatch, group, strict=False):+ monkeypatch.delenv("LAYA_API_KEY", raising=False)+ if group is None:+ monkeypatch.delenv("LAYA_TOURNAMENT_GROUP", raising=False)+ else:+ monkeypatch.setenv("LAYA_TOURNAMENT_GROUP", group)+ if strict:+ monkeypatch.setenv("LAYA_JEV_STRICT", "1")+ else:+ monkeypatch.delenv("LAYA_JEV_STRICT", raising=False)+ fake = EliminationRouter()+ return TestClient(create_app(router=fake)), fake+++def test_large_choice_is_refused_without_a_tournament(monkeypatch):+ client, fake = _tournament_client(monkeypatch, None)+ r = client.post("/v1/systemone", json=_large_req(128))+ assert r.status_code == 413+ assert "128 > 100" in r.json()["detail"]+ assert fake.calls == []+++def test_tournament_answers_a_large_choice_in_the_jev_shape(monkeypatch):+ client, fake = _tournament_client(monkeypatch, "16")+ r = client.post("/v1/systemone", json=_large_req(128))+ assert r.status_code == 200+ body = r.json()+ answer = body["answers"]["queue"]+ # every option is named, in criteria order, and the values are still a distribution+ assert list(answer["probabilities"]) == list(_queues(128))+ assert abs(sum(answer["probabilities"].values()) - 1.0) < 1e-9+ # the choice is still the argmax; eliminated options carry 0.0, finalists the final round's values+ assert answer["choice"] == "Queue 077"+ assert max(answer["probabilities"], key=answer["probabilities"].get) == "Queue 077"+ assert answer["x_tournament"]["rounds"] == 1+ finalists = answer["x_tournament"]["finalists"]+ assert len(finalists) == 8 and "Queue 077" in finalists+ assert all(p == 0.0 for label, p in answer["probabilities"].items() if label not in finalists)+ assert "tournament" not in body+ # one round of 8 group questions, then the final call; usage is summed over both+ assert [len(call) for call in fake.calls] == [8, 1]+ assert body["usage"]["input_tokens"] == 90+++def test_tournament_accepts_up_to_255_options(monkeypatch):+ client, _ = _tournament_client(monkeypatch, "16")+ assert client.post("/v1/systemone", json=_large_req(255)).status_code == 200+ r = client.post("/v1/systemone", json=_large_req(256))+ assert r.status_code == 413+ assert "256 > 255" in r.json()["detail"]+++def test_tournament_under_strict_keeps_only_the_contracted_keys(monkeypatch):+ client, _ = _tournament_client(monkeypatch, "16", strict=True)+ answer = client.post("/v1/systemone", json=_large_req(128)).json()["answers"]["queue"]+ assert set(answer) == {"type", "choice", "confidence", "probabilities"}+ assert len(answer["probabilities"]) == 128+++def test_small_choice_skips_the_tournament(monkeypatch):+ client, fake = _tournament_client(monkeypatch, "16")+ req = _large_req(16)+ answer = client.post("/v1/systemone", json=req).json()["answers"]["queue"]+ assert fake.calls == [req["questions"]]+ assert "x_tournament" not in answer+++def test_option_order_keeps_the_one_pass_path(monkeypatch):+ """Narrowing cannot carry an order over the full label set (#1062), so it is not attempted."""+ client, fake = _tournament_client(monkeypatch, "16")+ req = _large_req(32, option_order=list(range(32))[::-1])+ answer = client.post("/v1/systemone", json=req).json()["answers"]["queue"]+ assert len(fake.calls) == 1 and len(fake.calls[0]["queue"]["criteria"]) == 32+ assert "x_tournament" not in answer+++def test_batch_keeps_the_one_pass_cap_with_a_tournament_group(monkeypatch):+ client, _ = _tournament_client(monkeypatch, "16")+ req = _large_req(128)+ r = client.post("/v1/systemone/batch", json={"states": [req["state"]], "questions": req["questions"]})+ assert r.status_code == 413++[email protected]("raw, want", [(None, 0), ("", 0), ("abc", 0), ("0", 0), ("1", 0), ("-4", 0),+ ("2", 2), ("16", 16), ("255", 255),+ ("256", 0)])+def test_tournament_group_resolution(monkeypatch, raw, want):+ if raw is None:+ monkeypatch.delenv("LAYA_TOURNAMENT_GROUP", raising=False)+ else:+ monkeypatch.setenv("LAYA_TOURNAMENT_GROUP", raw)+ assert _resolve_tournament_group() == want++def test_known_model_is_honoured(monkeypatch):client, fake = _client(monkeypatch)client.post("/v1/systemone", json={**REQ, "model": "multilingual"})