Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(serve): narrow large choices on /v1/systemone/batch too (#302)
kmitul38921c00df9e
diff --git a/docs/http-api.md b/docs/http-api.mdindex 2cf9a5ac..999e4e44 100644--- a/docs/http-api.md+++ b/docs/http-api.md@@ -43,7 +43,7 @@ Everything is environment variables, so one image serves a laptop dev run and a| `LAYA_MAX_CONCURRENT` | requests admitted past auth at once; excess gets `503` | `16` || `LAYA_MAX_BATCH_TOKENS` | tokens one `/v1/systemone/batch` FORWARD PASS may collate (`states` x questions x row width); a larger batch is split into several passes, not refused | `131072` || `LAYA_JEV_STRICT` | serve the strict Jev wire contract: no root `routing`, no per-answer `action` / `answer_confidence`, no `confidence` on noul answers, and `usage` reduced to `input_tokens` + `output_tokens`. For clients that validate the response against the Jev contract with no extra fields | `0` |-| `LAYA_TOURNAMENT_GROUP` | answer a `choice` with more options than this by elimination, in groups of this size, on `/v1/systemone`, and raise its option cap from 100 to 255. See [Large choices](#large-choices-laya_tournament_group). Unset, or outside 2 to 100, is off | `0` |+| `LAYA_TOURNAMENT_GROUP` | answer a `choice` with more options than this by elimination, in groups of this size, on `/v1/systemone` and `/v1/systemone/batch`, and raise their option cap from 100 to 255. See [Large choices](#large-choices-laya_tournament_group). Unset, or outside 2 to 100, is off | `0` |For a deployment published under a prefix such as `/laya`, set `LAYA_ROOT_PATH=/laya`.FastAPI uses it when generating OpenAPI and Swagger UI URLs. Configure the reverse proxy to@@ -294,10 +294,10 @@ Successful responses also carry `Server-Timing: inference;dur=<ms>` and `X-InferOne pass reads every option of a `choice` inside one option budget, so past a few dozen optionsthey stop getting a token span each (`usage.options` reports when that happens), and past about 100-the request is refused. Jev accepts up to 255. With `LAYA_TOURNAMENT_GROUP=16`, `/v1/systemone`-answers a choice with more than 16 options the way `laya.predict_tournament` does: the options are+the request is refused. Jev accepts up to 255. With `LAYA_TOURNAMENT_GROUP=16`, both decision routes+answer a choice with more than 16 options the way `laya.predict_tournament` does: the options arecut into groups of at most 16, each group is answered in one shared call, and the group winners go-to a final call. Its option cap becomes 255. A choice of 16 or fewer, and every other question,+to a final call. Their option cap becomes 255. A choice of 16 or fewer, and every other question,takes the usual path unchanged. The group size is limited to 2 to 100, the one-pass cap, so everychoice the raised cap newly admits is always narrowed rather than read in one pass it cannot fit.A larger group is not a better one: each group is still read in one pass, and one pass loses@@ -314,9 +314,17 @@ The answer keeps the Jev shape, so the official SDK parses it:- `confidence` and `answer_confidence` are the final round's, over the finalists;- `usage.input_tokens` is summed over every call; the other `usage` keys are the final call's.-A choice that sets `option_order`, or uses a non-string label, keeps the one-pass path, since an order-over the full label set cannot be carried through a narrowing (#1062). `/v1/systemone/batch` is not-covered and keeps the 100-option cap.+Narrowing drops an `option_order`, since an order over the full label set no longer fits a group+(#1062). So a choice that sets one keeps the one-pass path while one pass can read it. Once a+request has a choice over 100 options, which only the raised cap admits, it is narrowed, and a+narrowed choice that carried an order reports `x_tournament.option_order_dropped: true`. A label+that is not a string cannot be mapped back onto the answer: such a choice keeps one pass, and a+request that has to be narrowed is refused with a `422` naming it.++On `/v1/systemone/batch` the question set is shared, so the decision is made once: a batch with+nothing to narrow is the single batched pass it has always been, and one with a large choice runs+the elimination state by state. `batch_size` and `sort_by_length` shape one batched pass, so they+do not apply on that path; `total_usage` still sums every call.Measured on the `english` checkpoint (CPU) with one test query per intent from CLINC150 (150intents, `random_state=0`), the question listing every intent and the instruction "Which intent does@@ -338,7 +346,7 @@ it answers `503` with `Retry-After: 1`.| `state` | 50,000 characters of the text the model is given -- the string itself for a string state, `json.dumps(state, ensure_ascii=False)` for an object or array || questions per request | 64 || `states` per batch request | 64 |-| options per `choice` question | 100; 255 on `/v1/systemone` with `LAYA_TOURNAMENT_GROUP` set |+| options per `choice` question | 100; 255 with `LAYA_TOURNAMENT_GROUP` set || levels per `score` question | 32 || options across all questions | 512 || concurrent admitted requests (`503`, not `413`) | `LAYA_MAX_CONCURRENT` (16) |diff --git a/laya/serve.py b/laya/serve.pyindex 12968bc7..c5eb2860 100644--- a/laya/serve.py+++ b/laya/serve.py@@ -55,8 +55,8 @@ env var meaning defaul`input_tokens` + `output_tokens```LAYA_TOURNAMENT_GROUP`` answer a choice with more options than this 0 (off)by elimination in groups of this size, on- ``/v1/systemone``; raises its option cap from- 100 to 255 (#302)+ both decision routes; raises their option+ cap from 100 to 255 (#302)========================= ============================================ =========``LAYA_DEVICE`` is a preference, not a guarantee: an ``Agent`` that asks for a@@ -76,7 +76,7 @@ import osimport reimport timefrom contextlib import asynccontextmanager-from typing import Any, Dict, Optional+from typing import Any, Dict, List, Optional# A failed inference is reported to the client as a fixed 500 so nothing about paths,# weights or memory state leaks, which leaves the server log as the only place the@@ -242,6 +242,39 @@ def _add_jev_confidence(result: Dict[str, Any]) -> Dict[str, Any]:return {**result, "answers": answers}+def _tournament_plan(questions: Any, group: int) -> Optional[Dict[str, List[str]]]:+ """The choices to narrow by elimination, as `{qid: labels}`, or None to answer in one pass.++ Every choice over `group` is narrowed. A choice with `option_order` keeps the one-pass path+ while one pass can still read it, since narrowing drops the order (#1062): but once any choice+ is over MAX_CHOICE_OPTIONS, which only the raised cap admits and one pass was never asked to+ read, the request is narrowed and an order on a narrowed choice is dropped as+ `predict_tournament` does. A label that is not a string cannot be mapped back onto the answer,+ so such a choice keeps one pass too, and is refused when one pass is not an option.+ """+ from .shortlist import _criteria_items+ if not isinstance(questions, dict):+ return None+ large: Dict[str, List[str]] = {}+ for qid, question in questions.items():+ if not (isinstance(question, dict) and question.get("type") == "choice"):+ continue+ criteria = question.get("criteria")+ if isinstance(criteria, (dict, list)) and len(criteria) > group:+ large[qid] = [label for label, _ in _criteria_items(criteria)]+ if not large:+ return None+ unmappable = [qid for qid, labels in large.items() if not all(isinstance(label, str) for label in labels)]+ ordered = [qid for qid in large if questions[qid].get("option_order") is not None]+ if any(len(labels) > MAX_CHOICE_OPTIONS for labels in large.values()):+ if unmappable:+ raise ValueError(+ "question %r: a choice is narrowed by elimination once a request has one over %d "+ "options, and that needs every label to be a string" % (unmappable[0], MAX_CHOICE_OPTIONS))+ return large+ return None if unmappable or ordered else large++def _predict_with_tournament(router: Any, state: Any, questions: Dict[str, Any], group: int,**predict_kwargs: Any) -> Dict[str, Any]:"""Answer `questions`, narrowing every choice over `group` options by elimination (#302).@@ -257,20 +290,12 @@ def _predict_with_tournament(router: Any, state: Any, questions: Dict[str, Any],the final round scored. `confidence` and `answer_confidence` are the final round's.`usage.input_tokens` is summed over every call; the other usage keys are the final call's.- A choice with `option_order`, or with a label that is not a string, keeps the one-pass path:- narrowing cannot carry an order over the full label set (#1062).+ Which choices are narrowed, and when an `option_order` is kept or dropped, is+ `_tournament_plan`'s; a dropped order is reported as `x_tournament.option_order_dropped`."""- from .shortlist import _criteria_items, predict_tournament- large = {}- for qid, question in questions.items():- if not (isinstance(question, dict) and question.get("type") == "choice"):- continue- criteria = question.get("criteria")- if isinstance(criteria, (dict, list)) and len(criteria) > group:- large[qid] = [label for label, _ in _criteria_items(criteria)]- if not large or any(questions[qid].get("option_order") is not None- or not all(isinstance(label, str) for label in labels)- for qid, labels in large.items()):+ from .shortlist import predict_tournament+ large = _tournament_plan(questions, group)+ if large is None:return router.predict(state, questions, **predict_kwargs)calls = []@@ -289,9 +314,12 @@ def _predict_with_tournament(router: Any, state: Any, questions: Dict[str, Any],if not isinstance(answer, dict) or not meta.get("rounds"):continuefinal = answer.get("probabilities") or {}+ report = {"finalists": list(meta["labels"]), "rounds": meta["rounds"]}+ if questions[qid].get("option_order") is not None:+ report["option_order_dropped"] = Trueanswers[qid] = {**answer,"probabilities": {label: final.get(label, 0.0) for label in labels},- "x_tournament": {"finalists": list(meta["labels"]), "rounds": meta["rounds"]}}+ "x_tournament": report}usage = dict(result.get("usage") or {})usage["input_tokens"] = sum((call.get("usage") or {}).get("input_tokens", 0) for call in calls)return {**result, "answers": answers, "usage": usage}@@ -908,7 +936,7 @@ def _batch_chunk_size(n_states: int, n_questions: int, max_len: Any = None) -> Oreturn max(1, min(n_states, rows_per_pass // n_questions))-def _check_batch_limits(states: Any, questions: Any) -> None:+def _check_batch_limits(states: Any, questions: Any, max_choice_options: int = MAX_CHOICE_OPTIONS) -> None:"""Validate batch states and questions before inference (400/413)."""from fastapi import HTTPException@@ -920,7 +948,7 @@ def _check_batch_limits(states: Any, questions: Any) -> None:detail="too many states in batch (%d > %d)" % (len(states), MAX_BATCH_STATES),)for state in states:- _check_request_limits(state, questions)+ _check_request_limits(state, questions, max_choice_options)# One surrogate code point in a Python string. Compiled once: a nested body can hold many# strings, and `re.search` scans each at C speed, where the per-character Python loop this# replaced cost ~170 ms on a near-cap body on the event loop.@@ -1359,7 +1387,8 @@ def create_app(router: Optional[Any] = None):)states = body["states"]questions = body["questions"]- _check_batch_limits(states, questions)+ _check_batch_limits(states, questions, MAX_TOURNAMENT_CHOICE_OPTIONS if tournament_group+ else MAX_CHOICE_OPTIONS)# Same refusal policy as the single endpoint: hooks and the two hook-execution knobs# belong to the deployment, not the caller. `predict_batch` does take `hooks_timeout`,# so without this call a batch body would silently hand a caller a shorter deadline for@@ -1437,7 +1466,14 @@ def create_app(router: Optional[Any] = None):loop = asyncio.get_running_loop()def _do_batch():- if hasattr(router, "predict_batch"):+ # A batch shares one question set, so whether it is narrowed is decided once.+ # Narrowing runs its rounds per state, so `batch_size` and `sort_by_length`,+ # which shape one batched pass, have nothing to apply to on that path.+ if tournament_group and _tournament_plan(questions, tournament_group) is not None:+ results = [_predict_with_tournament(router, s, questions, tournament_group,+ model=model, **predict_kwargs)+ for s in states]+ elif hasattr(router, "predict_batch"):reqs = [dict(state=s, questions=questions, model=model, **item_overrides)for s in states]results = router.predict_batch(reqs, **call_kwargs)diff --git a/nix/laya-serve.nix b/nix/laya-serve.nixindex 13fd8869..6835dfc9 100644--- a/nix/laya-serve.nix+++ b/nix/laya-serve.nix@@ -257,10 +257,11 @@ inexample = 16;description = ''Answer a `choice` with more options than this by elimination, in groups- of this size, on /v1/systemone (sets `LAYA_TOURNAMENT_GROUP`), and raise- its option cap from 100 to 255. Every option still comes back in- `probabilities`; one eliminated in an earlier round carries 0.0. null- leaves it off, so every choice is read in one pass as before.+ of this size, on /v1/systemone and /v1/systemone/batch (sets+ `LAYA_TOURNAMENT_GROUP`), and raise their option cap from 100 to 255.+ Every option still comes back in `probabilities`; one eliminated in an+ earlier round carries 0.0. null leaves it off, so every choice is read+ in one pass as before.'';};diff --git a/tests/test_serve.py b/tests/test_serve.pyindex 1d331330..a2294d87 100644--- a/tests/test_serve.py+++ b/tests/test_serve.py@@ -427,11 +427,96 @@ def test_option_order_keeps_the_one_pass_path(monkeypatch):assert "x_tournament" not in answer-def test_batch_keeps_the_one_pass_cap_with_a_tournament_group(monkeypatch):+def _large_batch(n, states=("I was charged twice.", "Please refund one of the charges."), **question):+ req = _large_req(n, **question)+ return {"states": list(states), "questions": req["questions"]}+++def test_batch_answers_a_large_choice_per_state(monkeypatch):+ client, fake = _tournament_client(monkeypatch, "16")+ r = client.post("/v1/systemone/batch", json=_large_batch(128))+ assert r.status_code == 200+ data = r.json()+ for item in data["results"]:+ answer = item["answers"]["queue"]+ assert list(answer["probabilities"]) == list(_queues(128))+ assert answer["choice"] == "Queue 077" and answer["x_tournament"]["rounds"] == 1+ assert item["usage"]["input_tokens"] == 90+ # each state ran its own round and final call, and the envelope sums every call+ assert [len(call) for call in fake.calls] == [8, 1, 8, 1]+ assert data["total_usage"] == {"input_tokens": 180, "output_tokens": 0}+++def test_batch_cap_follows_the_tournament_group(monkeypatch):+ client, _ = _tournament_client(monkeypatch, None)+ r = client.post("/v1/systemone/batch", json=_large_batch(128))+ assert r.status_code == 413 and "128 > 100" in r.json()["detail"]client, _ = _tournament_client(monkeypatch, "16")- req = _large_req(128)- r = client.post("/v1/systemone/batch", json={"states": [req["state"]], "questions": req["questions"]})- assert r.status_code == 413+ r = client.post("/v1/systemone/batch", json=_large_batch(256))+ assert r.status_code == 413 and "256 > 255" in r.json()["detail"]+++def test_batch_under_strict_keeps_only_the_contracted_keys(monkeypatch):+ client, _ = _tournament_client(monkeypatch, "16", strict=True)+ data = client.post("/v1/systemone/batch", json=_large_batch(128)).json()+ for item in data["results"]:+ answer = item["answers"]["queue"]+ assert set(answer) == {"type", "choice", "confidence", "probabilities"}+ assert len(answer["probabilities"]) == 128+++def test_batch_without_a_large_choice_keeps_the_batched_pass(monkeypatch):+ """A batch with nothing to narrow is the one `predict_batch` call it has always been."""++ class BatchingRouter(EliminationRouter):+ def __init__(self):+ super().__init__()+ self.batches = []++ def predict_batch(self, requests, **kwargs):+ self.batches.append(len(requests))+ return [self.predict(item["state"], item["questions"]) for item in requests]++ monkeypatch.setenv("LAYA_TOURNAMENT_GROUP", "16")+ monkeypatch.delenv("LAYA_API_KEY", raising=False)+ monkeypatch.delenv("LAYA_JEV_STRICT", raising=False)+ fake = BatchingRouter()+ client = TestClient(create_app(router=fake))+ data = client.post("/v1/systemone/batch", json=_large_batch(12)).json()+ assert fake.batches == [2]+ assert all("x_tournament" not in item["answers"]["queue"] for item in data["results"])+++def test_an_order_on_a_choice_over_the_one_pass_cap_is_dropped_and_reported(monkeypatch):+ """Over 100 options one pass was never an option, so the choice is narrowed and says so."""+ client, fake = _tournament_client(monkeypatch, "16")+ with pytest.warns(RuntimeWarning, match="option_order"):+ r = client.post("/v1/systemone", json=_large_req(150, option_order=list(range(150))[::-1]))+ assert r.status_code == 200+ report = r.json()["answers"]["queue"]["x_tournament"]+ assert report["option_order_dropped"] is True and report["rounds"] == 1+ assert len(fake.calls[0]) == 10+++def test_a_large_choice_narrows_an_ordered_choice_beside_it(monkeypatch):+ client, fake = _tournament_client(monkeypatch, "16")+ req = _large_req(150)+ req["questions"]["team"] = {"type": "choice", "instructions": "Which team?",+ "criteria": {"Team %02d" % i: None for i in range(32)},+ "option_order": list(range(32))[::-1]}+ with pytest.warns(RuntimeWarning, match="'team': 'option_order'"):+ answers = client.post("/v1/systemone", json=req).json()["answers"]+ assert answers["team"]["x_tournament"]["option_order_dropped"] is True+ assert "option_order_dropped" not in answers["queue"]["x_tournament"]+++def test_non_string_labels_over_the_one_pass_cap_are_refused(monkeypatch):+ client, fake = _tournament_client(monkeypatch, "16")+ req = {"state": "refund", "questions": {"code": {"type": "choice", "instructions": "Which code?",+ "criteria": list(range(150))}}}+ r = client.post("/v1/systemone", json=req)+ assert r.status_code == 422 and "string" in r.json()["detail"]+ assert fake.calls == []@pytest.mark.parametrize("raw, want", [(None, 0), ("", 0), ("abc", 0), ("0", 0), ("1", 0), ("-4", 0),