Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(serve): answer large choices by elimination behind LAYA_TOURNAMENT_GROUP (#302)

kmitulc3e631bcd936
diff --git a/docs/http-api.md b/docs/http-api.md
index a93644ab..18a8a88e 100644
--- a/docs/http-api.md
+++ b/docs/http-api.md
@@ -43,6 +43,7 @@ Everything is environment variables, so one image serves a laptop dev run and a
| `LAYA_MAX_CONCURRENT` | requests admitted past auth at once; excess gets `503` | `16` |
| `LAYA_MAX_BATCH_TOKENS` | tokens one `/v1/systemone/batch` FORWARD PASS may collate (`states` x questions x row width); a larger batch is split into several passes, not refused | `131072` |
| `LAYA_JEV_STRICT` | serve the strict Jev wire contract: no root `routing`, no per-answer `action` / `answer_confidence`, no `confidence` on noul answers, and `usage` reduced to `input_tokens` + `output_tokens`. For clients that validate the response against the Jev contract with no extra fields | `0` |
+| `LAYA_TOURNAMENT_GROUP` | answer a `choice` with more options than this by elimination, in groups of this size, on `/v1/systemone`, and raise its option cap from 100 to 255. See [Large choices](#large-choices-laya_tournament_group). Unset, or outside 2 to 255, is off | `0` |
For a deployment published under a prefix such as `/laya`, set `LAYA_ROOT_PATH=/laya`.
FastAPI uses it when generating OpenAPI and Swagger UI URLs. Configure the reverse proxy to
@@ -285,6 +286,37 @@ stringifies its own criteria may not match byte for byte.
Successful responses also carry `Server-Timing: inference;dur=<ms>` and `X-Inference-Time-Ms`.
+### Large choices: `LAYA_TOURNAMENT_GROUP`
+
+One pass reads every option of a `choice` inside one option budget, so past a few dozen options
+they stop getting a token span each (`usage.options` reports when that happens), and past about 100
+the request is refused. Jev accepts up to 255. With `LAYA_TOURNAMENT_GROUP=16`, `/v1/systemone`
+answers a choice with more than 16 options the way `laya.predict_tournament` does: the options are
+cut into groups of at most 16, each group is answered in one shared call, and the group winners go
+to a final call. Its option cap becomes 255. A choice of 16 or fewer, and every other question,
+takes the usual path unchanged.
+
+The answer keeps the Jev shape, so the official SDK parses it:
+
+- `probabilities` names every option, in criteria order. The finalists carry the final round's
+ probabilities and an option eliminated in an earlier round carries `0.0`, so `choice` is still
+ the argmax and the values still sum to 1;
+- `x_tournament` lists the `finalists` and the number of `rounds`, so a caller can tell an
+ eliminated option from one the final round scored. Under `LAYA_JEV_STRICT` it is not sent;
+- `confidence` and `answer_confidence` are the final round's, over the finalists;
+- `usage.input_tokens` is summed over every call; the other `usage` keys are the final call's.
+
+A choice that sets `option_order`, or uses a non-string label, keeps the one-pass path, since an order
+over the full label set cannot be carried through a narrowing (#1062). `/v1/systemone/batch` is not
+covered and keeps the 100-option cap.
+
+Measured on the `english` checkpoint (CPU) with one test query per intent from CLINC150 (150
+intents, `random_state=0`), the question listing every intent and the instruction "Which intent does
+this request express?": with the flag off the request is refused (`150 > 100`); with
+`LAYA_TOURNAMENT_GROUP=16` it is answered at 86.7% accuracy (130/150), median 0.77 s per request. On
+50-option subsets of the same queries, the tournament scored 94.0% against 82.7% for one pass. That
+is one dataset and one sample; measure on your own labels before relying on it.
+
## Limits
Request guardrails are checked before tokenization, so an oversized request costs the server
@@ -298,7 +330,7 @@ it answers `503` with `Retry-After: 1`.
| `state` | 50,000 characters of the text the model is given -- the string itself for a string state, `json.dumps(state, ensure_ascii=False)` for an object or array |
| questions per request | 64 |
| `states` per batch request | 64 |
-| options per `choice` question | 100 |
+| options per `choice` question | 100; 255 on `/v1/systemone` with `LAYA_TOURNAMENT_GROUP` set |
| levels per `score` question | 32 |
| options across all questions | 512 |
| concurrent admitted requests (`503`, not `413`) | `LAYA_MAX_CONCURRENT` (16) |
diff --git a/laya/serve.py b/laya/serve.py
index fd3da64d..549689a8 100644
--- a/laya/serve.py
+++ b/laya/serve.py
@@ -53,6 +53,10 @@ env var meaning defaul
`answer_confidence`, no `confidence` on noul
answers, and `usage` reduced to
`input_tokens` + `output_tokens`
+``LAYA_TOURNAMENT_GROUP`` answer a choice with more options than this 0 (off)
+ by elimination in groups of this size, on
+ ``/v1/systemone``; raises its option cap from
+ 100 to 255 (#302)
========================= ============================================ =========
``LAYA_DEVICE`` is a preference, not a guarantee: an ``Agent`` that asks for a
@@ -120,6 +124,10 @@ DEFAULT_MAX_BATCH_TOKENS = 131072
_BATCH_ROW_TOKENS_ASSUMED = 512
# HTTP-only amplification guard; the library keeps its head_max_len-aware budget.
MAX_CHOICE_OPTIONS = 100
+# With `LAYA_TOURNAMENT_GROUP` set, a choice is narrowed in groups rather than read in one pass, so
+# its cap is Jev's documented one instead (#302). The tournament reads every label inside a group's
+# option budget, so the one-pass budget that sets MAX_CHOICE_OPTIONS no longer binds.
+MAX_TOURNAMENT_CHOICE_OPTIONS = 255
MAX_SCORE_LEVELS = 32
MAX_TOTAL_OPTIONS = 512
@@ -226,6 +234,61 @@ def _add_jev_confidence(result: Dict[str, Any]) -> Dict[str, Any]:
return {**result, "answers": answers}
+def _predict_with_tournament(router: Any, state: Any, questions: Dict[str, Any], group: int,
+ **predict_kwargs: Any) -> Dict[str, Any]:
+ """Answer `questions`, narrowing every choice over `group` options by elimination (#302).
+
+ One pass reads every option inside a single option budget, and past a few dozen options they
+ no longer each get a token span of their own. `predict_tournament` reads them in groups of
+ `group` instead. When no choice is over `group`, this is exactly `router.predict`.
+
+ The answer keeps the Jev shape: `probabilities` names every option in criteria order, the
+ finalists carry the final round's probabilities, and an option eliminated in an earlier round
+ carries 0.0, so `choice` is still the argmax and the values still sum to 1. `x_tournament`
+ records the finalists and the round count, so a caller can tell an eliminated option from one
+ the final round scored. `confidence` and `answer_confidence` are the final round's.
+ `usage.input_tokens` is summed over every call; the other usage keys are the final call's.
+
+ A choice with `option_order`, or with a label that is not a string, keeps the one-pass path:
+ narrowing cannot carry an order over the full label set (#1062).
+ """
+ from .shortlist import _criteria_items, predict_tournament
+ large = {}
+ for qid, question in questions.items():
+ if not (isinstance(question, dict) and question.get("type") == "choice"):
+ continue
+ criteria = question.get("criteria")
+ if isinstance(criteria, (dict, list)) and len(criteria) > group:
+ large[qid] = [label for label, _ in _criteria_items(criteria)]
+ if not large or any(questions[qid].get("option_order") is not None
+ or not all(isinstance(label, str) for label in labels)
+ for qid, labels in large.items()):
+ return router.predict(state, questions, **predict_kwargs)
+
+ calls = []
+
+ class _Recorded:
+ def predict(self, *args, **kwargs):
+ result = router.predict(*args, **kwargs)
+ calls.append(result)
+ return result
+
+ result = dict(predict_tournament(_Recorded(), state, questions, group_size=group, **predict_kwargs))
+ rounds = result.pop("tournament", {})
+ answers = dict(result.get("answers") or {})
+ for qid, labels in large.items():
+ answer, meta = answers.get(qid), rounds.get(qid) or {}
+ if not isinstance(answer, dict) or not meta.get("rounds"):
+ continue
+ final = answer.get("probabilities") or {}
+ answers[qid] = {**answer,
+ "probabilities": {label: final.get(label, 0.0) for label in labels},
+ "x_tournament": {"finalists": list(meta["labels"]), "rounds": meta["rounds"]}}
+ usage = dict(result.get("usage") or {})
+ usage["input_tokens"] = sum((call.get("usage") or {}).get("input_tokens", 0) for call in calls)
+ return {**result, "answers": answers, "usage": usage}
+
+
def _published_model_ids() -> Dict[str, str]:
"""Public Hugging Face ids accepted so a client can name a checkpoint.
@@ -312,6 +375,23 @@ def _resolve_max_concurrent() -> int:
return n if n > 0 else DEFAULT_MAX_CONCURRENT
+def _resolve_tournament_group() -> int:
+ """Group size for choices answered by elimination, from LAYA_TOURNAMENT_GROUP; 0 is off (#302).
+
+ A choice with more options than this goes through `predict_tournament` in groups of this size.
+ Anything unset, unparseable, below 2 or above MAX_TOURNAMENT_CHOICE_OPTIONS leaves it off, so
+ the one-pass path is the default: no choice the cap admits has more options than such a group.
+ """
+ raw = os.environ.get("LAYA_TOURNAMENT_GROUP")
+ if not raw:
+ return 0
+ try:
+ n = int(raw)
+ except ValueError:
+ return 0
+ return n if 2 <= n <= MAX_TOURNAMENT_CHOICE_OPTIONS else 0
+
+
def _resolve_idle_unload_seconds() -> float:
"""Idle window from LAYA_IDLE_UNLOAD_SECONDS; zero disables unloading."""
raw = os.environ.get("LAYA_IDLE_UNLOAD_SECONDS", "").strip()
@@ -715,7 +795,7 @@ def _state_length(state: Any) -> int:
raise HTTPException(status_code=400, detail="'state' must be JSON-serializable")
-def _check_request_limits(state: Any, questions: Any) -> None:
+def _check_request_limits(state: Any, questions: Any, max_choice_options: int = MAX_CHOICE_OPTIONS) -> None:
"""Reject absent or oversized inference requests before tokenization (400/413)."""
from fastapi import HTTPException
@@ -742,10 +822,10 @@ def _check_request_limits(state: Any, questions: Any) -> None:
if qtype == "choice" and isinstance(crit, (dict, list)):
count = len(crit)
total_options += count
- if count > MAX_CHOICE_OPTIONS:
+ if count > max_choice_options:
raise HTTPException(
status_code=413,
- detail="too many choice options for %r (%d > %d)" % (qid, count, MAX_CHOICE_OPTIONS),
+ detail="too many choice options for %r (%d > %d)" % (qid, count, max_choice_options),
)
elif qtype == "score" and isinstance(crit, list):
count = len(crit)
@@ -981,6 +1061,7 @@ def create_app(router: Optional[Any] = None):
admission: Optional[asyncio.Semaphore] = None
idle_unload_seconds = _resolve_idle_unload_seconds()
+ tournament_group = _resolve_tournament_group()
last_request = time.monotonic()
def _mark_request():
@@ -1148,7 +1229,8 @@ def create_app(router: Optional[Any] = None):
raise HTTPException(status_code=400, detail="request body must be an object with a 'questions' field")
state = body.get("state")
questions = body["questions"]
- _check_request_limits(state, questions)
+ _check_request_limits(state, questions, MAX_TOURNAMENT_CHOICE_OPTIONS if tournament_group
+ else MAX_CHOICE_OPTIONS)
_refuse_body_refusals(body)
# After the size checks, so an oversized body is refused before anything walks it, and
# `MAX_STATE_CHARS`/`MAX_QUESTIONS` bound what the walk can reach. A `\udXXX` escape with
@@ -1194,8 +1276,12 @@ def create_app(router: Optional[Any] = None):
async with gate:
loop = asyncio.get_running_loop()
t0 = time.perf_counter()
- result = await loop.run_in_executor(
- pool, _run_inference, partial(router.predict, state, questions, model=model, **predict_kwargs))
+ if tournament_group:
+ run = partial(_predict_with_tournament, router, state, questions, tournament_group,
+ model=model, **predict_kwargs)
+ else:
+ run = partial(router.predict, state, questions, model=model, **predict_kwargs)
+ result = await loop.run_in_executor(pool, _run_inference, run)
infer_ms = (time.perf_counter() - t0) * 1000.0
if _env_bool("LAYA_JEV_STRICT", False):
result = _project_jev_strict(result)
diff --git a/nix/laya-serve.nix b/nix/laya-serve.nix
index 3ae5b63b..1efc4ad8 100644
--- a/nix/laya-serve.nix
+++ b/nix/laya-serve.nix
@@ -251,6 +251,19 @@ in
'';
};
+ tournamentGroup = lib.mkOption {
+ type = lib.types.nullOr (lib.types.ints.between 2 255);
+ default = null;
+ example = 16;
+ description = ''
+ Answer a `choice` with more options than this by elimination, in groups
+ of this size, on /v1/systemone (sets `LAYA_TOURNAMENT_GROUP`), and raise
+ its option cap from 100 to 255. Every option still comes back in
+ `probabilities`; one eliminated in an earlier round carries 0.0. null
+ leaves it off, so every choice is read in one pass as before.
+ '';
+ };
+
revision = lib.mkOption {
type = lib.types.nullOr (lib.types.strMatching "[A-Za-z0-9._/-]+");
default = null;
@@ -379,6 +392,8 @@ in
LAYA_MAX_TOKEN_BUDGET = toString cfg.maxTokenBudget;
} // lib.optionalAttrs (cfg.maxBatchTokens != null) {
LAYA_MAX_BATCH_TOKENS = toString cfg.maxBatchTokens;
+ } // lib.optionalAttrs (cfg.tournamentGroup != null) {
+ LAYA_TOURNAMENT_GROUP = toString cfg.tournamentGroup;
} // lib.optionalAttrs (cfg.revision != null) {
LAYA_REVISION = cfg.revision;
} // lib.optionalAttrs (cfg.extraModels != null) {
diff --git a/tests/test_packaging.py b/tests/test_packaging.py
index 7199fca4..a9cba7e2 100644
--- a/tests/test_packaging.py
+++ b/tests/test_packaging.py
@@ -594,7 +594,8 @@ def option_type(opt):
# Every new knob is opt-in: unset means the unit exports nothing and the runtime's own default
# applies, so a host that ignores them gets today's behaviour byte for byte.
for opt in ("rootPath", "logLevel", "maxConcurrent", "cudaAmp", "cpuAmp", "mpsAmpMinRows",
- "maxLoaded", "maxTokenBudget", "revision", "defaultModel", "idleUnloadSeconds"):
+ "maxLoaded", "maxTokenBudget", "revision", "defaultModel", "idleUnloadSeconds",
+ "tournamentGroup"):
_t = option_text(opt)
check_true("nix/module declares %s" % opt, _t != "", "option not found")
check_true("nix/%s is opt-in (nullOr, default null)" % opt,
diff --git a/tests/test_serve.py b/tests/test_serve.py
index 0850cd66..4b2aec90 100644
--- a/tests/test_serve.py
+++ b/tests/test_serve.py
@@ -38,6 +38,7 @@ from laya.serve import ( # noqa: E402
_resolve_idle_unload_seconds,
_resolve_max_loaded,
_resolve_model,
+ _resolve_tournament_group,
create_app,
)
@@ -288,6 +289,141 @@ def test_jev_strict_leaves_out_jev_confidence(monkeypatch):
assert all("x_jev_confidence" not in answer for answer in answers.values())
+# --- LAYA_TOURNAMENT_GROUP: large choices answered by elimination (#302) ----------------------
+
+def _queues(n):
+ return {"Queue %03d" % i: "Tickets routed to queue %d" % i for i in range(n)}
+
+
+def _large_req(n, **question):
+ return {"state": "I was charged twice, please refund one.",
+ "questions": {"queue": {"type": "choice", "instructions": "Which queue?",
+ "criteria": _queues(n), **question}}}
+
+
+class EliminationRouter:
+ """Answers every choice it is asked: `Queue 077` wins when offered, else the first label.
+
+ Records each call's question set and reports 10 input tokens per question, so a test can count
+ the calls a tournament made and check that their usage was summed.
+ """
+
+ loaded = ["english"]
+
+ def __init__(self):
+ self.calls = []
+
+ def predict(self, state, questions, model=None, **kwargs):
+ self.calls.append(questions)
+ answers = {}
+ for qid, q in questions.items():
+ labels = list(q["criteria"])
+ winner = "Queue 077" if "Queue 077" in labels else labels[0]
+ rest = (0.3 / (len(labels) - 1)) if len(labels) > 1 else 0.0
+ probabilities = {label: (0.7 if label == winner else rest) for label in labels}
+ if len(labels) == 1:
+ probabilities[winner] = 1.0
+ answers[qid] = {"type": "choice", "choice": winner, "probabilities": probabilities,
+ "confidence": 0.5, "answer_confidence": probabilities[winner]}
+ return {"model": "laya-rl-agent", "answers": answers,
+ "usage": {"input_tokens": 10 * len(questions), "output_tokens": 0},
+ "routing": {"model": "english", "reason": "English Latin text"}}
+
+
+def _tournament_client(monkeypatch, group, strict=False):
+ monkeypatch.delenv("LAYA_API_KEY", raising=False)
+ if group is None:
+ monkeypatch.delenv("LAYA_TOURNAMENT_GROUP", raising=False)
+ else:
+ monkeypatch.setenv("LAYA_TOURNAMENT_GROUP", group)
+ if strict:
+ monkeypatch.setenv("LAYA_JEV_STRICT", "1")
+ else:
+ monkeypatch.delenv("LAYA_JEV_STRICT", raising=False)
+ fake = EliminationRouter()
+ return TestClient(create_app(router=fake)), fake
+
+
+def test_large_choice_is_refused_without_a_tournament(monkeypatch):
+ client, fake = _tournament_client(monkeypatch, None)
+ r = client.post("/v1/systemone", json=_large_req(128))
+ assert r.status_code == 413
+ assert "128 > 100" in r.json()["detail"]
+ assert fake.calls == []
+
+
+def test_tournament_answers_a_large_choice_in_the_jev_shape(monkeypatch):
+ client, fake = _tournament_client(monkeypatch, "16")
+ r = client.post("/v1/systemone", json=_large_req(128))
+ assert r.status_code == 200
+ body = r.json()
+ answer = body["answers"]["queue"]
+ # every option is named, in criteria order, and the values are still a distribution
+ assert list(answer["probabilities"]) == list(_queues(128))
+ assert abs(sum(answer["probabilities"].values()) - 1.0) < 1e-9
+ # the choice is still the argmax; eliminated options carry 0.0, finalists the final round's values
+ assert answer["choice"] == "Queue 077"
+ assert max(answer["probabilities"], key=answer["probabilities"].get) == "Queue 077"
+ assert answer["x_tournament"]["rounds"] == 1
+ finalists = answer["x_tournament"]["finalists"]
+ assert len(finalists) == 8 and "Queue 077" in finalists
+ assert all(p == 0.0 for label, p in answer["probabilities"].items() if label not in finalists)
+ assert "tournament" not in body
+ # one round of 8 group questions, then the final call; usage is summed over both
+ assert [len(call) for call in fake.calls] == [8, 1]
+ assert body["usage"]["input_tokens"] == 90
+
+
+def test_tournament_accepts_up_to_255_options(monkeypatch):
+ client, _ = _tournament_client(monkeypatch, "16")
+ assert client.post("/v1/systemone", json=_large_req(255)).status_code == 200
+ r = client.post("/v1/systemone", json=_large_req(256))
+ assert r.status_code == 413
+ assert "256 > 255" in r.json()["detail"]
+
+
+def test_tournament_under_strict_keeps_only_the_contracted_keys(monkeypatch):
+ client, _ = _tournament_client(monkeypatch, "16", strict=True)
+ answer = client.post("/v1/systemone", json=_large_req(128)).json()["answers"]["queue"]
+ assert set(answer) == {"type", "choice", "confidence", "probabilities"}
+ assert len(answer["probabilities"]) == 128
+
+
+def test_small_choice_skips_the_tournament(monkeypatch):
+ client, fake = _tournament_client(monkeypatch, "16")
+ req = _large_req(16)
+ answer = client.post("/v1/systemone", json=req).json()["answers"]["queue"]
+ assert fake.calls == [req["questions"]]
+ assert "x_tournament" not in answer
+
+
+def test_option_order_keeps_the_one_pass_path(monkeypatch):
+ """Narrowing cannot carry an order over the full label set (#1062), so it is not attempted."""
+ client, fake = _tournament_client(monkeypatch, "16")
+ req = _large_req(32, option_order=list(range(32))[::-1])
+ answer = client.post("/v1/systemone", json=req).json()["answers"]["queue"]
+ assert len(fake.calls) == 1 and len(fake.calls[0]["queue"]["criteria"]) == 32
+ assert "x_tournament" not in answer
+
+
+def test_batch_keeps_the_one_pass_cap_with_a_tournament_group(monkeypatch):
+ client, _ = _tournament_client(monkeypatch, "16")
+ req = _large_req(128)
+ r = client.post("/v1/systemone/batch", json={"states": [req["state"]], "questions": req["questions"]})
+ assert r.status_code == 413
+
+
[email protected]("raw, want", [(None, 0), ("", 0), ("abc", 0), ("0", 0), ("1", 0), ("-4", 0),
+ ("2", 2), ("16", 16), ("255", 255),
+ ("256", 0)])
+def test_tournament_group_resolution(monkeypatch, raw, want):
+ if raw is None:
+ monkeypatch.delenv("LAYA_TOURNAMENT_GROUP", raising=False)
+ else:
+ monkeypatch.setenv("LAYA_TOURNAMENT_GROUP", raw)
+ assert _resolve_tournament_group() == want
+
+
def test_known_model_is_honoured(monkeypatch):
client, fake = _client(monkeypatch)
client.post("/v1/systemone", json={**REQ, "model": "multilingual"})