Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(serve): narrow large choices on /v1/systemone/batch too (#302)

kmitul38921c00df9e
diff --git a/docs/http-api.md b/docs/http-api.md
index 2cf9a5ac..999e4e44 100644
--- a/docs/http-api.md
+++ b/docs/http-api.md
@@ -43,7 +43,7 @@ Everything is environment variables, so one image serves a laptop dev run and a
| `LAYA_MAX_CONCURRENT` | requests admitted past auth at once; excess gets `503` | `16` |
| `LAYA_MAX_BATCH_TOKENS` | tokens one `/v1/systemone/batch` FORWARD PASS may collate (`states` x questions x row width); a larger batch is split into several passes, not refused | `131072` |
| `LAYA_JEV_STRICT` | serve the strict Jev wire contract: no root `routing`, no per-answer `action` / `answer_confidence`, no `confidence` on noul answers, and `usage` reduced to `input_tokens` + `output_tokens`. For clients that validate the response against the Jev contract with no extra fields | `0` |
-| `LAYA_TOURNAMENT_GROUP` | answer a `choice` with more options than this by elimination, in groups of this size, on `/v1/systemone`, and raise its option cap from 100 to 255. See [Large choices](#large-choices-laya_tournament_group). Unset, or outside 2 to 100, is off | `0` |
+| `LAYA_TOURNAMENT_GROUP` | answer a `choice` with more options than this by elimination, in groups of this size, on `/v1/systemone` and `/v1/systemone/batch`, and raise their option cap from 100 to 255. See [Large choices](#large-choices-laya_tournament_group). Unset, or outside 2 to 100, is off | `0` |
For a deployment published under a prefix such as `/laya`, set `LAYA_ROOT_PATH=/laya`.
FastAPI uses it when generating OpenAPI and Swagger UI URLs. Configure the reverse proxy to
@@ -294,10 +294,10 @@ Successful responses also carry `Server-Timing: inference;dur=<ms>` and `X-Infer
One pass reads every option of a `choice` inside one option budget, so past a few dozen options
they stop getting a token span each (`usage.options` reports when that happens), and past about 100
-the request is refused. Jev accepts up to 255. With `LAYA_TOURNAMENT_GROUP=16`, `/v1/systemone`
-answers a choice with more than 16 options the way `laya.predict_tournament` does: the options are
+the request is refused. Jev accepts up to 255. With `LAYA_TOURNAMENT_GROUP=16`, both decision routes
+answer a choice with more than 16 options the way `laya.predict_tournament` does: the options are
cut into groups of at most 16, each group is answered in one shared call, and the group winners go
-to a final call. Its option cap becomes 255. A choice of 16 or fewer, and every other question,
+to a final call. Their option cap becomes 255. A choice of 16 or fewer, and every other question,
takes the usual path unchanged. The group size is limited to 2 to 100, the one-pass cap, so every
choice the raised cap newly admits is always narrowed rather than read in one pass it cannot fit.
A larger group is not a better one: each group is still read in one pass, and one pass loses
@@ -314,9 +314,17 @@ The answer keeps the Jev shape, so the official SDK parses it:
- `confidence` and `answer_confidence` are the final round's, over the finalists;
- `usage.input_tokens` is summed over every call; the other `usage` keys are the final call's.
-A choice that sets `option_order`, or uses a non-string label, keeps the one-pass path, since an order
-over the full label set cannot be carried through a narrowing (#1062). `/v1/systemone/batch` is not
-covered and keeps the 100-option cap.
+Narrowing drops an `option_order`, since an order over the full label set no longer fits a group
+(#1062). So a choice that sets one keeps the one-pass path while one pass can read it. Once a
+request has a choice over 100 options, which only the raised cap admits, it is narrowed, and a
+narrowed choice that carried an order reports `x_tournament.option_order_dropped: true`. A label
+that is not a string cannot be mapped back onto the answer: such a choice keeps one pass, and a
+request that has to be narrowed is refused with a `422` naming it.
+
+On `/v1/systemone/batch` the question set is shared, so the decision is made once: a batch with
+nothing to narrow is the single batched pass it has always been, and one with a large choice runs
+the elimination state by state. `batch_size` and `sort_by_length` shape one batched pass, so they
+do not apply on that path; `total_usage` still sums every call.
Measured on the `english` checkpoint (CPU) with one test query per intent from CLINC150 (150
intents, `random_state=0`), the question listing every intent and the instruction "Which intent does
@@ -338,7 +346,7 @@ it answers `503` with `Retry-After: 1`.
| `state` | 50,000 characters of the text the model is given -- the string itself for a string state, `json.dumps(state, ensure_ascii=False)` for an object or array |
| questions per request | 64 |
| `states` per batch request | 64 |
-| options per `choice` question | 100; 255 on `/v1/systemone` with `LAYA_TOURNAMENT_GROUP` set |
+| options per `choice` question | 100; 255 with `LAYA_TOURNAMENT_GROUP` set |
| levels per `score` question | 32 |
| options across all questions | 512 |
| concurrent admitted requests (`503`, not `413`) | `LAYA_MAX_CONCURRENT` (16) |
diff --git a/laya/serve.py b/laya/serve.py
index 12968bc7..c5eb2860 100644
--- a/laya/serve.py
+++ b/laya/serve.py
@@ -55,8 +55,8 @@ env var meaning defaul
`input_tokens` + `output_tokens`
``LAYA_TOURNAMENT_GROUP`` answer a choice with more options than this 0 (off)
by elimination in groups of this size, on
- ``/v1/systemone``; raises its option cap from
- 100 to 255 (#302)
+ both decision routes; raises their option
+ cap from 100 to 255 (#302)
========================= ============================================ =========
``LAYA_DEVICE`` is a preference, not a guarantee: an ``Agent`` that asks for a
@@ -76,7 +76,7 @@ import os
import re
import time
from contextlib import asynccontextmanager
-from typing import Any, Dict, Optional
+from typing import Any, Dict, List, Optional
# A failed inference is reported to the client as a fixed 500 so nothing about paths,
# weights or memory state leaks, which leaves the server log as the only place the
@@ -242,6 +242,39 @@ def _add_jev_confidence(result: Dict[str, Any]) -> Dict[str, Any]:
return {**result, "answers": answers}
+def _tournament_plan(questions: Any, group: int) -> Optional[Dict[str, List[str]]]:
+ """The choices to narrow by elimination, as `{qid: labels}`, or None to answer in one pass.
+
+ Every choice over `group` is narrowed. A choice with `option_order` keeps the one-pass path
+ while one pass can still read it, since narrowing drops the order (#1062): but once any choice
+ is over MAX_CHOICE_OPTIONS, which only the raised cap admits and one pass was never asked to
+ read, the request is narrowed and an order on a narrowed choice is dropped as
+ `predict_tournament` does. A label that is not a string cannot be mapped back onto the answer,
+ so such a choice keeps one pass too, and is refused when one pass is not an option.
+ """
+ from .shortlist import _criteria_items
+ if not isinstance(questions, dict):
+ return None
+ large: Dict[str, List[str]] = {}
+ for qid, question in questions.items():
+ if not (isinstance(question, dict) and question.get("type") == "choice"):
+ continue
+ criteria = question.get("criteria")
+ if isinstance(criteria, (dict, list)) and len(criteria) > group:
+ large[qid] = [label for label, _ in _criteria_items(criteria)]
+ if not large:
+ return None
+ unmappable = [qid for qid, labels in large.items() if not all(isinstance(label, str) for label in labels)]
+ ordered = [qid for qid in large if questions[qid].get("option_order") is not None]
+ if any(len(labels) > MAX_CHOICE_OPTIONS for labels in large.values()):
+ if unmappable:
+ raise ValueError(
+ "question %r: a choice is narrowed by elimination once a request has one over %d "
+ "options, and that needs every label to be a string" % (unmappable[0], MAX_CHOICE_OPTIONS))
+ return large
+ return None if unmappable or ordered else large
+
+
def _predict_with_tournament(router: Any, state: Any, questions: Dict[str, Any], group: int,
**predict_kwargs: Any) -> Dict[str, Any]:
"""Answer `questions`, narrowing every choice over `group` options by elimination (#302).
@@ -257,20 +290,12 @@ def _predict_with_tournament(router: Any, state: Any, questions: Dict[str, Any],
the final round scored. `confidence` and `answer_confidence` are the final round's.
`usage.input_tokens` is summed over every call; the other usage keys are the final call's.
- A choice with `option_order`, or with a label that is not a string, keeps the one-pass path:
- narrowing cannot carry an order over the full label set (#1062).
+ Which choices are narrowed, and when an `option_order` is kept or dropped, is
+ `_tournament_plan`'s; a dropped order is reported as `x_tournament.option_order_dropped`.
"""
- from .shortlist import _criteria_items, predict_tournament
- large = {}
- for qid, question in questions.items():
- if not (isinstance(question, dict) and question.get("type") == "choice"):
- continue
- criteria = question.get("criteria")
- if isinstance(criteria, (dict, list)) and len(criteria) > group:
- large[qid] = [label for label, _ in _criteria_items(criteria)]
- if not large or any(questions[qid].get("option_order") is not None
- or not all(isinstance(label, str) for label in labels)
- for qid, labels in large.items()):
+ from .shortlist import predict_tournament
+ large = _tournament_plan(questions, group)
+ if large is None:
return router.predict(state, questions, **predict_kwargs)
calls = []
@@ -289,9 +314,12 @@ def _predict_with_tournament(router: Any, state: Any, questions: Dict[str, Any],
if not isinstance(answer, dict) or not meta.get("rounds"):
continue
final = answer.get("probabilities") or {}
+ report = {"finalists": list(meta["labels"]), "rounds": meta["rounds"]}
+ if questions[qid].get("option_order") is not None:
+ report["option_order_dropped"] = True
answers[qid] = {**answer,
"probabilities": {label: final.get(label, 0.0) for label in labels},
- "x_tournament": {"finalists": list(meta["labels"]), "rounds": meta["rounds"]}}
+ "x_tournament": report}
usage = dict(result.get("usage") or {})
usage["input_tokens"] = sum((call.get("usage") or {}).get("input_tokens", 0) for call in calls)
return {**result, "answers": answers, "usage": usage}
@@ -908,7 +936,7 @@ def _batch_chunk_size(n_states: int, n_questions: int, max_len: Any = None) -> O
return max(1, min(n_states, rows_per_pass // n_questions))
-def _check_batch_limits(states: Any, questions: Any) -> None:
+def _check_batch_limits(states: Any, questions: Any, max_choice_options: int = MAX_CHOICE_OPTIONS) -> None:
"""Validate batch states and questions before inference (400/413)."""
from fastapi import HTTPException
@@ -920,7 +948,7 @@ def _check_batch_limits(states: Any, questions: Any) -> None:
detail="too many states in batch (%d > %d)" % (len(states), MAX_BATCH_STATES),
)
for state in states:
- _check_request_limits(state, questions)
+ _check_request_limits(state, questions, max_choice_options)
# One surrogate code point in a Python string. Compiled once: a nested body can hold many
# strings, and `re.search` scans each at C speed, where the per-character Python loop this
# replaced cost ~170 ms on a near-cap body on the event loop.
@@ -1359,7 +1387,8 @@ def create_app(router: Optional[Any] = None):
)
states = body["states"]
questions = body["questions"]
- _check_batch_limits(states, questions)
+ _check_batch_limits(states, questions, MAX_TOURNAMENT_CHOICE_OPTIONS if tournament_group
+ else MAX_CHOICE_OPTIONS)
# Same refusal policy as the single endpoint: hooks and the two hook-execution knobs
# belong to the deployment, not the caller. `predict_batch` does take `hooks_timeout`,
# so without this call a batch body would silently hand a caller a shorter deadline for
@@ -1437,7 +1466,14 @@ def create_app(router: Optional[Any] = None):
loop = asyncio.get_running_loop()
def _do_batch():
- if hasattr(router, "predict_batch"):
+ # A batch shares one question set, so whether it is narrowed is decided once.
+ # Narrowing runs its rounds per state, so `batch_size` and `sort_by_length`,
+ # which shape one batched pass, have nothing to apply to on that path.
+ if tournament_group and _tournament_plan(questions, tournament_group) is not None:
+ results = [_predict_with_tournament(router, s, questions, tournament_group,
+ model=model, **predict_kwargs)
+ for s in states]
+ elif hasattr(router, "predict_batch"):
reqs = [dict(state=s, questions=questions, model=model, **item_overrides)
for s in states]
results = router.predict_batch(reqs, **call_kwargs)
diff --git a/nix/laya-serve.nix b/nix/laya-serve.nix
index 13fd8869..6835dfc9 100644
--- a/nix/laya-serve.nix
+++ b/nix/laya-serve.nix
@@ -257,10 +257,11 @@ in
example = 16;
description = ''
Answer a `choice` with more options than this by elimination, in groups
- of this size, on /v1/systemone (sets `LAYA_TOURNAMENT_GROUP`), and raise
- its option cap from 100 to 255. Every option still comes back in
- `probabilities`; one eliminated in an earlier round carries 0.0. null
- leaves it off, so every choice is read in one pass as before.
+ of this size, on /v1/systemone and /v1/systemone/batch (sets
+ `LAYA_TOURNAMENT_GROUP`), and raise their option cap from 100 to 255.
+ Every option still comes back in `probabilities`; one eliminated in an
+ earlier round carries 0.0. null leaves it off, so every choice is read
+ in one pass as before.
'';
};
diff --git a/tests/test_serve.py b/tests/test_serve.py
index 1d331330..a2294d87 100644
--- a/tests/test_serve.py
+++ b/tests/test_serve.py
@@ -427,11 +427,96 @@ def test_option_order_keeps_the_one_pass_path(monkeypatch):
assert "x_tournament" not in answer
-def test_batch_keeps_the_one_pass_cap_with_a_tournament_group(monkeypatch):
+def _large_batch(n, states=("I was charged twice.", "Please refund one of the charges."), **question):
+ req = _large_req(n, **question)
+ return {"states": list(states), "questions": req["questions"]}
+
+
+def test_batch_answers_a_large_choice_per_state(monkeypatch):
+ client, fake = _tournament_client(monkeypatch, "16")
+ r = client.post("/v1/systemone/batch", json=_large_batch(128))
+ assert r.status_code == 200
+ data = r.json()
+ for item in data["results"]:
+ answer = item["answers"]["queue"]
+ assert list(answer["probabilities"]) == list(_queues(128))
+ assert answer["choice"] == "Queue 077" and answer["x_tournament"]["rounds"] == 1
+ assert item["usage"]["input_tokens"] == 90
+ # each state ran its own round and final call, and the envelope sums every call
+ assert [len(call) for call in fake.calls] == [8, 1, 8, 1]
+ assert data["total_usage"] == {"input_tokens": 180, "output_tokens": 0}
+
+
+def test_batch_cap_follows_the_tournament_group(monkeypatch):
+ client, _ = _tournament_client(monkeypatch, None)
+ r = client.post("/v1/systemone/batch", json=_large_batch(128))
+ assert r.status_code == 413 and "128 > 100" in r.json()["detail"]
client, _ = _tournament_client(monkeypatch, "16")
- req = _large_req(128)
- r = client.post("/v1/systemone/batch", json={"states": [req["state"]], "questions": req["questions"]})
- assert r.status_code == 413
+ r = client.post("/v1/systemone/batch", json=_large_batch(256))
+ assert r.status_code == 413 and "256 > 255" in r.json()["detail"]
+
+
+def test_batch_under_strict_keeps_only_the_contracted_keys(monkeypatch):
+ client, _ = _tournament_client(monkeypatch, "16", strict=True)
+ data = client.post("/v1/systemone/batch", json=_large_batch(128)).json()
+ for item in data["results"]:
+ answer = item["answers"]["queue"]
+ assert set(answer) == {"type", "choice", "confidence", "probabilities"}
+ assert len(answer["probabilities"]) == 128
+
+
+def test_batch_without_a_large_choice_keeps_the_batched_pass(monkeypatch):
+ """A batch with nothing to narrow is the one `predict_batch` call it has always been."""
+
+ class BatchingRouter(EliminationRouter):
+ def __init__(self):
+ super().__init__()
+ self.batches = []
+
+ def predict_batch(self, requests, **kwargs):
+ self.batches.append(len(requests))
+ return [self.predict(item["state"], item["questions"]) for item in requests]
+
+ monkeypatch.setenv("LAYA_TOURNAMENT_GROUP", "16")
+ monkeypatch.delenv("LAYA_API_KEY", raising=False)
+ monkeypatch.delenv("LAYA_JEV_STRICT", raising=False)
+ fake = BatchingRouter()
+ client = TestClient(create_app(router=fake))
+ data = client.post("/v1/systemone/batch", json=_large_batch(12)).json()
+ assert fake.batches == [2]
+ assert all("x_tournament" not in item["answers"]["queue"] for item in data["results"])
+
+
+def test_an_order_on_a_choice_over_the_one_pass_cap_is_dropped_and_reported(monkeypatch):
+ """Over 100 options one pass was never an option, so the choice is narrowed and says so."""
+ client, fake = _tournament_client(monkeypatch, "16")
+ with pytest.warns(RuntimeWarning, match="option_order"):
+ r = client.post("/v1/systemone", json=_large_req(150, option_order=list(range(150))[::-1]))
+ assert r.status_code == 200
+ report = r.json()["answers"]["queue"]["x_tournament"]
+ assert report["option_order_dropped"] is True and report["rounds"] == 1
+ assert len(fake.calls[0]) == 10
+
+
+def test_a_large_choice_narrows_an_ordered_choice_beside_it(monkeypatch):
+ client, fake = _tournament_client(monkeypatch, "16")
+ req = _large_req(150)
+ req["questions"]["team"] = {"type": "choice", "instructions": "Which team?",
+ "criteria": {"Team %02d" % i: None for i in range(32)},
+ "option_order": list(range(32))[::-1]}
+ with pytest.warns(RuntimeWarning, match="'team': 'option_order'"):
+ answers = client.post("/v1/systemone", json=req).json()["answers"]
+ assert answers["team"]["x_tournament"]["option_order_dropped"] is True
+ assert "option_order_dropped" not in answers["queue"]["x_tournament"]
+
+
+def test_non_string_labels_over_the_one_pass_cap_are_refused(monkeypatch):
+ client, fake = _tournament_client(monkeypatch, "16")
+ req = {"state": "refund", "questions": {"code": {"type": "choice", "instructions": "Which code?",
+ "criteria": list(range(150))}}}
+ r = client.post("/v1/systemone", json=req)
+ assert r.status_code == 422 and "string" in r.json()["detail"]
+ assert fake.calls == []
@pytest.mark.parametrize("raw, want", [(None, 0), ("", 0), ("abc", 0), ("0", 0), ("1", 0), ("-4", 0),