Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(serve): split an oversized batch across forward passes instead of collating it whole
Adeshina Lasisidf3e2854ac32
diff --git a/README.md b/README.mdindex af65dae7..98cee5d3 100644--- a/README.md+++ b/README.md@@ -638,7 +638,7 @@ curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{}'```-To evaluate multiple states in a single call, send a `states` array to `POST /v1/systemone/batch` (capped at 64 states):+To evaluate multiple states in a single call, send a `states` array to `POST /v1/systemone/batch` (capped at 64 states; a batch larger than `LAYA_MAX_BATCH_TOKENS` is split across several forward passes rather than refused):```bashcurl -s localhost:8000/v1/systemone/batch -H 'content-type: application/json' -d '{diff --git a/docs/http-api.md b/docs/http-api.mdindex 6ab7bdb9..03914a4d 100644--- a/docs/http-api.md+++ b/docs/http-api.md@@ -37,6 +37,7 @@ Everything is environment variables, so one image serves a laptop dev run and a| `LAYA_API_KEY` | if set, require `Authorization: Bearer <key>` | none || `LAYA_LOG_LEVEL` | uvicorn log level | `info` || `LAYA_MAX_CONCURRENT` | requests admitted past auth at once; excess gets `503` | `16` |+| `LAYA_MAX_BATCH_TOKENS` | tokens one `/v1/systemone/batch` FORWARD PASS may collate (`states` x questions x row width); a larger batch is split into several passes, not refused | `131072` || `LAYA_JEV_STRICT` | serve the strict Jev wire contract: no root `routing`, no per-answer `action` / `answer_confidence`, no `confidence` on noul answers, and `usage` reduced to `input_tokens` + `output_tokens`. For clients that validate the response against the Jev contract with no extra fields | `0` |For a deployment published under a prefix such as `/laya`, set `LAYA_ROOT_PATH=/laya`.@@ -268,11 +269,30 @@ nothing but the bytes it read. Every one of them is a `413`; the `detail` says w| request body | 2 MiB, enforced while streaming -- a chunked or understated `Content-Length` cannot bypass it || `state` | 50,000 characters of the text the model is given -- the string itself for a string state, `json.dumps(state, ensure_ascii=False)` for an object or array || questions per request | 64 |+| `states` per batch request | 64 || options per `choice` question | 100 || levels per `score` question | 32 || options across all questions | 512 || concurrent admitted requests | `LAYA_MAX_CONCURRENT` (16) |+`/v1/systemone/batch` is bounded differently, and not by a refusal. It tokenizes each state once per+question and collates every row into a single tensor, so the field caps multiply: 64 states of 64+questions is 4096 rows, which every other limit on this page permits. What a row costs is its width,+and `max_len` is itself a request field, so the cost of a batch is `states x questions x width`.++Rather than refuse a large batch, the endpoint **splits** it: when that product exceeds+`LAYA_MAX_BATCH_TOKENS` (131,072 by default) it chooses a `batch_size` so each forward pass stays+inside the budget, and `Router.predict_batch` runs the batch in several passes. Every state is still+answered and the response is unchanged. A request whose rows already fit is passed no `batch_size` at+all, so it behaves exactly as before -- which matters because batch shape can move floating-point+results. A `batch_size` the caller sends always wins: it asked for a shape.++At the default, 256 rows go through in one pass -- 64 states of 4 questions, or 8 of 32. Larger+batches are split, and raising `max_len` makes each pass narrower rather than costing 16 times the+work. What this does not bound is how long one request occupies the server; that is+`LAYA_MAX_CONCURRENT` and the single inference worker, and is already true of one `/v1/systemone`+request over a 50,000-character state.+The option caps are HTTP-only amplification guards; the model itself fits option tokens into a`head_max_len=192` window, so a question inside the HTTP caps can still be refused as a `422` whenthe option texts together exceed that budget. The [Evaluation harness](evals.md) runs the samediff --git a/examples/server.py b/examples/server.pyindex 54e63c9c..cf5f095d 100644--- a/examples/server.py+++ b/examples/server.py@@ -70,6 +70,11 @@ MAX_CHOICE_OPTIONS = getattr(_laya_serve, "MAX_CHOICE_OPTIONS", 100)MAX_SCORE_LEVELS = getattr(_laya_serve, "MAX_SCORE_LEVELS", 32)MAX_TOTAL_OPTIONS = getattr(_laya_serve, "MAX_TOTAL_OPTIONS", 512)DEFAULT_MAX_TOKEN_BUDGET = getattr(_laya_serve, "DEFAULT_MAX_TOKEN_BUDGET", 8192)+# `/predict/batch` collates `states x questions` rows into one tensor and each row costs its width,+# so the field caps multiply and `max_len` multiplies again. Same getattr as the rest: a batch this+# demo answers but `laya.serve` refuses would break the parity its own test file states.+DEFAULT_MAX_BATCH_TOKENS = getattr(_laya_serve, "DEFAULT_MAX_BATCH_TOKENS", 131072)+_BATCH_ROW_TOKENS_ASSUMED = getattr(_laya_serve, "_BATCH_ROW_TOKENS_ASSUMED", 512)# The set of controls a client may put on the body, and the set that must be refused rather than# silently dropped -- both read from `laya.serve` so the demo cannot drift from the server it@@ -176,6 +181,24 @@ def _check_model(v: Optional[str]) -> Optional[str]:# detail a caller sees on `main` are the same as the shipped server's._resolve_max_token_budget = getattr(_laya_serve, "_resolve_max_token_budget",lambda: DEFAULT_MAX_TOKEN_BUDGET)+_resolve_max_batch_tokens = getattr(_laya_serve, "_resolve_max_batch_tokens",+ lambda: DEFAULT_MAX_BATCH_TOKENS)+++def _fallback_batch_chunk_size(n_states, n_questions, max_len=None):+ if n_states <= 0 or n_questions <= 0:+ return None+ width = (max_len if isinstance(max_len, int) and not isinstance(max_len, bool) and max_len > 0+ else _BATCH_ROW_TOKENS_ASSUMED)+ budget = _resolve_max_batch_tokens()+ if n_states * n_questions * width <= budget:+ return None+ return max(1, min(n_states, max(1, budget // width) // n_questions))+++# The planner itself comes from laya.serve when it is there, so the demo splits a batch exactly as the+# shipped server does rather than keeping a second copy of the arithmetic.+_batch_chunk_size = getattr(_laya_serve, "_batch_chunk_size", _fallback_batch_chunk_size)def _fallback_refuse_body_refusals(body: Dict[str, Any]) -> None:@@ -609,6 +632,13 @@ def predict_batch(req: BatchRequest) -> Dict[str, Any]:shape = {}if req.batch_size is not None:shape["batch_size"] = req.batch_size+ else:+ # Split the batch so one forward pass stays inside laya.serve's token budget, read by the same+ # getattr as every other bound here. None when the request already fits, so the call is+ # unchanged -- `predict_batch` warns that batch shapes can move floating-point results.+ planned = _batch_chunk_size(len(req.states), len(req.questions), req.max_len)+ if planned is not None:+ shape["batch_size"] = plannedif req.sort_by_length:shape["sort_by_length"] = Trueif req.min_confidence is not None:diff --git a/laya/serve.py b/laya/serve.pyindex 502d0bc1..79830bb5 100644--- a/laya/serve.py+++ b/laya/serve.py@@ -72,6 +72,37 @@ MAX_QUESTIONS = 64MAX_STATE_CHARS = 50000MAX_BATCH_STATES = 64MAX_BODY_BYTES = 2 * 1024 * 1024+# `/v1/systemone/batch` collates `states x questions` rows into ONE tensor, and each of the caps+# above bounds only one factor: 64 states of 64 questions is 4096 rows from a 4.1 KB body, with every+# documented limit satisfied, and `batch_size` defaults to `None`, which `predict_batch` documents as+# "sends them all in one pass". Measured on the english checkpoint, CPU: ~127-142 ms per row at its+# `max_len` of 512, roughly flat to 128 rows, so 4096 rows is minutes inside one forward pass while+# the single inference gate is held.+#+# What a row COSTS is its width, and `max_len` is a request field capped only by+# `DEFAULT_MAX_TOKEN_BUDGET` (8192) -- 16x the english checkpoint's own 512. 256 rows at 8192 is the+# same 2 097 152 tokens as 4096 rows at 512 and collates the same tensor, so a bound on rows is the+# wrong bound. The budget below is therefore in tokens.+#+# It CHUNKS rather than refuses. A refusal would need this number to be right about hardware serve+# cannot see: measured here, peak RSS did not move at all between 32, 64 and 128 rows (flat at+# 1151 MB), so the memory ceiling this would have been protecting was never reached at any size that+# could be measured, and a 4096-row batch may be entirely reasonable on a larger machine. Splitting+# the work bounds what one forward pass collates without refusing any request, which is the part that+# can be justified. What it does NOT bound is how long one request holds the inference gate -- that is+# `LAYA_MAX_CONCURRENT` and the single-worker pool's business, and is already true of a single+# `/v1/systemone` request over a 50 000-character state.+#+# 131072 is 256 rows at `_BATCH_ROW_TOKENS_ASSUMED`. Chosen so every shape that fits it goes through+# in ONE pass exactly as before -- byte-identical, since `batch_size` is then not passed at all -- and+# only larger shapes are split. That makes the number low-stakes: too low over-chunks, too high+# under-chunks, and neither refuses anything.+DEFAULT_MAX_BATCH_TOKENS = 131072+# The row width assumed when a request does not set `max_len`. serve cannot know the routed+# checkpoint's own `max_len` without loading it, so this is the shipped english value; multilingual+# is 1024, which this under-counts by 2x, meaning a multilingual batch chunks half as aggressively as+# the budget intends.+_BATCH_ROW_TOKENS_ASSUMED = 512# HTTP-only amplification guard; the library keeps its head_max_len-aware budget.MAX_CHOICE_OPTIONS = 100MAX_SCORE_LEVELS = 32@@ -226,6 +257,28 @@ def _resolve_max_token_budget() -> int:return n+def _resolve_max_batch_tokens() -> int:+ """Tokens one batch FORWARD PASS may collate, from LAYA_MAX_BATCH_TOKENS.++ Sizes the chunk, it does not refuse: see the constant's own comment for why a refusal would need+ this number to be right about hardware this process cannot see.+ """+ raw = os.environ.get("LAYA_MAX_BATCH_TOKENS")+ if not raw:+ return DEFAULT_MAX_BATCH_TOKENS+ try:+ n = int(str(raw).strip())+ except (TypeError, ValueError):+ _log.warning("invalid LAYA_MAX_BATCH_TOKENS %r; falling back to %d", raw,+ DEFAULT_MAX_BATCH_TOKENS)+ return DEFAULT_MAX_BATCH_TOKENS+ if n <= 0:+ _log.warning("LAYA_MAX_BATCH_TOKENS must be positive (got %d); falling back to %d", n,+ DEFAULT_MAX_BATCH_TOKENS)+ return DEFAULT_MAX_BATCH_TOKENS+ return n++def _validate_budget_param(body: Dict[str, Any], key: str, max_cap: int) -> Optional[int]:"""Validate optional integer budget parameters (max_len / head_max_len) from request body (422)."""from fastapi import HTTPException@@ -594,6 +647,28 @@ def _check_request_limits(state: Any, questions: Any) -> None:detail="state too large (%d > %d chars)" % (state_len, MAX_STATE_CHARS))+def _batch_chunk_size(n_states: int, n_questions: int, max_len: Any = None) -> Optional[int]:+ """States per forward pass so one pass stays inside the token budget, or None to change nothing.++ `None` means "send them all in one pass", which is `predict_batch`'s own default, so a request+ whose rows already fit is passed no `batch_size` at all and behaves byte-identically -- important+ because `predict_batch` warns that changing batch shapes can move floating-point results, and a+ request that works today must not start answering differently.++ A single state cannot be split further, so a batch chunks down to one state per pass and no+ lower: at that point one pass carries `n_questions` rows, which is exactly what one+ `/v1/systemone` request can already ask for.+ """+ if n_states <= 0 or n_questions <= 0:+ return None+ width = (max_len if isinstance(max_len, int) and not isinstance(max_len, bool) and max_len > 0+ else _BATCH_ROW_TOKENS_ASSUMED)+ if n_states * n_questions * width <= _resolve_max_batch_tokens():+ return None+ rows_per_pass = max(1, _resolve_max_batch_tokens() // width)+ return max(1, min(n_states, rows_per_pass // n_questions))++def _check_batch_limits(states: Any, questions: Any) -> None:"""Validate batch states and questions before inference (400/413)."""from fastapi import HTTPException@@ -1020,6 +1095,15 @@ def create_app(router: Optional[Any] = None):batch_size = _validate_batch_size_param(body)if batch_size is not None:call_kwargs["batch_size"] = batch_size+ else:+ # The caller's own `batch_size` always wins -- they asked for a shape. Otherwise split the+ # batch so one forward pass stays inside the token budget. `None` when it already fits, so+ # nothing is passed and the call is the one it has always been.+ planned = _batch_chunk_size(+ len(states), len(questions) if isinstance(questions, dict) else 0,+ body.get("max_len"))+ if planned is not None:+ call_kwargs["batch_size"] = plannedsort_by_length = _validate_sort_by_length_param(body)if sort_by_length:# Only `True` is forwarded: `predict_batch`'s own default is `False`, and an attacheddiff --git a/nix/laya-serve.nix b/nix/laya-serve.nixindex 3ef4ac07..719e5655 100644--- a/nix/laya-serve.nix+++ b/nix/laya-serve.nix@@ -227,6 +227,18 @@ in'';};+ maxBatchTokens = lib.mkOption {+ type = lib.types.nullOr lib.types.ints.positive;+ default = null;+ example = 262144;+ description = ''+ Tokens one /v1/systemone/batch FORWARD PASS may collate -- states x+ questions x the row width (sets `LAYA_MAX_BATCH_TOKENS`). A larger batch+ is split across several passes, not refused. null leaves the server's own+ default.+ '';+ };+revision = lib.mkOption {type = lib.types.nullOr (lib.types.strMatching "[A-Za-z0-9._/-]+");default = null;@@ -335,6 +347,8 @@ inLAYA_MAX_LOADED = toString cfg.maxLoaded;} // lib.optionalAttrs (cfg.maxTokenBudget != null) {LAYA_MAX_TOKEN_BUDGET = toString cfg.maxTokenBudget;+ } // lib.optionalAttrs (cfg.maxBatchTokens != null) {+ LAYA_MAX_BATCH_TOKENS = toString cfg.maxBatchTokens;} // lib.optionalAttrs (cfg.revision != null) {LAYA_REVISION = cfg.revision;} // lib.optionalAttrs (cfg.defaultModel != null) {diff --git a/tests/test_example_server_limits.py b/tests/test_example_server_limits.pyindex 639025f6..776c13d4 100644--- a/tests/test_example_server_limits.py+++ b/tests/test_example_server_limits.py@@ -290,6 +290,66 @@ def main():s, d = serve_verdict(st, qs), demo_verdict(st, qs)ok("parity with laya.serve: %s" % label, s == d, "serve=%s demo=%s" % (s, d))+ # The BATCH shape, which these parity cases could not reach: every case above drives+ # `_check_request_limits` with one state, so nothing compared how the two surfaces SIZE a batch.+ # `laya.serve` splits a batch so one forward pass stays inside its token budget rather than+ # refusing it, and the demo must split it identically -- it binds the planner from `laya.serve` by+ # getattr, so this checks the binding took AND that the plan reaches the call.+ serve_plan = _serve_mod._batch_chunk_size+ demo_plan = demo._batch_chunk_size++ tiny = {"q": {"type": "noul", "instructions": "True?"}}+ four = dict(("q%03d" % i, tiny["q"]) for i in range(4))+ thirty_two = dict(("q%03d" % i, tiny["q"]) for i in range(32))+ many = dict(("q%03d" % i, tiny["q"]) for i in range(MAX_QUESTIONS))+ wide = _serve_mod.DEFAULT_MAX_TOKEN_BUDGET+ for label, ns, nq, mlen in [+ ("an ordinary batch fits, no split", 2, len(four), None),+ ("64 states x 4 questions fits", 64, len(four), None),+ ("8 states x 32 questions fits", 8, len(thirty_two), None),+ ("9 states x 32 questions splits", 9, len(thirty_two), None),+ ("64 states x 64 questions splits", 64, len(many), None),+ ("a wide max_len splits harder", 8, len(thirty_two), wide),+ ("a wide max_len with few rows fits", 2, 1, wide)]:+ a, b = serve_plan(ns, nq, mlen), demo_plan(ns, nq, mlen)+ ok("batch plan parity with laya.serve: %s" % label, a == b, "serve=%r demo=%r" % (a, b))++ # Equal on both sides is not enough -- they bind the same function, so both could be wrong+ # together. The plan itself has to be the right shape.+ ok("batch plan/a fitting batch is not split", serve_plan(8, len(thirty_two)) is None,+ repr(serve_plan(8, len(thirty_two))))+ split = serve_plan(64, len(many))+ ok("batch plan/an oversized batch is split", split is not None and split >= 1, repr(split))+ ok("batch plan/one pass fits the budget",+ split * len(many) * _serve_mod._BATCH_ROW_TOKENS_ASSUMED+ <= _serve_mod.DEFAULT_MAX_BATCH_TOKENS,+ "%r states x %d questions" % (split, len(many)))++ # And the demo's route sends it, rather than computing it and dropping it.+ sent = {}+ real_router = demo._router++ class _Recorder:+ def predict_batch(self, requests, **kwargs):+ sent.update(kwargs)+ sent["n"] = len(requests)+ return [{"model": "m", "answers": {}, "usage": {"input_tokens": 1}} for _ in requests]++ demo._router = lambda: _Recorder()+ try:+ code = client.post("/predict/batch",+ json={"states": ["hi"] * 64, "questions": many}).status_code+ ok("batch plan/the demo answers an oversized batch instead of refusing it", code == 200, code)+ ok("batch plan/the demo forwards the planned batch_size",+ sent.get("batch_size") == serve_plan(64, len(many)),+ "sent=%r planned=%r" % (sent.get("batch_size"), serve_plan(64, len(many))))+ ok("batch plan/every state is still sent", sent.get("n") == 64, sent.get("n"))+ sent.clear()+ client.post("/predict/batch", json={"states": ["hi"] * 2, "questions": four})+ ok("batch plan/a fitting batch is passed no batch_size", "batch_size" not in sent, sent)+ finally:+ demo._router = real_router+# Parity alone cannot see a bug both surfaces share, and `getattr` guarantees they share one:# with `_state_length` measuring `str(state)` again, both agree on accepting a state that# serializes to 99 988 characters and every parity check above stays green (measured: 0 of themdiff --git a/tests/test_serve.py b/tests/test_serve.pyindex 33200783..40274159 100644--- a/tests/test_serve.py+++ b/tests/test_serve.py@@ -1390,6 +1390,237 @@ def test_batch_too_many_states_returns_413(monkeypatch):assert "too many states" in r.json()["detail"]+def _noul_questions(n):+ """`n` minimal questions, so a batch's row count is `len(states) * n` and nothing else."""+ return {("q%03d" % i): {"type": "noul", "instructions": "is it urgent?"} for i in range(n)}+++def _batch_env_client(monkeypatch, **env):+ """A batch client with the batch-token knob explicitly set, or explicitly unset.++ Every test below states the budget it measures against. Without this an exported+ `LAYA_MAX_BATCH_TOKENS` silently changes the chunk and these pass or fail for a reason that is not+ in the file -- there is no `conftest.py` in this repo to isolate the environment, and+ `_batch_client` clears only `LAYA_API_KEY`.+ """+ if "LAYA_MAX_BATCH_TOKENS" in env:+ monkeypatch.setenv("LAYA_MAX_BATCH_TOKENS", env["LAYA_MAX_BATCH_TOKENS"])+ else:+ monkeypatch.delenv("LAYA_MAX_BATCH_TOKENS", raising=False)+ return _batch_client(monkeypatch)+++def _sent_batch_size(fake):+ """The `batch_size` the route handed `predict_batch`, or the sentinel when it passed none."""+ _, kwargs = fake.batch_calls[0]+ return kwargs["batch_size"] if "batch_size" in kwargs else "<absent>"+++# --------------------------------------------------------------------- the batch token budget+#+# `/v1/systemone/batch` collates `states x questions` rows into one tensor and `batch_size` defaults to+# `None` -- "all in one pass" -- so the two field caps multiply: 64 states of 64 questions is 4096 rows+# from a 4.1 KB body. The budget SPLITS the work rather than refusing it, because a refusal would have+# to be right about hardware this process cannot see: measured on english, peak RSS did not move+# between 32, 64 and 128 rows. So the contract under test is "one pass stays inside the budget, and a+# request that already fits is passed no `batch_size` at all".++def test_batch_within_the_budget_is_passed_no_batch_size(monkeypatch):+ """A request that fits must behave byte-identically, which means not passing `batch_size`.++ `predict_batch` warns that changing batch shapes can move floating-point results, so a request+ that works today must not start answering differently. Not-passing is the only way to guarantee+ that: it leaves `predict_batch` on its own default.+ """+ import laya.serve as serve_mod++ client, fake = _batch_env_client(monkeypatch)+ rows = serve_mod.DEFAULT_MAX_BATCH_TOKENS // serve_mod._BATCH_ROW_TOKENS_ASSUMED+ r = client.post("/v1/systemone/batch",+ json={"states": ["hi"] * 8, "questions": _noul_questions(rows // 8)})+ assert r.status_code == 200, r.text+ requests, kwargs = fake.batch_calls[0]+ assert len(requests) * len(requests[0]["questions"]) == rows, "exactly at the budget"+ assert "batch_size" not in kwargs, kwargs+++def test_batch_over_the_budget_is_split_not_refused(monkeypatch):+ """The 4096-row shape still answers; it is chunked so one pass stays inside the budget."""+ import laya.serve as serve_mod++ client, fake = _batch_env_client(monkeypatch)+ r = client.post("/v1/systemone/batch",+ json={"states": ["hi"] * serve_mod.MAX_BATCH_STATES,+ "questions": _noul_questions(serve_mod.MAX_QUESTIONS)})+ assert r.status_code == 200, r.text+ rows_per_pass = serve_mod.DEFAULT_MAX_BATCH_TOKENS // serve_mod._BATCH_ROW_TOKENS_ASSUMED+ expected = max(1, rows_per_pass // serve_mod.MAX_QUESTIONS)+ assert _sent_batch_size(fake) == expected+ requests, _ = fake.batch_calls[0]+ assert len(requests) == serve_mod.MAX_BATCH_STATES, "every state is still sent"+ assert expected * serve_mod.MAX_QUESTIONS * serve_mod._BATCH_ROW_TOKENS_ASSUMED \+ <= serve_mod.DEFAULT_MAX_BATCH_TOKENS, "one pass must fit the budget"+++def test_batch_one_row_over_the_budget_is_split(monkeypatch):+ """One row over, with both factors inside their own caps, so the comparison cannot be off by one."""+ import laya.serve as serve_mod++ client, fake = _batch_env_client(monkeypatch)+ rows = serve_mod.DEFAULT_MAX_BATCH_TOKENS // serve_mod._BATCH_ROW_TOKENS_ASSUMED+ states, questions = 9, rows // 8+ assert states <= serve_mod.MAX_BATCH_STATES and questions <= serve_mod.MAX_QUESTIONS+ assert states * questions > rows, "must be over by at least one row"+ r = client.post("/v1/systemone/batch",+ json={"states": ["hi"] * states, "questions": _noul_questions(questions)})+ assert r.status_code == 200, r.text+ assert _sent_batch_size(fake) == max(1, rows // questions)+++def test_batch_a_raised_max_len_chunks_harder(monkeypatch):+ """A row cap would be the wrong bound, because `max_len` is a request field.++ `max_len` is capped only by `DEFAULT_MAX_TOKEN_BUDGET` (8192), 16x the english checkpoint's own+ 512, so 256 rows at 8192 is the same token count as 4096 rows at 512 and collates the same tensor.+ Counting tokens means a wider row buys proportionally fewer rows per pass.+ """+ import laya.serve as serve_mod++ wide = serve_mod.DEFAULT_MAX_TOKEN_BUDGET+ assert wide > serve_mod._BATCH_ROW_TOKENS_ASSUMED+ rows = serve_mod.DEFAULT_MAX_BATCH_TOKENS // serve_mod._BATCH_ROW_TOKENS_ASSUMED++ narrow_client, narrow = _batch_env_client(monkeypatch)+ narrow_client.post("/v1/systemone/batch",+ json={"states": ["hi"] * 8, "questions": _noul_questions(rows // 8)})+ assert "batch_size" not in narrow.batch_calls[0][1], "fits at the default width"++ wide_client, wide_fake = _batch_env_client(monkeypatch)+ r = wide_client.post("/v1/systemone/batch",+ json={"states": ["hi"] * 8, "questions": _noul_questions(rows // 8),+ "max_len": wide})+ assert r.status_code == 200, r.text+ rows_per_pass = serve_mod.DEFAULT_MAX_BATCH_TOKENS // wide+ assert _sent_batch_size(wide_fake) == max(1, rows_per_pass // (rows // 8))+++def test_batch_an_explicit_batch_size_is_never_overridden(monkeypatch):+ """The caller asked for a shape. Planning one on top would silently change what they requested."""+ import laya.serve as serve_mod++ client, fake = _batch_env_client(monkeypatch)+ r = client.post("/v1/systemone/batch",+ json={"states": ["hi"] * serve_mod.MAX_BATCH_STATES,+ "questions": _noul_questions(serve_mod.MAX_QUESTIONS),+ "batch_size": 7})+ assert r.status_code == 200, r.text+ assert _sent_batch_size(fake) == 7+++def test_batch_chunking_follows_a_raised_budget(monkeypatch):+ """A deployment whose hardware can take more stops being chunked."""+ import laya.serve as serve_mod++ client, fake = _batch_env_client(monkeypatch, LAYA_MAX_BATCH_TOKENS="2097152")+ r = client.post("/v1/systemone/batch",+ json={"states": ["hi"] * serve_mod.MAX_BATCH_STATES,+ "questions": _noul_questions(serve_mod.MAX_QUESTIONS)})+ assert r.status_code == 200, r.text+ assert "batch_size" not in fake.batch_calls[0][1], "4096 rows x 512 fits a 2 097 152 budget"+++def test_batch_chunking_follows_a_lowered_budget(monkeypatch):+ """And a LOWERED one -- the regime an operator on small hardware actually uses."""+ import laya.serve as serve_mod++ assert 2048 < serve_mod.DEFAULT_MAX_BATCH_TOKENS, "2048 must be the lowered regime"+ client, fake = _batch_env_client(monkeypatch, LAYA_MAX_BATCH_TOKENS="2048")+ r = client.post("/v1/systemone/batch",+ json={"states": ["hi"] * 8, "questions": _noul_questions(2)})+ assert r.status_code == 200, r.text+ # 2048 / 512 = 4 rows per pass, / 2 questions = 2 states per pass+ assert _sent_batch_size(fake) == 2++[email protected]("raw", ["nonsense", "0", "-5", "3.5"])+def test_batch_budget_warns_and_falls_back_on_an_unusable_value(monkeypatch, caplog, raw):+ """A bad knob must not disable the chunking, and must say so in the log."""+ import laya.serve as serve_mod++ monkeypatch.setenv("LAYA_MAX_BATCH_TOKENS", raw)+ with caplog.at_level(logging.WARNING, logger="laya.serve"):+ assert serve_mod._resolve_max_batch_tokens() == serve_mod.DEFAULT_MAX_BATCH_TOKENS+ assert "LAYA_MAX_BATCH_TOKENS" in caplog.text, caplog.text+ assert ("invalid" in caplog.text or "must be positive" in caplog.text), caplog.text++ client, fake = _batch_client(monkeypatch)+ r = client.post("/v1/systemone/batch",+ json={"states": ["hi"] * serve_mod.MAX_BATCH_STATES,+ "questions": _noul_questions(serve_mod.MAX_QUESTIONS)})+ assert r.status_code == 200, r.text+ assert _sent_batch_size(fake) != "<absent>", "the fallback budget must still chunk"+++def test_batch_an_empty_environment_value_falls_back_silently(monkeypatch):+ """Unset and empty both mean "not configured", which is not worth a warning."""+ import laya.serve as serve_mod++ monkeypatch.setenv("LAYA_MAX_BATCH_TOKENS", "")+ assert serve_mod._resolve_max_batch_tokens() == serve_mod.DEFAULT_MAX_BATCH_TOKENS+++def test_batch_chunk_plan_floors_at_one_state(monkeypatch):+ """A single state cannot be split, so the plan stops at one state per pass and never 0.++ At that floor one pass carries `len(questions)` rows, which is exactly what one `/v1/systemone`+ request can already ask for -- so the batch route's worst pass is the single route's worst pass.+ """+ import laya.serve as serve_mod++ monkeypatch.delenv("LAYA_MAX_BATCH_TOKENS", raising=False)+ plan = serve_mod._batch_chunk_size(serve_mod.MAX_BATCH_STATES, serve_mod.MAX_QUESTIONS,+ serve_mod.DEFAULT_MAX_TOKEN_BUDGET)+ assert plan == 1, plan+ assert serve_mod._batch_chunk_size(1, serve_mod.MAX_QUESTIONS,+ serve_mod.DEFAULT_MAX_TOKEN_BUDGET) == 1++[email protected]("states,questions", [(0, 4), (4, 0), (0, 0)])+def test_batch_chunk_plan_is_none_for_a_degenerate_shape(monkeypatch, states, questions):+ """Nothing to split, and no division by the question count."""+ import laya.serve as serve_mod++ monkeypatch.delenv("LAYA_MAX_BATCH_TOKENS", raising=False)+ assert serve_mod._batch_chunk_size(states, questions) is None+++def test_batch_an_empty_questions_body_is_not_chunked(monkeypatch):+ """Zero rows: no plan, no crash, and `predict_batch` short-circuits on its own."""+ client, fake = _batch_env_client(monkeypatch)+ r = client.post("/v1/systemone/batch", json={"states": ["hi"] * 8, "questions": {}})+ assert r.status_code == 200, r.text+ assert "batch_size" not in fake.batch_calls[0][1]+++def test_batch_a_non_object_questions_body_keeps_its_own_400(monkeypatch):+ """The planner must not turn a 400 into something else by calling len() on a list."""+ client, _ = _batch_env_client(monkeypatch)+ r = client.post("/v1/systemone/batch", json={"states": ["hi"] * 8, "questions": ["nope"] * 300})+ assert r.status_code == 400, r.text+ assert "'questions' must be an object" in r.json()["detail"], r.json()["detail"]+++def test_batch_too_many_questions_still_gets_its_own_413(monkeypatch):+ """Chunking does not replace the per-question cap: 300 questions is still refused, by name."""+ import laya.serve as serve_mod++ client, _ = _batch_env_client(monkeypatch)+ r = client.post("/v1/systemone/batch",+ json={"states": ["hi"], "questions": _noul_questions(serve_mod.MAX_QUESTIONS + 1)})+ assert r.status_code == 413, r.text+ assert "too many questions" in r.json()["detail"], r.json()["detail"]++def test_batch_individual_oversized_state_returns_413(monkeypatch):client, _ = _client(monkeypatch)from laya.serve import MAX_STATE_CHARS