Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(serve): split an oversized batch across forward passes instead of collating it whole

Adeshina Lasisidf3e2854ac32
diff --git a/README.md b/README.md
index af65dae7..98cee5d3 100644
--- a/README.md
+++ b/README.md
@@ -638,7 +638,7 @@ curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
}'
```
-To evaluate multiple states in a single call, send a `states` array to `POST /v1/systemone/batch` (capped at 64 states):
+To evaluate multiple states in a single call, send a `states` array to `POST /v1/systemone/batch` (capped at 64 states; a batch larger than `LAYA_MAX_BATCH_TOKENS` is split across several forward passes rather than refused):
```bash
curl -s localhost:8000/v1/systemone/batch -H 'content-type: application/json' -d '{
diff --git a/docs/http-api.md b/docs/http-api.md
index 6ab7bdb9..03914a4d 100644
--- a/docs/http-api.md
+++ b/docs/http-api.md
@@ -37,6 +37,7 @@ Everything is environment variables, so one image serves a laptop dev run and a
| `LAYA_API_KEY` | if set, require `Authorization: Bearer <key>` | none |
| `LAYA_LOG_LEVEL` | uvicorn log level | `info` |
| `LAYA_MAX_CONCURRENT` | requests admitted past auth at once; excess gets `503` | `16` |
+| `LAYA_MAX_BATCH_TOKENS` | tokens one `/v1/systemone/batch` FORWARD PASS may collate (`states` x questions x row width); a larger batch is split into several passes, not refused | `131072` |
| `LAYA_JEV_STRICT` | serve the strict Jev wire contract: no root `routing`, no per-answer `action` / `answer_confidence`, no `confidence` on noul answers, and `usage` reduced to `input_tokens` + `output_tokens`. For clients that validate the response against the Jev contract with no extra fields | `0` |
For a deployment published under a prefix such as `/laya`, set `LAYA_ROOT_PATH=/laya`.
@@ -268,11 +269,30 @@ nothing but the bytes it read. Every one of them is a `413`; the `detail` says w
| request body | 2 MiB, enforced while streaming -- a chunked or understated `Content-Length` cannot bypass it |
| `state` | 50,000 characters of the text the model is given -- the string itself for a string state, `json.dumps(state, ensure_ascii=False)` for an object or array |
| questions per request | 64 |
+| `states` per batch request | 64 |
| options per `choice` question | 100 |
| levels per `score` question | 32 |
| options across all questions | 512 |
| concurrent admitted requests | `LAYA_MAX_CONCURRENT` (16) |
+`/v1/systemone/batch` is bounded differently, and not by a refusal. It tokenizes each state once per
+question and collates every row into a single tensor, so the field caps multiply: 64 states of 64
+questions is 4096 rows, which every other limit on this page permits. What a row costs is its width,
+and `max_len` is itself a request field, so the cost of a batch is `states x questions x width`.
+
+Rather than refuse a large batch, the endpoint **splits** it: when that product exceeds
+`LAYA_MAX_BATCH_TOKENS` (131,072 by default) it chooses a `batch_size` so each forward pass stays
+inside the budget, and `Router.predict_batch` runs the batch in several passes. Every state is still
+answered and the response is unchanged. A request whose rows already fit is passed no `batch_size` at
+all, so it behaves exactly as before -- which matters because batch shape can move floating-point
+results. A `batch_size` the caller sends always wins: it asked for a shape.
+
+At the default, 256 rows go through in one pass -- 64 states of 4 questions, or 8 of 32. Larger
+batches are split, and raising `max_len` makes each pass narrower rather than costing 16 times the
+work. What this does not bound is how long one request occupies the server; that is
+`LAYA_MAX_CONCURRENT` and the single inference worker, and is already true of one `/v1/systemone`
+request over a 50,000-character state.
+
The option caps are HTTP-only amplification guards; the model itself fits option tokens into a
`head_max_len=192` window, so a question inside the HTTP caps can still be refused as a `422` when
the option texts together exceed that budget. The [Evaluation harness](evals.md) runs the same
diff --git a/examples/server.py b/examples/server.py
index 54e63c9c..cf5f095d 100644
--- a/examples/server.py
+++ b/examples/server.py
@@ -70,6 +70,11 @@ MAX_CHOICE_OPTIONS = getattr(_laya_serve, "MAX_CHOICE_OPTIONS", 100)
MAX_SCORE_LEVELS = getattr(_laya_serve, "MAX_SCORE_LEVELS", 32)
MAX_TOTAL_OPTIONS = getattr(_laya_serve, "MAX_TOTAL_OPTIONS", 512)
DEFAULT_MAX_TOKEN_BUDGET = getattr(_laya_serve, "DEFAULT_MAX_TOKEN_BUDGET", 8192)
+# `/predict/batch` collates `states x questions` rows into one tensor and each row costs its width,
+# so the field caps multiply and `max_len` multiplies again. Same getattr as the rest: a batch this
+# demo answers but `laya.serve` refuses would break the parity its own test file states.
+DEFAULT_MAX_BATCH_TOKENS = getattr(_laya_serve, "DEFAULT_MAX_BATCH_TOKENS", 131072)
+_BATCH_ROW_TOKENS_ASSUMED = getattr(_laya_serve, "_BATCH_ROW_TOKENS_ASSUMED", 512)
# The set of controls a client may put on the body, and the set that must be refused rather than
# silently dropped -- both read from `laya.serve` so the demo cannot drift from the server it
@@ -176,6 +181,24 @@ def _check_model(v: Optional[str]) -> Optional[str]:
# detail a caller sees on `main` are the same as the shipped server's.
_resolve_max_token_budget = getattr(_laya_serve, "_resolve_max_token_budget",
lambda: DEFAULT_MAX_TOKEN_BUDGET)
+_resolve_max_batch_tokens = getattr(_laya_serve, "_resolve_max_batch_tokens",
+ lambda: DEFAULT_MAX_BATCH_TOKENS)
+
+
+def _fallback_batch_chunk_size(n_states, n_questions, max_len=None):
+ if n_states <= 0 or n_questions <= 0:
+ return None
+ width = (max_len if isinstance(max_len, int) and not isinstance(max_len, bool) and max_len > 0
+ else _BATCH_ROW_TOKENS_ASSUMED)
+ budget = _resolve_max_batch_tokens()
+ if n_states * n_questions * width <= budget:
+ return None
+ return max(1, min(n_states, max(1, budget // width) // n_questions))
+
+
+# The planner itself comes from laya.serve when it is there, so the demo splits a batch exactly as the
+# shipped server does rather than keeping a second copy of the arithmetic.
+_batch_chunk_size = getattr(_laya_serve, "_batch_chunk_size", _fallback_batch_chunk_size)
def _fallback_refuse_body_refusals(body: Dict[str, Any]) -> None:
@@ -609,6 +632,13 @@ def predict_batch(req: BatchRequest) -> Dict[str, Any]:
shape = {}
if req.batch_size is not None:
shape["batch_size"] = req.batch_size
+ else:
+ # Split the batch so one forward pass stays inside laya.serve's token budget, read by the same
+ # getattr as every other bound here. None when the request already fits, so the call is
+ # unchanged -- `predict_batch` warns that batch shapes can move floating-point results.
+ planned = _batch_chunk_size(len(req.states), len(req.questions), req.max_len)
+ if planned is not None:
+ shape["batch_size"] = planned
if req.sort_by_length:
shape["sort_by_length"] = True
if req.min_confidence is not None:
diff --git a/laya/serve.py b/laya/serve.py
index 502d0bc1..79830bb5 100644
--- a/laya/serve.py
+++ b/laya/serve.py
@@ -72,6 +72,37 @@ MAX_QUESTIONS = 64
MAX_STATE_CHARS = 50000
MAX_BATCH_STATES = 64
MAX_BODY_BYTES = 2 * 1024 * 1024
+# `/v1/systemone/batch` collates `states x questions` rows into ONE tensor, and each of the caps
+# above bounds only one factor: 64 states of 64 questions is 4096 rows from a 4.1 KB body, with every
+# documented limit satisfied, and `batch_size` defaults to `None`, which `predict_batch` documents as
+# "sends them all in one pass". Measured on the english checkpoint, CPU: ~127-142 ms per row at its
+# `max_len` of 512, roughly flat to 128 rows, so 4096 rows is minutes inside one forward pass while
+# the single inference gate is held.
+#
+# What a row COSTS is its width, and `max_len` is a request field capped only by
+# `DEFAULT_MAX_TOKEN_BUDGET` (8192) -- 16x the english checkpoint's own 512. 256 rows at 8192 is the
+# same 2 097 152 tokens as 4096 rows at 512 and collates the same tensor, so a bound on rows is the
+# wrong bound. The budget below is therefore in tokens.
+#
+# It CHUNKS rather than refuses. A refusal would need this number to be right about hardware serve
+# cannot see: measured here, peak RSS did not move at all between 32, 64 and 128 rows (flat at
+# 1151 MB), so the memory ceiling this would have been protecting was never reached at any size that
+# could be measured, and a 4096-row batch may be entirely reasonable on a larger machine. Splitting
+# the work bounds what one forward pass collates without refusing any request, which is the part that
+# can be justified. What it does NOT bound is how long one request holds the inference gate -- that is
+# `LAYA_MAX_CONCURRENT` and the single-worker pool's business, and is already true of a single
+# `/v1/systemone` request over a 50 000-character state.
+#
+# 131072 is 256 rows at `_BATCH_ROW_TOKENS_ASSUMED`. Chosen so every shape that fits it goes through
+# in ONE pass exactly as before -- byte-identical, since `batch_size` is then not passed at all -- and
+# only larger shapes are split. That makes the number low-stakes: too low over-chunks, too high
+# under-chunks, and neither refuses anything.
+DEFAULT_MAX_BATCH_TOKENS = 131072
+# The row width assumed when a request does not set `max_len`. serve cannot know the routed
+# checkpoint's own `max_len` without loading it, so this is the shipped english value; multilingual
+# is 1024, which this under-counts by 2x, meaning a multilingual batch chunks half as aggressively as
+# the budget intends.
+_BATCH_ROW_TOKENS_ASSUMED = 512
# HTTP-only amplification guard; the library keeps its head_max_len-aware budget.
MAX_CHOICE_OPTIONS = 100
MAX_SCORE_LEVELS = 32
@@ -226,6 +257,28 @@ def _resolve_max_token_budget() -> int:
return n
+def _resolve_max_batch_tokens() -> int:
+ """Tokens one batch FORWARD PASS may collate, from LAYA_MAX_BATCH_TOKENS.
+
+ Sizes the chunk, it does not refuse: see the constant's own comment for why a refusal would need
+ this number to be right about hardware this process cannot see.
+ """
+ raw = os.environ.get("LAYA_MAX_BATCH_TOKENS")
+ if not raw:
+ return DEFAULT_MAX_BATCH_TOKENS
+ try:
+ n = int(str(raw).strip())
+ except (TypeError, ValueError):
+ _log.warning("invalid LAYA_MAX_BATCH_TOKENS %r; falling back to %d", raw,
+ DEFAULT_MAX_BATCH_TOKENS)
+ return DEFAULT_MAX_BATCH_TOKENS
+ if n <= 0:
+ _log.warning("LAYA_MAX_BATCH_TOKENS must be positive (got %d); falling back to %d", n,
+ DEFAULT_MAX_BATCH_TOKENS)
+ return DEFAULT_MAX_BATCH_TOKENS
+ return n
+
+
def _validate_budget_param(body: Dict[str, Any], key: str, max_cap: int) -> Optional[int]:
"""Validate optional integer budget parameters (max_len / head_max_len) from request body (422)."""
from fastapi import HTTPException
@@ -594,6 +647,28 @@ def _check_request_limits(state: Any, questions: Any) -> None:
detail="state too large (%d > %d chars)" % (state_len, MAX_STATE_CHARS))
+def _batch_chunk_size(n_states: int, n_questions: int, max_len: Any = None) -> Optional[int]:
+ """States per forward pass so one pass stays inside the token budget, or None to change nothing.
+
+ `None` means "send them all in one pass", which is `predict_batch`'s own default, so a request
+ whose rows already fit is passed no `batch_size` at all and behaves byte-identically -- important
+ because `predict_batch` warns that changing batch shapes can move floating-point results, and a
+ request that works today must not start answering differently.
+
+ A single state cannot be split further, so a batch chunks down to one state per pass and no
+ lower: at that point one pass carries `n_questions` rows, which is exactly what one
+ `/v1/systemone` request can already ask for.
+ """
+ if n_states <= 0 or n_questions <= 0:
+ return None
+ width = (max_len if isinstance(max_len, int) and not isinstance(max_len, bool) and max_len > 0
+ else _BATCH_ROW_TOKENS_ASSUMED)
+ if n_states * n_questions * width <= _resolve_max_batch_tokens():
+ return None
+ rows_per_pass = max(1, _resolve_max_batch_tokens() // width)
+ return max(1, min(n_states, rows_per_pass // n_questions))
+
+
def _check_batch_limits(states: Any, questions: Any) -> None:
"""Validate batch states and questions before inference (400/413)."""
from fastapi import HTTPException
@@ -1020,6 +1095,15 @@ def create_app(router: Optional[Any] = None):
batch_size = _validate_batch_size_param(body)
if batch_size is not None:
call_kwargs["batch_size"] = batch_size
+ else:
+ # The caller's own `batch_size` always wins -- they asked for a shape. Otherwise split the
+ # batch so one forward pass stays inside the token budget. `None` when it already fits, so
+ # nothing is passed and the call is the one it has always been.
+ planned = _batch_chunk_size(
+ len(states), len(questions) if isinstance(questions, dict) else 0,
+ body.get("max_len"))
+ if planned is not None:
+ call_kwargs["batch_size"] = planned
sort_by_length = _validate_sort_by_length_param(body)
if sort_by_length:
# Only `True` is forwarded: `predict_batch`'s own default is `False`, and an attached
diff --git a/nix/laya-serve.nix b/nix/laya-serve.nix
index 3ef4ac07..719e5655 100644
--- a/nix/laya-serve.nix
+++ b/nix/laya-serve.nix
@@ -227,6 +227,18 @@ in
'';
};
+ maxBatchTokens = lib.mkOption {
+ type = lib.types.nullOr lib.types.ints.positive;
+ default = null;
+ example = 262144;
+ description = ''
+ Tokens one /v1/systemone/batch FORWARD PASS may collate -- states x
+ questions x the row width (sets `LAYA_MAX_BATCH_TOKENS`). A larger batch
+ is split across several passes, not refused. null leaves the server's own
+ default.
+ '';
+ };
+
revision = lib.mkOption {
type = lib.types.nullOr (lib.types.strMatching "[A-Za-z0-9._/-]+");
default = null;
@@ -335,6 +347,8 @@ in
LAYA_MAX_LOADED = toString cfg.maxLoaded;
} // lib.optionalAttrs (cfg.maxTokenBudget != null) {
LAYA_MAX_TOKEN_BUDGET = toString cfg.maxTokenBudget;
+ } // lib.optionalAttrs (cfg.maxBatchTokens != null) {
+ LAYA_MAX_BATCH_TOKENS = toString cfg.maxBatchTokens;
} // lib.optionalAttrs (cfg.revision != null) {
LAYA_REVISION = cfg.revision;
} // lib.optionalAttrs (cfg.defaultModel != null) {
diff --git a/tests/test_example_server_limits.py b/tests/test_example_server_limits.py
index 639025f6..776c13d4 100644
--- a/tests/test_example_server_limits.py
+++ b/tests/test_example_server_limits.py
@@ -290,6 +290,66 @@ def main():
s, d = serve_verdict(st, qs), demo_verdict(st, qs)
ok("parity with laya.serve: %s" % label, s == d, "serve=%s demo=%s" % (s, d))
+ # The BATCH shape, which these parity cases could not reach: every case above drives
+ # `_check_request_limits` with one state, so nothing compared how the two surfaces SIZE a batch.
+ # `laya.serve` splits a batch so one forward pass stays inside its token budget rather than
+ # refusing it, and the demo must split it identically -- it binds the planner from `laya.serve` by
+ # getattr, so this checks the binding took AND that the plan reaches the call.
+ serve_plan = _serve_mod._batch_chunk_size
+ demo_plan = demo._batch_chunk_size
+
+ tiny = {"q": {"type": "noul", "instructions": "True?"}}
+ four = dict(("q%03d" % i, tiny["q"]) for i in range(4))
+ thirty_two = dict(("q%03d" % i, tiny["q"]) for i in range(32))
+ many = dict(("q%03d" % i, tiny["q"]) for i in range(MAX_QUESTIONS))
+ wide = _serve_mod.DEFAULT_MAX_TOKEN_BUDGET
+ for label, ns, nq, mlen in [
+ ("an ordinary batch fits, no split", 2, len(four), None),
+ ("64 states x 4 questions fits", 64, len(four), None),
+ ("8 states x 32 questions fits", 8, len(thirty_two), None),
+ ("9 states x 32 questions splits", 9, len(thirty_two), None),
+ ("64 states x 64 questions splits", 64, len(many), None),
+ ("a wide max_len splits harder", 8, len(thirty_two), wide),
+ ("a wide max_len with few rows fits", 2, 1, wide)]:
+ a, b = serve_plan(ns, nq, mlen), demo_plan(ns, nq, mlen)
+ ok("batch plan parity with laya.serve: %s" % label, a == b, "serve=%r demo=%r" % (a, b))
+
+ # Equal on both sides is not enough -- they bind the same function, so both could be wrong
+ # together. The plan itself has to be the right shape.
+ ok("batch plan/a fitting batch is not split", serve_plan(8, len(thirty_two)) is None,
+ repr(serve_plan(8, len(thirty_two))))
+ split = serve_plan(64, len(many))
+ ok("batch plan/an oversized batch is split", split is not None and split >= 1, repr(split))
+ ok("batch plan/one pass fits the budget",
+ split * len(many) * _serve_mod._BATCH_ROW_TOKENS_ASSUMED
+ <= _serve_mod.DEFAULT_MAX_BATCH_TOKENS,
+ "%r states x %d questions" % (split, len(many)))
+
+ # And the demo's route sends it, rather than computing it and dropping it.
+ sent = {}
+ real_router = demo._router
+
+ class _Recorder:
+ def predict_batch(self, requests, **kwargs):
+ sent.update(kwargs)
+ sent["n"] = len(requests)
+ return [{"model": "m", "answers": {}, "usage": {"input_tokens": 1}} for _ in requests]
+
+ demo._router = lambda: _Recorder()
+ try:
+ code = client.post("/predict/batch",
+ json={"states": ["hi"] * 64, "questions": many}).status_code
+ ok("batch plan/the demo answers an oversized batch instead of refusing it", code == 200, code)
+ ok("batch plan/the demo forwards the planned batch_size",
+ sent.get("batch_size") == serve_plan(64, len(many)),
+ "sent=%r planned=%r" % (sent.get("batch_size"), serve_plan(64, len(many))))
+ ok("batch plan/every state is still sent", sent.get("n") == 64, sent.get("n"))
+ sent.clear()
+ client.post("/predict/batch", json={"states": ["hi"] * 2, "questions": four})
+ ok("batch plan/a fitting batch is passed no batch_size", "batch_size" not in sent, sent)
+ finally:
+ demo._router = real_router
+
# Parity alone cannot see a bug both surfaces share, and `getattr` guarantees they share one:
# with `_state_length` measuring `str(state)` again, both agree on accepting a state that
# serializes to 99 988 characters and every parity check above stays green (measured: 0 of them
diff --git a/tests/test_serve.py b/tests/test_serve.py
index 33200783..40274159 100644
--- a/tests/test_serve.py
+++ b/tests/test_serve.py
@@ -1390,6 +1390,237 @@ def test_batch_too_many_states_returns_413(monkeypatch):
assert "too many states" in r.json()["detail"]
+def _noul_questions(n):
+ """`n` minimal questions, so a batch's row count is `len(states) * n` and nothing else."""
+ return {("q%03d" % i): {"type": "noul", "instructions": "is it urgent?"} for i in range(n)}
+
+
+def _batch_env_client(monkeypatch, **env):
+ """A batch client with the batch-token knob explicitly set, or explicitly unset.
+
+ Every test below states the budget it measures against. Without this an exported
+ `LAYA_MAX_BATCH_TOKENS` silently changes the chunk and these pass or fail for a reason that is not
+ in the file -- there is no `conftest.py` in this repo to isolate the environment, and
+ `_batch_client` clears only `LAYA_API_KEY`.
+ """
+ if "LAYA_MAX_BATCH_TOKENS" in env:
+ monkeypatch.setenv("LAYA_MAX_BATCH_TOKENS", env["LAYA_MAX_BATCH_TOKENS"])
+ else:
+ monkeypatch.delenv("LAYA_MAX_BATCH_TOKENS", raising=False)
+ return _batch_client(monkeypatch)
+
+
+def _sent_batch_size(fake):
+ """The `batch_size` the route handed `predict_batch`, or the sentinel when it passed none."""
+ _, kwargs = fake.batch_calls[0]
+ return kwargs["batch_size"] if "batch_size" in kwargs else "<absent>"
+
+
+# --------------------------------------------------------------------- the batch token budget
+#
+# `/v1/systemone/batch` collates `states x questions` rows into one tensor and `batch_size` defaults to
+# `None` -- "all in one pass" -- so the two field caps multiply: 64 states of 64 questions is 4096 rows
+# from a 4.1 KB body. The budget SPLITS the work rather than refusing it, because a refusal would have
+# to be right about hardware this process cannot see: measured on english, peak RSS did not move
+# between 32, 64 and 128 rows. So the contract under test is "one pass stays inside the budget, and a
+# request that already fits is passed no `batch_size` at all".
+
+def test_batch_within_the_budget_is_passed_no_batch_size(monkeypatch):
+ """A request that fits must behave byte-identically, which means not passing `batch_size`.
+
+ `predict_batch` warns that changing batch shapes can move floating-point results, so a request
+ that works today must not start answering differently. Not-passing is the only way to guarantee
+ that: it leaves `predict_batch` on its own default.
+ """
+ import laya.serve as serve_mod
+
+ client, fake = _batch_env_client(monkeypatch)
+ rows = serve_mod.DEFAULT_MAX_BATCH_TOKENS // serve_mod._BATCH_ROW_TOKENS_ASSUMED
+ r = client.post("/v1/systemone/batch",
+ json={"states": ["hi"] * 8, "questions": _noul_questions(rows // 8)})
+ assert r.status_code == 200, r.text
+ requests, kwargs = fake.batch_calls[0]
+ assert len(requests) * len(requests[0]["questions"]) == rows, "exactly at the budget"
+ assert "batch_size" not in kwargs, kwargs
+
+
+def test_batch_over_the_budget_is_split_not_refused(monkeypatch):
+ """The 4096-row shape still answers; it is chunked so one pass stays inside the budget."""
+ import laya.serve as serve_mod
+
+ client, fake = _batch_env_client(monkeypatch)
+ r = client.post("/v1/systemone/batch",
+ json={"states": ["hi"] * serve_mod.MAX_BATCH_STATES,
+ "questions": _noul_questions(serve_mod.MAX_QUESTIONS)})
+ assert r.status_code == 200, r.text
+ rows_per_pass = serve_mod.DEFAULT_MAX_BATCH_TOKENS // serve_mod._BATCH_ROW_TOKENS_ASSUMED
+ expected = max(1, rows_per_pass // serve_mod.MAX_QUESTIONS)
+ assert _sent_batch_size(fake) == expected
+ requests, _ = fake.batch_calls[0]
+ assert len(requests) == serve_mod.MAX_BATCH_STATES, "every state is still sent"
+ assert expected * serve_mod.MAX_QUESTIONS * serve_mod._BATCH_ROW_TOKENS_ASSUMED \
+ <= serve_mod.DEFAULT_MAX_BATCH_TOKENS, "one pass must fit the budget"
+
+
+def test_batch_one_row_over_the_budget_is_split(monkeypatch):
+ """One row over, with both factors inside their own caps, so the comparison cannot be off by one."""
+ import laya.serve as serve_mod
+
+ client, fake = _batch_env_client(monkeypatch)
+ rows = serve_mod.DEFAULT_MAX_BATCH_TOKENS // serve_mod._BATCH_ROW_TOKENS_ASSUMED
+ states, questions = 9, rows // 8
+ assert states <= serve_mod.MAX_BATCH_STATES and questions <= serve_mod.MAX_QUESTIONS
+ assert states * questions > rows, "must be over by at least one row"
+ r = client.post("/v1/systemone/batch",
+ json={"states": ["hi"] * states, "questions": _noul_questions(questions)})
+ assert r.status_code == 200, r.text
+ assert _sent_batch_size(fake) == max(1, rows // questions)
+
+
+def test_batch_a_raised_max_len_chunks_harder(monkeypatch):
+ """A row cap would be the wrong bound, because `max_len` is a request field.
+
+ `max_len` is capped only by `DEFAULT_MAX_TOKEN_BUDGET` (8192), 16x the english checkpoint's own
+ 512, so 256 rows at 8192 is the same token count as 4096 rows at 512 and collates the same tensor.
+ Counting tokens means a wider row buys proportionally fewer rows per pass.
+ """
+ import laya.serve as serve_mod
+
+ wide = serve_mod.DEFAULT_MAX_TOKEN_BUDGET
+ assert wide > serve_mod._BATCH_ROW_TOKENS_ASSUMED
+ rows = serve_mod.DEFAULT_MAX_BATCH_TOKENS // serve_mod._BATCH_ROW_TOKENS_ASSUMED
+
+ narrow_client, narrow = _batch_env_client(monkeypatch)
+ narrow_client.post("/v1/systemone/batch",
+ json={"states": ["hi"] * 8, "questions": _noul_questions(rows // 8)})
+ assert "batch_size" not in narrow.batch_calls[0][1], "fits at the default width"
+
+ wide_client, wide_fake = _batch_env_client(monkeypatch)
+ r = wide_client.post("/v1/systemone/batch",
+ json={"states": ["hi"] * 8, "questions": _noul_questions(rows // 8),
+ "max_len": wide})
+ assert r.status_code == 200, r.text
+ rows_per_pass = serve_mod.DEFAULT_MAX_BATCH_TOKENS // wide
+ assert _sent_batch_size(wide_fake) == max(1, rows_per_pass // (rows // 8))
+
+
+def test_batch_an_explicit_batch_size_is_never_overridden(monkeypatch):
+ """The caller asked for a shape. Planning one on top would silently change what they requested."""
+ import laya.serve as serve_mod
+
+ client, fake = _batch_env_client(monkeypatch)
+ r = client.post("/v1/systemone/batch",
+ json={"states": ["hi"] * serve_mod.MAX_BATCH_STATES,
+ "questions": _noul_questions(serve_mod.MAX_QUESTIONS),
+ "batch_size": 7})
+ assert r.status_code == 200, r.text
+ assert _sent_batch_size(fake) == 7
+
+
+def test_batch_chunking_follows_a_raised_budget(monkeypatch):
+ """A deployment whose hardware can take more stops being chunked."""
+ import laya.serve as serve_mod
+
+ client, fake = _batch_env_client(monkeypatch, LAYA_MAX_BATCH_TOKENS="2097152")
+ r = client.post("/v1/systemone/batch",
+ json={"states": ["hi"] * serve_mod.MAX_BATCH_STATES,
+ "questions": _noul_questions(serve_mod.MAX_QUESTIONS)})
+ assert r.status_code == 200, r.text
+ assert "batch_size" not in fake.batch_calls[0][1], "4096 rows x 512 fits a 2 097 152 budget"
+
+
+def test_batch_chunking_follows_a_lowered_budget(monkeypatch):
+ """And a LOWERED one -- the regime an operator on small hardware actually uses."""
+ import laya.serve as serve_mod
+
+ assert 2048 < serve_mod.DEFAULT_MAX_BATCH_TOKENS, "2048 must be the lowered regime"
+ client, fake = _batch_env_client(monkeypatch, LAYA_MAX_BATCH_TOKENS="2048")
+ r = client.post("/v1/systemone/batch",
+ json={"states": ["hi"] * 8, "questions": _noul_questions(2)})
+ assert r.status_code == 200, r.text
+ # 2048 / 512 = 4 rows per pass, / 2 questions = 2 states per pass
+ assert _sent_batch_size(fake) == 2
+
+
[email protected]("raw", ["nonsense", "0", "-5", "3.5"])
+def test_batch_budget_warns_and_falls_back_on_an_unusable_value(monkeypatch, caplog, raw):
+ """A bad knob must not disable the chunking, and must say so in the log."""
+ import laya.serve as serve_mod
+
+ monkeypatch.setenv("LAYA_MAX_BATCH_TOKENS", raw)
+ with caplog.at_level(logging.WARNING, logger="laya.serve"):
+ assert serve_mod._resolve_max_batch_tokens() == serve_mod.DEFAULT_MAX_BATCH_TOKENS
+ assert "LAYA_MAX_BATCH_TOKENS" in caplog.text, caplog.text
+ assert ("invalid" in caplog.text or "must be positive" in caplog.text), caplog.text
+
+ client, fake = _batch_client(monkeypatch)
+ r = client.post("/v1/systemone/batch",
+ json={"states": ["hi"] * serve_mod.MAX_BATCH_STATES,
+ "questions": _noul_questions(serve_mod.MAX_QUESTIONS)})
+ assert r.status_code == 200, r.text
+ assert _sent_batch_size(fake) != "<absent>", "the fallback budget must still chunk"
+
+
+def test_batch_an_empty_environment_value_falls_back_silently(monkeypatch):
+ """Unset and empty both mean "not configured", which is not worth a warning."""
+ import laya.serve as serve_mod
+
+ monkeypatch.setenv("LAYA_MAX_BATCH_TOKENS", "")
+ assert serve_mod._resolve_max_batch_tokens() == serve_mod.DEFAULT_MAX_BATCH_TOKENS
+
+
+def test_batch_chunk_plan_floors_at_one_state(monkeypatch):
+ """A single state cannot be split, so the plan stops at one state per pass and never 0.
+
+ At that floor one pass carries `len(questions)` rows, which is exactly what one `/v1/systemone`
+ request can already ask for -- so the batch route's worst pass is the single route's worst pass.
+ """
+ import laya.serve as serve_mod
+
+ monkeypatch.delenv("LAYA_MAX_BATCH_TOKENS", raising=False)
+ plan = serve_mod._batch_chunk_size(serve_mod.MAX_BATCH_STATES, serve_mod.MAX_QUESTIONS,
+ serve_mod.DEFAULT_MAX_TOKEN_BUDGET)
+ assert plan == 1, plan
+ assert serve_mod._batch_chunk_size(1, serve_mod.MAX_QUESTIONS,
+ serve_mod.DEFAULT_MAX_TOKEN_BUDGET) == 1
+
+
[email protected]("states,questions", [(0, 4), (4, 0), (0, 0)])
+def test_batch_chunk_plan_is_none_for_a_degenerate_shape(monkeypatch, states, questions):
+ """Nothing to split, and no division by the question count."""
+ import laya.serve as serve_mod
+
+ monkeypatch.delenv("LAYA_MAX_BATCH_TOKENS", raising=False)
+ assert serve_mod._batch_chunk_size(states, questions) is None
+
+
+def test_batch_an_empty_questions_body_is_not_chunked(monkeypatch):
+ """Zero rows: no plan, no crash, and `predict_batch` short-circuits on its own."""
+ client, fake = _batch_env_client(monkeypatch)
+ r = client.post("/v1/systemone/batch", json={"states": ["hi"] * 8, "questions": {}})
+ assert r.status_code == 200, r.text
+ assert "batch_size" not in fake.batch_calls[0][1]
+
+
+def test_batch_a_non_object_questions_body_keeps_its_own_400(monkeypatch):
+ """The planner must not turn a 400 into something else by calling len() on a list."""
+ client, _ = _batch_env_client(monkeypatch)
+ r = client.post("/v1/systemone/batch", json={"states": ["hi"] * 8, "questions": ["nope"] * 300})
+ assert r.status_code == 400, r.text
+ assert "'questions' must be an object" in r.json()["detail"], r.json()["detail"]
+
+
+def test_batch_too_many_questions_still_gets_its_own_413(monkeypatch):
+ """Chunking does not replace the per-question cap: 300 questions is still refused, by name."""
+ import laya.serve as serve_mod
+
+ client, _ = _batch_env_client(monkeypatch)
+ r = client.post("/v1/systemone/batch",
+ json={"states": ["hi"], "questions": _noul_questions(serve_mod.MAX_QUESTIONS + 1)})
+ assert r.status_code == 413, r.text
+ assert "too many questions" in r.json()["detail"], r.json()["detail"]
+
+
def test_batch_individual_oversized_state_returns_413(monkeypatch):
client, _ = _client(monkeypatch)
from laya.serve import MAX_STATE_CHARS