Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(laya): warn when Jev requests truncate shared state

Matt Van Horna89f83e6db32
diff --git a/README.md b/README.md
index 04959296..dee16061 100644
--- a/README.md
+++ b/README.md
@@ -741,6 +741,8 @@ curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
}'
```
+Questions in that request share the one `state`. Each is answered against the whole serialized state, so an instruction that names a record (`rows[i]`) still sees every record in it. One answer per record means one state per record, sent as `states` to `POST /v1/systemone/batch`. A state that does not fit the token room the questions leave is truncated before inference, and the server logs the dropped state tokens. The full response reports the same cut on `usage` (`truncated`, `state_tokens_dropped`, `truncated_questions`); `LAYA_JEV_STRICT=1` omits those fields, and the log line is the signal that remains.
+
To evaluate multiple states in a single call, send a `states` array to `POST /v1/systemone/batch` (capped at 64 states; a batch larger than `LAYA_MAX_BATCH_TOKENS` is split across several forward passes rather than refused):
```bash
@@ -754,7 +756,7 @@ curl -s localhost:8000/v1/systemone/batch -H 'content-type: application/json' -d
}'
```
-The endpoint shares forward passes via `Router.predict_batch` and returns an array of `results` in matching order along with aggregated `total_usage`.
+The endpoint shares forward passes via `Router.predict_batch` and returns an array of `results` in matching order along with aggregated `total_usage`. Each entry of `states` is its own context, which is the per-record call: a question on `/v1/systemone` sees the whole shared state, even when its text names `rows[i]`. When any result dropped state tokens, the server logs one warning for the batch — which results were cut, and how many tokens each lost — rather than one line per answer.
Configuration is by environment variable: `LAYA_HOST`, `LAYA_PORT`,
@@ -785,8 +787,9 @@ When publishing the server under a reverse-proxy prefix such as `/laya`, set
`LAYA_ROOT_PATH=/laya`. The proxy should strip that prefix before forwarding;
the setting controls FastAPI-generated URLs and leaves the internal routes unchanged.
-Three things differ from Jev when you port a client:
+Four things differ from Jev when you port a client:
+* **One shared context.** One `/v1/systemone` request is one state, and every question is decided on that whole state. Several records packed into it, with questions aimed at `rows[i]`, still share that context. Per-record answers are `states` on `/v1/systemone/batch`, one state per record. A state larger than the token room is truncated, and `laya-serve` logs the dropped tokens; under `LAYA_JEV_STRICT` that log is where the cut stays visible.
* **Options per question.** A question's options share the checkpoint's option budget, `head_max_len` (192 tokens on `laya`, 256 on the other two), not Jev's cap of 255 options. In addition, the HTTP server (`laya.serve`) enforces an amplification guard of at most 100 choice options per question (`MAX_CHOICE_OPTIONS = 100`, rejected with 413 before inference). Once options overflow the token budget, around 20 options with a short description each, every option is trimmed to fit, so long or similar labels can reach the model reading the same ([Where Jev leads](#where-jev-leads)). Once they no longer fit the window at all, the library rejects the request with 422. For more candidates, narrow them first with `predict_shortlist` ([Honest limits](#honest-limits)).
* **Score levels.** Every level needs a description. A `null` level is rejected with 422 rather than scored and echoed back in `legend`.
* **`confidence`** on `choice` and `score` answers is 1 minus normalised entropy, a measure of how concentrated the distribution is, not Jev's `(n·p_max − 1)/(n − 1)`. A threshold carried over from Jev does not transfer. For one calibrated number on every question type, gate on `answer_confidence`, the probability of the reported answer. Both values shift with the number of options, so fit and validate any cutoff on held-out data at the option counts your workload uses ([#394](https://github.com/NandhaKishorM/laya/issues/394)).
diff --git a/docs/http-api.md b/docs/http-api.md
index d49071e7..2fa55550 100644
--- a/docs/http-api.md
+++ b/docs/http-api.md
@@ -191,7 +191,10 @@ inference. Injecting the Router keeps the example independent of normal startup
### `POST /v1/systemone`
-One request carries a `state` and any number of questions over it:
+One request carries a `state` and any number of questions over it. Those questions share that
+state: each is answered against the whole serialized context, so a question whose instructions
+name `rows[i]` still sees every record. One answer per record is one entry of `states` on
+`POST /v1/systemone/batch`.
```bash
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
@@ -208,7 +211,7 @@ curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
| field | required | meaning |
|---|---|---|
-| `state` | yes | text, email, ticket or JSON document to decide on; a missing or `null` state is a `400` |
+| `state` | yes | the one context every question in the request is answered against: text, email, ticket or JSON; a missing or `null` state is a `400` |
| `questions` | yes | object keyed by question id; each question is `choice` / `score` / `noul` with `instructions` and `criteria` |
| `model` | no | names a checkpoint; a path or unpublished Hub id is a `422`, anything else is ignored (see below) |
| `task` | no | forces a checkpoint by workflow name instead of letting routing decide; an unknown name is a `422` naming it |
@@ -300,7 +303,8 @@ marks, it does not drop.
`usage` reports what the forward pass was built from. How much of a state the model reads is a token
budget, not a character count, and the budget moves with `max_len`, `head_max_len` and every
-question's own option prompt (#174), so these keys are the only place that fact is visible:
+question's own option prompt (#174), so these keys are the only place in the response
+that fact is visible:
| `usage` key | meaning |
|---|---|
@@ -313,7 +317,13 @@ question's own option prompt (#174), so these keys are the only place that fact
| `options` | present only when some question's options no longer have a token span each: keyed by question id, with `total` (the options that question defines), `distinct` (the spans that reached the sequence) and `tokens_per_option` |
A truncated answer is still an answer -- the head decides on the evidence it was given -- but a
-caller sizing states by character count cannot see the cut anywhere else in the response.
+caller sizing states by character count cannot see the cut anywhere else in the response. When
+`usage.truncated` is true or `state_tokens_dropped` is above zero, the server logs one warning
+for the request with that dropped-token count and the question ids. The warning is written on
+`/v1/systemone` and `/v1/systemone/batch` before `LAYA_JEV_STRICT` projects the payload, so a
+strict response that no longer carries these keys still leaves the cut in the server log. A
+batch logs once, with the indexes of the results that were cut and the dropped-token count of
+each, rather than once per answer.
`routing` records which checkpoint answered and why:
@@ -372,8 +382,9 @@ answering, on both `/v1/systemone` and `/v1/systemone/batch`:
The projection keeps only the contracted keys and recomputes nothing: every value is the one the
result already carries, so the probabilities and scores a strict client reads are identical to
the ones the full payload reports. The default stays the full payload, and a deployment that
-turns the flag on loses the truncation visibility `usage` provides -- a cut state is then
-visible in the logs, not in the response. Score `criteria` should stay plain strings under the
+turns the flag on loses the truncation visibility `usage` provides. The server has already
+logged the dropped state tokens by then, so a cut state is in the log, where the response no
+longer carries it. Score `criteria` should stay plain strings under the
strict contract: a strict client compares the returned `legend` against the criteria it sent,
and Laya renders a structured criterion with Python's JSON, which a JavaScript caller that
stringifies its own criteria may not match byte for byte.
@@ -543,8 +554,12 @@ as long as it keeps the socket open.
This server speaks one protocol on purpose. There is no OpenAI-compatible endpoint; run several
questions in one request instead, since they share a single forward pass per question set.
`POST /v1/systemone/batch` answers one `questions` set over an array of
-`states`. It has no section on this page yet -- its request shape is in the README's self-hosting
-section -- and every check above applies to it as it does to `POST /v1/systemone`: the same shape
-`400`s, the same unpaired-surrogate refusal, the same auth, admission, size limits, body-control
-validation and `500` mapping. The `laya` CLI and MCP server cover local use -- see the
+`states`. Each state is its own context, which is what a caller needs for one decision per
+record; questions in one `/v1/systemone` request see the whole shared state, including when
+their text names `rows[i]`. It has no section on this page yet -- its request shape is in the
+README's self-hosting section -- and every check above applies to it as it does to
+`POST /v1/systemone`: the same shape `400`s, the same unpaired-surrogate refusal, the same auth,
+admission, size limits, body-control validation and `500` mapping. A state that does not fit
+the token room is truncated the same way, and a batch with any truncated result logs one
+warning for the call. The `laya` CLI and MCP server cover local use -- see the
[README](https://github.com/NandhaKishorM/laya#readme).
diff --git a/laya/serve.py b/laya/serve.py
index 0e848293..b9a3d52d 100644
--- a/laya/serve.py
+++ b/laya/serve.py
@@ -281,6 +281,100 @@ def _add_jev_confidence(result: Dict[str, Any]) -> Dict[str, Any]:
return {**result, "answers": answers}
+def _log_field(value: Any) -> str:
+ """Caller-influenced text folded onto one log line.
+
+ Question ids are request keys, and `truncated_questions` copies them. A newline in
+ one would split the warning into a second record -- the same log injection the
+ inference-failure line already closes for `model`.
+ """
+ return str(value).replace("\n", "\\n").replace("\r", "\\r")
+
+
+def _question_count(questions: Any) -> int:
+ return len(questions) if isinstance(questions, dict) else 0
+
+
+def _state_truncation(result: Any) -> Optional[Dict[str, Any]]:
+ """Dropped-token signal on one result, or None when usage does not report a cut.
+
+ `truncated` is a bool on a normal predict and a window count on `predict_long`, so the
+ check is `> 0` and not `is True` (a count of 2 is a cut; `False` is not). Bool is an
+ int in Python, and a fake that puts `True` in `state_tokens_dropped` must not be read
+ as one dropped token. A missing `usage`, a non-dict, or a value that is not a flag or
+ a count is "no report": the fakes this server's tests inject, and any router that
+ predates the truncation fields, succeed without them, and a bad report must not turn
+ that success into a 500 (#1425).
+ """
+ if not isinstance(result, dict):
+ return None
+ usage = result.get("usage")
+ if not isinstance(usage, dict):
+ return None
+ dropped = usage.get("state_tokens_dropped")
+ count = dropped if isinstance(dropped, int) and not isinstance(dropped, bool) and dropped > 0 else 0
+ flag = usage.get("truncated")
+ cut = flag is True or (isinstance(flag, int) and not isinstance(flag, bool) and flag > 0)
+ if not cut and count == 0:
+ return None
+ raw_ids = usage.get("truncated_questions")
+ ids = ([_log_field(qid) for qid in raw_ids if isinstance(qid, str) and qid]
+ if isinstance(raw_ids, list) else [])
+ return {"state_tokens_dropped": count, "truncated_questions": ids}
+
+
+def _warn_state_truncated(result: Any, *, route: str, model: Any, questions: Any) -> None:
+ """Warn once when one decision result dropped state tokens (#1425).
+
+ Call before `_project_jev_strict`: that projection removes `truncated` and
+ `state_tokens_dropped`, and the log is then the only operator signal. The line
+ carries the route, the resolved model, how many questions shared the state, and
+ the dropped-token count -- not the state text.
+ """
+ report = _state_truncation(result)
+ if report is None:
+ return
+ _log.warning(
+ "state truncated on %s model=%s questions=%d state_tokens_dropped=%d truncated_questions=%s",
+ route, _log_field(model), _question_count(questions),
+ report["state_tokens_dropped"], report["truncated_questions"],
+ )
+
+
+def _warn_batch_state_truncated(results: Any, *, model: Any, questions: Any) -> None:
+ """Warn once for a batch, aggregating every item whose state was cut (#1425).
+
+ One line for the request, not one per answer and not one per truncated item: the
+ indexes and the per-item dropped counts are the context an operator needs to find
+ the records that never reached the model. Same ordering rule as
+ `_warn_state_truncated`: read `usage` before strict projection strips it.
+ """
+ if not isinstance(results, list):
+ return
+ hits = []
+ for index, item in enumerate(results):
+ report = _state_truncation(item)
+ if report is not None:
+ hits.append((index, report))
+ if not hits:
+ return
+ question_ids = []
+ seen = set()
+ for _, report in hits:
+ for qid in report["truncated_questions"]:
+ if qid not in seen:
+ seen.add(qid)
+ question_ids.append(qid)
+ _log.warning(
+ "state truncated on /v1/systemone/batch model=%s questions=%d results=%d/%d "
+ "indices=%s state_tokens_dropped=%s truncated_questions=%s",
+ _log_field(model), _question_count(questions), len(hits), len(results),
+ [index for index, _ in hits],
+ [report["state_tokens_dropped"] for _, report in hits],
+ question_ids,
+ )
+
+
def _tournament_plan(questions: Any, group: int) -> Optional[Dict[str, List[str]]]:
"""The choices to narrow by elimination, as `{qid: labels}`, or None to answer in one pass.
@@ -1554,6 +1648,9 @@ def create_app(router: Optional[Any] = None):
run = partial(router.predict, state, questions, model=model, **predict_kwargs)
result = await loop.run_in_executor(pool, _run_inference, run)
infer_ms = (time.perf_counter() - t0) * 1000.0
+ # Log while `usage` still carries the truncation report. `_project_jev_strict`
+ # drops it, and a strict client then has no other signal (#1425).
+ _warn_state_truncated(result, route="/v1/systemone", model=model, questions=questions)
if _env_bool("LAYA_JEV_STRICT", False):
result = _project_jev_strict(result)
else:
@@ -1707,6 +1804,11 @@ def create_app(router: Optional[Any] = None):
t0 = time.perf_counter()
batch_res = await loop.run_in_executor(pool, _run_inference, _do_batch)
infer_ms = (time.perf_counter() - t0) * 1000.0
+ # Same ordering as `/v1/systemone`: one aggregate warning, read off the
+ # items before strict projection removes their truncation fields (#1425).
+ _warn_batch_state_truncated(
+ batch_res.get("results") if isinstance(batch_res, dict) else None,
+ model=model, questions=questions)
if _env_bool("LAYA_JEV_STRICT", False):
batch_res["results"] = [_project_jev_strict(item) for item in batch_res["results"]]
else:
diff --git a/tests/test_serve.py b/tests/test_serve.py
index 774bb1c7..7e568038 100644
--- a/tests/test_serve.py
+++ b/tests/test_serve.py
@@ -280,6 +280,198 @@ def test_jev_strict_leaves_an_unknown_answer_shape_unchanged(monkeypatch):
assert "routing" not in body
+# A cut state is silent in the answer. The report lives on `usage`, and strict mode removes it,
+# so the server log is what an operator can alert on (#1425). These blocks are what `Agent`
+# publishes: `truncated` false with nothing dropped, and the same shape once tokens were lost.
+INTACT_USAGE = {"input_tokens": 42, "output_tokens": 0, "state_tokens": 12,
+ "state_tokens_dropped": 0, "truncated": False, "truncated_questions": []}
+CUT_USAGE = {"input_tokens": 42, "output_tokens": 0, "state_tokens": 935,
+ "state_tokens_dropped": 587, "truncated": True,
+ "truncated_questions": ["r0", "r1"]}
+
+
+class ScriptedUsageRouter(FakeRouter):
+ """FakeRouter that replaces `usage` from a per-call queue.
+
+ `None` drops the key. A state the queue does not cover keeps the base
+ `{input_tokens, output_tokens}` payload the other server tests assert.
+ """
+
+ def __init__(self, usages):
+ super().__init__()
+ self._usages = list(usages)
+
+ def predict(self, state, questions, model=None):
+ result = super().predict(state, questions, model=model)
+ if self._usages:
+ usage = self._usages.pop(0)
+ if usage is None:
+ result.pop("usage", None)
+ else:
+ result["usage"] = usage
+ return result
+
+ def predict_batch(self, requests, **kwargs):
+ return [self.predict(item["state"], item["questions"], model=item.get("model"))
+ for item in requests]
+
+
+def _state_truncated_logs(caplog):
+ return [r for r in caplog.records
+ if r.name == "laya.serve" and r.levelno == logging.WARNING
+ and "state truncated" in r.getMessage()]
+
+
+def test_truncated_state_is_logged_on_systemone(monkeypatch, caplog):
+ """A result that dropped state tokens warns once, and the JSON payload stays the result's."""
+ monkeypatch.delenv("LAYA_API_KEY", raising=False)
+ monkeypatch.delenv("LAYA_JEV_STRICT", raising=False)
+ client = TestClient(create_app(router=ScriptedUsageRouter([dict(CUT_USAGE)])))
+
+ with caplog.at_level(logging.WARNING, logger="laya.serve"):
+ response = client.post("/v1/systemone", json=REQ)
+
+ assert response.status_code == 200, response.text
+ body = response.json()
+ assert set(body) == {"model", "answers", "usage", "routing"}
+ assert body["usage"]["truncated"] is True
+ assert body["usage"]["state_tokens_dropped"] == 587
+ assert body["usage"]["truncated_questions"] == ["r0", "r1"]
+ assert body["answers"]["dept"]["choice"] == "billing"
+ logs = _state_truncated_logs(caplog)
+ assert len(logs) == 1, [r.getMessage() for r in caplog.records]
+ message = logs[0].getMessage()
+ assert message.startswith("state truncated on /v1/systemone ")
+ assert "state_tokens_dropped=587" in message
+ assert "r0" in message and "r1" in message
+ # two question ids, one line: not one warning per answer
+ assert message.count("state truncated") == 1
+
+
+def test_intact_responses_do_not_log_truncation(monkeypatch, caplog):
+ """An uncut result keeps today's payload and does not warn, on either decision route."""
+ monkeypatch.delenv("LAYA_JEV_STRICT", raising=False)
+ client, _ = _client(monkeypatch)
+ with caplog.at_level(logging.WARNING, logger="laya.serve"):
+ single = client.post("/v1/systemone", json=REQ)
+ batch = client.post("/v1/systemone/batch", json={"states": ["one", "two"],
+ "questions": REQ["questions"]})
+ assert single.status_code == 200 and batch.status_code == 200
+ assert single.json()["usage"] == {"input_tokens": 42, "output_tokens": 0}
+ assert [item["usage"] for item in batch.json()["results"]] == [
+ {"input_tokens": 42, "output_tokens": 0},
+ {"input_tokens": 42, "output_tokens": 0},
+ ]
+ assert batch.json()["total_usage"] == {"input_tokens": 84, "output_tokens": 0}
+ assert _state_truncated_logs(caplog) == []
+
+ # The fields the agent actually sends, when they say nothing was cut.
+ caplog.clear()
+ explicit = TestClient(create_app(router=ScriptedUsageRouter(
+ [dict(INTACT_USAGE), dict(INTACT_USAGE), dict(INTACT_USAGE)])))
+ with caplog.at_level(logging.WARNING, logger="laya.serve"):
+ kept = explicit.post("/v1/systemone", json=REQ)
+ kept_batch = explicit.post("/v1/systemone/batch", json={"states": ["one", "two"],
+ "questions": REQ["questions"]})
+ assert kept.status_code == 200 and kept_batch.status_code == 200
+ assert kept.json()["usage"]["truncated"] is False
+ assert kept.json()["usage"]["state_tokens_dropped"] == 0
+ assert all(item["usage"] == INTACT_USAGE for item in kept_batch.json()["results"])
+ assert _state_truncated_logs(caplog) == []
+
+
+def test_batch_logs_mixed_truncation_once(monkeypatch, caplog):
+ """One truncated item and one intact item produce one aggregate warning, not one per answer."""
+ monkeypatch.delenv("LAYA_API_KEY", raising=False)
+ monkeypatch.delenv("LAYA_JEV_STRICT", raising=False)
+ client = TestClient(create_app(router=ScriptedUsageRouter([dict(CUT_USAGE), dict(INTACT_USAGE)])))
+
+ with caplog.at_level(logging.WARNING, logger="laya.serve"):
+ response = client.post("/v1/systemone/batch", json={"states": ["cut", "kept"],
+ "questions": REQ["questions"]})
+
+ assert response.status_code == 200, response.text
+ data = response.json()
+ assert data["results"][0]["usage"]["truncated"] is True
+ assert data["results"][0]["usage"]["state_tokens_dropped"] == 587
+ assert data["results"][1]["usage"]["truncated"] is False
+ assert data["results"][1]["usage"]["state_tokens_dropped"] == 0
+ assert data["total_usage"] == {"input_tokens": 84, "output_tokens": 0}
+ logs = _state_truncated_logs(caplog)
+ assert len(logs) == 1, [r.getMessage() for r in caplog.records]
+ message = logs[0].getMessage()
+ assert message.startswith("state truncated on /v1/systemone/batch ")
+ assert "results=1/2" in message
+ assert "indices=[0]" in message
+ assert "state_tokens_dropped=[587]" in message
+ assert "r0" in message and "r1" in message
+
+
+def test_strict_mode_logs_truncation_before_projection(monkeypatch, caplog):
+ """The warning is read before LAYA_JEV_STRICT strips the truncation fields from the body."""
+ monkeypatch.delenv("LAYA_API_KEY", raising=False)
+ monkeypatch.setenv("LAYA_JEV_STRICT", "1")
+ client = TestClient(create_app(router=ScriptedUsageRouter([dict(CUT_USAGE), dict(CUT_USAGE), dict(INTACT_USAGE)])))
+
+ with caplog.at_level(logging.WARNING, logger="laya.serve"):
+ single = client.post("/v1/systemone", json=REQ)
+ assert single.status_code == 200, single.text
+ body = single.json()
+ assert set(body) == {"model", "answers", "usage"}
+ assert body["usage"] == {"input_tokens": 42, "output_tokens": 0}
+ assert "truncated" not in body["usage"]
+ assert "x_jev_confidence" not in body["answers"]["dept"]
+ logs = _state_truncated_logs(caplog)
+ assert len(logs) == 1, [r.getMessage() for r in caplog.records]
+ assert "state_tokens_dropped=587" in logs[0].getMessage()
+ assert "r0" in logs[0].getMessage() and "r1" in logs[0].getMessage()
+
+ caplog.clear()
+ with caplog.at_level(logging.WARNING, logger="laya.serve"):
+ batch = client.post("/v1/systemone/batch", json={"states": ["cut", "kept"],
+ "questions": REQ["questions"]})
+ assert batch.status_code == 200, batch.text
+ data = batch.json()
+ assert [item["usage"] for item in data["results"]] == [
+ {"input_tokens": 42, "output_tokens": 0},
+ {"input_tokens": 42, "output_tokens": 0},
+ ]
+ assert all("routing" not in item for item in data["results"])
+ assert data["total_usage"] == {"input_tokens": 84, "output_tokens": 0}
+ logs = _state_truncated_logs(caplog)
+ assert len(logs) == 1, [r.getMessage() for r in caplog.records]
+ message = logs[0].getMessage()
+ assert "results=1/2" in message
+ assert "indices=[0]" in message
+ assert "state_tokens_dropped=[587]" in message
+ assert "state_tokens_dropped" not in batch.text
+
+
[email protected]("usage", [None, {"input_tokens": 7, "output_tokens": 0,
+ "truncated": "yes", "state_tokens_dropped": "many"}])
+def test_missing_or_malformed_usage_stays_a_success(monkeypatch, caplog, usage):
+ """A router that omits the truncation report, or fills it with junk, still answers 200."""
+ monkeypatch.delenv("LAYA_API_KEY", raising=False)
+ monkeypatch.delenv("LAYA_JEV_STRICT", raising=False)
+ client = TestClient(create_app(router=ScriptedUsageRouter([usage, usage, usage])))
+
+ with caplog.at_level(logging.WARNING, logger="laya.serve"):
+ single = client.post("/v1/systemone", json=REQ)
+ batch = client.post("/v1/systemone/batch", json={"states": ["one", "two"],
+ "questions": REQ["questions"]})
+
+ assert single.status_code == 200, single.text
+ assert batch.status_code == 200, batch.text
+ if usage is None:
+ assert "usage" not in single.json()
+ assert all("usage" not in item for item in batch.json()["results"])
+ assert batch.json()["total_usage"] == {"input_tokens": 0, "output_tokens": 0}
+ else:
+ assert single.json()["usage"] == usage
+ assert [item["usage"] for item in batch.json()["results"]] == [usage, usage]
+ assert _state_truncated_logs(caplog) == []
+
+
def test_choice_and_score_answers_carry_jev_confidence(monkeypatch):
"""Each choice and score answer reports Jev's `confidence` beside Laya's own (#302)."""
client, _ = _strict_client(monkeypatch, None)