Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(structured): document decide_batch with measured batch speedups

Aashisha07400d9588d
diff --git a/README.md b/README.md
index 54c20cf9..ebc45751 100644
--- a/README.md
+++ b/README.md
@@ -686,6 +686,25 @@ agent.decide("I was charged twice, refund me.", schema=schema)
`decide` also works on a `Router`, accepts a pydantic model (install `laya[structured]`), and can
return per-field confidence with `return_details=True`. See [`docs/structured.md`](docs/structured.md).
+For throughput, `decide_batch` answers a list of states against the *same* schema through
+`predict_batch` (see [Batch Mode](#batch-mode-score-many-states-in-one-forward-pass)), so the schema
+is planned once and the states share forward passes:
+
+```python
+values = agent.decide_batch(ticket_texts, schema=schema) # values[i] matches ticket_texts[i]
+
+# On a Router the states may land on different checkpoints; keywords reach predict_batch:
+details = router.decide_batch(states, schema=schema, return_details=True, batch_size=64)
+```
+
+Measured on an Apple M-series (MPS), 8 English tickets through one checkpoint: 2624 ms one-by-one
+vs 723 ms batched (**3.6×**); 16 mixed English/German states through a `Router`: 2977 ms vs
+1903 ms (**1.6×**). On CPU the same workloads gave **1.6×** and **2.0×**. Projected values matched
+the one-by-one loop 8/8 and 16/16 on both devices, as expected wherever argmax is not at a
+threshold — verify against your own decision boundaries. `laya.decide_batch(runner, states, ...)`
+is the function form. `ONNXAgent` has no `predict_batch` yet, so it raises `TypeError` instead of
+silently looping.
+
---
## Built-in Workflow Presets
diff --git a/docs/structured.md b/docs/structured.md
index 97a3d3c2..37d75199 100644
--- a/docs/structured.md
+++ b/docs/structured.md
@@ -75,6 +75,7 @@ Limits: `MAX_PROPERTIES = 32`, `MAX_OPTIONS = 32`, `MAX_SCORE_LEVELS = 10`.
|---|---|
| `laya.decide(runner, state, schema=..., *, questions=..., return_details=..., **predict_kwargs)` | the free function, works for `Agent` and `Router` |
| `agent.decide(state, schema=..., ...)` / `router.decide(state, schema=..., ...)` | convenience methods |
+| `laya.decide_batch(runner, states, schema=..., ...)` / `runner.decide_batch(...)` | the same over many states, one batched call |
| `questions_from_json_schema(schema)` | schema to Laya questions |
| `questions_from_pydantic(model)` | pydantic model to questions (requires pydantic) |
| `answers_to_json(answers, schema)` | project raw answers onto schema values |
@@ -89,6 +90,24 @@ instead of projecting. Extra keyword arguments are forwarded to `predict`, so ho
router.decide(state, schema=Ticket, model="multilingual", hooks=[Metrics()])
```
+## Scoring many states
+
+`decide_batch` is the throughput form: the schema is planned once and its questions run over every
+state through `predict_batch`, so states share forward passes instead of one per call. Results come
+back in input order, projected exactly as `decide` projects, and `return_details=True` gives one
+`DecisionResult` per state:
+
+```python
+values = agent.decide_batch(ticket_texts, schema=Ticket) # values[i] matches ticket_texts[i]
+results = router.decide_batch(states, schema=Ticket, return_details=True, batch_size=64)
+```
+
+On a `Router` each state is still routed on its own, so one call can span checkpoints. Keyword
+arguments reach `predict_batch`, so `batch_size=`, `model=` and hooks work as they do for `decide`.
+A runner without `predict_batch` (such as `ONNXAgent`) raises `TypeError` rather than silently
+falling back to a loop — call `decide` per state there. Batching can shift borderline argmaxes the
+same way `predict_batch` does; the README records the measured speedups for both devices.
+
## Confidence and probabilities
By default `decide` returns only the values. Pass `return_details=True` for a `DecisionResult`