Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(langchain): document runnable batching with measured speedups
Aashishb7fdaff19852
diff --git a/README.md b/README.mdindex 54c20cf9..3e3b1143 100644--- a/README.md+++ b/README.md@@ -729,8 +729,19 @@ workflow.add_conditional_edges("triage", router)# 2. Inline prompt guardrailsguard = LayaGuardrail(action="raise") # raises LayaGuardrailError on jailbreak/injection++# 3. Score a backlog in one batched call, not one forward pass per input+routes = router.batch(["refund my invoice", "the app crashes", "change my password"])```+`batch()` and `abatch()` run the whole list through `predict_batch`, so `chain.batch(...)`,+`RunnableParallel` and LangGraph map-reduce nodes get Laya's shared forward passes instead of+LangChain's default one-call-per-input thread pool -- which races on MPS, where concurrent torch+forwards abort the process. Measured on Apple M-series (medians of three): **2.2x** on a 16-ticket+routing batch, **2.2x** across 24 mixed-language tickets through a `Router`, **1.8x** on the guard+preset, with every route label and guardrail flag unchanged. On CPU the same workloads are+2.2-2.4x over the one-by-one loop and 1.1-1.5x over the thread pool.+See [**`docs/langchain.md`**](docs/langchain.md) for full guide, support ticket triage nodes, and remote HTTP server configuration.---diff --git a/docs/langchain.md b/docs/langchain.mdindex a5bd8fd3..d846ee8c 100644--- a/docs/langchain.md+++ b/docs/langchain.md@@ -176,3 +176,42 @@ router = LayaRouter(```No local PyTorch or checkpoint downloads are required in remote mode.++---++## 5. Batching Many Inputs++Every Laya runnable implements `batch()` on Laya's shared forward passes, so a backlog costs one+batched call instead of one pass per input. LangChain calls this for you from `chain.batch(...)`,+`RunnableParallel`, and LangGraph map-reduce; you can also call it directly:++```python+routes = router.batch(["refund my invoice", "the app crashes", "change my password"])+# ["billing", "technical", "account"] -- one call, outputs in input order++graded = asyncio.run(evaluator.abatch(predictions)) # the async entry point, same batch+```++The outputs are the same as calling `invoke` on each input in turn, including the confidence+fallback on `LayaRouter` and the `action` (`raise` / `filter` / `annotate`) on `LayaGuardrail`. Two+differences are worth knowing:++- With `action="raise"`, the first violating input raises, so the batch stops there. Pass+ `return_exceptions=True` to get one outcome per input, exceptions included.+- `batch()` shares one forward pass, so a failure fails the batch; that is also why+ `return_exceptions=True` falls back to the per-input loop.++Remote mode (`base_url`) keeps the per-request loop, because `laya-serve` answers one decision per+`POST`. A runner you supply yourself only needs `predict_batch` to take the fast path; without it+the runnable behaves like any other `Runnable`.++This matters most on MPS: LangChain's default `batch` runs `invoke` concurrently on a thread pool,+and concurrent PyTorch MPS forwards abort the process+(`failed assertion _status < MTLCommandBufferStatusCommitted`). One batched call has no such race.+Measured on an Apple M-series GPU with a 4-way routing question, medians of three runs. 16 English+tickets through an `Agent`: 1320 ms invoking one by one vs **598 ms** batched (**2.2x**); 24 mixed+English/German tickets through a `Router`: 1805 ms vs **814 ms** (**2.2x**); 16 tickets through the+guard `LayaGuardrail`: 4173 ms vs **2329 ms** (**1.8x**). Route labels and guardrail flags were+identical to the one-by-one loop in every run (0/16 and 0/24 changes). On CPU the same workloads are+2.2x to 2.4x over the one-by-one loop, but only 1.1x to 1.5x over the thread pool, which already+overlaps cores -- the MPS case is where `batch()` was not just slower but unusable.