Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(langchain): when to widen the token budget, and when it costs you

Aashishbdc1d84253ca
diff --git a/README.md b/README.md
index 54c20cf9..8960d6a1 100644
--- a/README.md
+++ b/README.md
@@ -983,7 +983,7 @@ failure; it does not establish calibrated confidence.
* `laya` (English) defaults to 512 context (`head_max_len = 192`, ~320 tokens for state).
* `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`, ~768 tokens for state; mmBERT-base encoder supports up to 8,192 with RoPE).
At default settings, a 77-option question like Banking77 allocates only `(256 - 16) // 77` ≈ 3–4 tokens per label, which causes accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question:
- 1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct.
+ 1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. Both are also per-request: `predict(state, questions, head_max_len=512, max_len=1024)` widens one question without changing the agent for everyone else, and every LangChain node takes the same two arguments ([LangChain guide](docs/langchain.md)).
2. Or shortlist with embeddings and run one forward pass on the top `k` labels (`predict_shortlist`, example below). `predict` and `system_one` still score every criterion they are given.
3. Or split the label set yourself into a coarse question and a fine question.
diff --git a/docs/langchain.md b/docs/langchain.md
index a5bd8fd3..66c3ad48 100644
--- a/docs/langchain.md
+++ b/docs/langchain.md
@@ -176,3 +176,50 @@ router = LayaRouter(
```
No local PyTorch or checkpoint downloads are required in remote mode.
+
+---
+
+## 5. Widening the Token Budget for Many Options
+
+Every runnable takes `max_len` and `head_max_len`, the two per-request knobs the core API accepts.
+A choice question's options share the checkpoint's *option* budget -- `head_max_len`, 192 tokens on
+`laya` and 256 on `laya-multilingual` -- and each option carries its own description, so past
+roughly 20 options every label is trimmed to fit and similar labels start reaching the model as the
+same text. See the README's [Honest limits](https://github.com/NandhaKishorM/laya#honest-limits)
+for the same effect measured on Banking77.
+
+A routing node with many branches is the common case, and it had no way to ask for more room:
+
+```python
+router = LayaRouter(
+ criteria=queue_criteria, # 48 queues, each with a description
+ instructions="Which support queue owns this ticket?",
+ max_len=1024, # total window
+ head_max_len=512, # tokens shared by the option prompt
+)
+```
+
+Measured on `laya` (Apple silicon, one forward pass per state, scored on the chosen label) with
+queue labels a state names explicitly, so ground truth is exact. Each cell is the count over the
+full set, and all three repeats of every row gave the identical count:
+
+| Options | Default budget | `max_len=1024, head_max_len=512` |
+|---|---|---|
+| 24 | 24/24 | 20/24 |
+| 48 | 1/48 | 43/48 |
+| 72 | 1/72 | 63/72 |
+
+Both directions of that table matter. Past about 40 options the default budget collapses the
+decision, and widening it recovers most of it. Below that, widening it costs a few: at 24 options
+the labels already fit the default budget and four answers move. The docs do not claim to know why
+the wider collation changes those four -- it is enough that it can. That is why the two arguments
+are opt-in per node: set the knob to fix a question that does not fit, not to sharpen one that does.
+
+The same override applies to `LayaGuardrail`, `LayaTriage` and `LayaEvaluator`.
+It is per node, so a chain can give its wide routing step room while every other node keeps the
+checkpoint's defaults, which is the point of not raising `agent.cfg["head_max_len"]` process-wide.
+
+**Remote mode rejects it.** A budget override is applied by the local runner, and `laya-serve`
+neither accepts nor needs the field, so `max_len` or `head_max_len` on a node with a `base_url`
+raises `ValueError` instead of quietly doing nothing. Raise the budget on the server that runs
+inference.