Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(benchmarks): add guide to published results and limits
rahul05ranjan14750bc579cc
diff --git a/docs/benchmarks.md b/docs/benchmarks.mdnew file mode 100644index 00000000..05485511--- /dev/null+++ b/docs/benchmarks.md@@ -0,0 +1,78 @@+# Benchmarks and known limits++Laya's results depend on the checkpoint, task, question wording, option count and hardware. Use+the [full benchmark tables](https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md)+to find a comparable run before choosing a checkpoint or a confidence threshold. The+[research directory](https://github.com/NandhaKishorM/laya/tree/main/research) holds the scripts+and result files behind the main tables.++## Find the relevant measurement++| If you need to assess | Start with | Check before applying the result |+|---|---|---|+| Decisions across languages | The MASSIVE and XNLI tables in `BENCHMARKS.md` | Language, task, number of options and checkpoint |+| A workflow such as triage or moderation | The application-workflow table in `BENCHMARKS.md` | Whether the dataset was in the training mix or held out |+| The `laya-typed-decisions` checkpoint | The typed-decisions table in `BENCHMARKS.md` | It was fine-tuned on that benchmark's training split; the base checkpoints have separate rows |+| Response time | The T4, GB10, laptop CPU and server CPU sections in `BENCHMARKS.md` | Device, batch size, number of questions, warm-up and whether HTTP time is included |++The published Jev figures alongside the original Laya suites come from third-party studies+with different prompts and sample sizes. They are useful context, but are not a controlled+head-to-head run. See the comparison notes in+[`research/README.md`](https://github.com/NandhaKishorM/laya/blob/main/research/README.md).++Read accuracy together with the baseline and data split. For example, the typed-decisions+benchmark reports 0.766 accuracy for the fine-tuned checkpoint, while both base checkpoints+score below its 0.461 majority-class baseline. That result supports fine-tuning for a similar+task; it does not establish 0.766 accuracy for an untrained checkpoint or a new domain.++Read confidence separately from accuracy. Expected calibration error (ECE) measures how well+reported probabilities match observed correctness; lower is better. The 51-language sweep's+original ECE and mean-confidence columns predate the temperature clamp in #42. Its accuracy+columns still apply, but use the clamped rerun in `research/results/` when comparing current+confidence values. Even a lower ECE on one suite does not set a safe threshold for another task+or option count.++## Limits to check on your own data++- **Language routing:** The English checkpoint can be confident on text it handles badly outside+ English. Use `Router` for mixed-language inputs and check routing decisions on the languages+ you serve. The multilingual checkpoint also scores below the English checkpoint on the English+ MASSIVE and XNLI slices.+- **Many options:** Choice descriptions share a fixed token budget. The 77-label Banking77 run+ performs poorly at the default budget. Keep a single choice question to roughly 20 options,+ or evaluate a shortlist and a larger head budget on your own labels.+- **Calibration:** Both base checkpoints are over-confident on the published suites as shipped,+ yet a separate routing task was under-confident. Fit and evaluate temperatures on separate,+ held-out examples from your workflow before using a confidence gate.+- **Task transfer:** Held-out moderation is weak in the application benchmark, and ordinal+ `score` is the weakest primitive in the reported English suites. The multilingual checkpoint+ also has a measured bias against the first `score` level. Test the actual question type and+ data distribution you intend to serve.+- **Wording and order:** Option order can change a `choice` answer. Boolean-word choice labels+ and negated requests have also failed in documented examples; `noul` can follow its option+ labels instead of the state. Check alternate option orders and wording, especially when a wrong+ decision is costly.+- **Long documents:** The multilingual encoder can read up to 8,192 tokens when configured for+ that limit, but the [long-context benchmark](https://github.com/NandhaKishorM/laya/blob/main/research/results/long_context_multilingual.json)+ reports less reliable answers beyond about 4,000 tokens of preceding text. Measure accuracy at+ the lengths you expect in use.+- **Latency:** The T4 figures do not predict CPU or cold-load time. Measure warm and cold calls+ with your own checkpoint, device, input lengths and number of questions.++The [README's Honest limits section](https://github.com/NandhaKishorM/laya#honest-limits)+has examples and current workarounds. The benchmark tables give the dataset and hardware+behind each of the limits above.++## Reproduce or extend a result++Start with the [script and result map](https://github.com/NandhaKishorM/laya/blob/main/research/README.md)+and the run index at the top of `BENCHMARKS.md`. `research/scripts/bench_local.py` runs the+51-language CPU sweep, `bench_apps.py` covers application workflows, and+`bench_latency.py` measures routing and inference speed. The T4 notebook is generated from+`research/scripts/build_benchmark_nb.py`; edit the generator when changing that benchmark.++For a new deployment, keep a held-out set with the same states, questions and expected answers+for every checkpoint you compare. Record the checkpoint revision, Laya and library versions,+device, question count, option count and token budget with each run. Include a simple baseline+for accuracy and report latency after warm-up as well as first-use load time. This makes your+result comparable to the published runs and lets you revisit it after an upgrade.diff --git a/docs/index.md b/docs/index.mdindex 4c8228c4..d6f8742b 100644--- a/docs/index.md+++ b/docs/index.md@@ -24,7 +24,8 @@ is one copy to keep current.| run the SDK or the `laya-serve` HTTP API in a container, on CPU or an NVIDIA GPU | [Docker quickstart](docker.md) || build for ARM64 hosts or DGX Spark | [ARM64 and DGX Spark containers](docker-platforms.md) || specialise a checkpoint for your own decisions | [Browser-agent fine-tuning example](finetune_browser_agent.md) and the [fine-tuning notebook](https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb) |+| judge benchmark results and deployment limits | [Benchmarks and known limits](benchmarks.md) || look up a class, function or parameter | [Python API reference](reference/index.md) |-Routing, the HTTP API, the command line, the MCP server, confidence gating, benchmarks and known-limits are in the [README](https://github.com/NandhaKishorM/laya#readme) for now.+Routing, the HTTP API, the command line, the MCP server and confidence gating are in the+[README](https://github.com/NandhaKishorM/laya#readme) for now.