Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
Merge pull request #382 from emiliano-go/feat/eval-ci
Nandakishor2d23df4354ba
diff --cc README.mdindex 692960fb,cd55ee23..f80b6a2b--- a/README.md+++ b/README.md@@@ -764,9 -764,29 +764,29 @@@ text generation. Tests: `tests/test_mcp---+ ## Evaluation harness++ `laya.evals` scores a labelled dataset and gates a build on it, so a quality change is a+ reviewable diff instead of a hand-check. It is pure Python plus numpy, imports no torch, and+ needs no weights until you point it at a checkpoint.++ ```bash+ laya-evals validate research/evals/fixture.jsonl+ laya-evals run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language+ laya-evals run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json report.json+ ```++ `run` reports overall and per-slice metrics (`choice_accuracy`, `noul_accuracy`, `score_mae`,+ `ece`, `mean_confidence`, latency) and exits non-zero on a threshold or baseline failure, so it+ drops into CI unchanged. A weight-free job runs the metric and API tests on every PR, and a+ scheduled workflow evaluates the English checkpoint against the committed baseline. See+ [**`docs/evals.md`**](docs/evals.md) for the dataset format and the gate.++ ---+## Benchmarks-Community diagnostic: [Chinese workplace decisions (Feishu-style)](research/benchmarks/feishu_zh/README.md) · [中文说明](research/benchmarks/feishu_zh/README.zh-CN.md). Includes frozen synthetic cases, archived paired Laya/Jev responses, and an offline audit; separate from the benchmark suites below.+Community diagnostics: [Chinese workplace decisions (Feishu-style)](research/benchmarks/feishu_zh/README.md) · [中文说明](research/benchmarks/feishu_zh/README.zh-CN.md). Includes frozen synthetic cases, archived paired Laya/Jev responses, and an offline audit; separate from the benchmark suites below. Also [Chinese short-command routing](research/benchmarks/zh_short_commands/README.md) · [中文说明](research/benchmarks/zh_short_commands/README.zh-CN.md): 18 frozen commands and a seven-rung ablation of the documented prompt guidance, which locates the accuracy loss on the four-question path rather than the six-option one.**Full report: [`BENCHMARKS.md`](BENCHMARKS.md)** — every run consolidated, languages and themes, with per-language detail for all 51 languages.diff --cc docs/index.mdindex bf182506,215324e8..5af631e4--- a/docs/index.md+++ b/docs/index.md@@@ -25,8 -24,8 +25,9 @@@ is one copy to keep current| run the SDK or the `laya-serve` HTTP API in a container, on CPU or an NVIDIA GPU | [Docker quickstart](docker.md) || build for ARM64 hosts or DGX Spark | [ARM64 and DGX Spark containers](docker-platforms.md) || specialise a checkpoint for your own decisions | [Browser-agent fine-tuning example](finetune_browser_agent.md) and the [fine-tuning notebook](https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb) |+| judge benchmark results and deployment limits | [Benchmarks and known limits](benchmarks.md) || look up a class, function or parameter | [Python API reference](reference/index.md) |+ | score a labelled dataset, compare to a baseline, or gate a build on it | [Evaluation harness](evals.md) |-Routing, the HTTP API, the command line, the MCP server, confidence gating, benchmarks and known-limits are in the [README](https://github.com/NandhaKishorM/laya#readme) for now.+Routing, the HTTP API, the command line, the MCP server and confidence gating are in the+[README](https://github.com/NandhaKishorM/laya#readme) for now.