Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

Merge pull request #382 from emiliano-go/feat/eval-ci

Nandakishor2d23df4354ba
diff --cc README.md
index 692960fb,cd55ee23..f80b6a2b
--- a/README.md
+++ b/README.md
@@@ -764,9 -764,29 +764,29 @@@ text generation. Tests: `tests/test_mcp
---
+ ## Evaluation harness
+
+ `laya.evals` scores a labelled dataset and gates a build on it, so a quality change is a
+ reviewable diff instead of a hand-check. It is pure Python plus numpy, imports no torch, and
+ needs no weights until you point it at a checkpoint.
+
+ ```bash
+ laya-evals validate research/evals/fixture.jsonl
+ laya-evals run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language
+ laya-evals run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json report.json
+ ```
+
+ `run` reports overall and per-slice metrics (`choice_accuracy`, `noul_accuracy`, `score_mae`,
+ `ece`, `mean_confidence`, latency) and exits non-zero on a threshold or baseline failure, so it
+ drops into CI unchanged. A weight-free job runs the metric and API tests on every PR, and a
+ scheduled workflow evaluates the English checkpoint against the committed baseline. See
+ [**`docs/evals.md`**](docs/evals.md) for the dataset format and the gate.
+
+ ---
+
## Benchmarks
-Community diagnostic: [Chinese workplace decisions (Feishu-style)](research/benchmarks/feishu_zh/README.md) · [中文说明](research/benchmarks/feishu_zh/README.zh-CN.md). Includes frozen synthetic cases, archived paired Laya/Jev responses, and an offline audit; separate from the benchmark suites below.
+Community diagnostics: [Chinese workplace decisions (Feishu-style)](research/benchmarks/feishu_zh/README.md) · [中文说明](research/benchmarks/feishu_zh/README.zh-CN.md). Includes frozen synthetic cases, archived paired Laya/Jev responses, and an offline audit; separate from the benchmark suites below. Also [Chinese short-command routing](research/benchmarks/zh_short_commands/README.md) · [中文说明](research/benchmarks/zh_short_commands/README.zh-CN.md): 18 frozen commands and a seven-rung ablation of the documented prompt guidance, which locates the accuracy loss on the four-question path rather than the six-option one.
**Full report: [`BENCHMARKS.md`](BENCHMARKS.md)** — every run consolidated, languages and themes, with per-language detail for all 51 languages.
diff --cc docs/index.md
index bf182506,215324e8..5af631e4
--- a/docs/index.md
+++ b/docs/index.md
@@@ -25,8 -24,8 +25,9 @@@ is one copy to keep current
| run the SDK or the `laya-serve` HTTP API in a container, on CPU or an NVIDIA GPU | [Docker quickstart](docker.md) |
| build for ARM64 hosts or DGX Spark | [ARM64 and DGX Spark containers](docker-platforms.md) |
| specialise a checkpoint for your own decisions | [Browser-agent fine-tuning example](finetune_browser_agent.md) and the [fine-tuning notebook](https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb) |
+| judge benchmark results and deployment limits | [Benchmarks and known limits](benchmarks.md) |
| look up a class, function or parameter | [Python API reference](reference/index.md) |
+ | score a labelled dataset, compare to a baseline, or gate a build on it | [Evaluation harness](evals.md) |
-Routing, the HTTP API, the command line, the MCP server, confidence gating, benchmarks and known
-limits are in the [README](https://github.com/NandhaKishorM/laya#readme) for now.
+Routing, the HTTP API, the command line, the MCP server and confidence gating are in the
+[README](https://github.com/NandhaKishorM/laya#readme) for now.