Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
refactor(evals): name the console script laya-evals
Emiliano Gandini Outeda4097b7c58495
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 5e7f69ad..c6038a79 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -170,7 +170,7 @@ jobs:run: python tests/test_evals_api.py- name: Fixture dataset validates- run: laya-eval validate research/evals/fixture.jsonl+ run: laya-evals validate research/evals/fixture.jsonl# Every workflow in this repo ran on ubuntu-latest only, so a test that fails off Linux was# invisible until a user hit it -- which is what happened to tests/test_download.py (#140).diff --git a/README.md b/README.mdindex f29415c4..cd55ee23 100644--- a/README.md+++ b/README.md@@ -771,9 +771,9 @@ reviewable diff instead of a hand-check. It is pure Python plus numpy, imports nneeds no weights until you point it at a checkpoint.```bash-laya-eval validate research/evals/fixture.jsonl-laya-eval run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language-laya-eval run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json report.json+laya-evals validate research/evals/fixture.jsonl+laya-evals run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language+laya-evals run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json report.json````run` reports overall and per-slice metrics (`choice_accuracy`, `noul_accuracy`, `score_mae`,diff --git a/docs/evals.md b/docs/evals.mdindex 65855a0b..8c49967e 100644--- a/docs/evals.md+++ b/docs/evals.md@@ -11,15 +11,15 @@ takes its normal load time.```bash# check the format without a model-laya-eval validate research/evals/fixture.jsonl+laya-evals validate research/evals/fixture.jsonl# score a labelled set on one checkpoint, with thresholds and a baseline-laya-eval run data.jsonl --model english --device cpu \+laya-evals run data.jsonl --model english --device cpu \--min-accuracy 0.8 --max-ece 0.05 --slice language \--json report.json --markdown report.md# compare a saved report to a baseline-laya-eval compare report.json --baseline baseline.json --tolerance choice_accuracy=0.02+laya-evals compare report.json --baseline baseline.json --tolerance choice_accuracy=0.02````laya eval ...` is the same thing through the main CLI, so `laya eval validate data.jsonl`@@ -72,7 +72,7 @@ reading the aggregate.- Keep the dataset, a baseline report (`--json` output you have reviewed), and the tolerancestogether, committed, so a change is a reviewable diff. `--tolerance METRIC=VALUE` is themaximum absolute drift allowed for that metric.-- `laya-eval run ... --baseline baseline.json --tolerance ...` exits non-zero on drift, so it+- `laya-evals run ... --baseline baseline.json --tolerance ...` exits non-zero on drift, so itdrops into CI unchanged. `laya.evals.EvalReport.compare` and `assert_regression` expose thesame logic for tests.diff --git a/laya/cli.py b/laya/cli.pyindex a1789522..4beed490 100644--- a/laya/cli.py+++ b/laya/cli.py@@ -127,7 +127,7 @@ def interactive(args):def main(argv=None):argv = list(sys.argv[1:] if argv is None else argv)- if argv and argv[0] == "eval": # `laya eval ...` mirrors the `laya-eval` script+ if argv and argv[0] == "eval": # `laya eval ...` mirrors the `laya-evals` scriptfrom .evals_cli import main as eval_mainreturn eval_main(argv[1:])args = build_parser().parse_args(argv)diff --git a/laya/evals_cli.py b/laya/evals_cli.pyindex 49d7dd2b..e656eba6 100644--- a/laya/evals_cli.py+++ b/laya/evals_cli.py@@ -1,10 +1,10 @@-"""`laya-eval`: run a labelled evaluation, validate a dataset, or compare a report.+"""`laya-evals`: run a labelled evaluation, validate a dataset, or compare a report.Exit codes: 0 on success, 1 when a threshold or a baseline tolerance fails, 2 on a usage error.- laya-eval validate research/evals/fixture.jsonl- laya-eval run research/evals/fixture.jsonl --model english --min-accuracy 0.8 --max-ece 0.1- laya-eval run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json out.json+ laya-evals validate research/evals/fixture.jsonl+ laya-evals run research/evals/fixture.jsonl --model english --min-accuracy 0.8 --max-ece 0.1+ laya-evals run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json out.json`laya eval ...` dispatches here from the main CLI, so both spellings work."""@@ -48,7 +48,7 @@ def _parse_pairs(pairs: Optional[Sequence[str]]) -> Dict[str, float]:def _build_parser() -> argparse.ArgumentParser:- parser = argparse.ArgumentParser(prog="laya-eval",+ parser = argparse.ArgumentParser(prog="laya-evals",description="Evaluate a Laya checkpoint on a labelled dataset.")sub = parser.add_subparsers(dest="command", required=True)@@ -186,7 +186,7 @@ def main(argv: Optional[Sequence[str]] = None) -> int:return _cmd_run(args)return _cmd_compare(args)except EvalError as exc:- print("laya-eval: %s" % exc, file=sys.stderr)+ print("laya-evals: %s" % exc, file=sys.stderr)return 1diff --git a/pyproject.toml b/pyproject.tomlindex a40fad90..27b66880 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -67,7 +67,7 @@ langgraph = [[project.scripts]laya = "laya.cli:main"laya-serve = "laya.serve:main"-laya-eval = "laya.evals_cli:main"+laya-evals = "laya.evals_cli:main"laya-mcp-server = "laya.mcp.server:main"[project.urls]diff --git a/research/README.md b/research/README.mdindex 3bf63af7..dfc32401 100644--- a/research/README.md+++ b/research/README.md@@ -8,7 +8,7 @@ the numbers quoted in the main README — nothing here is imported by the `laya`- [`eval/`](eval/README.md): the independent per-language MASSIVE harness (`laya_eval.py`), the per-casereport behind the published numbers, plus the metamorphic and presentation checks.- [`evals/`](evals/README.md): the labelled datasets, thresholds and regression gate consumed by- `laya.evals` / `laya-eval` and the scheduled `evals` workflow.+ `laya.evals` / `laya-evals` and the scheduled `evals` workflow.## Community diagnosticsdiff --git a/research/evals/README.md b/research/evals/README.mdindex 16f65593..62790bc3 100644--- a/research/evals/README.md+++ b/research/evals/README.md@@ -1,6 +1,6 @@# research/evals-Datasets, baselines and thresholds for the `laya-eval` harness and its CI gate.+Datasets, baselines and thresholds for the `laya-evals` harness and its CI gate.- `fixture.jsonl`: a tiny hand-written set that exercises `choice`, `score` and `noul`across a few tags. It exists to prove the format and to run the harness in tests. It is@@ -27,9 +27,9 @@ One JSON object per line (JSONL). Blank lines and lines starting with `#` are ig## Use```bash-laya-eval validate research/evals/fixture.jsonl-laya-eval run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language-laya-eval run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json out.json+laya-evals validate research/evals/fixture.jsonl+laya-evals run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language+laya-evals run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json out.json````run` exits non-zero when a threshold or a baseline tolerance fails, so it drops into CI