Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

refactor(evals): name the console script laya-evals

Emiliano Gandini Outeda4097b7c58495
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 5e7f69ad..c6038a79 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -170,7 +170,7 @@ jobs:
run: python tests/test_evals_api.py
- name: Fixture dataset validates
- run: laya-eval validate research/evals/fixture.jsonl
+ run: laya-evals validate research/evals/fixture.jsonl
# Every workflow in this repo ran on ubuntu-latest only, so a test that fails off Linux was
# invisible until a user hit it -- which is what happened to tests/test_download.py (#140).
diff --git a/README.md b/README.md
index f29415c4..cd55ee23 100644
--- a/README.md
+++ b/README.md
@@ -771,9 +771,9 @@ reviewable diff instead of a hand-check. It is pure Python plus numpy, imports n
needs no weights until you point it at a checkpoint.
```bash
-laya-eval validate research/evals/fixture.jsonl
-laya-eval run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language
-laya-eval run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json report.json
+laya-evals validate research/evals/fixture.jsonl
+laya-evals run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language
+laya-evals run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json report.json
```
`run` reports overall and per-slice metrics (`choice_accuracy`, `noul_accuracy`, `score_mae`,
diff --git a/docs/evals.md b/docs/evals.md
index 65855a0b..8c49967e 100644
--- a/docs/evals.md
+++ b/docs/evals.md
@@ -11,15 +11,15 @@ takes its normal load time.
```bash
# check the format without a model
-laya-eval validate research/evals/fixture.jsonl
+laya-evals validate research/evals/fixture.jsonl
# score a labelled set on one checkpoint, with thresholds and a baseline
-laya-eval run data.jsonl --model english --device cpu \
+laya-evals run data.jsonl --model english --device cpu \
--min-accuracy 0.8 --max-ece 0.05 --slice language \
--json report.json --markdown report.md
# compare a saved report to a baseline
-laya-eval compare report.json --baseline baseline.json --tolerance choice_accuracy=0.02
+laya-evals compare report.json --baseline baseline.json --tolerance choice_accuracy=0.02
```
`laya eval ...` is the same thing through the main CLI, so `laya eval validate data.jsonl`
@@ -72,7 +72,7 @@ reading the aggregate.
- Keep the dataset, a baseline report (`--json` output you have reviewed), and the tolerances
together, committed, so a change is a reviewable diff. `--tolerance METRIC=VALUE` is the
maximum absolute drift allowed for that metric.
-- `laya-eval run ... --baseline baseline.json --tolerance ...` exits non-zero on drift, so it
+- `laya-evals run ... --baseline baseline.json --tolerance ...` exits non-zero on drift, so it
drops into CI unchanged. `laya.evals.EvalReport.compare` and `assert_regression` expose the
same logic for tests.
diff --git a/laya/cli.py b/laya/cli.py
index a1789522..4beed490 100644
--- a/laya/cli.py
+++ b/laya/cli.py
@@ -127,7 +127,7 @@ def interactive(args):
def main(argv=None):
argv = list(sys.argv[1:] if argv is None else argv)
- if argv and argv[0] == "eval": # `laya eval ...` mirrors the `laya-eval` script
+ if argv and argv[0] == "eval": # `laya eval ...` mirrors the `laya-evals` script
from .evals_cli import main as eval_main
return eval_main(argv[1:])
args = build_parser().parse_args(argv)
diff --git a/laya/evals_cli.py b/laya/evals_cli.py
index 49d7dd2b..e656eba6 100644
--- a/laya/evals_cli.py
+++ b/laya/evals_cli.py
@@ -1,10 +1,10 @@
-"""`laya-eval`: run a labelled evaluation, validate a dataset, or compare a report.
+"""`laya-evals`: run a labelled evaluation, validate a dataset, or compare a report.
Exit codes: 0 on success, 1 when a threshold or a baseline tolerance fails, 2 on a usage error.
- laya-eval validate research/evals/fixture.jsonl
- laya-eval run research/evals/fixture.jsonl --model english --min-accuracy 0.8 --max-ece 0.1
- laya-eval run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json out.json
+ laya-evals validate research/evals/fixture.jsonl
+ laya-evals run research/evals/fixture.jsonl --model english --min-accuracy 0.8 --max-ece 0.1
+ laya-evals run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json out.json
`laya eval ...` dispatches here from the main CLI, so both spellings work.
"""
@@ -48,7 +48,7 @@ def _parse_pairs(pairs: Optional[Sequence[str]]) -> Dict[str, float]:
def _build_parser() -> argparse.ArgumentParser:
- parser = argparse.ArgumentParser(prog="laya-eval",
+ parser = argparse.ArgumentParser(prog="laya-evals",
description="Evaluate a Laya checkpoint on a labelled dataset.")
sub = parser.add_subparsers(dest="command", required=True)
@@ -186,7 +186,7 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
return _cmd_run(args)
return _cmd_compare(args)
except EvalError as exc:
- print("laya-eval: %s" % exc, file=sys.stderr)
+ print("laya-evals: %s" % exc, file=sys.stderr)
return 1
diff --git a/pyproject.toml b/pyproject.toml
index a40fad90..27b66880 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -67,7 +67,7 @@ langgraph = [
[project.scripts]
laya = "laya.cli:main"
laya-serve = "laya.serve:main"
-laya-eval = "laya.evals_cli:main"
+laya-evals = "laya.evals_cli:main"
laya-mcp-server = "laya.mcp.server:main"
[project.urls]
diff --git a/research/README.md b/research/README.md
index 3bf63af7..dfc32401 100644
--- a/research/README.md
+++ b/research/README.md
@@ -8,7 +8,7 @@ the numbers quoted in the main README — nothing here is imported by the `laya`
- [`eval/`](eval/README.md): the independent per-language MASSIVE harness (`laya_eval.py`), the per-case
report behind the published numbers, plus the metamorphic and presentation checks.
- [`evals/`](evals/README.md): the labelled datasets, thresholds and regression gate consumed by
- `laya.evals` / `laya-eval` and the scheduled `evals` workflow.
+ `laya.evals` / `laya-evals` and the scheduled `evals` workflow.
## Community diagnostics
diff --git a/research/evals/README.md b/research/evals/README.md
index 16f65593..62790bc3 100644
--- a/research/evals/README.md
+++ b/research/evals/README.md
@@ -1,6 +1,6 @@
# research/evals
-Datasets, baselines and thresholds for the `laya-eval` harness and its CI gate.
+Datasets, baselines and thresholds for the `laya-evals` harness and its CI gate.
- `fixture.jsonl`: a tiny hand-written set that exercises `choice`, `score` and `noul`
across a few tags. It exists to prove the format and to run the harness in tests. It is
@@ -27,9 +27,9 @@ One JSON object per line (JSONL). Blank lines and lines starting with `#` are ig
## Use
```bash
-laya-eval validate research/evals/fixture.jsonl
-laya-eval run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language
-laya-eval run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json out.json
+laya-evals validate research/evals/fixture.jsonl
+laya-evals run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language
+laya-evals run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json out.json
```
`run` exits non-zero when a threshold or a baseline tolerance fails, so it drops into CI