Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(evals): a threshold names its metric, and a noul number is not choice_accuracy

Aashishb426eede5bf3
diff --git a/laya/evals_cli.py b/laya/evals_cli.py
index de3210fe..56c3b74e 100644
--- a/laya/evals_cli.py
+++ b/laya/evals_cli.py
@@ -302,9 +302,15 @@ def _load_report(path: str) -> Dict[str, Any]:
def _check_thresholds(overall: Dict[str, float], mins: Dict[str, float],
maxs: Dict[str, float]) -> List[str]:
+ """Each limit is checked against the metric it names, and a metric absent from the report fails.
+
+ A `noul_accuracy` standing in for `choice_accuracy` here would gate a choice threshold on the
+ accuracy of a different question type, so a noul-only run could clear a `--min choice_accuracy`
+ it never answered -- `docs/evals.md` requires the missing-metric failure instead. `--min-accuracy`
+ is the flag that means "whichever accuracy this dataset has", and `_cmd_run` resolves it to the
+ key the report actually carries.
+ """
failures: List[str] = []
- if "choice_accuracy" not in overall and "noul_accuracy" in overall:
- overall = dict(overall, choice_accuracy=overall["noul_accuracy"])
for name, limit in mins.items():
value = overall.get(name)
if value is None: