Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
research/evals
Datasets, baselines and thresholds for the laya-evals harness and its CI gate.
fixture.jsonl: a tiny hand-written set that exerciseschoice,scoreandnoulacross a few tags. It exists to prove the format and to run the harness in tests. It is not a quality claim and its accuracy has no meaning.dataset.template.jsonl: the format, with comments. Start here.thresholds.json: the tolerances the scheduled gate allows against the committed baselines inresearch/results/.check_regression.py: adapts aresearch/eval/laya_eval.pyreport intolaya.evals.EvalReportand compares it to a committed baseline.act_head_eval.py: scoresanswer.action.act_probabilityagainst labelled decisions in a report, as the bar a future act-head retraining has to beat. Pure arithmetic over a reportlaya-evals run --jsonalready wrote: no second model load, and no change to model, routing orlaya.evalscode. The current checkpoints never learned a useful act head, so this measures, it does not fix, and a number here is not a claim the signal is informative today.test_act_head_eval.py: weight-free tests for the above. Like theresearch/eval/test_*.pysuites, run it directly; CI does not collect it.
Format
One JSON object per line (JSONL). Blank lines and lines starting with # are ignored.
| field | required | meaning |
|---|---|---|
state |
yes | the text, email, ticket or JSON document to decide on |
questions |
yes | a Laya question dict, exactly as Router.predict accepts |
expected |
yes | ground truth keyed by question id: a label for choice, a number for score, true/false for noul |
tags |
no | strings to slice by |
language |
no | BCP-47-ish code, to slice by language |
model |
no | force a checkpoint for this row; --model overrides it |
Use
laya-evals validate research/evals/fixture.jsonl
laya-evals run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language
laya-evals run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json out.json
run exits non-zero when a threshold or a baseline tolerance fails, so it drops into CI
unchanged. laya eval ... is the same thing through the main CLI.
Adding a dataset: point --baseline at a report you have reviewed, keep the tolerances in
thresholds.json, and commit both beside the dataset, so a quality change is a reviewable
diff.
Act-head diagnostic
laya-evals run data.jsonl --model english --json report.json
python research/evals/act_head_eval.py report.json --markdown act.md
Answers one question: does act_probability rank correct decisions above incorrect ones?
It reports the scorable count, the correct/incorrect split, the tie-correct ROC-AUC of
act_probability, the same AUC for the calibrated confidence on exactly the same
decisions as a control, and the observed min/max/unique of act_probability so a signal
pinned at one value is visible rather than averaged away.
Cases are bucketed explicitly and nothing is folded into a denominator: a score answer has
no binary correct and is skipped with a count, a missing or non-numeric
act_probability is reported as missing rather than read as 0.0, and an AUC is null
with a reason when either class is absent, since ROC-AUC is undefined there.
The AUC is the Mann-Whitney U form with mid-ranks for ties, which agrees with
sklearn.metrics.roc_auc_score and needs only numpy. Ties are not a rounding detail here:
the current signal is constant, so a tie-correct implementation is the difference between
reporting a failure and reporting 0.5 as if it were a measurement.
The real labelled set
The maintainer's 396-decision English set and the 51-language sweep are the authoritative
numbers. They are not committed here yet; drop a JSONL in this directory and a baseline
report beside it and the gate will pick both up. The scheduled workflow currently runs the
MASSIVE English suite through research/eval/laya_eval.py against
research/results/eval_english_51_languages.json.