Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(evals): name every field the question fingerprint hashes
Aashishf91afdfcdf0d
diff --git a/docs/evals.md b/docs/evals.mdindex d63bb027..2900a9f0 100644--- a/docs/evals.md+++ b/docs/evals.md@@ -455,7 +455,7 @@ is reviewable on its own:| `schema` | the report shape, `laya-evals-report/1`, so a consumer can refuse one it cannot read || `dataset` | the path as typed -- a name, not a hash || `dataset_sha256` | the sha256 of the dataset bytes that were parsed |-| `questions_sha256` | a fingerprint of the question schema: every question's id, type, `instructions` and `criteria`, over the whole dataset |+| `questions_sha256` | a fingerprint of the question schema: every question's id, type, `instructions`, `labels` and `criteria`, plus `option_order` on any row that names it, over the whole dataset || `laya_version` | the `laya` that computed the numbers || `coverage_metric_definition` | which definition of `aurc` / `selective_accuracy@*` produced this report (see [coverage cuts](#coverage-cuts-and-ties)). A relative gate rule on either refuses a baseline recorded under a different one, rather than subtracting numbers that do not mean the same thing || `thresholds` | the gate this run applied: `min`, `max` and `baseline_tolerance` |@@ -487,6 +487,17 @@ a copy edit is treated as a new experiment. That is the honest default -- the alsimilarity heuristic standing between a run and its baseline, and no evaluation system in commonuse has one.+`option_order` is the one field hashed conditionally, and only when a row names it. It is a+caller-supplied key: `laya/mcp/server.py` advertises it, `Agent._check_question` and+`laya/mcp/tools.py::_check_option_order` validate it as a permutation of the option indices, and+`laya/common.py::build_sequence` and `unpermute_probs` read it to place each answer slot, so a row+that permutes its options asks a different question and two such datasets must not share a+fingerprint. `research/eval/metamorphic.py::permute_options` measures the same sensitivity from the+other side -- it exists to shuffle the option list and check that the answers follow. Hashing an+absent key as `null` would move the digest of every dataset that never uses the field, so an absent+key stays out of the hash and such a dataset keeps the fingerprint its committed baseline already+carries.+Nothing time-bearing is recorded, so a report is still byte-reproducible for a fixed runner.`REPORT_SCHEMA`, `questions_fingerprint(dataset)` and `file_fingerprint(path)` are public, so adiff --git a/laya/evals.py b/laya/evals.pyindex b33a9431..29dafc8a 100644--- a/laya/evals.py+++ b/laya/evals.py@@ -224,10 +224,10 @@ def questions_fingerprint(dataset: "Dataset") -> str:`option_order` is the fifth and last field a question body can carry, and it was the one leftout: `laya/agent.py::Agent._check_question` accepts only a permutation of the option indices,`Agent._to_internal` copies it into the internal question, and from there it reaches both the- tokenized head (`laya/common.py::build_sequence` via `build_head`) and the reported- probabilities (`laya/agent.py`'s `unpermute_probs` call). It is hashed only when a row names it,- so the digest of a question set that does not use the field stays the number its committed- baseline already carries.+ tokenized head (`laya/common.py::build_sequence` through `build_head`) and the reported+ probabilities (`laya/common.py::unpermute_probs`, applied in `Agent._decode_answers`). It is+ hashed only when a row names it, so the digest of a question set that does not use the field+ stays the number its committed baseline already carries."""schemas = set()for example in dataset.examples: