Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(evals): hash option_order into the question fingerprint
Aashish14a4cce3a553
diff --git a/laya/evals.py b/laya/evals.pyindex 1506d35e..b33a9431 100644--- a/laya/evals.py+++ b/laya/evals.py@@ -201,7 +201,10 @@ def questions_fingerprint(dataset: "Dataset") -> str:Laya already applies to its own question identity: `laya/router.py::_question_schema` and thismodule's batch grouping (`laya/evals.py::evaluate`) both hash the whole questions dict.`tests/test_router_batch.py:543` pins that rewording `instructions` alone moves a row into its- own batch group.+ own batch group. Those two need nothing when a question grows a field, because a whole-dict hash+ takes whatever the dict holds; this one lists its fields, so a sixth field+ `laya/agent.py::Agent._check_question` starts reading has to be added here as well or the+ identity goes blind to it again.It is a function of the decision space, not of the rows, so scoring more states on the samequestions leaves it unchanged.@@ -217,15 +220,29 @@ def questions_fingerprint(dataset: "Dataset") -> str:the internal question, and that resolver is what turns it into the option text the modelreads. Leaving it out let two question sets with different rendered options hash alike, whichis the one thing this function exists to prevent.++ `option_order` is the fifth and last field a question body can carry, and it was the one left+ out: `laya/agent.py::Agent._check_question` accepts only a permutation of the option indices,+ `Agent._to_internal` copies it into the internal question, and from there it reaches both the+ tokenized head (`laya/common.py::build_sequence` via `build_head`) and the reported+ probabilities (`laya/agent.py`'s `unpermute_probs` call). It is hashed only when a row names it,+ so the digest of a question set that does not use the field stays the number its committed+ baseline already carries."""schemas = set()for example in dataset.examples:for qid, question in example.questions.items():body = question if isinstance(question, dict) else {}- schemas.add(_canonical({"qid": qid, "type": body.get("type"),- "instructions": _as_tokenized_instructions(body),- "labels": body.get("labels"),- "criteria": body.get("criteria")}))+ hashed = {"qid": qid, "type": body.get("type"),+ "instructions": _as_tokenized_instructions(body),+ "labels": body.get("labels"),+ "criteria": body.get("criteria")}+ if "option_order" in body:+ schemas.add(_canonical(dict(hashed, option_order=body["option_order"])))+ else:+ # Left out rather than hashed as `null`, so a dataset that never names the field+ # keeps the digest its committed baseline already carries.+ schemas.add(_canonical(hashed))digest = hashlib.sha256()for schema in sorted(schemas):digest.update(schema.encode("utf-8"))diff --git a/tests/test_evals.py b/tests/test_evals.pyindex 38c9ca54..8ee41685 100644--- a/tests/test_evals.py+++ b/tests/test_evals.py@@ -2215,6 +2215,47 @@ def test_questions_fingerprint_keeps_criteria_order():evals.questions_fingerprint(with_criteria({"b": "beta", "a": "alpha"}))+def test_questions_fingerprint_covers_option_order():+ """`option_order` moves what the model reads and re-maps the answer, so it is a fifth field.++ `laya/agent.py::Agent._check_question` accepts only a permutation of the option indices,+ `Agent._to_internal` copies it into the internal question, and from there it reaches+ `laya/common.py::build_sequence` (the tokenized head) and `unpermute_probs` on the way back, so+ a permuted question is a different question in exactly the sense this key exists to notice.+ `research/eval/metamorphic.py::permute_options` is one of that harness's two transforms, and+ this suite's own `test_questions_fingerprint_follows_the_question_not_the_row` states "option+ order and label rename flip answers" while the hash looked past the first of the two.++ The literal digest at the end is the other half of the contract: a question set that never names+ the field keeps the digest its committed baseline already carries, so adding a field to the key+ moves no existing report's identity.+ """+ def question(order=None):+ q = {"type": "choice", "instructions": "Pick one", "criteria": {"a": "alpha", "b": "beta"}}+ if order is not None:+ q["option_order"] = order+ return {"pick": q}++ def with_order(order=None):+ return Dataset([Example("s", question(order), {"pick": "a"})])++ plain = evals.questions_fingerprint(with_order())+ assert evals.questions_fingerprint(with_order([1, 0])) != plain, \+ "a permutation of the options left the fingerprint unchanged"+ assert evals.questions_fingerprint(with_order([0, 1])) != plain, \+ "naming an identity permutation left the fingerprint unchanged"+ assert evals.questions_fingerprint(with_order([0, 1])) != \+ evals.questions_fingerprint(with_order([1, 0])), "two orders hashed alike"++ # The field is real on the way in, not an artefact of the hash: the engine carries it.+ from laya.agent import Agent+ assert "option_order" not in Agent._to_internal(question()["pick"])+ assert Agent._to_internal(question([1, 0])["pick"])["option_order"] == [1, 0]++ # Measured on `main` before `option_order` joined the key.+ assert plain == "8263dfd11d6593803f61d78fe6ccc629924f01e85e9e8a2bac5ce629ebb97ff4"++def test_a_top_level_schema_on_the_candidate_still_refuses(tmp_path, capsys):"""The identity fallback is two-sided.