Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(evals): the question fingerprint has to cover `instructions`
Bruce-Yii76e90d8368c0
diff --git a/docs/evals.md b/docs/evals.mdindex e8430636..1cfad0c9 100644--- a/docs/evals.md+++ b/docs/evals.md@@ -134,18 +134,35 @@ is reviewable on its own:| `schema` | the report shape, `laya-evals-report/1`, so a consumer can refuse one it cannot read || `dataset` | the path as typed -- a name, not a hash || `dataset_sha256` | the sha256 of the dataset bytes that were parsed |-| `questions_sha256` | a fingerprint of the question schema: every question's id, type and `criteria`, over the whole dataset |+| `questions_sha256` | a fingerprint of the question schema: every question's id, type, `instructions` and `criteria`, over the whole dataset || `laya_version` | the `laya` that computed the numbers || `thresholds` | the gate this run applied: `min`, `max` and `baseline_tolerance` || `revisions` | the commit each checkpoint that answered was loaded from (see [below](#baseline-and-ci-gate)) |`dataset` is a path, and a path is not an identity: a dataset can be edited in place, moved, orrefetched under the same name, and a CI cache can hand two runs the same filename and different-bytes. `questions_sha256` covers what was *asked* rather than how many rows there were, so scoring-more states on the same questions stays comparable to a baseline, while renaming a `choice` option-label does not -- `criteria` is the decision space, and the metamorphic checks in-`research/eval/metamorphic.py` exist because a label rename flips answers. `instructions` is-excluded on purpose: rewording prose changes the prompt, not the question being asked.+bytes. `questions_sha256` covers what was *asked* rather than how many rows there were, so adding+states to an unchanged question set leaves the fingerprint alone -- `dataset_sha256` still moves,+and adding a row is a change to the data, not to the question.++It covers `instructions` too, because the instruction text is the prompt. `build_sequence` renders+`"<type> question: <instructions>"` into the tokenized head, `Agent` refuses a question without one+("add the text the model should answer"), and Laya's own question identity already counts it:+`Router._question_schema` and this harness's batch grouping both key on the whole questions dict,+and `tests/test_router_batch.py` pins that rewording `instructions` alone moves a row into its own+batch group. So does a reworded instruction still compare equal to a baseline? No -- and that is+the point. "Judge whether a refund is justified" and "Be conservative and only approve explicit+refund requests" ask different questions, and the metric gate can only notice when the difference+happens to move a number further than the tolerance you named. Naming a `choice` option is the+same argument: `criteria` is the decision space, and the metamorphic checks in+`research/eval/metamorphic.py` exist because renaming a label flips answers.++Nothing about the instruction text is normalized except the one step the engine itself applies: a+non-string `instructions` is hashed as `json.dumps(ins, ensure_ascii=False)`, matching+`Agent._to_internal`. So whitespace and wording both count, and a rewording that a human considers+a copy edit is treated as a new experiment. That is the honest default -- the alternative is a+similarity heuristic standing between a run and its baseline, and no evaluation system in common+use has one.Nothing time-bearing is recorded, so a report is still byte-reproducible for a fixed runner.diff --git a/laya/evals.py b/laya/evals.pyindex a02243c2..4e3b133d 100644--- a/laya/evals.py+++ b/laya/evals.py@@ -130,6 +130,25 @@ def _canonical(value: Any) -> str:default=str)+def _as_tokenized_instructions(body: Dict[str, Any]) -> Any:+ """`instructions` exactly as `Agent._to_internal` (`laya/agent.py:736-748`) hands it over.++ A non-string becomes ``json.dumps(ins, ensure_ascii=False)``, and that step is not cosmetic:+ ``tests/test_criteria.py:229-251`` pins that the default ``ensure_ascii=True`` escaped+ non-ASCII to literal ``\\uXXXX``, the tokenizer read it as escape text, and one German+ question answered noul=0.1652 as a dict against 0.2650 as the identical plain string. So the+ fingerprint normalizes the same way, or it hashes the input's JSON shape instead of the text+ the model reads.++ Not mirrored: the tokenizer's mask-token strip in `common.build_sequence`, which needs+ `tok.mask_token` and would mean guessing which tokenizer a run loads. Instructions carrying+ a mask token therefore hash apart even though they render alike -- a false refusal, which is+ the safe direction to be wrong in.+ """+ ins = body.get("instructions")+ return ins if isinstance(ins, str) else json.dumps(ins, ensure_ascii=False)++def questions_fingerprint(dataset: "Dataset") -> str:"""A stable hash of *what was asked*, over every question in `dataset`.@@ -137,18 +156,27 @@ def questions_fingerprint(dataset: "Dataset") -> str:with the policy, and the part of the run identity a report can compute for itself: thedataset file hash is the CLI's, but the questions are parsed here.+ It covers every field the answer depends on, which is the same rule `examples/hooks/cache.py`+ states for a cache key ("has to cover everything the answer depends on") and the same one+ Laya already applies to its own question identity: `Router._question_schema`+ (`laya/router.py:141`) and this module's batch grouping (`laya/evals.py`) both hash the whole+ questions dict. `tests/test_router_batch.py:543` pins that rewording `instructions` alone+ moves a row into its own batch group.+It is a function of the decision space, not of the rows, so scoring more states on the same- questions stays comparable to a baseline. A `choice` option label lives in `criteria` and so- counts as a different question -- which is the case that matters, since- `research/eval/metamorphic.py` exists because renaming a label flips answers. Free-text- `instructions` are excluded: rewording them changes the prompt, not the decision space, and- a baseline should survive a copy edit.+ questions leaves it unchanged.++ `criteria` is hashed raw, without `Agent._to_internal`'s list-to-dict and lowercase-key+ rewrites. That is deliberate and asymmetric: two `noul` questions that render alike but are+ written differently would then be refused, which is the safe direction to be wrong in.+ Normalizing them would widen the change past what the identity needs."""schemas = set()for example in dataset.examples:for qid, question in example.questions.items():body = question if isinstance(question, dict) else {}schemas.add(_canonical({"qid": qid, "type": body.get("type"),+ "instructions": _as_tokenized_instructions(body),"criteria": body.get("criteria")}))digest = hashlib.sha256()for schema in sorted(schemas):diff --git a/tests/test_evals.py b/tests/test_evals.pyindex 2c227aca..d872259d 100644--- a/tests/test_evals.py+++ b/tests/test_evals.py@@ -626,6 +626,11 @@ def test_questions_fingerprint_follows_the_question_not_the_row():scores the same questions on more rows must still be comparable to it. Renaming a `choice`label is a different question, not a different row: `research/eval/metamorphic.py` existsbecause option order and label rename flip answers.++ The scope of the last claim is the fingerprint, not the gate. `dataset_sha256` is compared+ too, and adding a row changes the file's bytes, so through `laya-evals run` a longer dataset+ is correctly refused. The fingerprint being row-independent is what lets a programmatic+ caller see that the *questions* did not change."""base = Dataset([Example("s1", Q, {"intent": "a"})])more_rows = Dataset([Example("s1", Q, {"intent": "a"}), Example("s2", Q, {"intent": "a"})])@@ -641,6 +646,108 @@ def test_questions_fingerprint_follows_the_question_not_the_row():assert evals.questions_fingerprint(retyped) != evals.questions_fingerprint(base)+def test_questions_fingerprint_covers_the_instructions():+ """`instructions` is the prompt. Excluding it made the fingerprint say "same question" for+ two questions the model answers differently.++ `laya/common.py:158-159` renders `"%s question: %s" % (q["type"], instructions)` into the+ tokenized head, `laya/agent.py:633-634` makes the field mandatory ("add the text the model+ should answer"), and Laya's own identity key for sharing forward passes+ (`Router._question_schema`, `laya/router.py:141`) hashes the whole questions dict,+ instructions included -- as does this module's own batch grouping at `laya/evals.py:529`.+ `tests/test_router_batch.py:543` pins that rewording alone moves a row to its own batch group.+ Leaving it out made the one field this contract exists to protect the one field it ignored.+ """+ judged = Dataset([Example("s1", {"verdict": {"type": "choice", "instructions": "Judge whether a refund is justified",+ "criteria": {"yes": "approved", "no": "denied"}}},+ {"verdict": "yes"})])+ conservative = Dataset([Example("s1", {"verdict": {"type": "choice", "instructions": "Be conservative and only approve explicit refund requests",+ "criteria": {"yes": "approved", "no": "denied"}}},+ {"verdict": "yes"})])+ assert evals.questions_fingerprint(judged) != evals.questions_fingerprint(conservative)++ # Same wording, different question id, is still a different question.+ renamed_id = Dataset([Example("s1", {"decision": judged.examples[0].questions["verdict"]},+ {"decision": "yes"})])+ assert evals.questions_fingerprint(renamed_id) != evals.questions_fingerprint(judged)++ # Whitespace is not decoration: it is tokenized. `common.py:158` strips only the tokenizer's+ # mask token, and this module has no tokenizer to know what that is, so nothing else is+ # normalized away.+ spaced = Dataset([Example("s1", {"verdict": dict(judged.examples[0].questions["verdict"],+ instructions="Judge whether a refund is justified ")},+ {"verdict": "yes"})])+ assert evals.questions_fingerprint(spaced) != evals.questions_fingerprint(judged)+++def test_questions_fingerprint_normalizes_instructions_the_way_the_engine_does():+ """Two questions that render the same text are the same question, whatever their JSON shape.++ `Agent._to_internal` (`laya/agent.py:736-748`) turns a non-string `instructions` into+ `json.dumps(ins, ensure_ascii=False)` before tokenizing, and `tests/test_criteria.py:229-251`+ pins why: the default `ensure_ascii=True` escaped non-ASCII to literal `\\uXXXX` and a German+ question answered noul=0.1652 as a dict against 0.2650 as the identical plain string. The+ fingerprint has to mirror that step, or it hashes the input's JSON shape instead of the text+ the model reads.+ """+ criteria = {"yes": "approved", "no": "denied"}+ as_object = Dataset([Example("s1", {"verdict": {"type": "choice",+ "instructions": {"task": "Bittet der Kunde um eine Rückerstattung?"},+ "criteria": criteria}},+ {"verdict": "yes"})])+ as_text = Dataset([Example("s1", {"verdict": {"type": "choice",+ "instructions": '{"task": "Bittet der Kunde um eine Rückerstattung?"}',+ "criteria": criteria}},+ {"verdict": "yes"})])+ assert evals.questions_fingerprint(as_object) == evals.questions_fingerprint(as_text)+++def test_a_reworded_question_cannot_pass_the_gate():+ """The end-to-end contract: a different question is not a baseline drift, it is a new run.++ Reproduces the hole the fingerprint's exclusion opened. `evaluate(runner, Dataset(...))` has+ no dataset file to hash, so `questions_sha256` is the only identity a programmatic run has --+ and it used to be blind to the one field that decides the answer. The metric gate is a+ separate, tolerance-dependent safety net; this is the check that does not depend on having+ guessed the right tolerance.+ """+ criteria = {"yes": "approved", "no": "denied"}++ def twenty(instr):+ q = {"verdict": {"type": "choice", "instructions": instr, "criteria": criteria}}+ return Dataset([Example("s%d" % i, q, {"verdict": "yes"}) for i in range(20)])++ class OneRowFlips:+ """The answer depends on the instruction on exactly one state, so the metric delta can+ sit inside a realistic tolerance while the question asked is a different one."""+ def predict(self, state, questions, model=None):+ reworded = "conservative" in questions["verdict"]["instructions"]+ label = "no" if (reworded and state == "s19") else "yes"+ return {"model": "stub", "answers": {"verdict": {+ "type": "choice", "choice": label,+ "probabilities": {label: 0.9}, "confidence": 0.9}}}++ baseline = evaluate(OneRowFlips(), twenty("Judge whether a refund is justified"),+ evaluators=[ChoiceAccuracy()])+ candidate = evaluate(OneRowFlips(), twenty("Be conservative and only approve explicit refunds"),+ evaluators=[ChoiceAccuracy()])+ assert candidate.overall["choice_accuracy"] == pytest.approx(0.95)++ ok, reasons = candidate.comparable_to(baseline.to_json())+ assert not ok, "a reworded question is a different experiment, not an identical one"+ assert any("questions_sha256" in reason for reason in reasons), reasons++ # And the same question over more rows is still the same experiment, which is the property+ # that would be lost if the fingerprint were simply "the whole dataset".+ def same_question(rows):+ q = {"verdict": {"type": "choice", "instructions": "Judge whether a refund is justified",+ "criteria": criteria}}+ return Dataset([Example("s%d" % i, q, {"verdict": "yes"}) for i in range(rows)])++ longer = evaluate(OneRowFlips(), same_question(25), evaluators=[ChoiceAccuracy()])+ assert longer.comparable_to(baseline.to_json())[0], "more rows is not a different question"++def test_comparable_to_refuses_two_different_runs():"""`compare` reads `overall` only, so without this a gate passes two different experiments.