Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(evals): the question fingerprint has to cover `instructions`

Bruce-Yii76e90d8368c0
diff --git a/docs/evals.md b/docs/evals.md
index e8430636..1cfad0c9 100644
--- a/docs/evals.md
+++ b/docs/evals.md
@@ -134,18 +134,35 @@ is reviewable on its own:
| `schema` | the report shape, `laya-evals-report/1`, so a consumer can refuse one it cannot read |
| `dataset` | the path as typed -- a name, not a hash |
| `dataset_sha256` | the sha256 of the dataset bytes that were parsed |
-| `questions_sha256` | a fingerprint of the question schema: every question's id, type and `criteria`, over the whole dataset |
+| `questions_sha256` | a fingerprint of the question schema: every question's id, type, `instructions` and `criteria`, over the whole dataset |
| `laya_version` | the `laya` that computed the numbers |
| `thresholds` | the gate this run applied: `min`, `max` and `baseline_tolerance` |
| `revisions` | the commit each checkpoint that answered was loaded from (see [below](#baseline-and-ci-gate)) |
`dataset` is a path, and a path is not an identity: a dataset can be edited in place, moved, or
refetched under the same name, and a CI cache can hand two runs the same filename and different
-bytes. `questions_sha256` covers what was *asked* rather than how many rows there were, so scoring
-more states on the same questions stays comparable to a baseline, while renaming a `choice` option
-label does not -- `criteria` is the decision space, and the metamorphic checks in
-`research/eval/metamorphic.py` exist because a label rename flips answers. `instructions` is
-excluded on purpose: rewording prose changes the prompt, not the question being asked.
+bytes. `questions_sha256` covers what was *asked* rather than how many rows there were, so adding
+states to an unchanged question set leaves the fingerprint alone -- `dataset_sha256` still moves,
+and adding a row is a change to the data, not to the question.
+
+It covers `instructions` too, because the instruction text is the prompt. `build_sequence` renders
+`"<type> question: <instructions>"` into the tokenized head, `Agent` refuses a question without one
+("add the text the model should answer"), and Laya's own question identity already counts it:
+`Router._question_schema` and this harness's batch grouping both key on the whole questions dict,
+and `tests/test_router_batch.py` pins that rewording `instructions` alone moves a row into its own
+batch group. So does a reworded instruction still compare equal to a baseline? No -- and that is
+the point. "Judge whether a refund is justified" and "Be conservative and only approve explicit
+refund requests" ask different questions, and the metric gate can only notice when the difference
+happens to move a number further than the tolerance you named. Naming a `choice` option is the
+same argument: `criteria` is the decision space, and the metamorphic checks in
+`research/eval/metamorphic.py` exist because renaming a label flips answers.
+
+Nothing about the instruction text is normalized except the one step the engine itself applies: a
+non-string `instructions` is hashed as `json.dumps(ins, ensure_ascii=False)`, matching
+`Agent._to_internal`. So whitespace and wording both count, and a rewording that a human considers
+a copy edit is treated as a new experiment. That is the honest default -- the alternative is a
+similarity heuristic standing between a run and its baseline, and no evaluation system in common
+use has one.
Nothing time-bearing is recorded, so a report is still byte-reproducible for a fixed runner.
diff --git a/laya/evals.py b/laya/evals.py
index a02243c2..4e3b133d 100644
--- a/laya/evals.py
+++ b/laya/evals.py
@@ -130,6 +130,25 @@ def _canonical(value: Any) -> str:
default=str)
+def _as_tokenized_instructions(body: Dict[str, Any]) -> Any:
+ """`instructions` exactly as `Agent._to_internal` (`laya/agent.py:736-748`) hands it over.
+
+ A non-string becomes ``json.dumps(ins, ensure_ascii=False)``, and that step is not cosmetic:
+ ``tests/test_criteria.py:229-251`` pins that the default ``ensure_ascii=True`` escaped
+ non-ASCII to literal ``\\uXXXX``, the tokenizer read it as escape text, and one German
+ question answered noul=0.1652 as a dict against 0.2650 as the identical plain string. So the
+ fingerprint normalizes the same way, or it hashes the input's JSON shape instead of the text
+ the model reads.
+
+ Not mirrored: the tokenizer's mask-token strip in `common.build_sequence`, which needs
+ `tok.mask_token` and would mean guessing which tokenizer a run loads. Instructions carrying
+ a mask token therefore hash apart even though they render alike -- a false refusal, which is
+ the safe direction to be wrong in.
+ """
+ ins = body.get("instructions")
+ return ins if isinstance(ins, str) else json.dumps(ins, ensure_ascii=False)
+
+
def questions_fingerprint(dataset: "Dataset") -> str:
"""A stable hash of *what was asked*, over every question in `dataset`.
@@ -137,18 +156,27 @@ def questions_fingerprint(dataset: "Dataset") -> str:
with the policy, and the part of the run identity a report can compute for itself: the
dataset file hash is the CLI's, but the questions are parsed here.
+ It covers every field the answer depends on, which is the same rule `examples/hooks/cache.py`
+ states for a cache key ("has to cover everything the answer depends on") and the same one
+ Laya already applies to its own question identity: `Router._question_schema`
+ (`laya/router.py:141`) and this module's batch grouping (`laya/evals.py`) both hash the whole
+ questions dict. `tests/test_router_batch.py:543` pins that rewording `instructions` alone
+ moves a row into its own batch group.
+
It is a function of the decision space, not of the rows, so scoring more states on the same
- questions stays comparable to a baseline. A `choice` option label lives in `criteria` and so
- counts as a different question -- which is the case that matters, since
- `research/eval/metamorphic.py` exists because renaming a label flips answers. Free-text
- `instructions` are excluded: rewording them changes the prompt, not the decision space, and
- a baseline should survive a copy edit.
+ questions leaves it unchanged.
+
+ `criteria` is hashed raw, without `Agent._to_internal`'s list-to-dict and lowercase-key
+ rewrites. That is deliberate and asymmetric: two `noul` questions that render alike but are
+ written differently would then be refused, which is the safe direction to be wrong in.
+ Normalizing them would widen the change past what the identity needs.
"""
schemas = set()
for example in dataset.examples:
for qid, question in example.questions.items():
body = question if isinstance(question, dict) else {}
schemas.add(_canonical({"qid": qid, "type": body.get("type"),
+ "instructions": _as_tokenized_instructions(body),
"criteria": body.get("criteria")}))
digest = hashlib.sha256()
for schema in sorted(schemas):
diff --git a/tests/test_evals.py b/tests/test_evals.py
index 2c227aca..d872259d 100644
--- a/tests/test_evals.py
+++ b/tests/test_evals.py
@@ -626,6 +626,11 @@ def test_questions_fingerprint_follows_the_question_not_the_row():
scores the same questions on more rows must still be comparable to it. Renaming a `choice`
label is a different question, not a different row: `research/eval/metamorphic.py` exists
because option order and label rename flip answers.
+
+ The scope of the last claim is the fingerprint, not the gate. `dataset_sha256` is compared
+ too, and adding a row changes the file's bytes, so through `laya-evals run` a longer dataset
+ is correctly refused. The fingerprint being row-independent is what lets a programmatic
+ caller see that the *questions* did not change.
"""
base = Dataset([Example("s1", Q, {"intent": "a"})])
more_rows = Dataset([Example("s1", Q, {"intent": "a"}), Example("s2", Q, {"intent": "a"})])
@@ -641,6 +646,108 @@ def test_questions_fingerprint_follows_the_question_not_the_row():
assert evals.questions_fingerprint(retyped) != evals.questions_fingerprint(base)
+def test_questions_fingerprint_covers_the_instructions():
+ """`instructions` is the prompt. Excluding it made the fingerprint say "same question" for
+ two questions the model answers differently.
+
+ `laya/common.py:158-159` renders `"%s question: %s" % (q["type"], instructions)` into the
+ tokenized head, `laya/agent.py:633-634` makes the field mandatory ("add the text the model
+ should answer"), and Laya's own identity key for sharing forward passes
+ (`Router._question_schema`, `laya/router.py:141`) hashes the whole questions dict,
+ instructions included -- as does this module's own batch grouping at `laya/evals.py:529`.
+ `tests/test_router_batch.py:543` pins that rewording alone moves a row to its own batch group.
+ Leaving it out made the one field this contract exists to protect the one field it ignored.
+ """
+ judged = Dataset([Example("s1", {"verdict": {"type": "choice", "instructions": "Judge whether a refund is justified",
+ "criteria": {"yes": "approved", "no": "denied"}}},
+ {"verdict": "yes"})])
+ conservative = Dataset([Example("s1", {"verdict": {"type": "choice", "instructions": "Be conservative and only approve explicit refund requests",
+ "criteria": {"yes": "approved", "no": "denied"}}},
+ {"verdict": "yes"})])
+ assert evals.questions_fingerprint(judged) != evals.questions_fingerprint(conservative)
+
+ # Same wording, different question id, is still a different question.
+ renamed_id = Dataset([Example("s1", {"decision": judged.examples[0].questions["verdict"]},
+ {"decision": "yes"})])
+ assert evals.questions_fingerprint(renamed_id) != evals.questions_fingerprint(judged)
+
+ # Whitespace is not decoration: it is tokenized. `common.py:158` strips only the tokenizer's
+ # mask token, and this module has no tokenizer to know what that is, so nothing else is
+ # normalized away.
+ spaced = Dataset([Example("s1", {"verdict": dict(judged.examples[0].questions["verdict"],
+ instructions="Judge whether a refund is justified ")},
+ {"verdict": "yes"})])
+ assert evals.questions_fingerprint(spaced) != evals.questions_fingerprint(judged)
+
+
+def test_questions_fingerprint_normalizes_instructions_the_way_the_engine_does():
+ """Two questions that render the same text are the same question, whatever their JSON shape.
+
+ `Agent._to_internal` (`laya/agent.py:736-748`) turns a non-string `instructions` into
+ `json.dumps(ins, ensure_ascii=False)` before tokenizing, and `tests/test_criteria.py:229-251`
+ pins why: the default `ensure_ascii=True` escaped non-ASCII to literal `\\uXXXX` and a German
+ question answered noul=0.1652 as a dict against 0.2650 as the identical plain string. The
+ fingerprint has to mirror that step, or it hashes the input's JSON shape instead of the text
+ the model reads.
+ """
+ criteria = {"yes": "approved", "no": "denied"}
+ as_object = Dataset([Example("s1", {"verdict": {"type": "choice",
+ "instructions": {"task": "Bittet der Kunde um eine Rückerstattung?"},
+ "criteria": criteria}},
+ {"verdict": "yes"})])
+ as_text = Dataset([Example("s1", {"verdict": {"type": "choice",
+ "instructions": '{"task": "Bittet der Kunde um eine Rückerstattung?"}',
+ "criteria": criteria}},
+ {"verdict": "yes"})])
+ assert evals.questions_fingerprint(as_object) == evals.questions_fingerprint(as_text)
+
+
+def test_a_reworded_question_cannot_pass_the_gate():
+ """The end-to-end contract: a different question is not a baseline drift, it is a new run.
+
+ Reproduces the hole the fingerprint's exclusion opened. `evaluate(runner, Dataset(...))` has
+ no dataset file to hash, so `questions_sha256` is the only identity a programmatic run has --
+ and it used to be blind to the one field that decides the answer. The metric gate is a
+ separate, tolerance-dependent safety net; this is the check that does not depend on having
+ guessed the right tolerance.
+ """
+ criteria = {"yes": "approved", "no": "denied"}
+
+ def twenty(instr):
+ q = {"verdict": {"type": "choice", "instructions": instr, "criteria": criteria}}
+ return Dataset([Example("s%d" % i, q, {"verdict": "yes"}) for i in range(20)])
+
+ class OneRowFlips:
+ """The answer depends on the instruction on exactly one state, so the metric delta can
+ sit inside a realistic tolerance while the question asked is a different one."""
+ def predict(self, state, questions, model=None):
+ reworded = "conservative" in questions["verdict"]["instructions"]
+ label = "no" if (reworded and state == "s19") else "yes"
+ return {"model": "stub", "answers": {"verdict": {
+ "type": "choice", "choice": label,
+ "probabilities": {label: 0.9}, "confidence": 0.9}}}
+
+ baseline = evaluate(OneRowFlips(), twenty("Judge whether a refund is justified"),
+ evaluators=[ChoiceAccuracy()])
+ candidate = evaluate(OneRowFlips(), twenty("Be conservative and only approve explicit refunds"),
+ evaluators=[ChoiceAccuracy()])
+ assert candidate.overall["choice_accuracy"] == pytest.approx(0.95)
+
+ ok, reasons = candidate.comparable_to(baseline.to_json())
+ assert not ok, "a reworded question is a different experiment, not an identical one"
+ assert any("questions_sha256" in reason for reason in reasons), reasons
+
+ # And the same question over more rows is still the same experiment, which is the property
+ # that would be lost if the fingerprint were simply "the whole dataset".
+ def same_question(rows):
+ q = {"verdict": {"type": "choice", "instructions": "Judge whether a refund is justified",
+ "criteria": criteria}}
+ return Dataset([Example("s%d" % i, q, {"verdict": "yes"}) for i in range(rows)])
+
+ longer = evaluate(OneRowFlips(), same_question(25), evaluators=[ChoiceAccuracy()])
+ assert longer.comparable_to(baseline.to_json())[0], "more rows is not a different question"
+
+
def test_comparable_to_refuses_two_different_runs():
"""`compare` reads `overall` only, so without this a gate passes two different experiments.