Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(evals): name every field the question fingerprint hashes

Aashishf91afdfcdf0d
diff --git a/docs/evals.md b/docs/evals.md
index d63bb027..2900a9f0 100644
--- a/docs/evals.md
+++ b/docs/evals.md
@@ -455,7 +455,7 @@ is reviewable on its own:
| `schema` | the report shape, `laya-evals-report/1`, so a consumer can refuse one it cannot read |
| `dataset` | the path as typed -- a name, not a hash |
| `dataset_sha256` | the sha256 of the dataset bytes that were parsed |
-| `questions_sha256` | a fingerprint of the question schema: every question's id, type, `instructions` and `criteria`, over the whole dataset |
+| `questions_sha256` | a fingerprint of the question schema: every question's id, type, `instructions`, `labels` and `criteria`, plus `option_order` on any row that names it, over the whole dataset |
| `laya_version` | the `laya` that computed the numbers |
| `coverage_metric_definition` | which definition of `aurc` / `selective_accuracy@*` produced this report (see [coverage cuts](#coverage-cuts-and-ties)). A relative gate rule on either refuses a baseline recorded under a different one, rather than subtracting numbers that do not mean the same thing |
| `thresholds` | the gate this run applied: `min`, `max` and `baseline_tolerance` |
@@ -487,6 +487,17 @@ a copy edit is treated as a new experiment. That is the honest default -- the al
similarity heuristic standing between a run and its baseline, and no evaluation system in common
use has one.
+`option_order` is the one field hashed conditionally, and only when a row names it. It is a
+caller-supplied key: `laya/mcp/server.py` advertises it, `Agent._check_question` and
+`laya/mcp/tools.py::_check_option_order` validate it as a permutation of the option indices, and
+`laya/common.py::build_sequence` and `unpermute_probs` read it to place each answer slot, so a row
+that permutes its options asks a different question and two such datasets must not share a
+fingerprint. `research/eval/metamorphic.py::permute_options` measures the same sensitivity from the
+other side -- it exists to shuffle the option list and check that the answers follow. Hashing an
+absent key as `null` would move the digest of every dataset that never uses the field, so an absent
+key stays out of the hash and such a dataset keeps the fingerprint its committed baseline already
+carries.
+
Nothing time-bearing is recorded, so a report is still byte-reproducible for a fixed runner.
`REPORT_SCHEMA`, `questions_fingerprint(dataset)` and `file_fingerprint(path)` are public, so a
diff --git a/laya/evals.py b/laya/evals.py
index b33a9431..29dafc8a 100644
--- a/laya/evals.py
+++ b/laya/evals.py
@@ -224,10 +224,10 @@ def questions_fingerprint(dataset: "Dataset") -> str:
`option_order` is the fifth and last field a question body can carry, and it was the one left
out: `laya/agent.py::Agent._check_question` accepts only a permutation of the option indices,
`Agent._to_internal` copies it into the internal question, and from there it reaches both the
- tokenized head (`laya/common.py::build_sequence` via `build_head`) and the reported
- probabilities (`laya/agent.py`'s `unpermute_probs` call). It is hashed only when a row names it,
- so the digest of a question set that does not use the field stays the number its committed
- baseline already carries.
+ tokenized head (`laya/common.py::build_sequence` through `build_head`) and the reported
+ probabilities (`laya/common.py::unpermute_probs`, applied in `Agent._decode_answers`). It is
+ hashed only when a row names it, so the digest of a question set that does not use the field
+ stays the number its committed baseline already carries.
"""
schemas = set()
for example in dataset.examples: