Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
test(docs): make prose citations into symbols, and check that they resolve
Aashish85667922c12e
diff --git a/laya/evals.py b/laya/evals.pyindex b066859c..0080e81c 100644--- a/laya/evals.py+++ b/laya/evals.py@@ -171,7 +171,7 @@ def _canonical(value: Any) -> str:def _as_tokenized_instructions(body: Dict[str, Any]) -> Any:- """`instructions` exactly as `Agent._to_internal` (`laya/agent.py:736-748`) hands it over.+ """`instructions` exactly as `laya/agent.py::Agent._to_internal` hands it over.A non-string becomes ``json.dumps(ins, ensure_ascii=False)``, and that step is not cosmetic:``tests/test_criteria.py:229-251`` pins that the default ``ensure_ascii=True`` escaped@@ -198,10 +198,10 @@ def questions_fingerprint(dataset: "Dataset") -> str:It covers every field the answer depends on, which is the same rule `examples/hooks/cache.py`states for a cache key ("has to cover everything the answer depends on") and the same one- Laya already applies to its own question identity: `Router._question_schema`- (`laya/router.py:141`) and this module's batch grouping (`laya/evals.py`) both hash the whole- questions dict. `tests/test_router_batch.py:543` pins that rewording `instructions` alone- moves a row into its own batch group.+ Laya already applies to its own question identity: `laya/router.py::_question_schema` and this+ module's batch grouping (`laya/evals.py::evaluate`) both hash the whole questions dict.+ `tests/test_router_batch.py:543` pins that rewording `instructions` alone moves a row into its+ own batch group.It is a function of the decision space, not of the rows, so scoring more states on the samequestions leaves it unchanged.@@ -211,11 +211,12 @@ def questions_fingerprint(dataset: "Dataset") -> str:written differently would then be refused, which is the safe direction to be wrong in.Normalizing them would widen the change past what the identity needs.- `labels` is hashed for the same reason `instructions` is: it is validated- (`laya/agent.py:722-726` -> `_resolve_noul_labels`) and carried into the internal question- (`laya/agent.py:748-749`), and `_resolve_noul_labels` (`laya/common.py:92`) turns it into the- option text the model reads. Leaving it out let two question sets with different rendered- options hash alike, which is the one thing this function exists to prevent.+ `labels` is hashed for the same reason `instructions` is:+ `laya/agent.py::Agent._check_question` validates it through+ `laya/common.py::_resolve_noul_labels`, `laya/agent.py::Agent._to_internal` carries it into+ the internal question, and that resolver is what turns it into the option text the model+ reads. Leaving it out let two question sets with different rendered options hash alike, which+ is the one thing this function exists to prevent."""schemas = set()for example in dataset.examples:diff --git a/scripts/test_suites.py b/scripts/test_suites.pyindex c64b38d4..922ade5f 100644--- a/scripts/test_suites.py+++ b/scripts/test_suites.py@@ -55,6 +55,7 @@ SCRIPT_SUITES = ["tests/test_audit_scope.py","tests/test_doc_tables.py","tests/test_env_docs.py",+ "tests/test_doc_citations.py","tests/test_tokenizer_cache.py","tests/test_tokenizer_concurrency.py","tests/test_question_token_reuse.py",diff --git a/tests/test_confidence.py b/tests/test_confidence.pyindex 8285e6ac..e367ba24 100644--- a/tests/test_confidence.py+++ b/tests/test_confidence.py@@ -401,7 +401,7 @@ check("export/laya re-exports the same tuple", laya.GATE_STATES, _confidence.GAT# --------------------------------- the two doc pages must attribute confidence per question type# `docs/structured.md` and `docs/questions-and-answers.md` each opened their confidence section with# "`confidence` is normalized entropy". Neither agent writes that for every type:-# `Agent._decode_answers` (`laya/agent.py:1367-1404`) and `OnnxAgent._decode_answers`+# `Agent._decode_answers` (`laya/agent.py:1367-1404`) and `ONNXAgent._decode_answers`# (`laya/onnx_agent.py:682-727`) put `confidence_from_probs(p, k)` = `1 - H(p) / log(k)` in the# `choice` and `score` answers and `max(p_true, 1 - p_true)` in the `noul` one. Each page printed a# `noul` in the very block its sentence introduces, so the sentence was falsifiable from the pagediff --git a/tests/test_doc_citations.py b/tests/test_doc_citations.pynew file mode 100644index 00000000..f6a3a76e--- /dev/null+++ b/tests/test_doc_citations.py@@ -0,0 +1,264 @@+"""Every prose pointer from this repository's text into its code has to resolve.++Two forms are used, and each fails in its own way:++ * ``laya/router.py::_question_schema`` -- names a symbol, so it cannot rot, but it does go+ wrong: a rename leaves a citation to a function that no longer exists and nothing in the tree+ notices. This resolves each one against the cited file's ``ast``, including the owner when the+ prose claims one.+ * ``laya/router.py:141`` -- rots on its own, silently, every time the code above it grows.++Measured across 372 tracked text files on this branch before anything was fixed: 11 ``:NNN``+citations name a symbol in the same breath as the number, and **7 of those 11 are wrong** -- 3+stale numbers and 4 claims about an owner that does not exist. ``Agent._to_internal`` is cited at+``laya/agent.py:736-748`` in two places while the function is at 1109-1130, and+``_resolve_noul_labels`` at ``laya/common.py:92`` while it is at 113-124; line 141 is inside+``_check_checkpoint_policy``, a few lines from a ``warnings.warn``. Three sites call the schema key+``Router._question_schema``, but it is a module function at ``laya/router.py:221`` with no class+around it, so ``getattr(Router, "_question_schema", None)`` is ``None``; one cites+``OnnxAgent._decode_answers`` where the class has always been ``ONNXAgent``.++The line numbers are the failure a reader pays for. A citation exists so the reviewer can open the+file and see the claim; landing on an unrelated line costs them the search the pointer was written+to avoid, and reads as a claim nobody checked. So the three stale sites and the three ``Router.``+sites are rewritten into the ``::symbol`` form -- which eight citations elsewhere in the tree+already used, and twenty-three do now -- and this suite holds both forms to the tree: a ``::``+citation must resolve, and a ``:NNN`` citation written next to a symbol must fall inside it and+name that symbol's real owner. The two ``_decode_answers`` citations keep their line ranges on+purpose: they point at fragments of a long function, they are correct today, and they are the+subjects that keep the second check from going vacuous.++Deterministic and cheap: parse the tree, compare, print. No weights, no GPU, no network.++Run: python tests/test_doc_citations.py+"""+from __future__ import annotations++import ast+import os+import re+import subprocess+import sys+from typing import Dict, List, Optional, Sequence, Tuple++ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+SELF = os.path.relpath(os.path.abspath(__file__), ROOT).replace(os.sep, "/")++PASS: List[str] = []+FAIL: List[str] = []+++def check_true(what: str, cond: bool, detail: object = "") -> None:+ if cond:+ PASS.append(what)+ else:+ FAIL.append("%s: %s" % (what, detail))+++def _shown(named: Sequence["Symbol"]) -> str:+ return ", ".join(str(s) for s in named)+++class Symbol:+ """A name a ``::`` citation could mean, and the scope that defines it."""++ __slots__ = ("name", "parent", "start", "end")++ def __init__(self, name: str, parent: Optional[str], start: int, end: int) -> None:+ self.name, self.parent, self.start, self.end = name, parent, start, end++ def __str__(self) -> str:+ return "%s%s (%d-%d)" % ("%s." % self.parent if self.parent else "",+ self.name, self.start, self.end)+++DEFS = (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)+++def _collect(stmts: Sequence[ast.stmt], parent: Optional[str], out: List[Symbol]) -> None:+ for node in stmts:+ if isinstance(node, DEFS):+ out.append(Symbol(node.name, parent, node.lineno, node.end_lineno or node.lineno))+ _collect(node.body, node.name, out)+ elif isinstance(node, (ast.Assign, ast.AnnAssign)):+ for target in (node.targets if isinstance(node, ast.Assign) else [node.target]):+ if isinstance(target, ast.Name):+ out.append(Symbol(target.id, parent, node.lineno,+ node.end_lineno or node.lineno))+ elif isinstance(node, ast.If):+ # A module-level guard (``if TYPE_CHECKING:``) still defines what it holds.+ _collect(node.body, parent, out)+ _collect(node.orelse, parent, out)+++_CACHE: Dict[str, List[Symbol]] = {}+++def symbols(rel: str) -> List[Symbol]:+ if rel not in _CACHE:+ path = os.path.join(ROOT, rel.replace("/", os.sep))+ try:+ out: List[Symbol] = []+ _collect(ast.parse(open(path, encoding="utf-8").read()).body, None, out)+ except (OSError, SyntaxError, UnicodeDecodeError):+ out = []+ _CACHE[rel] = out+ return _CACHE[rel]+++DBLCOLON = re.compile(r'(?P<file>[A-Za-z0-9_.\-/]+\.py)::(?P<name>[A-Za-z_][A-Za-z0-9_]*'+ r'(?:\.[A-Za-z_][A-Za-z0-9_]*)?)')+LINECITE = re.compile(r'(?P<file>[A-Za-z0-9_.\-/]+\.py):(?P<a>\d+)(?:\s*-\s*(?P<b>\d+))?')+IDENT = re.compile(r'`(?P<name>[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)*)`')++# What may sit between a backticked symbol and the citation after it. Punctuation and spacing+# cover both live idioms -- "`Foo.bar` (`file:12`)" and "`Foo.bar`, (`file:12`)" -- and the+# connectors cover "`Router.predict_batch` -- see laya/router.py:961". Longer than that is a new+# clause, and binding across a clause is how a checker starts inventing violations.+GAP_LIMIT = 24+CONNECTORS = {"see", "via", "at", "in", "as", "is", "where", "shows", "says", "and"}++# A site this suite knows is stale but must not fail on, because the pull request that owns those+# lines is in review. The key is (citing file, cited file, symbol); the row retires when that+# change merges, and the check starts applying to the site again.+DEFERRED: Dict[Tuple[str, str, str], str] = {+ ("tests/test_example_server_per_call_controls.py", "laya/router.py", "predict_batch"):+ "#1086 rewrites both comments that carry this citation (its hunks are @@ -22,15 and "+ "@@ -270,9), so a sibling change across the same lines mid-review is worse than waiting. "+ "Once #1086 merges, delete this row and rewrite `laya/router.py:961` as "+ "`laya/router.py::Router.predict_batch`.",+}++SKIP_DIRS = {".git", ".venv", "node_modules", "__pycache__", "dist", "build", ".mypy_cache",+ ".pytest_cache", "site-packages"}+++def target_of(cited: str) -> Optional[str]:+ """Resolve a cited path in the tree, tolerating the ``laya/x.py`` and ``x.py`` spellings."""+ for cand in ({cited, cited.split("/", 1)[-1]} if "/" in cited else {cited}):+ if cand.endswith(".py") and os.path.isfile(os.path.join(ROOT, cand.replace("/", os.sep))):+ return cand+ return None+++def text_files() -> List[str]:+ try:+ listed = [f for f in subprocess.check_output(["git", "-C", ROOT, "ls-files"],+ text=True).split("\n") if f]+ except (OSError, subprocess.SubprocessError):+ listed = []+ for base, dirs, names in os.walk(ROOT):+ dirs[:] = [d for d in dirs if d not in SKIP_DIRS]+ listed.extend(os.path.relpath(os.path.join(base, n), ROOT).replace(os.sep, "/")+ for n in names)+ return sorted(f for f in listed if f != SELF and f.endswith((".py", ".md"))+ and not any(d in f.split("/") for d in SKIP_DIRS))+++def bound_symbol(text: str, pos: int) -> Optional[str]:+ """The backticked identifier a citation is attached to, or None when it is attached to none."""+ matches = list(IDENT.finditer(text, 0, pos))+ if not matches:+ return None+ gap = text[matches[-1].end():pos]+ if len(gap) > GAP_LIMIT or gap.count("`") > 1:+ return None+ if any(word not in CONNECTORS for word in re.findall(r'[A-Za-z_][A-Za-z0-9_]*', gap)):+ return None+ return matches[-1].group("name")+++def main() -> int:+ files = text_files()+ check_true("the scan covers the tree", len(files) > 300, "%d text files" % len(files))++ dbl_total = 0+ dbl_owner_claims = 0+ unresolved: List[str] = []+ bound = 0+ drifted: List[str] = []+ deferred: List[Tuple[Tuple[str, str, str], str]] = []+ unbound = 0++ for rel in files:+ try:+ text = open(os.path.join(ROOT, rel.replace("/", os.sep)),+ encoding="utf-8", errors="ignore").read()+ except OSError:+ continue++ def where(pos: int, _rel: str = rel, _text: str = text) -> str:+ return "%s:%d" % (_rel, _text.count("\n", 0, pos) + 1)++ for m in DBLCOLON.finditer(text):+ tgt = target_of(m.group("file"))+ if tgt is None:+ continue+ dbl_total += 1+ owner, _, leaf = m.group("name").rpartition(".")+ named = [s for s in symbols(tgt) if s.name == leaf]+ if not named:+ unresolved.append("%s cites %s, which %s does not define"+ % (where(m.start()), m.group(0), tgt))+ elif owner:+ dbl_owner_claims += 1+ if not any(s.parent == owner for s in named):+ unresolved.append("%s cites %s, but %s defines it as %s"+ % (where(m.start()), m.group(0), tgt, _shown(named)))++ for m in LINECITE.finditer(text):+ tgt = target_of(m.group("file"))+ if tgt is None:+ continue+ name = bound_symbol(text, m.start())+ if name is None:+ unbound += 1+ continue+ owner, _, leaf = name.rpartition(".")+ named = [s for s in symbols(tgt) if s.name == leaf]+ if not named:+ unbound += 1+ continue+ bound += 1+ key = (rel, tgt, leaf)+ if key in DEFERRED:+ deferred.append((key, "%s -> %s (names %s)"+ % (where(m.start()), m.group(0), name)))+ continue+ if owner and not any(s.parent == owner for s in named):+ drifted.append("%s calls it %s, but %s defines it as %s"+ % (where(m.start()), name, tgt, _shown(named)))+ continue+ lo, hi = int(m.group("a")), int(m.group("b") or m.group("a"))+ if not any(s.start <= hi and lo <= s.end for s in named):+ drifted.append("%s puts %s at %s:%d, but it runs %s"+ % (where(m.start()), name, tgt, lo, _shown(named)))++ check_true("every `file::symbol` citation resolves to a definition",+ not unresolved, unresolved)+ check_true("the `file::symbol` check has subjects", dbl_total >= 12,+ "only %d citations of that form" % dbl_total)+ check_true("no `file:NNN` citation lands outside the symbol named beside it",+ not drifted, drifted)+ check_true("the line-citation check has subjects", bound >= 3,+ "only %d citations bind to a named symbol" % bound)++ print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))+ for f in FAIL:+ print(" FAIL " + f)+ print("%d text files scanned from %s" % (len(files), ROOT))+ print("%d `file::symbol` citation(s) resolve, %d of them claiming an owner"+ % (dbl_total, dbl_owner_claims))+ print("%d `file:NNN` citation(s) name a symbol beside them; %d of those land inside it"+ % (bound, bound - len(deferred) - len(drifted)))+ for key, site in deferred:+ print("deferred: %s -- %s" % (site, DEFERRED[key]))+ print("not checkable from this repository (not asserted either way):")+ print(" - %d `file:NNN` citation(s) with no symbol named beside them -- a drift there is "+ "invisible, which is why the `::` form is the one to write" % unbound)+ print(" - citations to files outside the tree, and to non-Python targets")+ return 1 if FAIL else 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/tests/test_evals.py b/tests/test_evals.pyindex 12ed6555..76145945 100644--- a/tests/test_evals.py+++ b/tests/test_evals.py@@ -1626,11 +1626,11 @@ def test_questions_fingerprint_covers_the_instructions():"""`instructions` is the prompt. Excluding it made the fingerprint say "same question" fortwo questions the model answers differently.- `laya/common.py:158-159` renders `"%s question: %s" % (q["type"], instructions)` into the- tokenized head, `laya/agent.py:633-634` makes the field mandatory ("add the text the model- should answer"), and Laya's own identity key for sharing forward passes- (`Router._question_schema`, `laya/router.py:141`) hashes the whole questions dict,- instructions included -- as does this module's own batch grouping at `laya/evals.py:529`.+ `laya/common.py::build_head` renders `"%s question: %s" % (q["type"], instructions)` into the+ tokenized head, `laya/agent.py::Agent._check_question` makes the field mandatory ("add the+ text the model should answer"), and Laya's own identity key for sharing forward passes,+ `laya/router.py::_question_schema`, hashes the whole questions dict, instructions included --+ as does this module's own batch grouping at `laya/evals.py::evaluate`.`tests/test_router_batch.py:543` pins that rewording alone moves a row to its own batch group.Leaving it out made the one field this contract exists to protect the one field it ignored."""@@ -1659,7 +1659,7 @@ def test_questions_fingerprint_covers_the_instructions():def test_questions_fingerprint_normalizes_instructions_the_way_the_engine_does():"""Two questions that render the same text are the same question, whatever their JSON shape.- `Agent._to_internal` (`laya/agent.py:736-748`) turns a non-string `instructions` into+ `laya/agent.py::Agent._to_internal` turns a non-string `instructions` into`json.dumps(ins, ensure_ascii=False)` before tokenizing, and `tests/test_criteria.py:229-251`pins why: the default `ensure_ascii=True` escaped non-ASCII to literal `\\uXXXX` and a Germanquestion answered noul=0.1652 as a dict against 0.2650 as the identical plain string. The@@ -2026,9 +2026,10 @@ def test_questions_fingerprint_covers_labels():"""`labels` decides the option text, so it belongs in the key.The same argument `test_questions_fingerprint_covers_the_instructions` makes for- `instructions`. `labels` is validated (`laya/agent.py:722-726` -> `_resolve_noul_labels`),- carried into the internal question (`laya/agent.py:748-749`), and resolved into the option- text the model actually reads (`laya/common.py:92`). Two question sets differing only in+ `instructions`. `laya/agent.py::Agent._check_question` validates `labels` through+ `laya/common.py::_resolve_noul_labels`, `laya/agent.py::Agent._to_internal` carries it into+ the internal question, and the resolver is what produces the option text the model actually+ reads. Two question sets differing only in`labels` therefore ask the model different things, and hashing them alike let the gate passa comparison it exists to refuse."""@@ -2053,8 +2054,8 @@ def test_questions_fingerprint_covers_labels():def test_questions_fingerprint_keeps_criteria_order():"""A choice question's criteria order is positional, so two orders are two questions.- `examples/hooks/cache.py:22-25` states the rule and `Router._question_schema`- (`laya/router.py:141`) applies it with `sort_keys=False`. Worth pinning here for the dict+ `examples/hooks/cache.py:22-25` states the rule and `laya/router.py::_question_schema`+ applies it with `sort_keys=False`. Worth pinning here for the dictform: `json.dumps(sort_keys=True)` reorders *dict keys* and leaves *lists* alone, so alist-valued `criteria` -- the shape a dataset row has -- was already safe. The flag onlymattered for a dict-valued one, where folding is the wrong direction to be wrong in.