Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat: opt-in embedding shortlist for high-cardinality choice (#106)
Nithilan Kumaran62f4f158c968
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex d04d70c7..af24e1db 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -46,6 +46,7 @@ jobs:python tests/test_router.pypython tests/test_criteria.pypython tests/test_download.py+ python tests/test_shortlist.py- name: Email cleaning testsrun: python tests/test_email.pydiff --git a/README.md b/README.mdindex b41419ff..16744f96 100644--- a/README.md+++ b/README.md@@ -293,7 +293,7 @@ failure for anything branching on confidence.#### Where Jev leads-* **High-cardinality label spaces (>20 options at default settings):** On Banking77, Jev scores 0.870 (on 72 labels) while Laya scores 0.425 (on 77 labels at default 256-token head budget). This is an architectural token-budget constraint: options share a fixed `head_max_len` budget (192 tokens on English, 256 on multilingual), so 77 options receive only ~3 to 4 tokens per label, causing text to become indistinguishable. Jev supports up to 255 options out-of-the-box. While `laya-multilingual` supports 1,024 context (and up to 8,192 in the encoder) and you can raise `agent.cfg["head_max_len"] = 512` at runtime, Jev is currently better suited for 50+ options in a single prompt without tuning.+* **High-cardinality label spaces (>20 options at default settings):** On Banking77, Jev scores 0.870 (on 72 labels) while Laya scores 0.425 (on 77 labels at default 256-token head budget). This is an architectural token-budget constraint: options share a fixed `head_max_len` budget (192 tokens on English, 256 on multilingual), so 77 options receive only ~3 to 4 tokens per label, causing text to become indistinguishable. Jev supports up to 255 options out-of-the-box. While `laya-multilingual` supports 1,024 context (and up to 8,192 in the encoder) and you can raise `agent.cfg["head_max_len"] = 512` at runtime, Jev is currently better suited for 50+ options in a single prompt without tuning. `predict_shortlist` (see [Honest limits](#honest-limits)) keeps the top `k` labels with a caller-supplied embedding, then runs one forward pass on that shortlist.* **Soft distribution matching:** On typed-decisions, while Laya achieves higher argmax accuracy (0.766 vs 0.727), Jev achieves higher soft accuracy (0.580 vs 0.471) against the teacher's full probability distributions.* **Out-of-the-box raw calibration:** Before temperature scaling, the base checkpoint has higher raw ECE (0.213 vs 0.144). Laya achieves its 0.081 ECE after domain temperature fitting.@@ -368,7 +368,37 @@ temperatures at all, so fit them before relying on its probabilities.* `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`, ~768 tokens for state; mmBERT-base encoder supports up to 8,192 with RoPE).At default settings, a 77-option question like Banking77 allocates only `(256 - 16) // 77` ≈ 3–4 tokens per label, which causes accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question:1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct.- 2. Or split large option sets into a two-step coarse-to-fine hierarchical choice.+ 2. Or shortlist with embeddings and run one forward pass on the top `k` labels (`predict_shortlist`, example below). `predict` and `system_one` still score every criterion they are given.+ 3. Or split the label set yourself into a coarse question and a fine question.++```python+import laya++questions = {+ "intent": {+ "type": "choice",+ "instructions": "Which banking intent is this?",+ "criteria": {+ "card_arrival": "where is my card",+ "transfer_fee": "fee charged on a transfer",+ # ...the rest of a large label set+ },+ }+}+result = laya.predict_shortlist(+ agent,+ {"text": "I was charged twice for a transfer"},+ questions,+ embed_fn=laya.embed_fn_from_agent(agent), # or any callable: texts -> (n, dim)+ k=20,+)+result["shortlist"]["intent"]["labels"] # the top 20 labels sent to the model+```++`embed_fn(texts)` returns one vector per string. `embed_fn_from_agent` mean-pools the encoder already loaded on the agent; the decision head runs in the following `predict` / `system_one` call. Probabilities on a shortlisted choice are over those `k` labels. When `k` is at least the number of labels, the original question is passed through and `embed_fn` is not called.++[Issue #102](https://github.com/NandhaKishorM/laya/issues/102) reports that a top-20 zero-shot shortlist moved a BANKING77 run from 54.3% to 60.8% on the reporter's setup. Those figures are the reporter's; this repository has not remeasured them.+* Ordinal `score` questions are the weakest primitive (SST-5 0.372).* `laya` collapses outside English; `laya-multilingual` is weaker on English. Route, or pickdeliberately.diff --git a/laya/__init__.py b/laya/__init__.pyindex 1f1acb4f..64726091 100644--- a/laya/__init__.py+++ b/laya/__init__.py@@ -21,6 +21,7 @@ from .presets import (triage_questions,)from .router import DEFAULT_MODELS, RouteDecision, Router+from .shortlist import embed_fn_from_agent, predict_shortlist, shortlist_choice__version__ = "0.3.4"__all__ = [@@ -30,6 +31,9 @@ __all__ = ["Router","RouteDecision","DEFAULT_MODELS",+ "shortlist_choice",+ "predict_shortlist",+ "embed_fn_from_agent","detect_language","detect_script","is_english",diff --git a/laya/shortlist.py b/laya/shortlist.pynew file mode 100644index 00000000..b37638e9--- /dev/null+++ b/laya/shortlist.py@@ -0,0 +1,272 @@+"""Opt-in embedding shortlist for high-cardinality choice questions.++Choice options share one ``head_max_len`` budget, so a large label set leaves only a few+tokens per label. ``predict_shortlist`` embeds the state and each option with a+caller-supplied ``embed_fn``, keeps the top ``k``, and runs a single ``predict`` (or+``system_one``) on that reduced criteria set.++``Agent.predict`` and ``Agent.system_one`` are separate: they still score every criterion+they are given. This module does not change ``DecisionModel.forward`` and does not add a+second decision-model pass.++The coarse-to-fine pattern is the one the README recommends and the one reported in+https://github.com/NandhaKishorM/laya/issues/102. Ranking here is cosine similarity on+whatever vectors ``embed_fn`` returns. Issue #102's BANKING77 figures belong to that+report; this module does not measure them.+"""+import json+from typing import Any, Callable, Dict, List, Optional, Sequence++import numpy as np++from .common import render_options, serialize_state++DEFAULT_SHORTLIST_K = 20+++def shortlist_choice(+ state: Any,+ criteria: Any,+ embed_fn: Callable[[Sequence[str]], Any],+ k: int = DEFAULT_SHORTLIST_K,+ *,+ instructions: Optional[str] = None,+) -> List[Any]:+ """Return the top-``k`` choice labels for ``state``.++ ``embed_fn`` maps a list of strings to an array of shape ``(len(texts), dim)``.+ It is called once, with the query text first and then one string per option in+ criteria order. Option strings match ``render_options`` for a choice question.++ When ``k`` is at least the number of labels, every label is returned in its+ original order and ``embed_fn`` is not called.++ Ties keep the earlier label. A zero vector scores 0 and does not outrank a+ label that came before it.+ """+ labels, _scores, _passthrough, _n = _rank(state, criteria, embed_fn, k, instructions)+ return labels+++def predict_shortlist(+ agent: Any,+ state: Any,+ questions: Dict[str, Dict[str, Any]],+ embed_fn: Callable[[Sequence[str]], Any],+ k: int = DEFAULT_SHORTLIST_K,+ **predict_kwargs: Any,+) -> Dict[str, Any]:+ """Shortlist each choice question, then call ``predict`` or ``system_one`` once.++ Non-choice questions are forwarded unchanged. A choice whose label count is+ ``<= k`` is forwarded unchanged and does not call ``embed_fn``. The caller's+ ``questions`` dict is not mutated.++ The returned dict is the model result plus a ``shortlist`` entry. Probabilities+ on a shortlisted choice are over the kept labels only. ``shortlist[qid]`` holds+ ``labels`` (rank order), ``scores`` (cosine, or ``None`` when nothing was+ dropped), ``k``, ``n``, and ``passthrough``.++ Extra keyword arguments are forwarded to ``predict`` / ``system_one`` (for+ example ``model=`` on a ``Router``).+ """+ if not isinstance(questions, dict):+ raise TypeError("questions must be a dict of question id -> definition")+ checked = _check_k(k)+ reduced: Dict[str, Any] = {}+ meta: Dict[str, Dict[str, Any]] = {}+ for qid, qdef in questions.items():+ if not isinstance(qdef, dict) or qdef.get("type") != "choice":+ reduced[qid] = qdef+ continue+ if "criteria" not in qdef:+ raise ValueError("question %r is a choice but has no criteria" % (qid,))+ labels, scores, passthrough, n = _rank(+ state, qdef["criteria"], embed_fn, checked, qdef.get("instructions")+ )+ meta[qid] = {+ "labels": list(labels),+ "scores": scores,+ "k": checked,+ "n": n,+ "passthrough": passthrough,+ }+ if passthrough:+ reduced[qid] = qdef+ continue+ updated = dict(qdef)+ updated["criteria"] = _subset_criteria(qdef["criteria"], labels)+ reduced[qid] = updated++ result = _call_predict(agent, state, reduced, **predict_kwargs)+ if not isinstance(result, dict):+ raise TypeError(+ "predict/system_one must return a dict, got %s" % type(result).__name__+ )+ out = dict(result)+ out["shortlist"] = meta+ return out+++def embed_fn_from_agent(+ agent: Any,+ max_length: int = 512,+ batch_size: int = 32,+) -> Callable[[Sequence[str]], np.ndarray]:+ """Mean-pool the checkpoint encoder already loaded on ``agent``.++ The callable embeds a list of strings with ``agent.tok`` and ``agent.model.encoder``.+ It does not run the decision head and does not download weights. A dedicated+ bi-encoder passed as ``embed_fn`` will usually shortlist better; this helper is+ for callers who only have the Laya checkpoint in memory.++ Padding positions are excluded from the mean. The encoder's train/eval flag is+ left as the caller set it (a loaded ``Agent`` is already in eval).+ """+ if isinstance(max_length, bool) or not isinstance(max_length, int) or max_length < 1:+ raise ValueError("max_length must be a positive integer, got %r" % (max_length,))+ if isinstance(batch_size, bool) or not isinstance(batch_size, int) or batch_size < 1:+ raise ValueError("batch_size must be a positive integer, got %r" % (batch_size,))++ import torch++ tok = agent.tok+ encoder = agent.model.encoder+ device = agent.device++ def embed_fn(texts: Sequence[str]) -> np.ndarray:+ rows = ["" if text is None else str(text) for text in texts]+ hidden = _hidden_size(encoder)+ if not rows:+ return np.zeros((0, hidden), dtype=np.float32)+ parts: List[np.ndarray] = []+ for start in range(0, len(rows), batch_size):+ chunk = rows[start : start + batch_size]+ encoded = tok(+ chunk,+ padding=True,+ truncation=True,+ max_length=max_length,+ return_tensors="pt",+ )+ input_ids = encoded["input_ids"].to(device)+ attention_mask = encoded["attention_mask"].to(device)+ with torch.inference_mode():+ hidden_states = encoder(+ input_ids=input_ids, attention_mask=attention_mask+ ).last_hidden_state+ mask = attention_mask.unsqueeze(-1).to(dtype=hidden_states.dtype)+ pooled = (hidden_states * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1.0)+ parts.append(pooled.float().cpu().numpy())+ return np.concatenate(parts, axis=0)++ return embed_fn+++def _rank(state, criteria, embed_fn, k, instructions):+ checked = _check_k(k)+ items = _criteria_items(criteria)+ n = len(items)+ keys = [key for key, _value in items]+ if checked >= n:+ return list(keys), None, True, n+ query = _query_text(state, instructions)+ matrix = _embeddings(embed_fn, [query] + _option_texts(items))+ sims = _cosine(matrix[0], matrix[1:])+ order = np.argsort(-sims, kind="mergesort")[:checked]+ labels = [keys[int(i)] for i in order]+ scores = [float(sims[int(i)]) for i in order]+ return labels, scores, False, n+++def _check_k(k: int) -> int:+ if isinstance(k, bool) or not isinstance(k, int) or k < 1:+ raise ValueError("k must be a positive integer, got %r" % (k,))+ return k+++def _criteria_items(criteria):+ if isinstance(criteria, dict):+ items = list(criteria.items())+ elif isinstance(criteria, list):+ items = [(item, None) for item in criteria]+ else:+ raise TypeError(+ "choice criteria must be a dict or list, got %s" % type(criteria).__name__+ )+ if not items:+ raise ValueError("choice criteria must contain at least one option")+ seen = set()+ for key, _value in items:+ if key in seen:+ raise ValueError("choice criteria label %r is duplicated" % (key,))+ seen.add(key)+ return items+++def _option_texts(items) -> List[str]:+ crit = {key: value for key, value in items}+ rendered = render_options({"t": "choice", "ins": "", "crit": crit})+ texts = [piece if isinstance(piece, str) else str(piece) for piece in rendered]+ if len(texts) != len(items):+ raise ValueError("could not render every choice option")+ return texts+++def _query_text(state, instructions) -> str:+ body = serialize_state(state)+ if instructions is None or instructions == "":+ return body+ if not isinstance(instructions, str):+ instructions = json.dumps(instructions, ensure_ascii=False)+ return "%s\n%s" % (instructions, body)+++def _subset_criteria(criteria, labels):+ if isinstance(criteria, dict):+ return {label: criteria[label] for label in labels}+ return list(labels)+++def _embeddings(embed_fn, texts: Sequence[str]) -> np.ndarray:+ if not callable(embed_fn):+ raise TypeError("embed_fn must be callable")+ raw = embed_fn(list(texts))+ if hasattr(raw, "detach"):+ raw = raw.detach().float().cpu().numpy()+ arr = np.asarray(raw, dtype=np.float64)+ if arr.ndim != 2 or arr.shape[0] != len(texts) or arr.shape[1] < 1:+ raise ValueError(+ "embed_fn must return an array of shape (%d, dim), got %s"+ % (len(texts), tuple(arr.shape))+ )+ return np.nan_to_num(arr, copy=True, nan=0.0, posinf=0.0, neginf=0.0)+++def _cosine(query: np.ndarray, docs: np.ndarray) -> np.ndarray:+ qn = float(np.linalg.norm(query))+ dn = np.linalg.norm(docs, axis=1)+ sims = np.zeros(docs.shape[0], dtype=np.float64)+ if qn == 0.0:+ return sims+ denom = dn * qn+ ok = denom > 0.0+ if np.any(ok):+ sims[ok] = docs[ok] @ query / denom[ok]+ return sims+++def _call_predict(agent, state, questions, **predict_kwargs):+ fn = getattr(agent, "predict", None)+ if fn is None:+ fn = getattr(agent, "system_one", None)+ if fn is None:+ raise TypeError("agent must provide predict or system_one")+ return fn(state, questions, **predict_kwargs)+++def _hidden_size(encoder) -> int:+ size = getattr(getattr(encoder, "config", None), "hidden_size", None)+ if isinstance(size, bool) or not isinstance(size, int) or size < 1:+ return 0+ return sizediff --git a/tests/test_shortlist.py b/tests/test_shortlist.pynew file mode 100644index 00000000..362df649--- /dev/null+++ b/tests/test_shortlist.py@@ -0,0 +1,488 @@+"""Offline tests for the embedding shortlist. No checkpoint and no Hub download.++A fake ``embed_fn`` supplies vectors. ``predict`` / ``system_one`` are mocks, so the+decision model is never constructed.+"""+import inspect+import os+import sys++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))++import numpy as np # noqa: E402++import laya # noqa: E402+from laya.agent import Agent # noqa: E402+from laya.common import DecisionModel, render_options # noqa: E402+from laya.shortlist import ( # noqa: E402+ embed_fn_from_agent,+ predict_shortlist,+ shortlist_choice,+)++PASS, FAIL = [], []+++def check(name, got, want):+ if got == want:+ PASS.append(name)+ else:+ FAIL.append("%s: got %r, want %r" % (name, got, want))+++def check_true(name, cond, detail=""):+ if cond:+ PASS.append(name)+ else:+ FAIL.append("%s %s" % (name, detail))+++def check_raises(name, fn, exc=ValueError):+ try:+ fn()+ except exc:+ PASS.append(name)+ except Exception as e:+ FAIL.append("%s: raised %s (%s), want %s" % (name, type(e).__name__, e, exc.__name__))+ else:+ FAIL.append("%s: no exception" % name)+++class TableEmbed:+ def __init__(self, vectors):+ self.vectors = vectors+ self.calls = []++ def __call__(self, texts):+ self.calls.append(list(texts))+ missing = [t for t in texts if t not in self.vectors]+ if missing:+ raise AssertionError("unexpected texts %r" % (missing,))+ return [self.vectors[t] for t in texts]+++class BoomEmbed:+ def __call__(self, texts):+ raise AssertionError("embed_fn should not run when k >= n")+++class Recorder:+ """Stand-in for Agent. Records the questions handed to predict."""++ def __init__(self):+ self.calls = []+ self.system_one_calls = 0++ def predict(self, state, questions, **kwargs):+ self.calls.append((state, questions, kwargs))+ return {"model": "fake", "answers": _answers(questions)}++ def system_one(self, state, questions, **kwargs):+ self.system_one_calls += 1+ self.calls.append((state, questions, kwargs))+ return {"model": "fake", "answers": _answers(questions)}+++def _answers(questions):+ answers = {}+ for qid, qdef in questions.items():+ if isinstance(qdef, dict) and qdef.get("type") == "choice":+ crit = qdef["criteria"]+ keys = list(crit.keys()) if isinstance(crit, dict) else list(crit)+ answers[qid] = {"type": "choice", "choice": keys[0]}+ return answers+++# Vectors: query [1, 0]. alpha and delta tie at cosine 1; gamma is 0.6; beta is 0.+# Stable order must keep alpha ahead of delta.+VEC = {+ "pay me": [1.0, 0.0],+ "alpha": [1.0, 0.0],+ "beta": [0.0, 1.0],+ "gamma": [0.6, 0.8],+ "delta": [1.0, 0.0],+ "zero": [0.0, 0.0],+}+CRITERIA = {"alpha": None, "beta": "", "gamma": "mid", "delta": "same"}+# option texts follow render_options: bare key when value is None or ""+OPTION_TEXTS = {+ "alpha": [1.0, 0.0],+ "beta": [0.0, 1.0],+ "gamma: mid": [0.6, 0.8],+ "delta: same": [1.0, 0.0],+}+++def _embed_for(query_text, option_vectors):+ vectors = {query_text: [1.0, 0.0]}+ vectors.update(option_vectors)+ return TableEmbed(vectors)+++# ---------------------------------------------------------------- exports and default path+check_true("export/shortlist_choice", laya.shortlist_choice is shortlist_choice)+check_true("export/predict_shortlist", laya.predict_shortlist is predict_shortlist)+check_true("export/embed_fn_from_agent", laya.embed_fn_from_agent is embed_fn_from_agent)+for _name in ("shortlist_choice", "predict_shortlist", "embed_fn_from_agent"):+ check_true("all/%s" % _name, _name in laya.__all__)++_predict_src = inspect.getsource(Agent.system_one)+check_true("default/predict is system_one", Agent.predict is Agent.system_one)+check_true("default/system_one has no shortlist", "shortlist" not in _predict_src and "embed_fn" not in _predict_src)+_forward_src = inspect.getsource(DecisionModel.forward)+check_true("default/forward has no shortlist", "shortlist" not in _forward_src and "embed_fn" not in _forward_src)+++# ---------------------------------------------------------------- deterministic top-k+embed = _embed_for("pay me", OPTION_TEXTS)+labels = shortlist_choice("pay me", CRITERIA, embed, k=2)+check("topk/k=2 keeps the cosine tie in input order", labels, ["alpha", "delta"])+check("topk/one embed call", len(embed.calls), 1)+check("topk/query is the first text", embed.calls[0][0], "pay me")+check(+ "topk/option texts match render_options",+ embed.calls[0][1:],+ render_options({"t": "choice", "ins": "", "crit": CRITERIA}),+)+check("topk/k=1 is the earliest max", shortlist_choice("pay me", CRITERIA, embed, k=1), ["alpha"])+check(+ "topk/k=3 appends the next cosine",+ shortlist_choice("pay me", CRITERIA, embed, k=3),+ ["alpha", "delta", "gamma"],+)++# all-zero query: every cosine is 0, so the earliest labels win+zero_q = TableEmbed({+ "pay me": [0.0, 0.0],+ "alpha": [1.0, 0.0],+ "beta": [0.0, 1.0],+ "gamma: mid": [0.6, 0.8],+ "delta: same": [3.0, 4.0],+})+check(+ "topk/zero query keeps original order",+ shortlist_choice("pay me", CRITERIA, zero_q, k=2),+ ["alpha", "beta"],+)++# non-finite option vector is treated as 0 and loses to a real match+nan_embed = TableEmbed({+ "pay me": [1.0, 0.0],+ "alpha": [float("nan"), float("nan")],+ "beta": [1.0, 0.0],+})+check(+ "topk/nan vector sorts behind a finite match",+ shortlist_choice("pay me", {"alpha": None, "beta": None}, nan_embed, k=1),+ ["beta"],+)+++# ---------------------------------------------------------------- list criteria and instructions+list_embed = TableEmbed({+ "Classify\npay me": [0.0, 1.0],+ "alpha": [1.0, 0.0],+ "beta": [0.0, 1.0],+ "gamma": [0.0, 0.2],+})+check(+ "list/instructions change the query and the winner",+ shortlist_choice("pay me", ["alpha", "beta", "gamma"], list_embed, k=2, instructions="Classify"),+ ["beta", "gamma"],+)+check("list/query text includes instructions", list_embed.calls[0][0], "Classify\npay me")+check("list/option texts are the labels", list_embed.calls[0][1:], ["alpha", "beta", "gamma"])++state = {"text": "hi"}+dict_embed = TableEmbed({+ 'Classify\n{"text": "hi"}': [1.0, 0.0],+ "alpha": [1.0, 0.0],+ "beta": [0.0, 1.0],+})+check(+ "query/dict state is serialized",+ shortlist_choice(state, ["alpha", "beta"], dict_embed, k=1, instructions="Classify"),+ ["alpha"],+)+check("query/json matches serialize_state", dict_embed.calls[0][0], 'Classify\n{"text": "hi"}')++# 0 and False are real criterion values, so they are part of the embedded text+rich = {"zero": 0, "no": False, "bare": None, "named": {"desc": "payments"}}+rich_rendered = render_options({"t": "choice", "ins": "", "crit": rich})+rich_embed = TableEmbed({"pay me": [1.0, 0.0], **{text: [1.0, 0.0] for text in rich_rendered}})+shortlist_choice("pay me", rich, rich_embed, k=1)+check("render/0 and False stay in the option text", rich_embed.calls[0][1:], rich_rendered)+++# ---------------------------------------------------------------- k >= n pass-through+boom = BoomEmbed()+check("pass/k == n returns every label in order", shortlist_choice("pay me", CRITERIA, boom, k=4), list(CRITERIA))+check("pass/k > n returns every label in order", shortlist_choice("pay me", CRITERIA, boom, k=20), list(CRITERIA))+++# ---------------------------------------------------------------- mock predict sees only k criteria+sentinel = {"desc": "payments"}+full = {"billing": sentinel, "tech": "bugs", "sales": None, "other": "misc"}+full_vectors = {+ "Which desk?\nI was charged twice": [1.0, 0.0],+ 'billing: {"desc": "payments"}': [0.0, 1.0],+ "tech: bugs": [1.0, 0.0],+ "sales": [0.2, 0.2],+ "other: misc": [0.0, 1.0],+}+# cosine vs [1, 0]: tech=1, sales=0.707, billing=0, other=0. k=2 -> tech, sales.+agent = Recorder()+score_q = {"type": "score", "instructions": "How urgent?", "criteria": ["low", "mid", "high", "now"]}+noul_q = {"type": "noul", "instructions": "Is a refund requested?"}+questions = {+ "intent": {+ "type": "choice",+ "instructions": "Which desk?",+ "criteria": full,+ },+ "urgency": score_q,+ "refund": noul_q,+ "note": "leave me alone",+}+state = "I was charged twice"+result = predict_shortlist(agent, state, questions, TableEmbed(full_vectors), k=2, model="english")++check("predict/called once", len(agent.calls), 1)+check("predict/system_one not used when predict exists", agent.system_one_calls, 0)+got_state, got_questions, got_kwargs = agent.calls[0]+check_true("predict/state is the same object", got_state is state)+check("predict/kwargs forwarded", got_kwargs, {"model": "english"})+check("predict/choice criteria are only the top 2", list(got_questions["intent"]["criteria"]), ["tech", "sales"])+check_true(+ "predict/kept description is the original object",+ got_questions["intent"]["criteria"]["tech"] == "bugs",+)+check_true("predict/score question is the same object", got_questions["urgency"] is score_q)+check_true("predict/noul question is the same object", got_questions["refund"] is noul_q)+check_true("predict/non-dict question is the same object", got_questions["note"] is questions["note"])+check("caller/criteria unchanged", questions["intent"]["criteria"], full)+check_true("caller/criteria object unchanged", questions["intent"]["criteria"] is full)+check("caller/sentinel intact", full["billing"], sentinel)++internal = Agent._to_internal(got_questions["intent"])+check("predict/internal choice keys are the shortlist", list(internal["crit"]), ["tech", "sales"])+check("result/choice is the mock's first shortlisted label", result["answers"]["intent"]["choice"], "tech")+check("result/shortlist labels", result["shortlist"]["intent"]["labels"], ["tech", "sales"])+check_true(+ "result/scores descend",+ result["shortlist"]["intent"]["scores"][0] > result["shortlist"]["intent"]["scores"][1] > 0,+)+check("result/shortlist k and n", (result["shortlist"]["intent"]["k"], result["shortlist"]["intent"]["n"]), (2, 4))+check("result/not a passthrough", result["shortlist"]["intent"]["passthrough"], False)+check_true("result/non-choice questions are absent from shortlist meta", "urgency" not in result["shortlist"])+# The predict return is copied before shortlist is attached.+held = {}+++class Holding:+ def predict(self, state, questions, **kwargs):+ held["questions"] = questions+ held["result"] = {"model": "fake", "answers": {}}+ return held["result"]+++out = predict_shortlist(Holding(), "pay me", {"intent": {"type": "choice", "criteria": CRITERIA}}, BoomEmbed(), k=4)+check_true("result/shortlist key is on the copy", "shortlist" in out and "shortlist" not in held["result"])+++# ---------------------------------------------------------------- k >= n pass-through reaches predict unchanged+passthrough_agent = Recorder()+original_q = {"type": "choice", "instructions": "Which desk?", "criteria": full}+out = predict_shortlist(+ passthrough_agent,+ "I was charged twice",+ {"intent": original_q},+ BoomEmbed(),+ k=4,+)+check_true("pass/predict received the original question", passthrough_agent.calls[0][1]["intent"] is original_q)+check("pass/metadata labels are the full set", out["shortlist"]["intent"]["labels"], list(full))+check("pass/scores omitted", out["shortlist"]["intent"]["scores"], None)+check("pass/flag", out["shortlist"]["intent"]["passthrough"], True)+check("pass/k > n flag", predict_shortlist(+ Recorder(), "x", {"intent": original_q}, BoomEmbed(), k=99+)["shortlist"]["intent"]["passthrough"], True)++# list criteria stay a list, in rank order, and Agent._to_internal accepts them+list_agent = Recorder()+list_q_embed = TableEmbed({+ "Which?\nhello": [1.0, 0.0],+ "alpha": [0.0, 1.0],+ "beta": [1.0, 0.0],+ "gamma": [0.0, 0.0],+})+list_questions = {"intent": {"type": "choice", "instructions": "Which?", "criteria": ["alpha", "beta", "gamma"]}}+predict_shortlist(list_agent, "hello", list_questions, list_q_embed, k=2)+received = list_agent.calls[0][1]["intent"]["criteria"]+check("list/predict receives a list of k labels", received, ["beta", "alpha"])+check("list/caller criteria unchanged", list_questions["intent"]["criteria"], ["alpha", "beta", "gamma"])+check(+ "list/internal keys follow the shortlist",+ list(Agent._to_internal(list_agent.calls[0][1]["intent"])["crit"]),+ ["beta", "alpha"],+)+++# ---------------------------------------------------------------- system_one-only agent+class SystemOneOnly:+ def __init__(self):+ self.questions = None++ def system_one(self, state, questions):+ self.questions = questions+ return {"answers": {"intent": {"choice": "beta"}}}+++only = SystemOneOnly()+out = predict_shortlist(only, "hello", list_questions, list_q_embed, k=1)+check("system_one/used when predict is absent", out["answers"]["intent"]["choice"], "beta")+check("system_one/criteria length is k", len(only.questions["intent"]["criteria"]), 1)+++# ---------------------------------------------------------------- errors+check_raises("err/k=0", lambda: shortlist_choice("pay me", CRITERIA, embed, k=0))+check_raises("err/k negative", lambda: shortlist_choice("pay me", CRITERIA, embed, k=-3))+check_raises("err/k bool", lambda: shortlist_choice("pay me", CRITERIA, embed, k=True))+check_raises("err/k float", lambda: shortlist_choice("pay me", CRITERIA, embed, k=1.5))+check_raises("err/k str", lambda: shortlist_choice("pay me", CRITERIA, embed, k="2"))+check_raises("err/empty dict", lambda: shortlist_choice("pay me", {}, embed, k=1))+check_raises("err/empty list", lambda: shortlist_choice("pay me", [], embed, k=1))+check_raises("err/criteria tuple", lambda: shortlist_choice("pay me", ("alpha", "beta"), embed, k=1), TypeError)+check_raises("err/duplicate label", lambda: shortlist_choice("pay me", ["alpha", "alpha"], embed, k=1))+check_raises(+ "err/missing criteria",+ lambda: predict_shortlist(Recorder(), "pay me", {"intent": {"type": "choice"}}, embed, k=1),+)+check_raises("err/questions not a dict", lambda: predict_shortlist(Recorder(), "pay me", [], embed, k=1), TypeError)+++def _bad_shape(texts):+ return np.zeros((1, 4))+++bad_agent = Recorder()+check_raises(+ "err/bad embed shape",+ lambda: predict_shortlist(bad_agent, "pay me", {"intent": original_q}, _bad_shape, k=2),+)+check("err/bad shape does not call predict", len(bad_agent.calls), 0)+++def _torch_rows(texts):+ import torch+ rows = []+ for i, _text in enumerate(texts):+ rows.append([1.0, 0.0] if i == 0 or i == 1 else [0.0, 1.0])+ return torch.tensor(rows)+++check(+ "embed/torch tensor return is accepted",+ shortlist_choice("pay me", {"alpha": None, "beta": None}, _torch_rows, k=1),+ ["alpha"],+)+++# ---------------------------------------------------------------- encoder mean-pool, no Hub+import torch # noqa: E402+++class _Out:+ def __init__(self, hidden):+ self.last_hidden_state = hidden+++class TinyEncoder(torch.nn.Module):+ def __init__(self):+ super().__init__()+ self.config = type("Cfg", (), {"hidden_size": 2})()+ self.forwards = 0++ def forward(self, input_ids, attention_mask):+ self.forwards += 1+ # Channel 0 is the token id; channel 1 is 1. Padding must drop out of the mean.+ ids = input_ids.float()+ hidden = torch.stack([ids, torch.ones_like(ids)], dim=-1)+ return _Out(hidden)+++class TinyTok:+ def __init__(self):+ self.kwargs = None+ self.batches = []++ def __call__(self, texts, padding=True, truncation=True, max_length=256, return_tensors="pt"):+ self.kwargs = {"padding": padding, "truncation": truncation, "max_length": max_length}+ self.batches.append(list(texts))+ rows = [[(ord(ch) % 5) + 1 for ch in text][:max_length] or [1] for text in texts]+ width = max(len(row) for row in rows)+ input_ids, mask = [], []+ for row in rows:+ pad = width - len(row)+ input_ids.append(row + [0] * pad)+ mask.append([1] * len(row) + [0] * pad)+ return {+ "input_ids": torch.tensor(input_ids, dtype=torch.long),+ "attention_mask": torch.tensor(mask, dtype=torch.long),+ }+++class TinyAgent:+ def __init__(self):+ self.tok = TinyTok()+ self.encoder = TinyEncoder()+ self.model = type("M", (), {"encoder": self.encoder})()+ self.device = torch.device("cpu")+ self.forward_calls = 0++ def forward(self, *args, **kwargs):+ self.forward_calls += 1+ raise AssertionError("decision forward must not run during embedding")+++tiny = TinyAgent()+fn = embed_fn_from_agent(tiny, max_length=32, batch_size=2)+first = fn(["ab", "a"])+second = fn(["ab", "a"])+check("encoder/shape", first.shape, (2, 2))+check_true("encoder/deterministic", np.array_equal(first, second))+check("encoder/one batch for both strings", tiny.tok.batches, [["ab", "a"], ["ab", "a"]])+check("encoder/max_length forwarded", tiny.tok.kwargs["max_length"], 32)+check("encoder/decision forward not used", tiny.forward_calls, 0)++# "a" is one real token (id = ord('a') % 5 + 1) plus padding. Masked mean keeps that id.+token_id = (ord("a") % 5) + 1+check("encoder/padding excluded from mean", first[1].tolist(), [float(token_id), 1.0])+# "ab" has two real tokens and no reliance on padding once both are kept.+ids = [(ord(ch) % 5) + 1 for ch in "ab"]+check("encoder/mean of real tokens", first[0].tolist(), [sum(ids) / 2.0, 1.0])++empty = fn([])+check("encoder/empty input shape", empty.shape, (0, 2))+check_raises("encoder/bad max_length", lambda: embed_fn_from_agent(tiny, max_length=0))+check_raises("encoder/bad batch_size", lambda: embed_fn_from_agent(tiny, batch_size=True))+++# ---------------------------------------------------------------- end-to-end helper: shortlist then the real question normalizer only+# Confirms a reduced choice still has one criterion per kept label, which is what+# system_one would score. No weights, no forward.+pipe_agent = Recorder()+pipe_q = {+ "intent": {"type": "choice", "instructions": "Which desk?", "criteria": full},+ "urgency": score_q,+}+piped = predict_shortlist(pipe_agent, "I was charged twice", pipe_q, TableEmbed(full_vectors), k=2)+scored = Agent._to_internal(pipe_agent.calls[0][1]["intent"])+check("pipe/marker count equals k", len(scored["crit"]), 2)+check("pipe/answer choice is inside the shortlist", piped["answers"]["intent"]["choice"] in piped["shortlist"]["intent"]["labels"], True)+++print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))+for f in FAIL:+ print(" FAIL " + f)+sys.exit(1 if FAIL else 0)