Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat: opt-in embedding shortlist for high-cardinality choice (#106)

Nithilan Kumaran62f4f158c968
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index d04d70c7..af24e1db 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -46,6 +46,7 @@ jobs:
python tests/test_router.py
python tests/test_criteria.py
python tests/test_download.py
+ python tests/test_shortlist.py
- name: Email cleaning tests
run: python tests/test_email.py
diff --git a/README.md b/README.md
index b41419ff..16744f96 100644
--- a/README.md
+++ b/README.md
@@ -293,7 +293,7 @@ failure for anything branching on confidence.
#### Where Jev leads
-* **High-cardinality label spaces (>20 options at default settings):** On Banking77, Jev scores 0.870 (on 72 labels) while Laya scores 0.425 (on 77 labels at default 256-token head budget). This is an architectural token-budget constraint: options share a fixed `head_max_len` budget (192 tokens on English, 256 on multilingual), so 77 options receive only ~3 to 4 tokens per label, causing text to become indistinguishable. Jev supports up to 255 options out-of-the-box. While `laya-multilingual` supports 1,024 context (and up to 8,192 in the encoder) and you can raise `agent.cfg["head_max_len"] = 512` at runtime, Jev is currently better suited for 50+ options in a single prompt without tuning.
+* **High-cardinality label spaces (>20 options at default settings):** On Banking77, Jev scores 0.870 (on 72 labels) while Laya scores 0.425 (on 77 labels at default 256-token head budget). This is an architectural token-budget constraint: options share a fixed `head_max_len` budget (192 tokens on English, 256 on multilingual), so 77 options receive only ~3 to 4 tokens per label, causing text to become indistinguishable. Jev supports up to 255 options out-of-the-box. While `laya-multilingual` supports 1,024 context (and up to 8,192 in the encoder) and you can raise `agent.cfg["head_max_len"] = 512` at runtime, Jev is currently better suited for 50+ options in a single prompt without tuning. `predict_shortlist` (see [Honest limits](#honest-limits)) keeps the top `k` labels with a caller-supplied embedding, then runs one forward pass on that shortlist.
* **Soft distribution matching:** On typed-decisions, while Laya achieves higher argmax accuracy (0.766 vs 0.727), Jev achieves higher soft accuracy (0.580 vs 0.471) against the teacher's full probability distributions.
* **Out-of-the-box raw calibration:** Before temperature scaling, the base checkpoint has higher raw ECE (0.213 vs 0.144). Laya achieves its 0.081 ECE after domain temperature fitting.
@@ -368,7 +368,37 @@ temperatures at all, so fit them before relying on its probabilities.
* `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`, ~768 tokens for state; mmBERT-base encoder supports up to 8,192 with RoPE).
At default settings, a 77-option question like Banking77 allocates only `(256 - 16) // 77` ≈ 3–4 tokens per label, which causes accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question:
1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct.
- 2. Or split large option sets into a two-step coarse-to-fine hierarchical choice.
+ 2. Or shortlist with embeddings and run one forward pass on the top `k` labels (`predict_shortlist`, example below). `predict` and `system_one` still score every criterion they are given.
+ 3. Or split the label set yourself into a coarse question and a fine question.
+
+```python
+import laya
+
+questions = {
+ "intent": {
+ "type": "choice",
+ "instructions": "Which banking intent is this?",
+ "criteria": {
+ "card_arrival": "where is my card",
+ "transfer_fee": "fee charged on a transfer",
+ # ...the rest of a large label set
+ },
+ }
+}
+result = laya.predict_shortlist(
+ agent,
+ {"text": "I was charged twice for a transfer"},
+ questions,
+ embed_fn=laya.embed_fn_from_agent(agent), # or any callable: texts -> (n, dim)
+ k=20,
+)
+result["shortlist"]["intent"]["labels"] # the top 20 labels sent to the model
+```
+
+`embed_fn(texts)` returns one vector per string. `embed_fn_from_agent` mean-pools the encoder already loaded on the agent; the decision head runs in the following `predict` / `system_one` call. Probabilities on a shortlisted choice are over those `k` labels. When `k` is at least the number of labels, the original question is passed through and `embed_fn` is not called.
+
+[Issue #102](https://github.com/NandhaKishorM/laya/issues/102) reports that a top-20 zero-shot shortlist moved a BANKING77 run from 54.3% to 60.8% on the reporter's setup. Those figures are the reporter's; this repository has not remeasured them.
+
* Ordinal `score` questions are the weakest primitive (SST-5 0.372).
* `laya` collapses outside English; `laya-multilingual` is weaker on English. Route, or pick
deliberately.
diff --git a/laya/__init__.py b/laya/__init__.py
index 1f1acb4f..64726091 100644
--- a/laya/__init__.py
+++ b/laya/__init__.py
@@ -21,6 +21,7 @@ from .presets import (
triage_questions,
)
from .router import DEFAULT_MODELS, RouteDecision, Router
+from .shortlist import embed_fn_from_agent, predict_shortlist, shortlist_choice
__version__ = "0.3.4"
__all__ = [
@@ -30,6 +31,9 @@ __all__ = [
"Router",
"RouteDecision",
"DEFAULT_MODELS",
+ "shortlist_choice",
+ "predict_shortlist",
+ "embed_fn_from_agent",
"detect_language",
"detect_script",
"is_english",
diff --git a/laya/shortlist.py b/laya/shortlist.py
new file mode 100644
index 00000000..b37638e9
--- /dev/null
+++ b/laya/shortlist.py
@@ -0,0 +1,272 @@
+"""Opt-in embedding shortlist for high-cardinality choice questions.
+
+Choice options share one ``head_max_len`` budget, so a large label set leaves only a few
+tokens per label. ``predict_shortlist`` embeds the state and each option with a
+caller-supplied ``embed_fn``, keeps the top ``k``, and runs a single ``predict`` (or
+``system_one``) on that reduced criteria set.
+
+``Agent.predict`` and ``Agent.system_one`` are separate: they still score every criterion
+they are given. This module does not change ``DecisionModel.forward`` and does not add a
+second decision-model pass.
+
+The coarse-to-fine pattern is the one the README recommends and the one reported in
+https://github.com/NandhaKishorM/laya/issues/102. Ranking here is cosine similarity on
+whatever vectors ``embed_fn`` returns. Issue #102's BANKING77 figures belong to that
+report; this module does not measure them.
+"""
+import json
+from typing import Any, Callable, Dict, List, Optional, Sequence
+
+import numpy as np
+
+from .common import render_options, serialize_state
+
+DEFAULT_SHORTLIST_K = 20
+
+
+def shortlist_choice(
+ state: Any,
+ criteria: Any,
+ embed_fn: Callable[[Sequence[str]], Any],
+ k: int = DEFAULT_SHORTLIST_K,
+ *,
+ instructions: Optional[str] = None,
+) -> List[Any]:
+ """Return the top-``k`` choice labels for ``state``.
+
+ ``embed_fn`` maps a list of strings to an array of shape ``(len(texts), dim)``.
+ It is called once, with the query text first and then one string per option in
+ criteria order. Option strings match ``render_options`` for a choice question.
+
+ When ``k`` is at least the number of labels, every label is returned in its
+ original order and ``embed_fn`` is not called.
+
+ Ties keep the earlier label. A zero vector scores 0 and does not outrank a
+ label that came before it.
+ """
+ labels, _scores, _passthrough, _n = _rank(state, criteria, embed_fn, k, instructions)
+ return labels
+
+
+def predict_shortlist(
+ agent: Any,
+ state: Any,
+ questions: Dict[str, Dict[str, Any]],
+ embed_fn: Callable[[Sequence[str]], Any],
+ k: int = DEFAULT_SHORTLIST_K,
+ **predict_kwargs: Any,
+) -> Dict[str, Any]:
+ """Shortlist each choice question, then call ``predict`` or ``system_one`` once.
+
+ Non-choice questions are forwarded unchanged. A choice whose label count is
+ ``<= k`` is forwarded unchanged and does not call ``embed_fn``. The caller's
+ ``questions`` dict is not mutated.
+
+ The returned dict is the model result plus a ``shortlist`` entry. Probabilities
+ on a shortlisted choice are over the kept labels only. ``shortlist[qid]`` holds
+ ``labels`` (rank order), ``scores`` (cosine, or ``None`` when nothing was
+ dropped), ``k``, ``n``, and ``passthrough``.
+
+ Extra keyword arguments are forwarded to ``predict`` / ``system_one`` (for
+ example ``model=`` on a ``Router``).
+ """
+ if not isinstance(questions, dict):
+ raise TypeError("questions must be a dict of question id -> definition")
+ checked = _check_k(k)
+ reduced: Dict[str, Any] = {}
+ meta: Dict[str, Dict[str, Any]] = {}
+ for qid, qdef in questions.items():
+ if not isinstance(qdef, dict) or qdef.get("type") != "choice":
+ reduced[qid] = qdef
+ continue
+ if "criteria" not in qdef:
+ raise ValueError("question %r is a choice but has no criteria" % (qid,))
+ labels, scores, passthrough, n = _rank(
+ state, qdef["criteria"], embed_fn, checked, qdef.get("instructions")
+ )
+ meta[qid] = {
+ "labels": list(labels),
+ "scores": scores,
+ "k": checked,
+ "n": n,
+ "passthrough": passthrough,
+ }
+ if passthrough:
+ reduced[qid] = qdef
+ continue
+ updated = dict(qdef)
+ updated["criteria"] = _subset_criteria(qdef["criteria"], labels)
+ reduced[qid] = updated
+
+ result = _call_predict(agent, state, reduced, **predict_kwargs)
+ if not isinstance(result, dict):
+ raise TypeError(
+ "predict/system_one must return a dict, got %s" % type(result).__name__
+ )
+ out = dict(result)
+ out["shortlist"] = meta
+ return out
+
+
+def embed_fn_from_agent(
+ agent: Any,
+ max_length: int = 512,
+ batch_size: int = 32,
+) -> Callable[[Sequence[str]], np.ndarray]:
+ """Mean-pool the checkpoint encoder already loaded on ``agent``.
+
+ The callable embeds a list of strings with ``agent.tok`` and ``agent.model.encoder``.
+ It does not run the decision head and does not download weights. A dedicated
+ bi-encoder passed as ``embed_fn`` will usually shortlist better; this helper is
+ for callers who only have the Laya checkpoint in memory.
+
+ Padding positions are excluded from the mean. The encoder's train/eval flag is
+ left as the caller set it (a loaded ``Agent`` is already in eval).
+ """
+ if isinstance(max_length, bool) or not isinstance(max_length, int) or max_length < 1:
+ raise ValueError("max_length must be a positive integer, got %r" % (max_length,))
+ if isinstance(batch_size, bool) or not isinstance(batch_size, int) or batch_size < 1:
+ raise ValueError("batch_size must be a positive integer, got %r" % (batch_size,))
+
+ import torch
+
+ tok = agent.tok
+ encoder = agent.model.encoder
+ device = agent.device
+
+ def embed_fn(texts: Sequence[str]) -> np.ndarray:
+ rows = ["" if text is None else str(text) for text in texts]
+ hidden = _hidden_size(encoder)
+ if not rows:
+ return np.zeros((0, hidden), dtype=np.float32)
+ parts: List[np.ndarray] = []
+ for start in range(0, len(rows), batch_size):
+ chunk = rows[start : start + batch_size]
+ encoded = tok(
+ chunk,
+ padding=True,
+ truncation=True,
+ max_length=max_length,
+ return_tensors="pt",
+ )
+ input_ids = encoded["input_ids"].to(device)
+ attention_mask = encoded["attention_mask"].to(device)
+ with torch.inference_mode():
+ hidden_states = encoder(
+ input_ids=input_ids, attention_mask=attention_mask
+ ).last_hidden_state
+ mask = attention_mask.unsqueeze(-1).to(dtype=hidden_states.dtype)
+ pooled = (hidden_states * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1.0)
+ parts.append(pooled.float().cpu().numpy())
+ return np.concatenate(parts, axis=0)
+
+ return embed_fn
+
+
+def _rank(state, criteria, embed_fn, k, instructions):
+ checked = _check_k(k)
+ items = _criteria_items(criteria)
+ n = len(items)
+ keys = [key for key, _value in items]
+ if checked >= n:
+ return list(keys), None, True, n
+ query = _query_text(state, instructions)
+ matrix = _embeddings(embed_fn, [query] + _option_texts(items))
+ sims = _cosine(matrix[0], matrix[1:])
+ order = np.argsort(-sims, kind="mergesort")[:checked]
+ labels = [keys[int(i)] for i in order]
+ scores = [float(sims[int(i)]) for i in order]
+ return labels, scores, False, n
+
+
+def _check_k(k: int) -> int:
+ if isinstance(k, bool) or not isinstance(k, int) or k < 1:
+ raise ValueError("k must be a positive integer, got %r" % (k,))
+ return k
+
+
+def _criteria_items(criteria):
+ if isinstance(criteria, dict):
+ items = list(criteria.items())
+ elif isinstance(criteria, list):
+ items = [(item, None) for item in criteria]
+ else:
+ raise TypeError(
+ "choice criteria must be a dict or list, got %s" % type(criteria).__name__
+ )
+ if not items:
+ raise ValueError("choice criteria must contain at least one option")
+ seen = set()
+ for key, _value in items:
+ if key in seen:
+ raise ValueError("choice criteria label %r is duplicated" % (key,))
+ seen.add(key)
+ return items
+
+
+def _option_texts(items) -> List[str]:
+ crit = {key: value for key, value in items}
+ rendered = render_options({"t": "choice", "ins": "", "crit": crit})
+ texts = [piece if isinstance(piece, str) else str(piece) for piece in rendered]
+ if len(texts) != len(items):
+ raise ValueError("could not render every choice option")
+ return texts
+
+
+def _query_text(state, instructions) -> str:
+ body = serialize_state(state)
+ if instructions is None or instructions == "":
+ return body
+ if not isinstance(instructions, str):
+ instructions = json.dumps(instructions, ensure_ascii=False)
+ return "%s\n%s" % (instructions, body)
+
+
+def _subset_criteria(criteria, labels):
+ if isinstance(criteria, dict):
+ return {label: criteria[label] for label in labels}
+ return list(labels)
+
+
+def _embeddings(embed_fn, texts: Sequence[str]) -> np.ndarray:
+ if not callable(embed_fn):
+ raise TypeError("embed_fn must be callable")
+ raw = embed_fn(list(texts))
+ if hasattr(raw, "detach"):
+ raw = raw.detach().float().cpu().numpy()
+ arr = np.asarray(raw, dtype=np.float64)
+ if arr.ndim != 2 or arr.shape[0] != len(texts) or arr.shape[1] < 1:
+ raise ValueError(
+ "embed_fn must return an array of shape (%d, dim), got %s"
+ % (len(texts), tuple(arr.shape))
+ )
+ return np.nan_to_num(arr, copy=True, nan=0.0, posinf=0.0, neginf=0.0)
+
+
+def _cosine(query: np.ndarray, docs: np.ndarray) -> np.ndarray:
+ qn = float(np.linalg.norm(query))
+ dn = np.linalg.norm(docs, axis=1)
+ sims = np.zeros(docs.shape[0], dtype=np.float64)
+ if qn == 0.0:
+ return sims
+ denom = dn * qn
+ ok = denom > 0.0
+ if np.any(ok):
+ sims[ok] = docs[ok] @ query / denom[ok]
+ return sims
+
+
+def _call_predict(agent, state, questions, **predict_kwargs):
+ fn = getattr(agent, "predict", None)
+ if fn is None:
+ fn = getattr(agent, "system_one", None)
+ if fn is None:
+ raise TypeError("agent must provide predict or system_one")
+ return fn(state, questions, **predict_kwargs)
+
+
+def _hidden_size(encoder) -> int:
+ size = getattr(getattr(encoder, "config", None), "hidden_size", None)
+ if isinstance(size, bool) or not isinstance(size, int) or size < 1:
+ return 0
+ return size
diff --git a/tests/test_shortlist.py b/tests/test_shortlist.py
new file mode 100644
index 00000000..362df649
--- /dev/null
+++ b/tests/test_shortlist.py
@@ -0,0 +1,488 @@
+"""Offline tests for the embedding shortlist. No checkpoint and no Hub download.
+
+A fake ``embed_fn`` supplies vectors. ``predict`` / ``system_one`` are mocks, so the
+decision model is never constructed.
+"""
+import inspect
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import numpy as np # noqa: E402
+
+import laya # noqa: E402
+from laya.agent import Agent # noqa: E402
+from laya.common import DecisionModel, render_options # noqa: E402
+from laya.shortlist import ( # noqa: E402
+ embed_fn_from_agent,
+ predict_shortlist,
+ shortlist_choice,
+)
+
+PASS, FAIL = [], []
+
+
+def check(name, got, want):
+ if got == want:
+ PASS.append(name)
+ else:
+ FAIL.append("%s: got %r, want %r" % (name, got, want))
+
+
+def check_true(name, cond, detail=""):
+ if cond:
+ PASS.append(name)
+ else:
+ FAIL.append("%s %s" % (name, detail))
+
+
+def check_raises(name, fn, exc=ValueError):
+ try:
+ fn()
+ except exc:
+ PASS.append(name)
+ except Exception as e:
+ FAIL.append("%s: raised %s (%s), want %s" % (name, type(e).__name__, e, exc.__name__))
+ else:
+ FAIL.append("%s: no exception" % name)
+
+
+class TableEmbed:
+ def __init__(self, vectors):
+ self.vectors = vectors
+ self.calls = []
+
+ def __call__(self, texts):
+ self.calls.append(list(texts))
+ missing = [t for t in texts if t not in self.vectors]
+ if missing:
+ raise AssertionError("unexpected texts %r" % (missing,))
+ return [self.vectors[t] for t in texts]
+
+
+class BoomEmbed:
+ def __call__(self, texts):
+ raise AssertionError("embed_fn should not run when k >= n")
+
+
+class Recorder:
+ """Stand-in for Agent. Records the questions handed to predict."""
+
+ def __init__(self):
+ self.calls = []
+ self.system_one_calls = 0
+
+ def predict(self, state, questions, **kwargs):
+ self.calls.append((state, questions, kwargs))
+ return {"model": "fake", "answers": _answers(questions)}
+
+ def system_one(self, state, questions, **kwargs):
+ self.system_one_calls += 1
+ self.calls.append((state, questions, kwargs))
+ return {"model": "fake", "answers": _answers(questions)}
+
+
+def _answers(questions):
+ answers = {}
+ for qid, qdef in questions.items():
+ if isinstance(qdef, dict) and qdef.get("type") == "choice":
+ crit = qdef["criteria"]
+ keys = list(crit.keys()) if isinstance(crit, dict) else list(crit)
+ answers[qid] = {"type": "choice", "choice": keys[0]}
+ return answers
+
+
+# Vectors: query [1, 0]. alpha and delta tie at cosine 1; gamma is 0.6; beta is 0.
+# Stable order must keep alpha ahead of delta.
+VEC = {
+ "pay me": [1.0, 0.0],
+ "alpha": [1.0, 0.0],
+ "beta": [0.0, 1.0],
+ "gamma": [0.6, 0.8],
+ "delta": [1.0, 0.0],
+ "zero": [0.0, 0.0],
+}
+CRITERIA = {"alpha": None, "beta": "", "gamma": "mid", "delta": "same"}
+# option texts follow render_options: bare key when value is None or ""
+OPTION_TEXTS = {
+ "alpha": [1.0, 0.0],
+ "beta": [0.0, 1.0],
+ "gamma: mid": [0.6, 0.8],
+ "delta: same": [1.0, 0.0],
+}
+
+
+def _embed_for(query_text, option_vectors):
+ vectors = {query_text: [1.0, 0.0]}
+ vectors.update(option_vectors)
+ return TableEmbed(vectors)
+
+
+# ---------------------------------------------------------------- exports and default path
+check_true("export/shortlist_choice", laya.shortlist_choice is shortlist_choice)
+check_true("export/predict_shortlist", laya.predict_shortlist is predict_shortlist)
+check_true("export/embed_fn_from_agent", laya.embed_fn_from_agent is embed_fn_from_agent)
+for _name in ("shortlist_choice", "predict_shortlist", "embed_fn_from_agent"):
+ check_true("all/%s" % _name, _name in laya.__all__)
+
+_predict_src = inspect.getsource(Agent.system_one)
+check_true("default/predict is system_one", Agent.predict is Agent.system_one)
+check_true("default/system_one has no shortlist", "shortlist" not in _predict_src and "embed_fn" not in _predict_src)
+_forward_src = inspect.getsource(DecisionModel.forward)
+check_true("default/forward has no shortlist", "shortlist" not in _forward_src and "embed_fn" not in _forward_src)
+
+
+# ---------------------------------------------------------------- deterministic top-k
+embed = _embed_for("pay me", OPTION_TEXTS)
+labels = shortlist_choice("pay me", CRITERIA, embed, k=2)
+check("topk/k=2 keeps the cosine tie in input order", labels, ["alpha", "delta"])
+check("topk/one embed call", len(embed.calls), 1)
+check("topk/query is the first text", embed.calls[0][0], "pay me")
+check(
+ "topk/option texts match render_options",
+ embed.calls[0][1:],
+ render_options({"t": "choice", "ins": "", "crit": CRITERIA}),
+)
+check("topk/k=1 is the earliest max", shortlist_choice("pay me", CRITERIA, embed, k=1), ["alpha"])
+check(
+ "topk/k=3 appends the next cosine",
+ shortlist_choice("pay me", CRITERIA, embed, k=3),
+ ["alpha", "delta", "gamma"],
+)
+
+# all-zero query: every cosine is 0, so the earliest labels win
+zero_q = TableEmbed({
+ "pay me": [0.0, 0.0],
+ "alpha": [1.0, 0.0],
+ "beta": [0.0, 1.0],
+ "gamma: mid": [0.6, 0.8],
+ "delta: same": [3.0, 4.0],
+})
+check(
+ "topk/zero query keeps original order",
+ shortlist_choice("pay me", CRITERIA, zero_q, k=2),
+ ["alpha", "beta"],
+)
+
+# non-finite option vector is treated as 0 and loses to a real match
+nan_embed = TableEmbed({
+ "pay me": [1.0, 0.0],
+ "alpha": [float("nan"), float("nan")],
+ "beta": [1.0, 0.0],
+})
+check(
+ "topk/nan vector sorts behind a finite match",
+ shortlist_choice("pay me", {"alpha": None, "beta": None}, nan_embed, k=1),
+ ["beta"],
+)
+
+
+# ---------------------------------------------------------------- list criteria and instructions
+list_embed = TableEmbed({
+ "Classify\npay me": [0.0, 1.0],
+ "alpha": [1.0, 0.0],
+ "beta": [0.0, 1.0],
+ "gamma": [0.0, 0.2],
+})
+check(
+ "list/instructions change the query and the winner",
+ shortlist_choice("pay me", ["alpha", "beta", "gamma"], list_embed, k=2, instructions="Classify"),
+ ["beta", "gamma"],
+)
+check("list/query text includes instructions", list_embed.calls[0][0], "Classify\npay me")
+check("list/option texts are the labels", list_embed.calls[0][1:], ["alpha", "beta", "gamma"])
+
+state = {"text": "hi"}
+dict_embed = TableEmbed({
+ 'Classify\n{"text": "hi"}': [1.0, 0.0],
+ "alpha": [1.0, 0.0],
+ "beta": [0.0, 1.0],
+})
+check(
+ "query/dict state is serialized",
+ shortlist_choice(state, ["alpha", "beta"], dict_embed, k=1, instructions="Classify"),
+ ["alpha"],
+)
+check("query/json matches serialize_state", dict_embed.calls[0][0], 'Classify\n{"text": "hi"}')
+
+# 0 and False are real criterion values, so they are part of the embedded text
+rich = {"zero": 0, "no": False, "bare": None, "named": {"desc": "payments"}}
+rich_rendered = render_options({"t": "choice", "ins": "", "crit": rich})
+rich_embed = TableEmbed({"pay me": [1.0, 0.0], **{text: [1.0, 0.0] for text in rich_rendered}})
+shortlist_choice("pay me", rich, rich_embed, k=1)
+check("render/0 and False stay in the option text", rich_embed.calls[0][1:], rich_rendered)
+
+
+# ---------------------------------------------------------------- k >= n pass-through
+boom = BoomEmbed()
+check("pass/k == n returns every label in order", shortlist_choice("pay me", CRITERIA, boom, k=4), list(CRITERIA))
+check("pass/k > n returns every label in order", shortlist_choice("pay me", CRITERIA, boom, k=20), list(CRITERIA))
+
+
+# ---------------------------------------------------------------- mock predict sees only k criteria
+sentinel = {"desc": "payments"}
+full = {"billing": sentinel, "tech": "bugs", "sales": None, "other": "misc"}
+full_vectors = {
+ "Which desk?\nI was charged twice": [1.0, 0.0],
+ 'billing: {"desc": "payments"}': [0.0, 1.0],
+ "tech: bugs": [1.0, 0.0],
+ "sales": [0.2, 0.2],
+ "other: misc": [0.0, 1.0],
+}
+# cosine vs [1, 0]: tech=1, sales=0.707, billing=0, other=0. k=2 -> tech, sales.
+agent = Recorder()
+score_q = {"type": "score", "instructions": "How urgent?", "criteria": ["low", "mid", "high", "now"]}
+noul_q = {"type": "noul", "instructions": "Is a refund requested?"}
+questions = {
+ "intent": {
+ "type": "choice",
+ "instructions": "Which desk?",
+ "criteria": full,
+ },
+ "urgency": score_q,
+ "refund": noul_q,
+ "note": "leave me alone",
+}
+state = "I was charged twice"
+result = predict_shortlist(agent, state, questions, TableEmbed(full_vectors), k=2, model="english")
+
+check("predict/called once", len(agent.calls), 1)
+check("predict/system_one not used when predict exists", agent.system_one_calls, 0)
+got_state, got_questions, got_kwargs = agent.calls[0]
+check_true("predict/state is the same object", got_state is state)
+check("predict/kwargs forwarded", got_kwargs, {"model": "english"})
+check("predict/choice criteria are only the top 2", list(got_questions["intent"]["criteria"]), ["tech", "sales"])
+check_true(
+ "predict/kept description is the original object",
+ got_questions["intent"]["criteria"]["tech"] == "bugs",
+)
+check_true("predict/score question is the same object", got_questions["urgency"] is score_q)
+check_true("predict/noul question is the same object", got_questions["refund"] is noul_q)
+check_true("predict/non-dict question is the same object", got_questions["note"] is questions["note"])
+check("caller/criteria unchanged", questions["intent"]["criteria"], full)
+check_true("caller/criteria object unchanged", questions["intent"]["criteria"] is full)
+check("caller/sentinel intact", full["billing"], sentinel)
+
+internal = Agent._to_internal(got_questions["intent"])
+check("predict/internal choice keys are the shortlist", list(internal["crit"]), ["tech", "sales"])
+check("result/choice is the mock's first shortlisted label", result["answers"]["intent"]["choice"], "tech")
+check("result/shortlist labels", result["shortlist"]["intent"]["labels"], ["tech", "sales"])
+check_true(
+ "result/scores descend",
+ result["shortlist"]["intent"]["scores"][0] > result["shortlist"]["intent"]["scores"][1] > 0,
+)
+check("result/shortlist k and n", (result["shortlist"]["intent"]["k"], result["shortlist"]["intent"]["n"]), (2, 4))
+check("result/not a passthrough", result["shortlist"]["intent"]["passthrough"], False)
+check_true("result/non-choice questions are absent from shortlist meta", "urgency" not in result["shortlist"])
+# The predict return is copied before shortlist is attached.
+held = {}
+
+
+class Holding:
+ def predict(self, state, questions, **kwargs):
+ held["questions"] = questions
+ held["result"] = {"model": "fake", "answers": {}}
+ return held["result"]
+
+
+out = predict_shortlist(Holding(), "pay me", {"intent": {"type": "choice", "criteria": CRITERIA}}, BoomEmbed(), k=4)
+check_true("result/shortlist key is on the copy", "shortlist" in out and "shortlist" not in held["result"])
+
+
+# ---------------------------------------------------------------- k >= n pass-through reaches predict unchanged
+passthrough_agent = Recorder()
+original_q = {"type": "choice", "instructions": "Which desk?", "criteria": full}
+out = predict_shortlist(
+ passthrough_agent,
+ "I was charged twice",
+ {"intent": original_q},
+ BoomEmbed(),
+ k=4,
+)
+check_true("pass/predict received the original question", passthrough_agent.calls[0][1]["intent"] is original_q)
+check("pass/metadata labels are the full set", out["shortlist"]["intent"]["labels"], list(full))
+check("pass/scores omitted", out["shortlist"]["intent"]["scores"], None)
+check("pass/flag", out["shortlist"]["intent"]["passthrough"], True)
+check("pass/k > n flag", predict_shortlist(
+ Recorder(), "x", {"intent": original_q}, BoomEmbed(), k=99
+)["shortlist"]["intent"]["passthrough"], True)
+
+# list criteria stay a list, in rank order, and Agent._to_internal accepts them
+list_agent = Recorder()
+list_q_embed = TableEmbed({
+ "Which?\nhello": [1.0, 0.0],
+ "alpha": [0.0, 1.0],
+ "beta": [1.0, 0.0],
+ "gamma": [0.0, 0.0],
+})
+list_questions = {"intent": {"type": "choice", "instructions": "Which?", "criteria": ["alpha", "beta", "gamma"]}}
+predict_shortlist(list_agent, "hello", list_questions, list_q_embed, k=2)
+received = list_agent.calls[0][1]["intent"]["criteria"]
+check("list/predict receives a list of k labels", received, ["beta", "alpha"])
+check("list/caller criteria unchanged", list_questions["intent"]["criteria"], ["alpha", "beta", "gamma"])
+check(
+ "list/internal keys follow the shortlist",
+ list(Agent._to_internal(list_agent.calls[0][1]["intent"])["crit"]),
+ ["beta", "alpha"],
+)
+
+
+# ---------------------------------------------------------------- system_one-only agent
+class SystemOneOnly:
+ def __init__(self):
+ self.questions = None
+
+ def system_one(self, state, questions):
+ self.questions = questions
+ return {"answers": {"intent": {"choice": "beta"}}}
+
+
+only = SystemOneOnly()
+out = predict_shortlist(only, "hello", list_questions, list_q_embed, k=1)
+check("system_one/used when predict is absent", out["answers"]["intent"]["choice"], "beta")
+check("system_one/criteria length is k", len(only.questions["intent"]["criteria"]), 1)
+
+
+# ---------------------------------------------------------------- errors
+check_raises("err/k=0", lambda: shortlist_choice("pay me", CRITERIA, embed, k=0))
+check_raises("err/k negative", lambda: shortlist_choice("pay me", CRITERIA, embed, k=-3))
+check_raises("err/k bool", lambda: shortlist_choice("pay me", CRITERIA, embed, k=True))
+check_raises("err/k float", lambda: shortlist_choice("pay me", CRITERIA, embed, k=1.5))
+check_raises("err/k str", lambda: shortlist_choice("pay me", CRITERIA, embed, k="2"))
+check_raises("err/empty dict", lambda: shortlist_choice("pay me", {}, embed, k=1))
+check_raises("err/empty list", lambda: shortlist_choice("pay me", [], embed, k=1))
+check_raises("err/criteria tuple", lambda: shortlist_choice("pay me", ("alpha", "beta"), embed, k=1), TypeError)
+check_raises("err/duplicate label", lambda: shortlist_choice("pay me", ["alpha", "alpha"], embed, k=1))
+check_raises(
+ "err/missing criteria",
+ lambda: predict_shortlist(Recorder(), "pay me", {"intent": {"type": "choice"}}, embed, k=1),
+)
+check_raises("err/questions not a dict", lambda: predict_shortlist(Recorder(), "pay me", [], embed, k=1), TypeError)
+
+
+def _bad_shape(texts):
+ return np.zeros((1, 4))
+
+
+bad_agent = Recorder()
+check_raises(
+ "err/bad embed shape",
+ lambda: predict_shortlist(bad_agent, "pay me", {"intent": original_q}, _bad_shape, k=2),
+)
+check("err/bad shape does not call predict", len(bad_agent.calls), 0)
+
+
+def _torch_rows(texts):
+ import torch
+ rows = []
+ for i, _text in enumerate(texts):
+ rows.append([1.0, 0.0] if i == 0 or i == 1 else [0.0, 1.0])
+ return torch.tensor(rows)
+
+
+check(
+ "embed/torch tensor return is accepted",
+ shortlist_choice("pay me", {"alpha": None, "beta": None}, _torch_rows, k=1),
+ ["alpha"],
+)
+
+
+# ---------------------------------------------------------------- encoder mean-pool, no Hub
+import torch # noqa: E402
+
+
+class _Out:
+ def __init__(self, hidden):
+ self.last_hidden_state = hidden
+
+
+class TinyEncoder(torch.nn.Module):
+ def __init__(self):
+ super().__init__()
+ self.config = type("Cfg", (), {"hidden_size": 2})()
+ self.forwards = 0
+
+ def forward(self, input_ids, attention_mask):
+ self.forwards += 1
+ # Channel 0 is the token id; channel 1 is 1. Padding must drop out of the mean.
+ ids = input_ids.float()
+ hidden = torch.stack([ids, torch.ones_like(ids)], dim=-1)
+ return _Out(hidden)
+
+
+class TinyTok:
+ def __init__(self):
+ self.kwargs = None
+ self.batches = []
+
+ def __call__(self, texts, padding=True, truncation=True, max_length=256, return_tensors="pt"):
+ self.kwargs = {"padding": padding, "truncation": truncation, "max_length": max_length}
+ self.batches.append(list(texts))
+ rows = [[(ord(ch) % 5) + 1 for ch in text][:max_length] or [1] for text in texts]
+ width = max(len(row) for row in rows)
+ input_ids, mask = [], []
+ for row in rows:
+ pad = width - len(row)
+ input_ids.append(row + [0] * pad)
+ mask.append([1] * len(row) + [0] * pad)
+ return {
+ "input_ids": torch.tensor(input_ids, dtype=torch.long),
+ "attention_mask": torch.tensor(mask, dtype=torch.long),
+ }
+
+
+class TinyAgent:
+ def __init__(self):
+ self.tok = TinyTok()
+ self.encoder = TinyEncoder()
+ self.model = type("M", (), {"encoder": self.encoder})()
+ self.device = torch.device("cpu")
+ self.forward_calls = 0
+
+ def forward(self, *args, **kwargs):
+ self.forward_calls += 1
+ raise AssertionError("decision forward must not run during embedding")
+
+
+tiny = TinyAgent()
+fn = embed_fn_from_agent(tiny, max_length=32, batch_size=2)
+first = fn(["ab", "a"])
+second = fn(["ab", "a"])
+check("encoder/shape", first.shape, (2, 2))
+check_true("encoder/deterministic", np.array_equal(first, second))
+check("encoder/one batch for both strings", tiny.tok.batches, [["ab", "a"], ["ab", "a"]])
+check("encoder/max_length forwarded", tiny.tok.kwargs["max_length"], 32)
+check("encoder/decision forward not used", tiny.forward_calls, 0)
+
+# "a" is one real token (id = ord('a') % 5 + 1) plus padding. Masked mean keeps that id.
+token_id = (ord("a") % 5) + 1
+check("encoder/padding excluded from mean", first[1].tolist(), [float(token_id), 1.0])
+# "ab" has two real tokens and no reliance on padding once both are kept.
+ids = [(ord(ch) % 5) + 1 for ch in "ab"]
+check("encoder/mean of real tokens", first[0].tolist(), [sum(ids) / 2.0, 1.0])
+
+empty = fn([])
+check("encoder/empty input shape", empty.shape, (0, 2))
+check_raises("encoder/bad max_length", lambda: embed_fn_from_agent(tiny, max_length=0))
+check_raises("encoder/bad batch_size", lambda: embed_fn_from_agent(tiny, batch_size=True))
+
+
+# ---------------------------------------------------------------- end-to-end helper: shortlist then the real question normalizer only
+# Confirms a reduced choice still has one criterion per kept label, which is what
+# system_one would score. No weights, no forward.
+pipe_agent = Recorder()
+pipe_q = {
+ "intent": {"type": "choice", "instructions": "Which desk?", "criteria": full},
+ "urgency": score_q,
+}
+piped = predict_shortlist(pipe_agent, "I was charged twice", pipe_q, TableEmbed(full_vectors), k=2)
+scored = Agent._to_internal(pipe_agent.calls[0][1]["intent"])
+check("pipe/marker count equals k", len(scored["crit"]), 2)
+check("pipe/answer choice is inside the shortlist", piped["answers"]["intent"]["choice"] in piped["shortlist"]["intent"]["labels"], True)
+
+
+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
+for f in FAIL:
+ print(" FAIL " + f)
+sys.exit(1 if FAIL else 0)