Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(rust): add embedding shortlist for high-cardinality choice
kevin-gatimu24ff83862ae6
diff --git a/laya-rust/src/laya_shortlist.rs b/laya-rust/src/laya_shortlist.rsnew file mode 100644index 00000000..8d2f90b1--- /dev/null+++ b/laya-rust/src/laya_shortlist.rs@@ -0,0 +1,323 @@+//! Opt-in embedding shortlist for high-cardinality choice questions. Ports `laya/shortlist.py`.+//!+//! Choice options share one `head_max_len` token budget, so a large label set leaves only a few+//! tokens per label. [`LayaShortlist::predict`] embeds the state and each option with a+//! caller-supplied `embed_fn`, keeps the top `k`, and runs a single predict on that reduced+//! criteria set. A non-choice question is forwarded unchanged, and a choice whose label count is+//! already `<= k` is forwarded unchanged too — `embed_fn` is never called for it.+//!+//! `embed_fn_from_agent` (Python: mean-pools the checkpoint encoder already loaded on an `Agent`)+//! is deliberately not ported: this port only accepts a caller-supplied embedding function (a+//! real deployment should bring a dedicated bi-encoder), matching the plan for this port.+//!+//! ## Why `embed_fn` is a generic `Fn` bound rather than a boxed trait object+//!+//! `shortlist_choice`/`predict` take `embed_fn: F where F: Fn(&[String]) -> Result<Vec<Vec<f64>>>`+//! instead of `embed_fn: &dyn Fn(...)` or `Box<dyn Fn(...)>`. A shortlist call embeds a handful of+//! short strings once per choice question — not a hot loop — so the extra dynamic-dispatch+//! indirection a trait object would add buys nothing; a generic parameter still accepts an+//! ordinary closure, a `fn` item, or a `&F`/`Box<F>` coerced by the caller, and it lets the+//! compiler monomorphize and inline the common "one closure, called a few times" case. The+//! function signature itself (`&[String] -> Result<Vec<Vec<f64>>>`) mirrors Python's+//! `Callable[[Sequence[str]], Any]`: a list of texts in, one embedding vector per text out, in+//! the same order, fallible (a real embedder can hit a network or model error, unlike this+//! crate's own deterministic test embedder).++use crate::answers::LayaResult;+use crate::error::{LayaError, Result};+use crate::laya_router::LayaPredictor;+use crate::python_json::PythonJson;+use crate::questions::{Question, QuestionSet};+use crate::sequence_builder::SequenceBuilder;+use indexmap::IndexMap;+use serde_json::Value;+use std::collections::HashSet;++/// Default shortlist size, matching Python's `DEFAULT_SHORTLIST_K`.+pub const DEFAULT_SHORTLIST_K: usize = 20;++/// What [`LayaShortlist::predict`] recorded for one shortlisted choice question.+#[derive(Debug, Clone, PartialEq)]+pub struct ShortlistInfo {+ /// Kept labels, in rank order (highest cosine similarity first).+ pub labels: Vec<String>,+ /// Cosine similarity per kept label, in the same order as `labels`. `None` when this+ /// question passed through unchanged (`k` was already `>=` the label count, so nothing was+ /// ranked and `embed_fn` was never called).+ pub scores: Option<Vec<f64>>,+ /// The `k` this question was shortlisted with.+ pub k: usize,+ /// How many labels the question had before shortlisting.+ pub n: usize,+ /// Whether this question passed through unchanged.+ pub passthrough: bool,+}++/// Namespace for the embedding shortlist. See the module docs for the source this ports.+pub struct LayaShortlist;++impl LayaShortlist {+ /// Return the top-`k` choice labels for `state`.+ ///+ /// `embed_fn` is called once, with the query text first and then one string per option in+ /// criteria order; option strings match [`SequenceBuilder::render_options`] for a choice+ /// question. When `k` is at least the number of labels, every label is returned in its+ /// original order and `embed_fn` is not called.+ ///+ /// Ties keep the earlier label. A zero vector scores 0 and does not outrank a label that came+ /// before it.+ ///+ /// `criteria` is an ordered `(label, description)` sequence — the same shape+ /// [`Question::choice`] takes, covering both Python's dict criteria (label -> description)+ /// and list criteria (bare labels, each with a `Value::Null` description) uniformly. Rejects+ /// an empty or duplicate-labelled criteria set the same way `Question::choice` does.+ pub fn shortlist_choice<F, L, D>(+ state: &Value,+ criteria: impl IntoIterator<Item = (L, D)>,+ embed_fn: F,+ k: usize,+ instructions: Option<&Value>,+ ) -> Result<Vec<String>>+ where+ F: Fn(&[String]) -> Result<Vec<Vec<f64>>>,+ L: Into<String>,+ D: Into<Value>,+ {+ let items: Vec<(String, Value)> = criteria+ .into_iter()+ .map(|(l, d)| (l.into(), d.into()))+ .collect();+ check_k(k)?;+ check_criteria(&items)?;+ let query = query_text(state, instructions);+ Ok(rank(query, &items, &embed_fn, k)?.labels)+ }++ /// Shortlist each choice question in `questions`, then call `predictor.predict` once on the+ /// reduced set. Non-choice questions are forwarded unchanged; a choice question whose label+ /// count is already `<= k` is forwarded unchanged and does not call `embed_fn`.+ ///+ /// The returned [`LayaResult`] carries [`LayaResult::shortlist`] recording, per shortlisted+ /// choice question id, the kept labels (rank order), their cosine scores (`None` for a+ /// passthrough), `k`, `n` and whether it passed through. `predictor` may be a bare+ /// [`crate::LayaEngine`] or a [`crate::laya_router::LayaRouter`] — they compose, since a+ /// router's own [`LayaPredictor::predict`] already attaches its routing decision to the+ /// result this then adds shortlist metadata to.+ pub fn predict<F>(+ predictor: &dyn LayaPredictor,+ state: impl Into<Value>,+ questions: &QuestionSet,+ embed_fn: F,+ k: usize,+ ) -> Result<LayaResult>+ where+ F: Fn(&[String]) -> Result<Vec<Vec<f64>>>,+ {+ check_k(k)?;+ let state = state.into();+ let mut reduced = QuestionSet::new();+ let mut meta: IndexMap<String, ShortlistInfo> = IndexMap::new();++ for (id, question) in questions.iter() {+ match question {+ Question::Choice(choice) => {+ let items = choice.options().to_vec();+ let query = query_text(&state, Some(choice.instructions()));+ let outcome = rank(query, &items, &embed_fn, k)?;++ meta.insert(+ id.to_string(),+ ShortlistInfo {+ labels: outcome.labels.clone(),+ scores: outcome.scores,+ k,+ n: outcome.n,+ passthrough: outcome.passthrough,+ },+ );++ if outcome.passthrough {+ reduced.insert(id, question.clone())?;+ } else {+ // Rebuild with only the kept labels, in rank order: Python's+ // `_subset_criteria` builds `{label: criteria[label] for label in+ // labels}`, which is exactly a re-keyed, re-ordered option list here.+ let kept: Vec<(String, Value)> = outcome+ .labels+ .into_iter()+ .map(|label| {+ let description = items+ .iter()+ .find(|(l, _)| *l == label)+ .map(|(_, d)| d.clone())+ .expect("a ranked label always came from `items`");+ (label, description)+ })+ .collect();+ let reduced_choice = Question::choice(choice.instructions().clone(), kept)?;+ reduced.insert(id, reduced_choice)?;+ }+ }+ other => {+ reduced.insert(id, other.clone())?;+ }+ }+ }++ let result = predictor.predict(state, &reduced)?;+ Ok(result.with_shortlist(meta))+ }+}++fn check_k(k: usize) -> Result<()> {+ if k == 0 {+ return Err(LayaError::InvalidQuestion(+ "k must be a positive integer, got 0".to_string(),+ ));+ }+ Ok(())+}++fn check_criteria(items: &[(String, Value)]) -> Result<()> {+ if items.is_empty() {+ return Err(LayaError::InvalidQuestion(+ "choice criteria must contain at least one option".to_string(),+ ));+ }+ let mut seen = HashSet::with_capacity(items.len());+ for (label, _) in items {+ if !seen.insert(label.as_str()) {+ return Err(LayaError::InvalidQuestion(format!(+ "choice criteria label '{label}' is duplicated"+ )));+ }+ }+ Ok(())+}++/// The shortlist ranking's query text: `serialize_state(state)`, or `"instructions\n" ++/// serialize_state(state)` when non-blank instructions are given. Blank is `None` or the empty+/// string, matching Python's `instructions is None or instructions == ""` (a non-string+/// instructions value is never equal to `""`, so only an actual empty *string* counts).+fn query_text(state: &Value, instructions: Option<&Value>) -> String {+ let body = PythonJson::state(state);+ match instructions {+ Some(v) if !is_blank_instructions(v) => {+ format!("{}\n{body}", PythonJson::shortlist_instructions(v))+ }+ _ => body,+ }+}++fn is_blank_instructions(value: &Value) -> bool {+ matches!(value, Value::Null) || matches!(value, Value::String(s) if s.is_empty())+}++/// What [`rank`] found: the same four values [`ShortlistInfo`] eventually carries for one+/// question, plus the bare label list [`LayaShortlist::shortlist_choice`] returns.+struct RankOutcome {+ labels: Vec<String>,+ scores: Option<Vec<f64>>,+ passthrough: bool,+ n: usize,+}++/// Rank `items` against `query` with `embed_fn`. `query` is the already-built query text (not the+/// raw state), so both [`LayaShortlist::shortlist_choice`] and [`LayaShortlist::predict`] can+/// share this after building their own query text (a bare `instructions` kwarg for the former, a+/// question's own `instructions` field for the latter).+fn rank<F>(query: String, items: &[(String, Value)], embed_fn: &F, k: usize) -> Result<RankOutcome>+where+ F: Fn(&[String]) -> Result<Vec<Vec<f64>>>,+{+ let n = items.len();+ let keys: Vec<String> = items.iter().map(|(label, _)| label.clone()).collect();+ if k >= n {+ return Ok(RankOutcome {+ labels: keys,+ scores: None,+ passthrough: true,+ n,+ });+ }++ // A throwaway choice question purely to reuse `SequenceBuilder::render_options`'s exact+ // rendering rule (bare label when the description is blank, `"label: description"`+ // otherwise) rather than duplicating it here.+ let temp = Question::choice(Value::Null, items.iter().cloned())?;+ let option_texts = SequenceBuilder::render_options(&temp);++ let mut texts = Vec::with_capacity(1 + option_texts.len());+ texts.push(query);+ texts.extend(option_texts);++ let matrix = embed_matrix(embed_fn, &texts)?;+ let sims = cosine(&matrix[0], &matrix[1..]);++ let mut order: Vec<usize> = (0..sims.len()).collect();+ // Stable sort descending by score: comparing `(b, a)` instead of `(a, b)` reverses the sense+ // of the comparison while stability still preserves the original (ascending-index) order+ // among ties, matching `np.argsort(-sims, kind="mergesort")`.+ order.sort_by(|&a, &b| sims[b].total_cmp(&sims[a]));+ order.truncate(k);++ let labels: Vec<String> = order.iter().map(|&i| keys[i].clone()).collect();+ let scores: Vec<f64> = order.iter().map(|&i| sims[i]).collect();+ Ok(RankOutcome {+ labels,+ scores: Some(scores),+ passthrough: false,+ n,+ })+}++/// Calls `embed_fn`, checks the returned matrix has one row per text and a consistent, non-zero+/// row width, and zeroes non-finite components (`np.nan_to_num`: only the offending components,+/// not the whole row — `[NaN, 1.0]` becomes `[0.0, 1.0]`, a real vector, not a zero vector).+fn embed_matrix<F>(embed_fn: &F, texts: &[String]) -> Result<Vec<Vec<f64>>>+where+ F: Fn(&[String]) -> Result<Vec<Vec<f64>>>,+{+ let raw = embed_fn(texts)?;+ if raw.len() != texts.len() {+ return Err(LayaError::InvalidQuestion(format!(+ "embed_fn must return one row per text: got {} rows for {} texts",+ raw.len(),+ texts.len()+ )));+ }+ let dim = raw.first().map(Vec::len).unwrap_or(0);+ if dim == 0 || raw.iter().any(|row| row.len() != dim) {+ return Err(LayaError::InvalidQuestion(+ "embed_fn must return rows of equal, non-zero length".to_string(),+ ));+ }+ Ok(raw+ .into_iter()+ .map(|row| {+ row.into_iter()+ .map(|v| if v.is_finite() { v } else { 0.0 })+ .collect()+ })+ .collect())+}++/// Cosine similarity of `query` against every row of `docs`. A zero-norm query scores every+/// document 0 (matching Python: the whole `sims` array is returned as-is, all zero, without even+/// looking at the documents). A zero-norm document scores 0 against a non-zero query.+fn cosine(query: &[f64], docs: &[Vec<f64>]) -> Vec<f64> {+ let query_norm = query.iter().map(|v| v * v).sum::<f64>().sqrt();+ let mut sims = vec![0.0; docs.len()];+ if query_norm == 0.0 {+ return sims;+ }+ for (i, doc) in docs.iter().enumerate() {+ let doc_norm = doc.iter().map(|v| v * v).sum::<f64>().sqrt();+ let denom = doc_norm * query_norm;+ if denom > 0.0 {+ let dot: f64 = doc.iter().zip(query).map(|(a, b)| a * b).sum();+ sims[i] = dot / denom;+ }+ }+ sims+}diff --git a/laya-rust/tests/common/hashing_embedder.rs b/laya-rust/tests/common/hashing_embedder.rsnew file mode 100644index 00000000..f1c67183--- /dev/null+++ b/laya-rust/tests/common/hashing_embedder.rs@@ -0,0 +1,65 @@+//! Deterministic, dependency-free demo embedder for the shortlist golden vectors and end-to-end+//! tests. Ports `tools/hashing_embedder.py` bit-for-bit — see that file's docstring for the exact+//! algorithm spec this follows step by step. **Not** a real embedding model: a hashed+//! character-trigram bag-of-features vector, chosen only because it is a pure function of the+//! input text that is trivial to reproduce identically in every language.+//!+//! Every step below is precision-sensitive:+//! - Lowercasing uses full Unicode case mapping (`str::to_lowercase`), matching Python's+//! `str.lower()` — not a per-UTF-16-unit simple mapping.+//! - The 3-item sliding window is over Unicode scalar values (`char`, which in Rust already *is*+//! a Unicode scalar value), matching Python's per-code-point iteration; an astral character+//! (outside the BMP) is one window position, not two.+//! - The window is re-encoded to UTF-8 on its own each time (not sliced out of one pre-encoded+//! byte buffer), because a multi-byte character's encoding does not change with position, but+//! slicing a shared buffer at the wrong byte offset would corrupt it.+//! - The FNV-1a multiply wraps at 32 bits (`u32::wrapping_mul`), matching Python's explicit+//! `& 0xFFFFFFFF`.++use laya::error::Result as LayaResult;++/// Output vector width.+pub const DIM: usize = 64;++const FNV_OFFSET_BASIS: u32 = 0x811c_9dc5;+const FNV_PRIME: u32 = 0x0100_0193;++/// 32-bit FNV-1a over raw bytes, wrapping like an unsigned 32-bit integer.+fn fnv1a_32(data: &[u8]) -> u32 {+ let mut hash = FNV_OFFSET_BASIS;+ for &byte in data {+ hash ^= u32::from(byte);+ hash = hash.wrapping_mul(FNV_PRIME);+ }+ hash+}++fn embed_one(text: &str) -> Vec<f64> {+ let mut vector = vec![0.0f64; DIM];+ let padded: Vec<char> = format!(" {} ", text.to_lowercase()).chars().collect();+ let n = padded.len();+ // Padding alone is 4 characters, so `n >= 4` always and this subtraction never underflows;+ // the guard is defensive documentation, not a reachable branch.+ if n < 3 {+ return vector;+ }+ for start in 0..=(n - 3) {+ let window: String = padded[start..start + 3].iter().collect();+ let hash = fnv1a_32(window.as_bytes());+ let bucket = (hash % DIM as u32) as usize;+ let sign = if (hash >> 31) & 1 == 0 { 1.0 } else { -1.0 };+ vector[bucket] += sign;+ }+ vector+}++/// Embed a list of strings into one `(len(texts), 64)` matrix, in input order. See the module+/// docs for the exact algorithm; `None` inputs have no Rust equivalent (the signature already+/// requires `&str`), matching Python's `"" if t is None else str(t)` only in that an empty string+/// embeds the same way an absent one would have.+///+/// Returns [`laya::error::Result`] so it can be used directly wherever `LayaShortlist` expects an+/// `embed_fn` closure, even though this particular embedder never fails.+pub fn embed(texts: &[String]) -> LayaResult<Vec<Vec<f64>>> {+ Ok(texts.iter().map(|t| embed_one(t)).collect())+}diff --git a/laya-rust/tests/hashing_embedder_tests.rs b/laya-rust/tests/hashing_embedder_tests.rsnew file mode 100644index 00000000..4628f7b0--- /dev/null+++ b/laya-rust/tests/hashing_embedder_tests.rs@@ -0,0 +1,47 @@+//! Tier 1: the deterministic hashing embedder (`tests/common/hashing_embedder.rs`) against every+//! entry of the routing golden's `hashing_embedder_probe.json`. This is a test-helper probe, not+//! a check on `laya` itself — the embedder is reimplemented per-SDK, per the plan, so the Rust+//! test helper must reproduce `tools/hashing_embedder.py` bit-for-bit.++mod common;++use common::{hashing_embedder, routing_golden};++#[test]+fn hashing_embedder_matches_every_probe_entry() {+ let Some(cases) = routing_golden::load_array("hashing_embedder_probe.json") else {+ eprintln!("skip: routing golden not found (tests/golden/routing)");+ return;+ };++ for case in &cases {+ let text = case["text"]+ .as_str()+ .expect("probe entry needs a 'text' string");+ let expected: Vec<f64> = case["vector"]+ .as_array()+ .expect("probe entry needs a 'vector' array")+ .iter()+ .map(|v| v.as_f64().expect("vector component must be a number"))+ .collect();++ let got = hashing_embedder::embed(&[text.to_string()]).expect("embed never fails");+ assert_eq!(got.len(), 1);+ assert_eq!(got[0].len(), expected.len(), "text {text:?}: vector length");+ for (i, (&g, &e)) in got[0].iter().zip(expected.iter()).enumerate() {+ assert_eq!(g, e, "text {text:?}: vector[{i}]");+ }+ }+ eprintln!(+ "hashing_embedder_matches_every_probe_entry: checked {} texts",+ cases.len()+ );+}++#[test]+fn hashing_embedder_is_order_preserving_and_batches_independently() {+ let a = hashing_embedder::embed(&["alpha".to_string()]).unwrap();+ let b = hashing_embedder::embed(&["beta".to_string(), "alpha".to_string()]).unwrap();+ // Each text is embedded independently of its neighbours in the batch.+ assert_eq!(a[0], b[1]);+}diff --git a/laya-rust/tests/laya_shortlist_tests.rs b/laya-rust/tests/laya_shortlist_tests.rsnew file mode 100644index 00000000..14d9c4be--- /dev/null+++ b/laya-rust/tests/laya_shortlist_tests.rs@@ -0,0 +1,515 @@+//! Tier 1: the embedding shortlist. Ports the non-torch checks from `tests/test_shortlist.py`+//! (the torch mean-pool block — `embed_fn_from_agent` — is out of scope: this port only accepts a+//! caller-supplied `embed_fn`) plus every entry of the routing golden's `shortlist_probe.json`,+//! and the hashing embedder vs its own probe (see `hashing_embedder_tests.rs`).++mod common;++use common::routing_golden;+use indexmap::IndexMap;+use laya::answers::{ActionInfo, Answer, ChoiceAnswer, LayaResult, Usage};+use laya::error::{LayaError, Result};+use laya::laya_router::LayaPredictor;+use laya::laya_shortlist::{LayaShortlist, ShortlistInfo};+use laya::questions::{Question, QuestionSet};+use serde_json::{Value, json};+use std::collections::HashMap;++/// A predictor that never inspects `state`: it just answers every choice question with its first+/// (post-shortlist) label, so `LayaShortlist::predict` tests can check the shortlist metadata and+/// the reduced question set without needing a real engine.+struct FirstLabelPredictor;++impl LayaPredictor for FirstLabelPredictor {+ fn predict(&self, _state: Value, questions: &QuestionSet) -> Result<LayaResult> {+ let order: Vec<String> = questions.ids().map(String::from).collect();+ let mut by_id = HashMap::new();+ for id in &order {+ if let Question::Choice(c) = questions.get(id).unwrap() {+ let first = c.labels().next().unwrap_or("").to_string();+ let mut probabilities = IndexMap::new();+ probabilities.insert(first.clone(), 1.0);+ by_id.insert(+ id.clone(),+ Answer::Choice(ChoiceAnswer::new(+ first,+ probabilities,+ 1.0,+ 1.0,+ ActionInfo::default(),+ )),+ );+ }+ }+ Ok(LayaResult::new("mock", order, by_id, Usage::default()))+ }+}++fn table_embed(table: &[(&str, [f64; 2])]) -> impl Fn(&[String]) -> Result<Vec<Vec<f64>>> {+ let table: Vec<(String, Vec<f64>)> = table+ .iter()+ .map(|(k, v)| (k.to_string(), v.to_vec()))+ .collect();+ move |texts: &[String]| {+ texts+ .iter()+ .map(|t| {+ table+ .iter()+ .find(|(k, _)| k == t)+ .map(|(_, v)| v.clone())+ .ok_or_else(|| LayaError::InvalidQuestion(format!("unexpected text {t:?}")))+ })+ .collect()+ }+}++fn boom_embed(_texts: &[String]) -> Result<Vec<Vec<f64>>> {+ panic!("embed_fn should not run when k >= n")+}++// ---------------------------------------------------------------------- named cases (test_shortlist.py)++#[test]+fn deterministic_topk_and_ties() {+ // Vectors: query [1, 0]. alpha and delta tie at cosine 1; gamma is 0.6; beta is 0.+ // Stable order must keep alpha ahead of delta.+ let embed = table_embed(&[+ ("pay me", [1.0, 0.0]),+ ("alpha", [1.0, 0.0]),+ ("beta", [0.0, 1.0]),+ ("gamma: mid", [0.6, 0.8]),+ ("delta: same", [1.0, 0.0]),+ ]);+ let criteria = [+ ("alpha", Value::Null),+ ("beta", json!("")),+ ("gamma", json!("mid")),+ ("delta", json!("same")),+ ];+ let state = json!("pay me");++ assert_eq!(+ LayaShortlist::shortlist_choice(&state, criteria.clone(), &embed, 2, None).unwrap(),+ vec!["alpha", "delta"],+ "topk/k=2 keeps the cosine tie in input order"+ );+ assert_eq!(+ LayaShortlist::shortlist_choice(&state, criteria.clone(), &embed, 1, None).unwrap(),+ vec!["alpha"],+ "topk/k=1 is the earliest max"+ );+ assert_eq!(+ LayaShortlist::shortlist_choice(&state, criteria.clone(), &embed, 3, None).unwrap(),+ vec!["alpha", "delta", "gamma"],+ "topk/k=3 appends the next cosine"+ );++ // All-zero query: every cosine is 0, so the earliest labels win.+ let zero_q = table_embed(&[+ ("pay me", [0.0, 0.0]),+ ("alpha", [1.0, 0.0]),+ ("beta", [0.0, 1.0]),+ ("gamma: mid", [0.6, 0.8]),+ ("delta: same", [3.0, 4.0]),+ ]);+ assert_eq!(+ LayaShortlist::shortlist_choice(&state, criteria.clone(), &zero_q, 2, None).unwrap(),+ vec!["alpha", "beta"],+ "topk/zero query keeps original order"+ );++ // A non-finite option vector is treated as 0 and loses to a real match.+ let nan_embed = |texts: &[String]| -> Result<Vec<Vec<f64>>> {+ Ok(texts+ .iter()+ .map(|t| match t.as_str() {+ "pay me" => vec![1.0, 0.0],+ "alpha" => vec![f64::NAN, f64::NAN],+ "beta" => vec![1.0, 0.0],+ other => panic!("unexpected text {other:?}"),+ })+ .collect())+ };+ assert_eq!(+ LayaShortlist::shortlist_choice(+ &state,+ [("alpha", Value::Null), ("beta", Value::Null)],+ nan_embed,+ 1,+ None+ )+ .unwrap(),+ vec!["beta"],+ "topk/nan vector sorts behind a finite match"+ );+}++#[test]+fn list_criteria_and_instructions_change_the_query() {+ let embed = table_embed(&[+ ("Classify\npay me", [0.0, 1.0]),+ ("alpha", [1.0, 0.0]),+ ("beta", [0.0, 1.0]),+ ("gamma", [0.0, 0.2]),+ ]);+ let state = json!("pay me");+ let instructions = json!("Classify");+ let got = LayaShortlist::shortlist_choice(+ &state,+ [+ ("alpha", Value::Null),+ ("beta", Value::Null),+ ("gamma", Value::Null),+ ],+ &embed,+ 2,+ Some(&instructions),+ )+ .unwrap();+ assert_eq!(+ got,+ vec!["beta", "gamma"],+ "list/instructions change the winner"+ );+}++#[test]+fn dict_state_is_serialized_into_the_query() {+ let embed = table_embed(&[+ ("Classify\n{\"text\": \"hi\"}", [1.0, 0.0]),+ ("alpha", [1.0, 0.0]),+ ("beta", [0.0, 1.0]),+ ]);+ let state = json!({"text": "hi"});+ let instructions = json!("Classify");+ let got = LayaShortlist::shortlist_choice(+ &state,+ [("alpha", Value::Null), ("beta", Value::Null)],+ &embed,+ 1,+ Some(&instructions),+ )+ .unwrap();+ assert_eq!(got, vec!["alpha"], "query/dict state is serialized");+}++#[test]+fn k_ge_n_passes_through_without_calling_embed_fn() {+ let criteria = [+ ("alpha", Value::Null),+ ("beta", json!("")),+ ("gamma", json!("mid")),+ ("delta", json!("same")),+ ];+ let state = json!("pay me");+ assert_eq!(+ LayaShortlist::shortlist_choice(&state, criteria.clone(), boom_embed, 4, None).unwrap(),+ vec!["alpha", "beta", "gamma", "delta"],+ "pass/k == n returns every label in order"+ );+ assert_eq!(+ LayaShortlist::shortlist_choice(&state, criteria, boom_embed, 20, None).unwrap(),+ vec!["alpha", "beta", "gamma", "delta"],+ "pass/k > n returns every label in order"+ );+}++#[test]+fn predict_reduces_choice_and_forwards_other_questions_unchanged() {+ // cosine vs [1, 0]: tech=1, sales≈0.707, billing=0, other=0. k=2 -> tech, sales.+ let embed = table_embed(&[+ ("Which desk?\nI was charged twice", [1.0, 0.0]),+ ("billing: {\"desc\": \"payments\"}", [0.0, 1.0]),+ ("tech: bugs", [1.0, 0.0]),+ ("sales", [0.2, 0.2]),+ ("other: misc", [0.0, 1.0]),+ ]);+ let full_criteria = [+ ("billing", json!({"desc": "payments"})),+ ("tech", json!("bugs")),+ ("sales", Value::Null),+ ("other", json!("misc")),+ ];+ let intent = Question::choice("Which desk?", full_criteria).unwrap();+ let urgency = Question::score("How urgent?", ["low", "mid", "high", "now"]).unwrap();+ let refund = Question::noul("Is a refund requested?", Value::Null, Value::Null);+ let mut questions = QuestionSet::new();+ questions = questions.with("intent", intent).unwrap();+ questions = questions.with("urgency", urgency).unwrap();+ questions = questions.with("refund", refund).unwrap();++ let state = json!("I was charged twice");+ let result = LayaShortlist::predict(&FirstLabelPredictor, state, &questions, embed, 2).unwrap();++ let shortlist = result.shortlist().expect("shortlist metadata");+ let intent_meta = &shortlist["intent"];+ assert_eq!(+ intent_meta.labels,+ vec!["tech", "sales"],+ "kept top 2, rank order"+ );+ assert_eq!(intent_meta.k, 2);+ assert_eq!(intent_meta.n, 4);+ assert!(!intent_meta.passthrough);+ let scores = intent_meta.scores.as_ref().unwrap();+ assert!(+ scores[0] > scores[1] && scores[1] > 0.0,+ "result/scores descend"+ );+ assert!(+ !shortlist.contains_key("urgency"),+ "non-choice questions absent from shortlist meta"+ );++ assert_eq!(+ result.get("intent").unwrap().as_choice().unwrap().choice,+ "tech",+ "mock's first shortlisted label wins"+ );+}++#[test]+fn predict_passthrough_forwards_the_original_question() {+ let full_criteria = [+ ("billing", json!({"desc": "payments"})),+ ("tech", json!("bugs")),+ ("sales", Value::Null),+ ("other", json!("misc")),+ ];+ let intent = Question::choice("Which desk?", full_criteria).unwrap();+ let mut questions = QuestionSet::new();+ questions = questions.with("intent", intent).unwrap();++ let result = LayaShortlist::predict(+ &FirstLabelPredictor,+ json!("I was charged twice"),+ &questions,+ boom_embed,+ 4,+ )+ .unwrap();+ let meta = &result.shortlist().unwrap()["intent"];+ assert!(meta.passthrough, "pass/flag");+ assert_eq!(meta.scores, None, "pass/scores omitted");+ assert_eq!(meta.labels, vec!["billing", "tech", "sales", "other"]);+}++#[test]+fn zero_and_false_criteria_values_are_real_descriptions() {+ let rich = [+ ("zero", json!(0)),+ ("no", json!(false)),+ ("bare", Value::Null),+ ("named", json!({"desc": "payments"})),+ ];+ let state = json!("pay me");+ // Passthrough (k >= n): does not call embed_fn, so this only exercises validation + ordering.+ let got = LayaShortlist::shortlist_choice(&state, rich, boom_embed, 4, None).unwrap();+ assert_eq!(got, vec!["zero", "no", "bare", "named"]);+}++#[test]+fn errors_are_rejected() {+ let state = json!("pay me");+ let embed = |_: &[String]| -> Result<Vec<Vec<f64>>> { Ok(vec![]) };++ assert!(matches!(+ LayaShortlist::shortlist_choice(&state, [("a", Value::Null)], embed, 0, None),+ Err(LayaError::InvalidQuestion(_))+ ));+ assert!(matches!(+ LayaShortlist::shortlist_choice(&state, Vec::<(&str, Value)>::new(), embed, 1, None),+ Err(LayaError::InvalidQuestion(_))+ ));+ assert!(matches!(+ LayaShortlist::shortlist_choice(+ &state,+ [("alpha", Value::Null), ("alpha", Value::Null)],+ embed,+ 1,+ None+ ),+ Err(LayaError::InvalidQuestion(_))+ ));++ // A shape mismatch from embed_fn must fail, and must not have called `predict`.+ let bad_shape = |_: &[String]| -> Result<Vec<Vec<f64>>> { Ok(vec![vec![0.0; 4]]) };+ struct Counting(std::sync::atomic::AtomicU32);+ impl LayaPredictor for Counting {+ fn predict(&self, _state: Value, questions: &QuestionSet) -> Result<LayaResult> {+ self.0.fetch_add(1, std::sync::atomic::Ordering::SeqCst);+ Ok(LayaResult::new(+ "mock",+ questions.ids().map(String::from).collect(),+ HashMap::new(),+ Usage::default(),+ ))+ }+ }+ let counting = Counting(std::sync::atomic::AtomicU32::new(0));+ let original = Question::choice(+ "Which desk?",+ [("billing", Value::Null), ("tech", Value::Null)],+ )+ .unwrap();+ let mut questions = QuestionSet::new();+ questions = questions.with("intent", original).unwrap();+ let err = LayaShortlist::predict(&counting, json!("pay me"), &questions, bad_shape, 1);+ assert!(err.is_err(), "err/bad embed shape");+ assert_eq!(+ counting.0.load(std::sync::atomic::Ordering::SeqCst),+ 0,+ "err/bad shape does not call predict"+ );+}++// ---------------------------------------------------------------------- shortlist_probe.json++fn parse_component(v: &Value) -> f64 {+ match v {+ Value::String(s) => match s.as_str() {+ "nan" => f64::NAN,+ "inf" => f64::INFINITY,+ "-inf" => f64::NEG_INFINITY,+ other => other+ .parse()+ .unwrap_or_else(|_| panic!("unexpected string component {other:?}")),+ },+ _ => v+ .as_f64()+ .expect("expected a numeric (or nan/inf-string) component"),+ }+}++fn parse_criteria(v: &Value) -> Vec<(String, Value)> {+ match v {+ Value::Object(map) => map+ .iter()+ .map(|(k, val)| (k.clone(), val.clone()))+ .collect(),+ Value::Array(items) => items+ .iter()+ .map(|item| {+ (+ item.as_str()+ .expect("list criteria item must be a string")+ .to_string(),+ Value::Null,+ )+ })+ .collect(),+ other => panic!("unexpected criteria shape: {other:?}"),+ }+}++fn parse_vectors(v: &Value) -> Option<Vec<Vec<f64>>> {+ let arr = v.as_array()?;+ Some(+ arr.iter()+ .map(|row| {+ row.as_array()+ .unwrap()+ .iter()+ .map(parse_component)+ .collect()+ })+ .collect(),+ )+}++#[test]+fn shortlist_probe_matches_every_entry() {+ let Some(cases) = routing_golden::load_array("shortlist_probe.json") else {+ eprintln!("skip: routing golden not found (tests/golden/routing)");+ return;+ };++ for case in &cases {+ let label = case["label"].as_str().unwrap();+ let state = case["state"].clone();+ let criteria = parse_criteria(&case["criteria"]);+ let k = case["k"].as_u64().unwrap() as usize;+ let instructions = &case["instructions"];+ let vectors = parse_vectors(&case["vectors"]);+ let expected_labels: Vec<String> = case["labels"]+ .as_array()+ .unwrap()+ .iter()+ .map(|v| v.as_str().unwrap().to_string())+ .collect();+ let expected_scores: Option<Vec<f64>> = case["scores"]+ .as_array()+ .map(|arr| arr.iter().map(|v| v.as_f64().unwrap()).collect());+ let expected_passthrough = case["passthrough"].as_bool().unwrap();+ let expected_n = case["n"].as_u64().unwrap() as usize;++ // shortlist_choice: the direct, label-only entry point.+ let vectors_for_choice = vectors.clone();+ let choice_embed = move |_texts: &[String]| -> Result<Vec<Vec<f64>>> {+ match &vectors_for_choice {+ Some(v) => Ok(v.clone()),+ None => panic!("{label}: embed_fn should not run when k >= n (shortlist_choice)"),+ }+ };+ let ins_opt = if instructions.is_null() {+ None+ } else {+ Some(instructions)+ };+ let got_labels =+ LayaShortlist::shortlist_choice(&state, criteria.clone(), choice_embed, k, ins_opt)+ .unwrap_or_else(|e| panic!("{label}: shortlist_choice failed: {e}"));+ assert_eq!(+ got_labels, expected_labels,+ "{label}: shortlist_choice labels"+ );++ // predict: the full ShortlistInfo (labels + scores + passthrough + n), reached by putting+ // the same criteria/instructions on one choice question.+ let question = Question::choice(instructions.clone(), criteria.clone()).unwrap();+ let mut questions = QuestionSet::new();+ questions = questions.with("q", question).unwrap();++ let vectors_for_predict = vectors.clone();+ let predict_embed = move |_texts: &[String]| -> Result<Vec<Vec<f64>>> {+ match &vectors_for_predict {+ Some(v) => Ok(v.clone()),+ None => panic!("{label}: embed_fn should not run when k >= n (predict)"),+ }+ };+ let result = LayaShortlist::predict(+ &FirstLabelPredictor,+ state.clone(),+ &questions,+ predict_embed,+ k,+ )+ .unwrap_or_else(|e| panic!("{label}: predict failed: {e}"));+ let info: &ShortlistInfo = &result.shortlist().unwrap()["q"];+ assert_eq!(info.labels, expected_labels, "{label}: predict labels");+ assert_eq!(+ info.passthrough, expected_passthrough,+ "{label}: passthrough"+ );+ assert_eq!(info.n, expected_n, "{label}: n");+ assert_eq!(info.k, k, "{label}: k");+ match (&info.scores, &expected_scores) {+ (None, None) => {}+ (Some(got), Some(want)) => {+ assert_eq!(got.len(), want.len(), "{label}: scores length");+ for (i, (g, w)) in got.iter().zip(want.iter()).enumerate() {+ assert!(+ (g - w).abs() < 1e-9,+ "{label}: scores[{i}] got {g}, want {w}"+ );+ }+ }+ (got, want) => panic!("{label}: scores mismatch, got {got:?}, want {want:?}"),+ }+ }+ eprintln!(+ "shortlist_probe_matches_every_entry: checked {} cases",+ cases.len()+ );+}diff --git a/laya-rust/tests/shortlist_end_to_end_tests.rs b/laya-rust/tests/shortlist_end_to_end_tests.rsnew file mode 100644index 00000000..020f3da4--- /dev/null+++ b/laya-rust/tests/shortlist_end_to_end_tests.rs@@ -0,0 +1,117 @@+//! Tier 2 (needs ONNX artifacts): the shortlist driving a real engine end to end, for all three+//! checkpoints. Ports the `shortlist_many_options` golden case (`"kind": "shortlist"` in each+//! checkpoint's `index.json`, kept out of the ordinary predict-parity cases by the `golden_data`+//! fix described in the routing-port notes).+//!+//! Run the full suite with:+//! `LAYA_ONNX_ROOT=<parent-of-checkpoints> cargo test --release -- --test-threads=1`++mod common;++use common::{engines, golden_data::CheckpointGoldenData, hashing_embedder, result_compare};+use laya::laya_options::LayaCheckpoint;+use laya::laya_shortlist::LayaShortlist;++#[test]+fn shortlist_predict_matches_the_recorded_result_for_every_checkpoint() {+ let mut checked_any = false;++ for checkpoint in engines::CHECKPOINTS {+ let Some(engine) = engines::load(checkpoint) else {+ continue;+ };+ let golden = CheckpointGoldenData::for_checkpoint(checkpoint.subdir());+ if !golden.available() {+ eprintln!("skip: no golden data for '{}'", checkpoint.subdir());+ continue;+ }++ for case_info in golden.shortlist_cases() {+ checked_any = true;+ let root = golden.load_case(&case_info);+ let where_case = format!("{}/{case_info}", checkpoint.subdir());++ let state = root["state"].clone();+ let full_questions = common::golden_data::build_questions(&root["questions"])+ .unwrap_or_else(|e| panic!("{where_case}: build full questions: {e}"));+ let k = root["shortlist_k"]+ .as_u64()+ .unwrap_or_else(|| panic!("{where_case}: missing 'shortlist_k'"))+ as usize;++ let result =+ LayaShortlist::predict(&engine, state, &full_questions, hashing_embedder::embed, k)+ .unwrap_or_else(|e| panic!("{where_case}: LayaShortlist::predict failed: {e}"));++ // The kept, rank-ordered labels for "intent" must match Python's recorded reduction+ // exactly (`reduced_questions.intent.criteria`'s key order).+ let expected_labels: Vec<&str> = root["reduced_questions"]["intent"]["criteria"]+ .as_object()+ .unwrap_or_else(|| {+ panic!("{where_case}: 'reduced_questions.intent.criteria' must be an object")+ })+ .keys()+ .map(String::as_str)+ .collect();+ let shortlist_meta = result.shortlist().unwrap_or_else(|| {+ panic!("{where_case}: LayaResult::shortlist() must be populated")+ });+ let intent_meta = shortlist_meta+ .get("intent")+ .unwrap_or_else(|| panic!("{where_case}: no shortlist metadata for 'intent'"));+ assert_eq!(+ intent_meta.labels, expected_labels,+ "{where_case}: shortlisted labels vs reduced_questions"+ );+ assert!(+ !intent_meta.passthrough,+ "{where_case}: 40 options with k={k} must not pass through"+ );+ assert_eq!(intent_meta.n, 40, "{where_case}: option count");++ // The answer itself (kept labels' probabilities, chosen answer, confidence) must+ // match the recorded predict-on-the-reduced-question result.+ result_compare::assert_result_matches(&where_case, &root["result"], &result);+ }+ }++ if !checked_any {+ eprintln!("skip: no shortlist golden data + artifacts found for any checkpoint");+ }+}++/// Sanity check that every checkpoint's shortlist case actually reduced from 40 to 20 options+/// (the number the plan's combined-sample section names), not some other pair that happens to+/// still pass the assertions above.+#[test]+fn shortlist_many_options_case_shape_is_forty_reduced_to_twenty() {+ let Some(engine) = engines::load(LayaCheckpoint::Multilingual) else {+ eprintln!("skip: no multilingual artifacts");+ return;+ };+ let golden = CheckpointGoldenData::for_checkpoint(LayaCheckpoint::Multilingual.subdir());+ if !golden.available() {+ eprintln!("skip: no golden data for 'multilingual'");+ return;+ }+ let Some(case_info) = golden.shortlist_cases().into_iter().next() else {+ eprintln!("skip: no shortlist case recorded");+ return;+ };+ let root = golden.load_case(&case_info);+ let full_questions =+ common::golden_data::build_questions(&root["questions"]).expect("build questions");+ assert_eq!(root["shortlist_k"].as_u64().unwrap(), 20);+ let result = LayaShortlist::predict(+ &engine,+ root["state"].clone(),+ &full_questions,+ hashing_embedder::embed,+ 20,+ )+ .expect("predict");+ let meta = &result.shortlist().unwrap()["intent"];+ assert_eq!(meta.n, 40);+ assert_eq!(meta.k, 20);+ assert_eq!(meta.labels.len(), 20);+}diff --git a/laya-rust/tools/hashing_embedder.py b/laya-rust/tools/hashing_embedder.pynew file mode 100644index 00000000..0fec90d1--- /dev/null+++ b/laya-rust/tools/hashing_embedder.py@@ -0,0 +1,101 @@+"""Deterministic, dependency-free demo embedder for the shortlist golden vectors and samples.++This is **not** a real embedding model: it is a hashed character-trigram bag-of-features+vector, chosen only because it is a pure function of the input text and is trivial to+reimplement bit-for-bit in another language with nothing but a string, UTF-8 encoding and+32-bit integer arithmetic. It is deliberately named and documented as a demo: any real+deployment should pass `embed_fn_from_agent` (Python) or a real bi-encoder to the shortlist,+not this.++Every other-language port MUST reproduce this exact algorithm, because+`hashing_embedder_probe.json` and `case_shortlist_many_options.json` in the golden vectors+were both generated with it, and the samples in every language use it so that their+stdout can be diffed line for line.++Algorithm (this docstring is the spec -- read it precisely before porting):++1. Lowercase the input with a full Unicode case mapping (Python's `str.lower()`; other+ languages must use their own full Unicode case-folding function on the *string*, NOT a+ per-UTF-16-unit simple mapping. This matters for characters whose lowercase form is+ more than one code point, e.g. `"İ".lower() == "i̇"` (Turkish capital I with dot+ above lowercases to 2 code points: 'i' + combining dot above).+2. Pad the lowercased text with two ASCII space characters (U+0020) on each side:+ `" " + text + " "`. Padding is added even when `text` is empty, so the padded text is+ never shorter than 4 code points and there are always at least 2 windows (see step 3):+ this is what lets short inputs like `""` or `"a"` still produce a nonzero, well-defined+ vector instead of an empty one.+3. Slide a window of exactly 3 **Unicode code points** (not UTF-16 code units, not grapheme+ clusters, not bytes) across the padded text, one code point at a time. For padded text of+ length `n` code points there are `n - 2` windows, at start offsets `0, 1, ..., n - 3`+ inclusive. A code point outside the Basic Multilingual Plane (an "astral" character, e.g.+ an emoji) counts as exactly one code point in this count -- a port that iterates UTF-16+ code units instead of Unicode scalar values will split a surrogate pair into two units and+ get a different window count and different window contents; it must enumerate Unicode+ scalar values instead. Walking grapheme clusters is NOT correct either -- it must be+ rune/code-point enumeration (Rust's `char` already IS a Unicode scalar value, so iterating+ `str::chars()` is correct there).+4. Encode each 3-code-point window as UTF-8 bytes (the window's own bytes, not the whole+ padded text's bytes -- re-encode the 3-code-point substring on its own each time).+5. Hash those UTF-8 bytes with 32-bit FNV-1a:++ hash = 0x811c9dc5 # FNV offset basis+ for byte in utf8_bytes:+ hash ^= byte+ hash = (hash * 0x01000193) & 0xFFFFFFFF # FNV prime, wrapped to 32 bits++ `hash` is an unsigned 32-bit integer throughout; the multiply must wrap (mask to+ 0xFFFFFFFF / use unchecked/wrapping arithmetic in the target language) exactly like+ Python's explicit `& 0xFFFFFFFF` above.+6. `bucket = hash % 64` (0..63, unsigned).+7. `sign = +1.0 if (hash >> 31) & 1 == 0 else -1.0` (the hash's top bit, as an unsigned+ 32-bit value -- this is a sign-hashing trick to reduce feature collisions, not a+ statement about the hash being negative).+8. Accumulate `sign` into `vector[bucket]` for every window (i.e. `vector[bucket] += sign`;+ multiple windows can and do land in the same bucket and their signed contributions add).+ The accumulator is float64 throughout.+9. No normalisation is applied. The shortlist's cosine similarity handles scale on its own,+ and leaving the raw counts in the golden vectors makes them easier to hand-check.++`embed(texts)` returns one such vector per input string, as a `(len(texts), 64)` float64+NumPy array, in input order. It does not batch, cache, or special-case any input; every text+is processed independently and only via the steps above.+"""+from typing import Sequence++import numpy as np++DIM = 64+_FNV_OFFSET_BASIS = 0x811C9DC5+_FNV_PRIME = 0x01000193+_MASK32 = 0xFFFFFFFF+++def _fnv1a_32(data: bytes) -> int:+ """32-bit FNV-1a over raw bytes, wrapping like an unsigned 32-bit integer."""+ h = _FNV_OFFSET_BASIS+ for b in data:+ h ^= b+ h = (h * _FNV_PRIME) & _MASK32+ return h+++def _embed_one(text: str) -> np.ndarray:+ vec = np.zeros(DIM, dtype=np.float64)+ padded = " " + text.lower() + " "+ n = len(padded)+ for i in range(n - 2):+ window = padded[i : i + 3]+ h = _fnv1a_32(window.encode("utf-8"))+ bucket = h % DIM+ sign = 1.0 if (h >> 31) & 1 == 0 else -1.0+ vec[bucket] += sign+ return vec+++def embed(texts: Sequence[str]) -> np.ndarray:+ """Embed a list of strings into a `(len(texts), 64)` float64 array. See the module docstring+ for the exact algorithm; every other-language port must match it bit-for-bit."""+ rows = [_embed_one("" if t is None else str(t)) for t in texts]+ if not rows:+ return np.zeros((0, DIM), dtype=np.float64)+ return np.stack(rows, axis=0)