Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(rust): add embedding shortlist for high-cardinality choice

kevin-gatimu24ff83862ae6
diff --git a/laya-rust/src/laya_shortlist.rs b/laya-rust/src/laya_shortlist.rs
new file mode 100644
index 00000000..8d2f90b1
--- /dev/null
+++ b/laya-rust/src/laya_shortlist.rs
@@ -0,0 +1,323 @@
+//! Opt-in embedding shortlist for high-cardinality choice questions. Ports `laya/shortlist.py`.
+//!
+//! Choice options share one `head_max_len` token budget, so a large label set leaves only a few
+//! tokens per label. [`LayaShortlist::predict`] embeds the state and each option with a
+//! caller-supplied `embed_fn`, keeps the top `k`, and runs a single predict on that reduced
+//! criteria set. A non-choice question is forwarded unchanged, and a choice whose label count is
+//! already `<= k` is forwarded unchanged too — `embed_fn` is never called for it.
+//!
+//! `embed_fn_from_agent` (Python: mean-pools the checkpoint encoder already loaded on an `Agent`)
+//! is deliberately not ported: this port only accepts a caller-supplied embedding function (a
+//! real deployment should bring a dedicated bi-encoder), matching the plan for this port.
+//!
+//! ## Why `embed_fn` is a generic `Fn` bound rather than a boxed trait object
+//!
+//! `shortlist_choice`/`predict` take `embed_fn: F where F: Fn(&[String]) -> Result<Vec<Vec<f64>>>`
+//! instead of `embed_fn: &dyn Fn(...)` or `Box<dyn Fn(...)>`. A shortlist call embeds a handful of
+//! short strings once per choice question — not a hot loop — so the extra dynamic-dispatch
+//! indirection a trait object would add buys nothing; a generic parameter still accepts an
+//! ordinary closure, a `fn` item, or a `&F`/`Box<F>` coerced by the caller, and it lets the
+//! compiler monomorphize and inline the common "one closure, called a few times" case. The
+//! function signature itself (`&[String] -> Result<Vec<Vec<f64>>>`) mirrors Python's
+//! `Callable[[Sequence[str]], Any]`: a list of texts in, one embedding vector per text out, in
+//! the same order, fallible (a real embedder can hit a network or model error, unlike this
+//! crate's own deterministic test embedder).
+
+use crate::answers::LayaResult;
+use crate::error::{LayaError, Result};
+use crate::laya_router::LayaPredictor;
+use crate::python_json::PythonJson;
+use crate::questions::{Question, QuestionSet};
+use crate::sequence_builder::SequenceBuilder;
+use indexmap::IndexMap;
+use serde_json::Value;
+use std::collections::HashSet;
+
+/// Default shortlist size, matching Python's `DEFAULT_SHORTLIST_K`.
+pub const DEFAULT_SHORTLIST_K: usize = 20;
+
+/// What [`LayaShortlist::predict`] recorded for one shortlisted choice question.
+#[derive(Debug, Clone, PartialEq)]
+pub struct ShortlistInfo {
+ /// Kept labels, in rank order (highest cosine similarity first).
+ pub labels: Vec<String>,
+ /// Cosine similarity per kept label, in the same order as `labels`. `None` when this
+ /// question passed through unchanged (`k` was already `>=` the label count, so nothing was
+ /// ranked and `embed_fn` was never called).
+ pub scores: Option<Vec<f64>>,
+ /// The `k` this question was shortlisted with.
+ pub k: usize,
+ /// How many labels the question had before shortlisting.
+ pub n: usize,
+ /// Whether this question passed through unchanged.
+ pub passthrough: bool,
+}
+
+/// Namespace for the embedding shortlist. See the module docs for the source this ports.
+pub struct LayaShortlist;
+
+impl LayaShortlist {
+ /// Return the top-`k` choice labels for `state`.
+ ///
+ /// `embed_fn` is called once, with the query text first and then one string per option in
+ /// criteria order; option strings match [`SequenceBuilder::render_options`] for a choice
+ /// question. When `k` is at least the number of labels, every label is returned in its
+ /// original order and `embed_fn` is not called.
+ ///
+ /// Ties keep the earlier label. A zero vector scores 0 and does not outrank a label that came
+ /// before it.
+ ///
+ /// `criteria` is an ordered `(label, description)` sequence — the same shape
+ /// [`Question::choice`] takes, covering both Python's dict criteria (label -> description)
+ /// and list criteria (bare labels, each with a `Value::Null` description) uniformly. Rejects
+ /// an empty or duplicate-labelled criteria set the same way `Question::choice` does.
+ pub fn shortlist_choice<F, L, D>(
+ state: &Value,
+ criteria: impl IntoIterator<Item = (L, D)>,
+ embed_fn: F,
+ k: usize,
+ instructions: Option<&Value>,
+ ) -> Result<Vec<String>>
+ where
+ F: Fn(&[String]) -> Result<Vec<Vec<f64>>>,
+ L: Into<String>,
+ D: Into<Value>,
+ {
+ let items: Vec<(String, Value)> = criteria
+ .into_iter()
+ .map(|(l, d)| (l.into(), d.into()))
+ .collect();
+ check_k(k)?;
+ check_criteria(&items)?;
+ let query = query_text(state, instructions);
+ Ok(rank(query, &items, &embed_fn, k)?.labels)
+ }
+
+ /// Shortlist each choice question in `questions`, then call `predictor.predict` once on the
+ /// reduced set. Non-choice questions are forwarded unchanged; a choice question whose label
+ /// count is already `<= k` is forwarded unchanged and does not call `embed_fn`.
+ ///
+ /// The returned [`LayaResult`] carries [`LayaResult::shortlist`] recording, per shortlisted
+ /// choice question id, the kept labels (rank order), their cosine scores (`None` for a
+ /// passthrough), `k`, `n` and whether it passed through. `predictor` may be a bare
+ /// [`crate::LayaEngine`] or a [`crate::laya_router::LayaRouter`] — they compose, since a
+ /// router's own [`LayaPredictor::predict`] already attaches its routing decision to the
+ /// result this then adds shortlist metadata to.
+ pub fn predict<F>(
+ predictor: &dyn LayaPredictor,
+ state: impl Into<Value>,
+ questions: &QuestionSet,
+ embed_fn: F,
+ k: usize,
+ ) -> Result<LayaResult>
+ where
+ F: Fn(&[String]) -> Result<Vec<Vec<f64>>>,
+ {
+ check_k(k)?;
+ let state = state.into();
+ let mut reduced = QuestionSet::new();
+ let mut meta: IndexMap<String, ShortlistInfo> = IndexMap::new();
+
+ for (id, question) in questions.iter() {
+ match question {
+ Question::Choice(choice) => {
+ let items = choice.options().to_vec();
+ let query = query_text(&state, Some(choice.instructions()));
+ let outcome = rank(query, &items, &embed_fn, k)?;
+
+ meta.insert(
+ id.to_string(),
+ ShortlistInfo {
+ labels: outcome.labels.clone(),
+ scores: outcome.scores,
+ k,
+ n: outcome.n,
+ passthrough: outcome.passthrough,
+ },
+ );
+
+ if outcome.passthrough {
+ reduced.insert(id, question.clone())?;
+ } else {
+ // Rebuild with only the kept labels, in rank order: Python's
+ // `_subset_criteria` builds `{label: criteria[label] for label in
+ // labels}`, which is exactly a re-keyed, re-ordered option list here.
+ let kept: Vec<(String, Value)> = outcome
+ .labels
+ .into_iter()
+ .map(|label| {
+ let description = items
+ .iter()
+ .find(|(l, _)| *l == label)
+ .map(|(_, d)| d.clone())
+ .expect("a ranked label always came from `items`");
+ (label, description)
+ })
+ .collect();
+ let reduced_choice = Question::choice(choice.instructions().clone(), kept)?;
+ reduced.insert(id, reduced_choice)?;
+ }
+ }
+ other => {
+ reduced.insert(id, other.clone())?;
+ }
+ }
+ }
+
+ let result = predictor.predict(state, &reduced)?;
+ Ok(result.with_shortlist(meta))
+ }
+}
+
+fn check_k(k: usize) -> Result<()> {
+ if k == 0 {
+ return Err(LayaError::InvalidQuestion(
+ "k must be a positive integer, got 0".to_string(),
+ ));
+ }
+ Ok(())
+}
+
+fn check_criteria(items: &[(String, Value)]) -> Result<()> {
+ if items.is_empty() {
+ return Err(LayaError::InvalidQuestion(
+ "choice criteria must contain at least one option".to_string(),
+ ));
+ }
+ let mut seen = HashSet::with_capacity(items.len());
+ for (label, _) in items {
+ if !seen.insert(label.as_str()) {
+ return Err(LayaError::InvalidQuestion(format!(
+ "choice criteria label '{label}' is duplicated"
+ )));
+ }
+ }
+ Ok(())
+}
+
+/// The shortlist ranking's query text: `serialize_state(state)`, or `"instructions\n" +
+/// serialize_state(state)` when non-blank instructions are given. Blank is `None` or the empty
+/// string, matching Python's `instructions is None or instructions == ""` (a non-string
+/// instructions value is never equal to `""`, so only an actual empty *string* counts).
+fn query_text(state: &Value, instructions: Option<&Value>) -> String {
+ let body = PythonJson::state(state);
+ match instructions {
+ Some(v) if !is_blank_instructions(v) => {
+ format!("{}\n{body}", PythonJson::shortlist_instructions(v))
+ }
+ _ => body,
+ }
+}
+
+fn is_blank_instructions(value: &Value) -> bool {
+ matches!(value, Value::Null) || matches!(value, Value::String(s) if s.is_empty())
+}
+
+/// What [`rank`] found: the same four values [`ShortlistInfo`] eventually carries for one
+/// question, plus the bare label list [`LayaShortlist::shortlist_choice`] returns.
+struct RankOutcome {
+ labels: Vec<String>,
+ scores: Option<Vec<f64>>,
+ passthrough: bool,
+ n: usize,
+}
+
+/// Rank `items` against `query` with `embed_fn`. `query` is the already-built query text (not the
+/// raw state), so both [`LayaShortlist::shortlist_choice`] and [`LayaShortlist::predict`] can
+/// share this after building their own query text (a bare `instructions` kwarg for the former, a
+/// question's own `instructions` field for the latter).
+fn rank<F>(query: String, items: &[(String, Value)], embed_fn: &F, k: usize) -> Result<RankOutcome>
+where
+ F: Fn(&[String]) -> Result<Vec<Vec<f64>>>,
+{
+ let n = items.len();
+ let keys: Vec<String> = items.iter().map(|(label, _)| label.clone()).collect();
+ if k >= n {
+ return Ok(RankOutcome {
+ labels: keys,
+ scores: None,
+ passthrough: true,
+ n,
+ });
+ }
+
+ // A throwaway choice question purely to reuse `SequenceBuilder::render_options`'s exact
+ // rendering rule (bare label when the description is blank, `"label: description"`
+ // otherwise) rather than duplicating it here.
+ let temp = Question::choice(Value::Null, items.iter().cloned())?;
+ let option_texts = SequenceBuilder::render_options(&temp);
+
+ let mut texts = Vec::with_capacity(1 + option_texts.len());
+ texts.push(query);
+ texts.extend(option_texts);
+
+ let matrix = embed_matrix(embed_fn, &texts)?;
+ let sims = cosine(&matrix[0], &matrix[1..]);
+
+ let mut order: Vec<usize> = (0..sims.len()).collect();
+ // Stable sort descending by score: comparing `(b, a)` instead of `(a, b)` reverses the sense
+ // of the comparison while stability still preserves the original (ascending-index) order
+ // among ties, matching `np.argsort(-sims, kind="mergesort")`.
+ order.sort_by(|&a, &b| sims[b].total_cmp(&sims[a]));
+ order.truncate(k);
+
+ let labels: Vec<String> = order.iter().map(|&i| keys[i].clone()).collect();
+ let scores: Vec<f64> = order.iter().map(|&i| sims[i]).collect();
+ Ok(RankOutcome {
+ labels,
+ scores: Some(scores),
+ passthrough: false,
+ n,
+ })
+}
+
+/// Calls `embed_fn`, checks the returned matrix has one row per text and a consistent, non-zero
+/// row width, and zeroes non-finite components (`np.nan_to_num`: only the offending components,
+/// not the whole row — `[NaN, 1.0]` becomes `[0.0, 1.0]`, a real vector, not a zero vector).
+fn embed_matrix<F>(embed_fn: &F, texts: &[String]) -> Result<Vec<Vec<f64>>>
+where
+ F: Fn(&[String]) -> Result<Vec<Vec<f64>>>,
+{
+ let raw = embed_fn(texts)?;
+ if raw.len() != texts.len() {
+ return Err(LayaError::InvalidQuestion(format!(
+ "embed_fn must return one row per text: got {} rows for {} texts",
+ raw.len(),
+ texts.len()
+ )));
+ }
+ let dim = raw.first().map(Vec::len).unwrap_or(0);
+ if dim == 0 || raw.iter().any(|row| row.len() != dim) {
+ return Err(LayaError::InvalidQuestion(
+ "embed_fn must return rows of equal, non-zero length".to_string(),
+ ));
+ }
+ Ok(raw
+ .into_iter()
+ .map(|row| {
+ row.into_iter()
+ .map(|v| if v.is_finite() { v } else { 0.0 })
+ .collect()
+ })
+ .collect())
+}
+
+/// Cosine similarity of `query` against every row of `docs`. A zero-norm query scores every
+/// document 0 (matching Python: the whole `sims` array is returned as-is, all zero, without even
+/// looking at the documents). A zero-norm document scores 0 against a non-zero query.
+fn cosine(query: &[f64], docs: &[Vec<f64>]) -> Vec<f64> {
+ let query_norm = query.iter().map(|v| v * v).sum::<f64>().sqrt();
+ let mut sims = vec![0.0; docs.len()];
+ if query_norm == 0.0 {
+ return sims;
+ }
+ for (i, doc) in docs.iter().enumerate() {
+ let doc_norm = doc.iter().map(|v| v * v).sum::<f64>().sqrt();
+ let denom = doc_norm * query_norm;
+ if denom > 0.0 {
+ let dot: f64 = doc.iter().zip(query).map(|(a, b)| a * b).sum();
+ sims[i] = dot / denom;
+ }
+ }
+ sims
+}
diff --git a/laya-rust/tests/common/hashing_embedder.rs b/laya-rust/tests/common/hashing_embedder.rs
new file mode 100644
index 00000000..f1c67183
--- /dev/null
+++ b/laya-rust/tests/common/hashing_embedder.rs
@@ -0,0 +1,65 @@
+//! Deterministic, dependency-free demo embedder for the shortlist golden vectors and end-to-end
+//! tests. Ports `tools/hashing_embedder.py` bit-for-bit — see that file's docstring for the exact
+//! algorithm spec this follows step by step. **Not** a real embedding model: a hashed
+//! character-trigram bag-of-features vector, chosen only because it is a pure function of the
+//! input text that is trivial to reproduce identically in every language.
+//!
+//! Every step below is precision-sensitive:
+//! - Lowercasing uses full Unicode case mapping (`str::to_lowercase`), matching Python's
+//! `str.lower()` — not a per-UTF-16-unit simple mapping.
+//! - The 3-item sliding window is over Unicode scalar values (`char`, which in Rust already *is*
+//! a Unicode scalar value), matching Python's per-code-point iteration; an astral character
+//! (outside the BMP) is one window position, not two.
+//! - The window is re-encoded to UTF-8 on its own each time (not sliced out of one pre-encoded
+//! byte buffer), because a multi-byte character's encoding does not change with position, but
+//! slicing a shared buffer at the wrong byte offset would corrupt it.
+//! - The FNV-1a multiply wraps at 32 bits (`u32::wrapping_mul`), matching Python's explicit
+//! `& 0xFFFFFFFF`.
+
+use laya::error::Result as LayaResult;
+
+/// Output vector width.
+pub const DIM: usize = 64;
+
+const FNV_OFFSET_BASIS: u32 = 0x811c_9dc5;
+const FNV_PRIME: u32 = 0x0100_0193;
+
+/// 32-bit FNV-1a over raw bytes, wrapping like an unsigned 32-bit integer.
+fn fnv1a_32(data: &[u8]) -> u32 {
+ let mut hash = FNV_OFFSET_BASIS;
+ for &byte in data {
+ hash ^= u32::from(byte);
+ hash = hash.wrapping_mul(FNV_PRIME);
+ }
+ hash
+}
+
+fn embed_one(text: &str) -> Vec<f64> {
+ let mut vector = vec![0.0f64; DIM];
+ let padded: Vec<char> = format!(" {} ", text.to_lowercase()).chars().collect();
+ let n = padded.len();
+ // Padding alone is 4 characters, so `n >= 4` always and this subtraction never underflows;
+ // the guard is defensive documentation, not a reachable branch.
+ if n < 3 {
+ return vector;
+ }
+ for start in 0..=(n - 3) {
+ let window: String = padded[start..start + 3].iter().collect();
+ let hash = fnv1a_32(window.as_bytes());
+ let bucket = (hash % DIM as u32) as usize;
+ let sign = if (hash >> 31) & 1 == 0 { 1.0 } else { -1.0 };
+ vector[bucket] += sign;
+ }
+ vector
+}
+
+/// Embed a list of strings into one `(len(texts), 64)` matrix, in input order. See the module
+/// docs for the exact algorithm; `None` inputs have no Rust equivalent (the signature already
+/// requires `&str`), matching Python's `"" if t is None else str(t)` only in that an empty string
+/// embeds the same way an absent one would have.
+///
+/// Returns [`laya::error::Result`] so it can be used directly wherever `LayaShortlist` expects an
+/// `embed_fn` closure, even though this particular embedder never fails.
+pub fn embed(texts: &[String]) -> LayaResult<Vec<Vec<f64>>> {
+ Ok(texts.iter().map(|t| embed_one(t)).collect())
+}
diff --git a/laya-rust/tests/hashing_embedder_tests.rs b/laya-rust/tests/hashing_embedder_tests.rs
new file mode 100644
index 00000000..4628f7b0
--- /dev/null
+++ b/laya-rust/tests/hashing_embedder_tests.rs
@@ -0,0 +1,47 @@
+//! Tier 1: the deterministic hashing embedder (`tests/common/hashing_embedder.rs`) against every
+//! entry of the routing golden's `hashing_embedder_probe.json`. This is a test-helper probe, not
+//! a check on `laya` itself — the embedder is reimplemented per-SDK, per the plan, so the Rust
+//! test helper must reproduce `tools/hashing_embedder.py` bit-for-bit.
+
+mod common;
+
+use common::{hashing_embedder, routing_golden};
+
+#[test]
+fn hashing_embedder_matches_every_probe_entry() {
+ let Some(cases) = routing_golden::load_array("hashing_embedder_probe.json") else {
+ eprintln!("skip: routing golden not found (tests/golden/routing)");
+ return;
+ };
+
+ for case in &cases {
+ let text = case["text"]
+ .as_str()
+ .expect("probe entry needs a 'text' string");
+ let expected: Vec<f64> = case["vector"]
+ .as_array()
+ .expect("probe entry needs a 'vector' array")
+ .iter()
+ .map(|v| v.as_f64().expect("vector component must be a number"))
+ .collect();
+
+ let got = hashing_embedder::embed(&[text.to_string()]).expect("embed never fails");
+ assert_eq!(got.len(), 1);
+ assert_eq!(got[0].len(), expected.len(), "text {text:?}: vector length");
+ for (i, (&g, &e)) in got[0].iter().zip(expected.iter()).enumerate() {
+ assert_eq!(g, e, "text {text:?}: vector[{i}]");
+ }
+ }
+ eprintln!(
+ "hashing_embedder_matches_every_probe_entry: checked {} texts",
+ cases.len()
+ );
+}
+
+#[test]
+fn hashing_embedder_is_order_preserving_and_batches_independently() {
+ let a = hashing_embedder::embed(&["alpha".to_string()]).unwrap();
+ let b = hashing_embedder::embed(&["beta".to_string(), "alpha".to_string()]).unwrap();
+ // Each text is embedded independently of its neighbours in the batch.
+ assert_eq!(a[0], b[1]);
+}
diff --git a/laya-rust/tests/laya_shortlist_tests.rs b/laya-rust/tests/laya_shortlist_tests.rs
new file mode 100644
index 00000000..14d9c4be
--- /dev/null
+++ b/laya-rust/tests/laya_shortlist_tests.rs
@@ -0,0 +1,515 @@
+//! Tier 1: the embedding shortlist. Ports the non-torch checks from `tests/test_shortlist.py`
+//! (the torch mean-pool block — `embed_fn_from_agent` — is out of scope: this port only accepts a
+//! caller-supplied `embed_fn`) plus every entry of the routing golden's `shortlist_probe.json`,
+//! and the hashing embedder vs its own probe (see `hashing_embedder_tests.rs`).
+
+mod common;
+
+use common::routing_golden;
+use indexmap::IndexMap;
+use laya::answers::{ActionInfo, Answer, ChoiceAnswer, LayaResult, Usage};
+use laya::error::{LayaError, Result};
+use laya::laya_router::LayaPredictor;
+use laya::laya_shortlist::{LayaShortlist, ShortlistInfo};
+use laya::questions::{Question, QuestionSet};
+use serde_json::{Value, json};
+use std::collections::HashMap;
+
+/// A predictor that never inspects `state`: it just answers every choice question with its first
+/// (post-shortlist) label, so `LayaShortlist::predict` tests can check the shortlist metadata and
+/// the reduced question set without needing a real engine.
+struct FirstLabelPredictor;
+
+impl LayaPredictor for FirstLabelPredictor {
+ fn predict(&self, _state: Value, questions: &QuestionSet) -> Result<LayaResult> {
+ let order: Vec<String> = questions.ids().map(String::from).collect();
+ let mut by_id = HashMap::new();
+ for id in &order {
+ if let Question::Choice(c) = questions.get(id).unwrap() {
+ let first = c.labels().next().unwrap_or("").to_string();
+ let mut probabilities = IndexMap::new();
+ probabilities.insert(first.clone(), 1.0);
+ by_id.insert(
+ id.clone(),
+ Answer::Choice(ChoiceAnswer::new(
+ first,
+ probabilities,
+ 1.0,
+ 1.0,
+ ActionInfo::default(),
+ )),
+ );
+ }
+ }
+ Ok(LayaResult::new("mock", order, by_id, Usage::default()))
+ }
+}
+
+fn table_embed(table: &[(&str, [f64; 2])]) -> impl Fn(&[String]) -> Result<Vec<Vec<f64>>> {
+ let table: Vec<(String, Vec<f64>)> = table
+ .iter()
+ .map(|(k, v)| (k.to_string(), v.to_vec()))
+ .collect();
+ move |texts: &[String]| {
+ texts
+ .iter()
+ .map(|t| {
+ table
+ .iter()
+ .find(|(k, _)| k == t)
+ .map(|(_, v)| v.clone())
+ .ok_or_else(|| LayaError::InvalidQuestion(format!("unexpected text {t:?}")))
+ })
+ .collect()
+ }
+}
+
+fn boom_embed(_texts: &[String]) -> Result<Vec<Vec<f64>>> {
+ panic!("embed_fn should not run when k >= n")
+}
+
+// ---------------------------------------------------------------------- named cases (test_shortlist.py)
+
+#[test]
+fn deterministic_topk_and_ties() {
+ // Vectors: query [1, 0]. alpha and delta tie at cosine 1; gamma is 0.6; beta is 0.
+ // Stable order must keep alpha ahead of delta.
+ let embed = table_embed(&[
+ ("pay me", [1.0, 0.0]),
+ ("alpha", [1.0, 0.0]),
+ ("beta", [0.0, 1.0]),
+ ("gamma: mid", [0.6, 0.8]),
+ ("delta: same", [1.0, 0.0]),
+ ]);
+ let criteria = [
+ ("alpha", Value::Null),
+ ("beta", json!("")),
+ ("gamma", json!("mid")),
+ ("delta", json!("same")),
+ ];
+ let state = json!("pay me");
+
+ assert_eq!(
+ LayaShortlist::shortlist_choice(&state, criteria.clone(), &embed, 2, None).unwrap(),
+ vec!["alpha", "delta"],
+ "topk/k=2 keeps the cosine tie in input order"
+ );
+ assert_eq!(
+ LayaShortlist::shortlist_choice(&state, criteria.clone(), &embed, 1, None).unwrap(),
+ vec!["alpha"],
+ "topk/k=1 is the earliest max"
+ );
+ assert_eq!(
+ LayaShortlist::shortlist_choice(&state, criteria.clone(), &embed, 3, None).unwrap(),
+ vec!["alpha", "delta", "gamma"],
+ "topk/k=3 appends the next cosine"
+ );
+
+ // All-zero query: every cosine is 0, so the earliest labels win.
+ let zero_q = table_embed(&[
+ ("pay me", [0.0, 0.0]),
+ ("alpha", [1.0, 0.0]),
+ ("beta", [0.0, 1.0]),
+ ("gamma: mid", [0.6, 0.8]),
+ ("delta: same", [3.0, 4.0]),
+ ]);
+ assert_eq!(
+ LayaShortlist::shortlist_choice(&state, criteria.clone(), &zero_q, 2, None).unwrap(),
+ vec!["alpha", "beta"],
+ "topk/zero query keeps original order"
+ );
+
+ // A non-finite option vector is treated as 0 and loses to a real match.
+ let nan_embed = |texts: &[String]| -> Result<Vec<Vec<f64>>> {
+ Ok(texts
+ .iter()
+ .map(|t| match t.as_str() {
+ "pay me" => vec![1.0, 0.0],
+ "alpha" => vec![f64::NAN, f64::NAN],
+ "beta" => vec![1.0, 0.0],
+ other => panic!("unexpected text {other:?}"),
+ })
+ .collect())
+ };
+ assert_eq!(
+ LayaShortlist::shortlist_choice(
+ &state,
+ [("alpha", Value::Null), ("beta", Value::Null)],
+ nan_embed,
+ 1,
+ None
+ )
+ .unwrap(),
+ vec!["beta"],
+ "topk/nan vector sorts behind a finite match"
+ );
+}
+
+#[test]
+fn list_criteria_and_instructions_change_the_query() {
+ let embed = table_embed(&[
+ ("Classify\npay me", [0.0, 1.0]),
+ ("alpha", [1.0, 0.0]),
+ ("beta", [0.0, 1.0]),
+ ("gamma", [0.0, 0.2]),
+ ]);
+ let state = json!("pay me");
+ let instructions = json!("Classify");
+ let got = LayaShortlist::shortlist_choice(
+ &state,
+ [
+ ("alpha", Value::Null),
+ ("beta", Value::Null),
+ ("gamma", Value::Null),
+ ],
+ &embed,
+ 2,
+ Some(&instructions),
+ )
+ .unwrap();
+ assert_eq!(
+ got,
+ vec!["beta", "gamma"],
+ "list/instructions change the winner"
+ );
+}
+
+#[test]
+fn dict_state_is_serialized_into_the_query() {
+ let embed = table_embed(&[
+ ("Classify\n{\"text\": \"hi\"}", [1.0, 0.0]),
+ ("alpha", [1.0, 0.0]),
+ ("beta", [0.0, 1.0]),
+ ]);
+ let state = json!({"text": "hi"});
+ let instructions = json!("Classify");
+ let got = LayaShortlist::shortlist_choice(
+ &state,
+ [("alpha", Value::Null), ("beta", Value::Null)],
+ &embed,
+ 1,
+ Some(&instructions),
+ )
+ .unwrap();
+ assert_eq!(got, vec!["alpha"], "query/dict state is serialized");
+}
+
+#[test]
+fn k_ge_n_passes_through_without_calling_embed_fn() {
+ let criteria = [
+ ("alpha", Value::Null),
+ ("beta", json!("")),
+ ("gamma", json!("mid")),
+ ("delta", json!("same")),
+ ];
+ let state = json!("pay me");
+ assert_eq!(
+ LayaShortlist::shortlist_choice(&state, criteria.clone(), boom_embed, 4, None).unwrap(),
+ vec!["alpha", "beta", "gamma", "delta"],
+ "pass/k == n returns every label in order"
+ );
+ assert_eq!(
+ LayaShortlist::shortlist_choice(&state, criteria, boom_embed, 20, None).unwrap(),
+ vec!["alpha", "beta", "gamma", "delta"],
+ "pass/k > n returns every label in order"
+ );
+}
+
+#[test]
+fn predict_reduces_choice_and_forwards_other_questions_unchanged() {
+ // cosine vs [1, 0]: tech=1, sales≈0.707, billing=0, other=0. k=2 -> tech, sales.
+ let embed = table_embed(&[
+ ("Which desk?\nI was charged twice", [1.0, 0.0]),
+ ("billing: {\"desc\": \"payments\"}", [0.0, 1.0]),
+ ("tech: bugs", [1.0, 0.0]),
+ ("sales", [0.2, 0.2]),
+ ("other: misc", [0.0, 1.0]),
+ ]);
+ let full_criteria = [
+ ("billing", json!({"desc": "payments"})),
+ ("tech", json!("bugs")),
+ ("sales", Value::Null),
+ ("other", json!("misc")),
+ ];
+ let intent = Question::choice("Which desk?", full_criteria).unwrap();
+ let urgency = Question::score("How urgent?", ["low", "mid", "high", "now"]).unwrap();
+ let refund = Question::noul("Is a refund requested?", Value::Null, Value::Null);
+ let mut questions = QuestionSet::new();
+ questions = questions.with("intent", intent).unwrap();
+ questions = questions.with("urgency", urgency).unwrap();
+ questions = questions.with("refund", refund).unwrap();
+
+ let state = json!("I was charged twice");
+ let result = LayaShortlist::predict(&FirstLabelPredictor, state, &questions, embed, 2).unwrap();
+
+ let shortlist = result.shortlist().expect("shortlist metadata");
+ let intent_meta = &shortlist["intent"];
+ assert_eq!(
+ intent_meta.labels,
+ vec!["tech", "sales"],
+ "kept top 2, rank order"
+ );
+ assert_eq!(intent_meta.k, 2);
+ assert_eq!(intent_meta.n, 4);
+ assert!(!intent_meta.passthrough);
+ let scores = intent_meta.scores.as_ref().unwrap();
+ assert!(
+ scores[0] > scores[1] && scores[1] > 0.0,
+ "result/scores descend"
+ );
+ assert!(
+ !shortlist.contains_key("urgency"),
+ "non-choice questions absent from shortlist meta"
+ );
+
+ assert_eq!(
+ result.get("intent").unwrap().as_choice().unwrap().choice,
+ "tech",
+ "mock's first shortlisted label wins"
+ );
+}
+
+#[test]
+fn predict_passthrough_forwards_the_original_question() {
+ let full_criteria = [
+ ("billing", json!({"desc": "payments"})),
+ ("tech", json!("bugs")),
+ ("sales", Value::Null),
+ ("other", json!("misc")),
+ ];
+ let intent = Question::choice("Which desk?", full_criteria).unwrap();
+ let mut questions = QuestionSet::new();
+ questions = questions.with("intent", intent).unwrap();
+
+ let result = LayaShortlist::predict(
+ &FirstLabelPredictor,
+ json!("I was charged twice"),
+ &questions,
+ boom_embed,
+ 4,
+ )
+ .unwrap();
+ let meta = &result.shortlist().unwrap()["intent"];
+ assert!(meta.passthrough, "pass/flag");
+ assert_eq!(meta.scores, None, "pass/scores omitted");
+ assert_eq!(meta.labels, vec!["billing", "tech", "sales", "other"]);
+}
+
+#[test]
+fn zero_and_false_criteria_values_are_real_descriptions() {
+ let rich = [
+ ("zero", json!(0)),
+ ("no", json!(false)),
+ ("bare", Value::Null),
+ ("named", json!({"desc": "payments"})),
+ ];
+ let state = json!("pay me");
+ // Passthrough (k >= n): does not call embed_fn, so this only exercises validation + ordering.
+ let got = LayaShortlist::shortlist_choice(&state, rich, boom_embed, 4, None).unwrap();
+ assert_eq!(got, vec!["zero", "no", "bare", "named"]);
+}
+
+#[test]
+fn errors_are_rejected() {
+ let state = json!("pay me");
+ let embed = |_: &[String]| -> Result<Vec<Vec<f64>>> { Ok(vec![]) };
+
+ assert!(matches!(
+ LayaShortlist::shortlist_choice(&state, [("a", Value::Null)], embed, 0, None),
+ Err(LayaError::InvalidQuestion(_))
+ ));
+ assert!(matches!(
+ LayaShortlist::shortlist_choice(&state, Vec::<(&str, Value)>::new(), embed, 1, None),
+ Err(LayaError::InvalidQuestion(_))
+ ));
+ assert!(matches!(
+ LayaShortlist::shortlist_choice(
+ &state,
+ [("alpha", Value::Null), ("alpha", Value::Null)],
+ embed,
+ 1,
+ None
+ ),
+ Err(LayaError::InvalidQuestion(_))
+ ));
+
+ // A shape mismatch from embed_fn must fail, and must not have called `predict`.
+ let bad_shape = |_: &[String]| -> Result<Vec<Vec<f64>>> { Ok(vec![vec![0.0; 4]]) };
+ struct Counting(std::sync::atomic::AtomicU32);
+ impl LayaPredictor for Counting {
+ fn predict(&self, _state: Value, questions: &QuestionSet) -> Result<LayaResult> {
+ self.0.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
+ Ok(LayaResult::new(
+ "mock",
+ questions.ids().map(String::from).collect(),
+ HashMap::new(),
+ Usage::default(),
+ ))
+ }
+ }
+ let counting = Counting(std::sync::atomic::AtomicU32::new(0));
+ let original = Question::choice(
+ "Which desk?",
+ [("billing", Value::Null), ("tech", Value::Null)],
+ )
+ .unwrap();
+ let mut questions = QuestionSet::new();
+ questions = questions.with("intent", original).unwrap();
+ let err = LayaShortlist::predict(&counting, json!("pay me"), &questions, bad_shape, 1);
+ assert!(err.is_err(), "err/bad embed shape");
+ assert_eq!(
+ counting.0.load(std::sync::atomic::Ordering::SeqCst),
+ 0,
+ "err/bad shape does not call predict"
+ );
+}
+
+// ---------------------------------------------------------------------- shortlist_probe.json
+
+fn parse_component(v: &Value) -> f64 {
+ match v {
+ Value::String(s) => match s.as_str() {
+ "nan" => f64::NAN,
+ "inf" => f64::INFINITY,
+ "-inf" => f64::NEG_INFINITY,
+ other => other
+ .parse()
+ .unwrap_or_else(|_| panic!("unexpected string component {other:?}")),
+ },
+ _ => v
+ .as_f64()
+ .expect("expected a numeric (or nan/inf-string) component"),
+ }
+}
+
+fn parse_criteria(v: &Value) -> Vec<(String, Value)> {
+ match v {
+ Value::Object(map) => map
+ .iter()
+ .map(|(k, val)| (k.clone(), val.clone()))
+ .collect(),
+ Value::Array(items) => items
+ .iter()
+ .map(|item| {
+ (
+ item.as_str()
+ .expect("list criteria item must be a string")
+ .to_string(),
+ Value::Null,
+ )
+ })
+ .collect(),
+ other => panic!("unexpected criteria shape: {other:?}"),
+ }
+}
+
+fn parse_vectors(v: &Value) -> Option<Vec<Vec<f64>>> {
+ let arr = v.as_array()?;
+ Some(
+ arr.iter()
+ .map(|row| {
+ row.as_array()
+ .unwrap()
+ .iter()
+ .map(parse_component)
+ .collect()
+ })
+ .collect(),
+ )
+}
+
+#[test]
+fn shortlist_probe_matches_every_entry() {
+ let Some(cases) = routing_golden::load_array("shortlist_probe.json") else {
+ eprintln!("skip: routing golden not found (tests/golden/routing)");
+ return;
+ };
+
+ for case in &cases {
+ let label = case["label"].as_str().unwrap();
+ let state = case["state"].clone();
+ let criteria = parse_criteria(&case["criteria"]);
+ let k = case["k"].as_u64().unwrap() as usize;
+ let instructions = &case["instructions"];
+ let vectors = parse_vectors(&case["vectors"]);
+ let expected_labels: Vec<String> = case["labels"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .map(|v| v.as_str().unwrap().to_string())
+ .collect();
+ let expected_scores: Option<Vec<f64>> = case["scores"]
+ .as_array()
+ .map(|arr| arr.iter().map(|v| v.as_f64().unwrap()).collect());
+ let expected_passthrough = case["passthrough"].as_bool().unwrap();
+ let expected_n = case["n"].as_u64().unwrap() as usize;
+
+ // shortlist_choice: the direct, label-only entry point.
+ let vectors_for_choice = vectors.clone();
+ let choice_embed = move |_texts: &[String]| -> Result<Vec<Vec<f64>>> {
+ match &vectors_for_choice {
+ Some(v) => Ok(v.clone()),
+ None => panic!("{label}: embed_fn should not run when k >= n (shortlist_choice)"),
+ }
+ };
+ let ins_opt = if instructions.is_null() {
+ None
+ } else {
+ Some(instructions)
+ };
+ let got_labels =
+ LayaShortlist::shortlist_choice(&state, criteria.clone(), choice_embed, k, ins_opt)
+ .unwrap_or_else(|e| panic!("{label}: shortlist_choice failed: {e}"));
+ assert_eq!(
+ got_labels, expected_labels,
+ "{label}: shortlist_choice labels"
+ );
+
+ // predict: the full ShortlistInfo (labels + scores + passthrough + n), reached by putting
+ // the same criteria/instructions on one choice question.
+ let question = Question::choice(instructions.clone(), criteria.clone()).unwrap();
+ let mut questions = QuestionSet::new();
+ questions = questions.with("q", question).unwrap();
+
+ let vectors_for_predict = vectors.clone();
+ let predict_embed = move |_texts: &[String]| -> Result<Vec<Vec<f64>>> {
+ match &vectors_for_predict {
+ Some(v) => Ok(v.clone()),
+ None => panic!("{label}: embed_fn should not run when k >= n (predict)"),
+ }
+ };
+ let result = LayaShortlist::predict(
+ &FirstLabelPredictor,
+ state.clone(),
+ &questions,
+ predict_embed,
+ k,
+ )
+ .unwrap_or_else(|e| panic!("{label}: predict failed: {e}"));
+ let info: &ShortlistInfo = &result.shortlist().unwrap()["q"];
+ assert_eq!(info.labels, expected_labels, "{label}: predict labels");
+ assert_eq!(
+ info.passthrough, expected_passthrough,
+ "{label}: passthrough"
+ );
+ assert_eq!(info.n, expected_n, "{label}: n");
+ assert_eq!(info.k, k, "{label}: k");
+ match (&info.scores, &expected_scores) {
+ (None, None) => {}
+ (Some(got), Some(want)) => {
+ assert_eq!(got.len(), want.len(), "{label}: scores length");
+ for (i, (g, w)) in got.iter().zip(want.iter()).enumerate() {
+ assert!(
+ (g - w).abs() < 1e-9,
+ "{label}: scores[{i}] got {g}, want {w}"
+ );
+ }
+ }
+ (got, want) => panic!("{label}: scores mismatch, got {got:?}, want {want:?}"),
+ }
+ }
+ eprintln!(
+ "shortlist_probe_matches_every_entry: checked {} cases",
+ cases.len()
+ );
+}
diff --git a/laya-rust/tests/shortlist_end_to_end_tests.rs b/laya-rust/tests/shortlist_end_to_end_tests.rs
new file mode 100644
index 00000000..020f3da4
--- /dev/null
+++ b/laya-rust/tests/shortlist_end_to_end_tests.rs
@@ -0,0 +1,117 @@
+//! Tier 2 (needs ONNX artifacts): the shortlist driving a real engine end to end, for all three
+//! checkpoints. Ports the `shortlist_many_options` golden case (`"kind": "shortlist"` in each
+//! checkpoint's `index.json`, kept out of the ordinary predict-parity cases by the `golden_data`
+//! fix described in the routing-port notes).
+//!
+//! Run the full suite with:
+//! `LAYA_ONNX_ROOT=<parent-of-checkpoints> cargo test --release -- --test-threads=1`
+
+mod common;
+
+use common::{engines, golden_data::CheckpointGoldenData, hashing_embedder, result_compare};
+use laya::laya_options::LayaCheckpoint;
+use laya::laya_shortlist::LayaShortlist;
+
+#[test]
+fn shortlist_predict_matches_the_recorded_result_for_every_checkpoint() {
+ let mut checked_any = false;
+
+ for checkpoint in engines::CHECKPOINTS {
+ let Some(engine) = engines::load(checkpoint) else {
+ continue;
+ };
+ let golden = CheckpointGoldenData::for_checkpoint(checkpoint.subdir());
+ if !golden.available() {
+ eprintln!("skip: no golden data for '{}'", checkpoint.subdir());
+ continue;
+ }
+
+ for case_info in golden.shortlist_cases() {
+ checked_any = true;
+ let root = golden.load_case(&case_info);
+ let where_case = format!("{}/{case_info}", checkpoint.subdir());
+
+ let state = root["state"].clone();
+ let full_questions = common::golden_data::build_questions(&root["questions"])
+ .unwrap_or_else(|e| panic!("{where_case}: build full questions: {e}"));
+ let k = root["shortlist_k"]
+ .as_u64()
+ .unwrap_or_else(|| panic!("{where_case}: missing 'shortlist_k'"))
+ as usize;
+
+ let result =
+ LayaShortlist::predict(&engine, state, &full_questions, hashing_embedder::embed, k)
+ .unwrap_or_else(|e| panic!("{where_case}: LayaShortlist::predict failed: {e}"));
+
+ // The kept, rank-ordered labels for "intent" must match Python's recorded reduction
+ // exactly (`reduced_questions.intent.criteria`'s key order).
+ let expected_labels: Vec<&str> = root["reduced_questions"]["intent"]["criteria"]
+ .as_object()
+ .unwrap_or_else(|| {
+ panic!("{where_case}: 'reduced_questions.intent.criteria' must be an object")
+ })
+ .keys()
+ .map(String::as_str)
+ .collect();
+ let shortlist_meta = result.shortlist().unwrap_or_else(|| {
+ panic!("{where_case}: LayaResult::shortlist() must be populated")
+ });
+ let intent_meta = shortlist_meta
+ .get("intent")
+ .unwrap_or_else(|| panic!("{where_case}: no shortlist metadata for 'intent'"));
+ assert_eq!(
+ intent_meta.labels, expected_labels,
+ "{where_case}: shortlisted labels vs reduced_questions"
+ );
+ assert!(
+ !intent_meta.passthrough,
+ "{where_case}: 40 options with k={k} must not pass through"
+ );
+ assert_eq!(intent_meta.n, 40, "{where_case}: option count");
+
+ // The answer itself (kept labels' probabilities, chosen answer, confidence) must
+ // match the recorded predict-on-the-reduced-question result.
+ result_compare::assert_result_matches(&where_case, &root["result"], &result);
+ }
+ }
+
+ if !checked_any {
+ eprintln!("skip: no shortlist golden data + artifacts found for any checkpoint");
+ }
+}
+
+/// Sanity check that every checkpoint's shortlist case actually reduced from 40 to 20 options
+/// (the number the plan's combined-sample section names), not some other pair that happens to
+/// still pass the assertions above.
+#[test]
+fn shortlist_many_options_case_shape_is_forty_reduced_to_twenty() {
+ let Some(engine) = engines::load(LayaCheckpoint::Multilingual) else {
+ eprintln!("skip: no multilingual artifacts");
+ return;
+ };
+ let golden = CheckpointGoldenData::for_checkpoint(LayaCheckpoint::Multilingual.subdir());
+ if !golden.available() {
+ eprintln!("skip: no golden data for 'multilingual'");
+ return;
+ }
+ let Some(case_info) = golden.shortlist_cases().into_iter().next() else {
+ eprintln!("skip: no shortlist case recorded");
+ return;
+ };
+ let root = golden.load_case(&case_info);
+ let full_questions =
+ common::golden_data::build_questions(&root["questions"]).expect("build questions");
+ assert_eq!(root["shortlist_k"].as_u64().unwrap(), 20);
+ let result = LayaShortlist::predict(
+ &engine,
+ root["state"].clone(),
+ &full_questions,
+ hashing_embedder::embed,
+ 20,
+ )
+ .expect("predict");
+ let meta = &result.shortlist().unwrap()["intent"];
+ assert_eq!(meta.n, 40);
+ assert_eq!(meta.k, 20);
+ assert_eq!(meta.labels.len(), 20);
+}
diff --git a/laya-rust/tools/hashing_embedder.py b/laya-rust/tools/hashing_embedder.py
new file mode 100644
index 00000000..0fec90d1
--- /dev/null
+++ b/laya-rust/tools/hashing_embedder.py
@@ -0,0 +1,101 @@
+"""Deterministic, dependency-free demo embedder for the shortlist golden vectors and samples.
+
+This is **not** a real embedding model: it is a hashed character-trigram bag-of-features
+vector, chosen only because it is a pure function of the input text and is trivial to
+reimplement bit-for-bit in another language with nothing but a string, UTF-8 encoding and
+32-bit integer arithmetic. It is deliberately named and documented as a demo: any real
+deployment should pass `embed_fn_from_agent` (Python) or a real bi-encoder to the shortlist,
+not this.
+
+Every other-language port MUST reproduce this exact algorithm, because
+`hashing_embedder_probe.json` and `case_shortlist_many_options.json` in the golden vectors
+were both generated with it, and the samples in every language use it so that their
+stdout can be diffed line for line.
+
+Algorithm (this docstring is the spec -- read it precisely before porting):
+
+1. Lowercase the input with a full Unicode case mapping (Python's `str.lower()`; other
+ languages must use their own full Unicode case-folding function on the *string*, NOT a
+ per-UTF-16-unit simple mapping. This matters for characters whose lowercase form is
+ more than one code point, e.g. `"İ".lower() == "i̇"` (Turkish capital I with dot
+ above lowercases to 2 code points: 'i' + combining dot above).
+2. Pad the lowercased text with two ASCII space characters (U+0020) on each side:
+ `" " + text + " "`. Padding is added even when `text` is empty, so the padded text is
+ never shorter than 4 code points and there are always at least 2 windows (see step 3):
+ this is what lets short inputs like `""` or `"a"` still produce a nonzero, well-defined
+ vector instead of an empty one.
+3. Slide a window of exactly 3 **Unicode code points** (not UTF-16 code units, not grapheme
+ clusters, not bytes) across the padded text, one code point at a time. For padded text of
+ length `n` code points there are `n - 2` windows, at start offsets `0, 1, ..., n - 3`
+ inclusive. A code point outside the Basic Multilingual Plane (an "astral" character, e.g.
+ an emoji) counts as exactly one code point in this count -- a port that iterates UTF-16
+ code units instead of Unicode scalar values will split a surrogate pair into two units and
+ get a different window count and different window contents; it must enumerate Unicode
+ scalar values instead. Walking grapheme clusters is NOT correct either -- it must be
+ rune/code-point enumeration (Rust's `char` already IS a Unicode scalar value, so iterating
+ `str::chars()` is correct there).
+4. Encode each 3-code-point window as UTF-8 bytes (the window's own bytes, not the whole
+ padded text's bytes -- re-encode the 3-code-point substring on its own each time).
+5. Hash those UTF-8 bytes with 32-bit FNV-1a:
+
+ hash = 0x811c9dc5 # FNV offset basis
+ for byte in utf8_bytes:
+ hash ^= byte
+ hash = (hash * 0x01000193) & 0xFFFFFFFF # FNV prime, wrapped to 32 bits
+
+ `hash` is an unsigned 32-bit integer throughout; the multiply must wrap (mask to
+ 0xFFFFFFFF / use unchecked/wrapping arithmetic in the target language) exactly like
+ Python's explicit `& 0xFFFFFFFF` above.
+6. `bucket = hash % 64` (0..63, unsigned).
+7. `sign = +1.0 if (hash >> 31) & 1 == 0 else -1.0` (the hash's top bit, as an unsigned
+ 32-bit value -- this is a sign-hashing trick to reduce feature collisions, not a
+ statement about the hash being negative).
+8. Accumulate `sign` into `vector[bucket]` for every window (i.e. `vector[bucket] += sign`;
+ multiple windows can and do land in the same bucket and their signed contributions add).
+ The accumulator is float64 throughout.
+9. No normalisation is applied. The shortlist's cosine similarity handles scale on its own,
+ and leaving the raw counts in the golden vectors makes them easier to hand-check.
+
+`embed(texts)` returns one such vector per input string, as a `(len(texts), 64)` float64
+NumPy array, in input order. It does not batch, cache, or special-case any input; every text
+is processed independently and only via the steps above.
+"""
+from typing import Sequence
+
+import numpy as np
+
+DIM = 64
+_FNV_OFFSET_BASIS = 0x811C9DC5
+_FNV_PRIME = 0x01000193
+_MASK32 = 0xFFFFFFFF
+
+
+def _fnv1a_32(data: bytes) -> int:
+ """32-bit FNV-1a over raw bytes, wrapping like an unsigned 32-bit integer."""
+ h = _FNV_OFFSET_BASIS
+ for b in data:
+ h ^= b
+ h = (h * _FNV_PRIME) & _MASK32
+ return h
+
+
+def _embed_one(text: str) -> np.ndarray:
+ vec = np.zeros(DIM, dtype=np.float64)
+ padded = " " + text.lower() + " "
+ n = len(padded)
+ for i in range(n - 2):
+ window = padded[i : i + 3]
+ h = _fnv1a_32(window.encode("utf-8"))
+ bucket = h % DIM
+ sign = 1.0 if (h >> 31) & 1 == 0 else -1.0
+ vec[bucket] += sign
+ return vec
+
+
+def embed(texts: Sequence[str]) -> np.ndarray:
+ """Embed a list of strings into a `(len(texts), 64)` float64 array. See the module docstring
+ for the exact algorithm; every other-language port must match it bit-for-bit."""
+ rows = [_embed_one("" if t is None else str(t)) for t in texts]
+ if not rows:
+ return np.zeros((0, DIM), dtype=np.float64)
+ return np.stack(rows, axis=0)