Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(rust): match Python result and truncation error schemas

kevin-gatimu284e170a0e1e
diff --git a/laya-rust/CHANGELOG.md b/laya-rust/CHANGELOG.md
index e1edbaa6..4496ba77 100644
--- a/laya-rust/CHANGELOG.md
+++ b/laya-rust/CHANGELOG.md
@@ -6,12 +6,18 @@ All notable changes to the `laya-onnx` crate are documented here. The format fol
## [0.1.0] - unreleased
-First release: a port of the Laya Python SDK, tested against laya (Python) 0.3.21.
+First release: a port of the Laya Python SDK, checked against regenerated laya (Python) 0.4.0
+goldens for the supported paths (see the known gaps below).
### Added
- Advisory parity CI on every push and PR: regenerate fixtures from the current Python checkout,
test all checkpoints, and reject skipped tests. No golden cache or crate publication.
+- `LayaResult::state_usage()` reports state-token truncation across question budgets, matching
+ Python 0.4.0 without changing the existing `Usage` totals or result constructor.
+- Engine score legends render structured criteria as strings, matching Python 0.4.0.
+- `NoMarkers` errors include actual/requested marker counts and the total sequence budget,
+ matching Python's actionable truncation diagnostic.
- `LayaEngine`: `choice`, `score` and `noul` predictions over the exported ONNX checkpoints
(english, multilingual, typed-decisions); all questions for a state run in one batched forward pass.
- Loads both ONNX layouts: the fused `model.onnx` and the split `encoder.onnx` + `head.onnx`.
@@ -26,5 +32,5 @@ First release: a port of the Laya Python SDK, tested against laya (Python) 0.3.2
- Routing, script counting and fused-line email disclaimers follow the regenerated Python 0.4.0
probes. New mixed-field language heuristics are not yet ported.
-- Not yet ported from Python 0.3.21: abstention (`min_confidence`), `predict_long` and
- `lang_temperatures`.
+- Not yet ported: abstention (`min_confidence`, including per-bucket thresholds),
+ histogram-binning calibration, `predict_long` and `lang_temperatures`.
diff --git a/laya-rust/README.md b/laya-rust/README.md
index 1298ca33..1845f1a9 100644
--- a/laya-rust/README.md
+++ b/laya-rust/README.md
@@ -558,6 +558,10 @@ Goldens are never cached. A `skip:` diagnostic or ignored test makes the lane re
Cargo counts a missing-model early return as a pass. Job-level `continue-on-error` keeps Python
improvements unblocked; Rust drift needs a follow-up port. This workflow never publishes crates.
+Engine results expose Python 0.4.0's state-budget diagnostics via `LayaResult::state_usage()`:
+original state tokens, worst-case dropped tokens, whether truncation occurred, and the affected
+question ids. The existing `Usage` totals and result constructor remain source-compatible.
+
To reproduce it, install CPU torch and [`tools/requirements-regen.txt`](tools/requirements-regen.txt),
then run from the repository root:
@@ -651,6 +655,7 @@ Rust port groups the corresponding functions into a unit struct with associated
and checked against freshly regenerated probes. New mixed-field language heuristics remain
a follow-up; these probes are not a claim of full coverage of every new Python feature.
-- **Opt-in features added in Python 0.3.21 are not ported yet:** abstention (`min_confidence`),
- `predict_long`, and per-language temperatures (`lang_temperatures`). They are off by default in
+- **Opt-in features are not ported yet:** abstention (`min_confidence`, including per-bucket
+ thresholds), histogram-binning calibration, `predict_long`, and per-language temperatures
+ (`lang_temperatures`). They are off by default in
Python, so default predictions are unaffected. They are planned as follow-ups.
diff --git a/laya-rust/src/answers.rs b/laya-rust/src/answers.rs
index bc909276..bb483c67 100644
--- a/laya-rust/src/answers.rs
+++ b/laya-rust/src/answers.rs
@@ -67,7 +67,8 @@ pub struct ScoreAnswer {
/// Expected level: the probability-weighted mean of the level indices, a fractional value
/// between 0 and `legend.len() - 1`. Rounded to 4 decimals.
pub score: f64,
- /// What each level index means, as supplied on the question.
+ /// What each level index means. Engine-produced legends render each criterion as a string,
+ /// matching Python 0.4.0, including structured dict/list levels.
pub legend: Vec<Value>,
/// Probability per level index. Rounded to 4 decimals.
pub probabilities: Vec<f64>,
@@ -220,6 +221,19 @@ impl Answer {
}
}
+/// State-budget diagnostics reported by Python's `usage` since 0.4.0.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct StateUsage {
+ /// Tokens in the untruncated state, counted once rather than once per question.
+ pub state_tokens: usize,
+ /// The largest state-token loss across all question heads.
+ pub state_tokens_dropped: usize,
+ /// Whether at least one question lost state tokens.
+ pub truncated: bool,
+ /// Ids of questions that lost state tokens, in question order.
+ pub truncated_questions: Vec<String>,
+}
+
/// The result of one `predict` call: one answer per question, plus token usage.
#[derive(Debug, Clone)]
pub struct LayaResult {
@@ -229,6 +243,7 @@ pub struct LayaResult {
by_id: HashMap<String, Answer>,
routing: Option<RouteDecision>,
shortlist: Option<IndexMap<String, ShortlistInfo>>,
+ state_usage: Option<StateUsage>,
}
impl LayaResult {
@@ -250,6 +265,7 @@ impl LayaResult {
by_id,
routing: None,
shortlist: None,
+ state_usage: None,
}
}
@@ -290,6 +306,17 @@ impl LayaResult {
self.usage
}
+ /// State-budget diagnostics for engine-produced answers. Manually constructed results
+ /// retain `None`, preserving the existing constructor and the `Copy` token-total `Usage`.
+ pub fn state_usage(&self) -> Option<&StateUsage> {
+ self.state_usage.as_ref()
+ }
+
+ pub(crate) fn with_state_usage(mut self, usage: StateUsage) -> Self {
+ self.state_usage = Some(usage);
+ self
+ }
+
/// The question ids, in the order they were asked.
pub fn ids(&self) -> &[String] {
&self.order
diff --git a/laya-rust/src/error.rs b/laya-rust/src/error.rs
index 9b99f2c2..3f306504 100644
--- a/laya-rust/src/error.rs
+++ b/laya-rust/src/error.rs
@@ -84,11 +84,44 @@ pub enum LayaError {
/// Ports the "silently answering a truncated question" failure mode called out in
/// `sequence-construction.md`, and matches Python's wording exactly (parity tests compare
/// this text against golden `error_message` values byte-for-byte).
- #[error("question '{id}' options exceed head_max_len={head_max_len}")]
+ #[error(
+ "question {}: only {actual_markers} of its {expected_markers} option markers fit in \
+ max_len={max_len} with head_max_len={head_max_len} spent on the question; \
+ lower head_max_len, raise max_len, or use fewer options",
+ crate::laya_router::python_repr_str(.id)
+ )]
NoMarkers {
/// The question id that lost one or more markers.
id: String,
/// The configured head budget the question's options did not fit within.
head_max_len: usize,
+ /// Markers that survived the total sequence budget.
+ actual_markers: usize,
+ /// Markers requested by the question.
+ expected_markers: usize,
+ /// The configured total sequence budget.
+ max_len: usize,
},
}
+
+#[cfg(test)]
+mod tests {
+ use super::LayaError;
+
+ #[test]
+ fn marker_truncation_error_matches_python_and_quotes_question_ids() {
+ let error = LayaError::NoMarkers {
+ id: "user's question".to_string(),
+ head_max_len: 192,
+ actual_markers: 125,
+ expected_markers: 300,
+ max_len: 512,
+ };
+ assert_eq!(
+ error.to_string(),
+ "question \"user's question\": only 125 of its 300 option markers fit in max_len=512 \
+ with head_max_len=192 spent on the question; lower head_max_len, raise max_len, or \
+ use fewer options"
+ );
+ }
+}
diff --git a/laya-rust/src/laya_engine.rs b/laya-rust/src/laya_engine.rs
index e90d68b1..0bb14304 100644
--- a/laya-rust/src/laya_engine.rs
+++ b/laya-rust/src/laya_engine.rs
@@ -13,7 +13,7 @@
//! Tokenization needs no such lock: `tokenizers::Tokenizer::encode` takes `&self`.
use crate::answers::{
- ActionInfo, Answer, ChoiceAnswer, LayaResult, NoulAnswer, ScoreAnswer, Usage,
+ ActionInfo, Answer, ChoiceAnswer, LayaResult, NoulAnswer, ScoreAnswer, StateUsage, Usage,
};
use crate::calibration::Calibration;
use crate::collator::{CollatedBatch, Collator, SequenceItem};
@@ -175,6 +175,12 @@ impl LayaEngine {
let ids: Vec<String> = questions.ids().map(str::to_string).collect();
let mut items = Vec::with_capacity(ids.len());
+ let mut state_usage = StateUsage {
+ state_tokens: 0,
+ state_tokens_dropped: 0,
+ truncated: false,
+ truncated_questions: Vec::new(),
+ };
// A conversation-turn list is serialized newest-last (see `PythonJson::state`), so the
// default right-truncation of an over-long state would silently drop the newest turn.
@@ -186,7 +192,7 @@ impl LayaEngine {
// The tokenizer here needs no lock: tokenizers::Tokenizer::encode takes &self.
for id in &ids {
let question = &questions[id.as_str()];
- let (seq, markers) = SequenceBuilder::build(
+ let (seq, markers, stats) = SequenceBuilder::build_with_stats(
self.tokenizer.as_ref(),
&state,
question,
@@ -194,13 +200,23 @@ impl LayaEngine {
self.config.head_max_len,
truncate_left,
)?;
+ state_usage.state_tokens = stats.tokens;
+ state_usage.state_tokens_dropped = state_usage.state_tokens_dropped.max(stats.dropped);
+ if stats.dropped > 0 {
+ state_usage.truncated = true;
+ state_usage.truncated_questions.push(id.clone());
+ }
// A marker dropped for landing past max_len means an option is not in the sequence at
// all, so its logit would score whatever token happens to sit at position 0.
- if markers.len() != SequenceBuilder::render_options(question).len() {
+ let expected_markers = SequenceBuilder::render_options(question).len();
+ if markers.len() != expected_markers {
return Err(LayaError::NoMarkers {
id: id.clone(),
head_max_len: self.config.head_max_len,
+ actual_markers: markers.len(),
+ expected_markers,
+ max_len: self.config.max_len,
});
}
@@ -241,7 +257,8 @@ impl LayaEngine {
input_tokens: batch.input_tokens as u32,
output_tokens: 0,
},
- ))
+ )
+ .with_state_usage(state_usage))
}
/// One forward pass over a collated batch, returning the raw graph outputs. See the
@@ -526,7 +543,9 @@ fn build_answer(question: &Question, p: &[f64], action: ActionInfo) -> Answer {
let expected = Calibration::round4(Calibration::expected_score(p));
Answer::Score(ScoreAnswer::new(
expected,
- score.levels().to_vec(),
+ score.levels().iter()
+ .map(|level| Value::String(crate::python_json::PythonJson::criterion(level)))
+ .collect(),
probabilities,
confidence,
answer_confidence,
@@ -661,6 +680,28 @@ mod tests {
use super::*;
use serde_json::json;
+ #[test]
+ fn score_legends_render_structured_levels_as_strings() {
+ let question = Question::score(
+ "severity",
+ [json!({"label": "info", "cvss": 0.0}), json!({"label": "high", "cvss": 7.5})],
+ ).unwrap();
+ let Answer::Score(answer) = build_answer(
+ &question,
+ &[0.25, 0.75],
+ ActionInfo { act_probability: 1.0 },
+ ) else {
+ panic!("score question must produce a score answer");
+ };
+ assert_eq!(
+ answer.legend,
+ vec![
+ json!("{\"label\": \"info\", \"cvss\": 0.0}"),
+ json!("{\"label\": \"high\", \"cvss\": 7.5}"),
+ ],
+ );
+ }
+
/// A tiny two-row batch, just enough shape/data to exercise `value_for_input` without a real
/// ONNX session.
fn tiny_batch() -> CollatedBatch {
diff --git a/laya-rust/src/laya_router.rs b/laya-rust/src/laya_router.rs
index ad3ab50b..edd1c1c6 100644
--- a/laya-rust/src/laya_router.rs
+++ b/laya-rust/src/laya_router.rs
@@ -569,7 +569,7 @@ impl LayaPredictor for LayaRouter {
/// the common control characters are escaped. Every model/task/lang name recorded in the routing
/// golden vectors is plain ASCII, so this covers the cases that matter without attempting a full
/// `repr()` (which would also need to classify every Unicode character as printable or not).
-fn python_repr_str(s: &str) -> String {
+pub(crate) fn python_repr_str(s: &str) -> String {
let quote = if s.contains('\'') && !s.contains('"') {
'"'
} else {
diff --git a/laya-rust/src/sequence_builder.rs b/laya-rust/src/sequence_builder.rs
index 9f1e8bc7..38519fd2 100644
--- a/laya-rust/src/sequence_builder.rs
+++ b/laya-rust/src/sequence_builder.rs
@@ -16,6 +16,12 @@ use serde_json::Value;
/// Namespace for sequence construction. See the module docs for the source this ports.
pub struct SequenceBuilder;
+#[derive(Debug, Clone, Copy)]
+pub(crate) struct StateTokenStats {
+ pub tokens: usize,
+ pub dropped: usize,
+}
+
impl SequenceBuilder {
/// One option fragment is capped at this many tokens, before the marker is prepended.
const MAX_OPTION_TOKENS: usize = 48;
@@ -125,6 +131,18 @@ impl SequenceBuilder {
head_max_len: usize,
truncate_left: bool,
) -> Result<(Vec<u32>, Vec<usize>)> {
+ Self::build_with_stats(tokenizer, state, question, max_len, head_max_len, truncate_left)
+ .map(|(ids, markers, _)| (ids, markers))
+ }
+
+ pub(crate) fn build_with_stats(
+ tokenizer: &dyn LayaTokenizer,
+ state: &Value,
+ question: &Question,
+ max_len: usize,
+ head_max_len: usize,
+ truncate_left: bool,
+ ) -> Result<(Vec<u32>, Vec<usize>, StateTokenStats)> {
let special = tokenizer.special();
let mask_token = special.mask_token.as_str();
@@ -201,6 +219,10 @@ impl SequenceBuilder {
}
markers.retain(|&m| m < max_len);
- Ok((ids, markers))
+ let stats = StateTokenStats {
+ tokens: state_ids.len(),
+ dropped: state_ids.len() - sliced.len(),
+ };
+ Ok((ids, markers, stats))
}
}
diff --git a/laya-rust/tests/common/result_compare.rs b/laya-rust/tests/common/result_compare.rs
index e320fc68..9be5925a 100644
--- a/laya-rust/tests/common/result_compare.rs
+++ b/laya-rust/tests/common/result_compare.rs
@@ -5,7 +5,6 @@
use laya::LayaResult;
use laya::answers::{Answer, ChoiceAnswer, NoulAnswer, ScoreAnswer};
-use laya::python_json::PythonJson;
use laya::sequence_builder::SequenceBuilder;
use serde_json::Value;
@@ -26,6 +25,7 @@ fn assert_close(expected: f64, actual: f64, tolerance: f64, what: &str) {
/// action/choice/score) matches the recorded golden `expected` (a case's `"result"` field) within
/// the standard parity tolerances. `where_case` prefixes every assertion message.
pub fn assert_result_matches(where_case: &str, expected: &Value, result: &LayaResult) {
+ assert_state_usage_matches(where_case, expected, result);
assert_eq!(
expected["model"].as_str().unwrap(),
result.model(),
@@ -36,6 +36,11 @@ pub fn assert_result_matches(where_case: &str, expected: &Value, result: &LayaRe
result.usage().input_tokens,
"{where_case}: usage.input_tokens"
);
+ assert_eq!(
+ expected["usage"]["output_tokens"].as_u64().unwrap() as u32,
+ result.usage().output_tokens,
+ "{where_case}: usage.output_tokens"
+ );
let answers = expected["answers"].as_object().unwrap();
let expected_ids: Vec<&str> = answers.keys().map(String::as_str).collect();
@@ -62,6 +67,12 @@ pub fn assert_result_matches(where_case: &str, expected: &Value, result: &LayaRe
PROB_TOLERANCE,
&format!("{where_q}.confidence"),
);
+ assert_close(
+ want["answer_confidence"].as_f64().unwrap(),
+ got.answer_confidence(),
+ PROB_TOLERANCE,
+ &format!("{where_q}.answer_confidence"),
+ );
assert_close(
want["action"]["act_probability"].as_f64().unwrap(),
got.action().act_probability,
@@ -77,6 +88,29 @@ pub fn assert_result_matches(where_case: &str, expected: &Value, result: &LayaRe
}
}
+pub fn assert_state_usage_matches(where_case: &str, expected: &Value, result: &LayaResult) {
+ let usage = result.state_usage().expect("engine result must report state usage");
+ let want = &expected["usage"];
+ assert_eq!(want.as_object().unwrap().len(), 6,
+ "{where_case}: new Python usage fields need a Rust port");
+ assert_eq!(
+ want["state_tokens"].as_u64().unwrap() as usize, usage.state_tokens,
+ "{where_case}: usage.state_tokens"
+ );
+ assert_eq!(
+ want["state_tokens_dropped"].as_u64().unwrap() as usize, usage.state_tokens_dropped,
+ "{where_case}: usage.state_tokens_dropped"
+ );
+ assert_eq!(
+ want["truncated"].as_bool().unwrap(), usage.truncated,
+ "{where_case}: usage.truncated"
+ );
+ let ids: Vec<&str> = want["truncated_questions"].as_array().unwrap()
+ .iter().map(|id| id.as_str().unwrap()).collect();
+ let got: Vec<&str> = usage.truncated_questions.iter().map(String::as_str).collect();
+ assert_eq!(ids, got, "{where_case}: usage.truncated_questions");
+}
+
fn assert_choice(want: &Value, got: &ChoiceAnswer, where_q: &str) {
let probabilities = want["probabilities"].as_object().unwrap();
let expected_labels: Vec<&str> = probabilities.keys().map(String::as_str).collect();
@@ -106,8 +140,8 @@ fn assert_score(want: &Value, got: &ScoreAnswer, where_q: &str) {
for i in 0..got.legend.len() {
let expected_level = &legend[&i.to_string()];
assert_eq!(
- PythonJson::criterion(expected_level),
- PythonJson::criterion(&got.legend[i]),
+ expected_level,
+ &got.legend[i],
"{where_q}: legend[{i}]"
);
}
diff --git a/laya-rust/tests/predict_parity_tests.rs b/laya-rust/tests/predict_parity_tests.rs
index f4466875..36cac94a 100644
--- a/laya-rust/tests/predict_parity_tests.rs
+++ b/laya-rust/tests/predict_parity_tests.rs
@@ -18,7 +18,6 @@ use common::golden_data::{self, CheckpointGoldenData};
use laya::answers::{Answer, ChoiceAnswer, NoulAnswer, ScoreAnswer};
use laya::error::LayaError;
use laya::laya_options::LayaCheckpoint;
-use laya::python_json::PythonJson;
use laya::questions::QuestionSet;
use laya::sequence_builder::SequenceBuilder;
use serde_json::Value;
@@ -74,6 +73,7 @@ fn reproduces_the_recorded_answers() {
.unwrap_or_else(|e| panic!("{}/{case}: predict failed: {e}", checkpoint.subdir()));
let expected = &root["result"];
let where_case = format!("{}/{case}", checkpoint.subdir());
+ common::result_compare::assert_state_usage_matches(&where_case, expected, &result);
assert_eq!(
expected["model"].as_str().unwrap(),
@@ -119,6 +119,12 @@ fn reproduces_the_recorded_answers() {
PROB_TOLERANCE,
&format!("{where_q}.confidence"),
);
+ assert_close(
+ want["answer_confidence"].as_f64().unwrap(),
+ got.answer_confidence(),
+ PROB_TOLERANCE,
+ &format!("{where_q}.answer_confidence"),
+ );
assert_close(
want["action"]["act_probability"].as_f64().unwrap(),
got.action().act_probability,
@@ -176,6 +182,7 @@ fn reproduces_the_recorded_answers_for_the_split_layout() {
.unwrap_or_else(|e| panic!("{}/{case}: predict failed: {e}", checkpoint.subdir()));
let expected = &root["result"];
let where_case = format!("{}/{case} (split layout)", checkpoint.subdir());
+ common::result_compare::assert_state_usage_matches(&where_case, expected, &result);
assert_eq!(
expected["model"].as_str().unwrap(),
@@ -204,6 +211,12 @@ fn reproduces_the_recorded_answers_for_the_split_layout() {
&format!("{where_q}.confidence"),
);
+ assert_close(
+ want["answer_confidence"].as_f64().unwrap(),
+ got.answer_confidence(),
+ PROB_TOLERANCE,
+ &format!("{where_q}.answer_confidence"),
+ );
match got {
Answer::Choice(choice) => assert_choice(want, choice, &where_q),
Answer::Score(score) => assert_score(want, score, &where_q),
@@ -428,8 +441,8 @@ fn assert_score(want: &Value, got: &ScoreAnswer, where_q: &str) {
for i in 0..got.legend.len() {
let expected_level = &legend[&i.to_string()];
assert_eq!(
- PythonJson::criterion(expected_level),
- PythonJson::criterion(&got.legend[i]),
+ expected_level,
+ &got.legend[i],
"{where_q}: legend[{i}]"
);
}