Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(rust): port the min_confidence abstention gate and jev_confidence
kevin-gatimu846d65228d2f
diff --git a/laya-rust/CHANGELOG.md b/laya-rust/CHANGELOG.mdindex 2495da09..06982bf3 100644--- a/laya-rust/CHANGELOG.md+++ b/laya-rust/CHANGELOG.md@@ -11,6 +11,21 @@ regenerated from laya (Python) 0.4.14 for the supported paths (see the known gap### Added+- The abstention gate, ported from `laya/confidence.py`: `PredictOptions::min_confidence` with+ `LayaEngine::predict_with_options` and `LayaRouter::predict_with_options`, and+ `BatchOptions::min_confidence` for `predict_batch` on the engine and the router (one threshold+ for the whole batch, applied to every result and no part of the router's grouping).+ `MinConfidence` is a scalar or a per-bucket map, built with `MinConfidence::scalar` and+ `MinConfidence::map`, which refuse what `check_min_confidence` refuses with Python's `ValueError`+ text (`LayaError::InvalidMinConfidence`); the router validates before it routes or loads, and the+ engine before it looks at the states. With a threshold set, `Answer::{low_confidence, abstention,+ abstention_threshold}` report the gate (`GATE_STATES`: passed, abstained, unevaluated); without+ one they report nothing. `Answer::jev_confidence` is Python's `jev_confidence` and+ `bucket_keys()` its `bucket_keys`. `predict_long` takes no threshold, as in Python. Checked+ against `routing/confidence_probe.json`, which Python's own `confidence.py` writes. Two API+ notes: `BatchOptions` no longer derives `Eq` (it now holds floats), and `LayaError` is not+ `#[non_exhaustive]`, so an exhaustive `match` on it outside the crate needs an arm for+ `InvalidMinConfidence`.- `LangGuess` and `LayaRouter::{route_with_lang_guess, predict_with_lang_guess,predict_long_with_lang_guess, with_lang_guess}`: Python's `lang_guess` hint (a language code or acallback taking the state), per call or installed on the router, checked after an explicit `lang`@@ -26,7 +41,7 @@ regenerated from laya (Python) 0.4.14 for the supported paths (see the known gaplonger than one sequence in overlapping token windows and keep the most confident window perquestion, with `Answer::window()` attribution, `LayaResult::scan_usage()` and summed`state_usage()`. Matches Python's `ONNXAgent.predict_long` and the .NET SDK; hooks, per-language- temperatures and the confidence gate are not part of it. New `LongOptions`, `LayaLongPredictor`,+ temperatures are not part of it, and, as in Python, it takes no `min_confidence`. New `LongOptions`, `LayaLongPredictor`,`LayaError::{UnsupportedAggregate, InvalidWindow, NotSupported}` and a default`LayaTokenizer::decode`. `LayaError` is not `#[non_exhaustive]`, so an exhaustive `match` on itoutside the crate needs arms for the three new variants.@@ -124,5 +139,4 @@ regenerated from laya (Python) 0.4.14 for the supported paths (see the known gap- Language detection, routing and email cleaning follow the regenerated Python 0.4.14 probes; alater Python change needs the probes regenerated.-- Not yet ported: abstention (`min_confidence`, including per-bucket thresholds),- histogram-binning calibration and `lang_temperatures`.+- Not yet ported: histogram-binning calibration and `lang_temperatures`.diff --git a/laya-rust/README.md b/laya-rust/README.mdindex 4ccaa001..91f90c1a 100644--- a/laya-rust/README.md+++ b/laya-rust/README.md@@ -34,6 +34,8 @@ Mirrors the [Laya Python SDK](https://github.com/NandhaKishorM/laya/blob/main/RE- [Built-in presets](#built-in-presets)- [Reading results](#reading-results)- [Long prediction](#long-prediction)+- [Batch prediction](#batch-prediction)+- [Abstention (`min_confidence`)](#abstention-min_confidence)- [Thread safety](#thread-safety)- [Execution providers and cargo features](#execution-providers-and-cargo-features)- [Async usage](#async-usage)@@ -488,8 +490,8 @@ if let Some(window) = answer.window() { // None for a single-pass statestride are sent, with Python's exact text, to the [`log`](https://docs.rs/log) facade at `warn`level with target `laya`. Install any `log` backend (for example `env_logger`) to see them; withnone installed they are dropped.-- **Not ported.** Per-call hooks, per-language temperatures and the confidence gate are not part of- `predict_long` here.+- **Not ported.** Per-call hooks and per-language temperatures are not part of `predict_long`+ here. As in Python, it takes no `min_confidence`.---@@ -524,12 +526,59 @@ let results = engine.predict_batch(&states, &questions, &options)?; // resultseach with its `routing`. `LayaRouter::route_batch(&requests)` returns only the decisions and loadsnothing. A predictor without a batch capability (see `LayaPredictor::as_batch_predictor`) gives`LayaError::NotSupported`.-- **Not ported.** Per-call hooks, `min_confidence`, `lang`, per-request `max_len` / `head_max_len`- and the router's `diagnostics` report. `BatchOptions` and `BatchRequest` are `#[non_exhaustive]`+- **`min_confidence`.** `BatchOptions::min_confidence` gates every result of the call; see+ [Abstention](#abstention-min_confidence).+- **Not ported.** Per-call hooks, `lang`, per-request `max_len` / `head_max_len` and the router's+ `diagnostics` report. `BatchOptions` and `BatchRequest` are `#[non_exhaustive]`so those can be added without breaking callers.---+## Abstention (`min_confidence`)++An opt-in gate on `answer_confidence` (`max(p)`, the quantity the calibration figures describe). It+ports `min_confidence` from Python's `system_one`, `predict_batch` and `Router.predict`. A threshold+is a `MinConfidence`: one number for every answer, or a map from option-count bucket to threshold,+because one level does not transfer across option counts.++```rust,no_run+# use laya::{LayaEngine, MinConfidence, PredictOptions, QuestionSet};+# let engine = LayaEngine::from_directory("x")?; let questions = QuestionSet::new();+let gate = MinConfidence::map([("choice:2", 0.8), ("choice:3-5", 0.6), ("default", 0.5)])?;+let options = PredictOptions::new().min_confidence(gate);+let result = engine.predict_with_options("I want a refund", &questions, &options)?;+for (_id, answer) in result.iter() {+ // "passed", "abstained" or "unevaluated"; None when no threshold was set.+ println!("{:?} below {:?}: {}", answer.abstention(), answer.abstention_threshold(), answer.low_confidence());+}+# Ok::<(), laya::LayaError>(())+```++- **Buckets.** Keys are `choice`, `score` or `noul` plus `:2`, `:3-5`, `:6-10` or `:11+`+ (`laya::bucket_keys()`), and `"default"` for any bucket the map does not name. A bucket the map+ does not name, with no `"default"`, is gated at `0.0`, which gates nothing.+- **Verdict.** With a threshold set every answer reports `abstention` (`laya::GATE_STATES`) and+ `abstention_threshold`; `low_confidence()` is true only for `"abstained"`. An answer with no+ usable confidence is `"unevaluated"`, not a pass. A threshold of `0.0` was set, so it still+ reports states. With no threshold nothing is reported (`abstention()` is `None`).+- **Validation.** `MinConfidence::scalar` and `MinConfidence::map` refuse what Python's+ `check_min_confidence` refuses (out of `[0.0, 1.0]`, NaN, an empty map, an unknown bucket key)+ with Python's `ValueError` text as `LayaError::InvalidMinConfidence`. The variants are public, so+ the engine and the router validate again where Python does: the router before it routes or loads+ anything, the engine before it looks at the states.+- **Router.** `LayaRouter::predict_with_options` and `BatchOptions::min_confidence` on+ `LayaRouter::predict_batch` (one threshold for the whole batch, not part of the grouping). The+ router applies the gate itself, so an attached predictor needs no support for it.+- **`predict_long`.** Takes no threshold, as in Python.+- **`Answer::jev_confidence()`.** The `confidence` Jev would report for the answer's probabilities+ (choice and score only; `None` for noul or malformed probabilities). It is a different formula+ from Laya's normalized entropy, so a threshold carried over from Jev selects about twice as much+ on Laya.+- **Python 3.12 arithmetic.** `jev_confidence` sums with Python's compensated `sum()` (CPython+ 3.12 and later), so its last digits match a 3.12+ Python, not 3.11 or earlier.++---+## Routing, language detection, email and shortlistRunnable end-to-end example: [`samples/laya-routing`](https://github.com/NandhaKishorM/laya/tree/main/laya-rust/samples/laya-routing).@@ -742,6 +791,8 @@ invocation that has the `download` feature enabled (the default) with no externa| `src/laya_engine.rs` | Loads the ONNX model and answers questions | `LayaEngine` || `src/long_prediction.rs` | Window plan and aggregation for `predict_long` | `LongOptions` || `src/batch_prediction.rs` | Chunking, sort windows and order restoration for `predict_batch` | `BatchOptions` |+| `src/confidence.rs` | The abstention gate and Jev's `confidence` | `MinConfidence`, `GATE_STATES`, `bucket_keys` |+| `src/predict_options.rs` | Per-call options for `predict_with_options` | `PredictOptions` || `src/error.rs` | Error type | `LayaError`, `Result<T>` — every fallible call in this crate returns `Result<T, LayaError>` |Where the equivalent Python code is a module of related functions (`common.py`, `agent.py`), the@@ -803,8 +854,7 @@ Rust port groups the corresponding functions into a unit struct with associatedrouting probes and a seeded differential corpus regenerated at that version(`tools/regen_golden.py --checkpoint routing`). A later Python change needs them regenerated.-- **Opt-in features are not ported yet:** abstention (`min_confidence`, including per-bucket- thresholds), histogram-binning calibration, and per-language temperatures- (`lang_temperatures`). `predict_long` and `predict_batch` are ported, without hooks,- per-language temperatures or the confidence gate. They are off by default in+- **Opt-in features are not ported yet:** histogram-binning calibration and per-language+ temperatures (`lang_temperatures`). `predict_long`, `predict_batch` and the `min_confidence`+ gate are ported, without hooks or per-language temperatures. They are off by default inPython, so default predictions are unaffected. They are planned as follow-ups.diff --git a/laya-rust/src/answers.rs b/laya-rust/src/answers.rsindex cf182121..9bcdc657 100644--- a/laya-rust/src/answers.rs+++ b/laya-rust/src/answers.rs@@ -1,6 +1,7 @@//! The typed result of one `predict` call. Pure data — nothing here runs the model; `engine.rs`//! (a later phase) is what builds these from calibrated probabilities.+use crate::confidence::{self, Gate};use crate::laya_router::RouteDecision;use crate::laya_shortlist::ShortlistInfo;use crate::questions::QuestionType;@@ -50,6 +51,7 @@ pub struct ChoiceAnswer {answer_confidence: f64,action: ActionInfo,window: Option<WindowInfo>,+ gate: Gate,}impl ChoiceAnswer {@@ -68,6 +70,7 @@ impl ChoiceAnswer {answer_confidence,action,window: None,+ gate: Gate::default(),}}@@ -92,6 +95,7 @@ pub struct ScoreAnswer {answer_confidence: f64,action: ActionInfo,window: Option<WindowInfo>,+ gate: Gate,}impl ScoreAnswer {@@ -112,6 +116,7 @@ impl ScoreAnswer {answer_confidence,action,window: None,+ gate: Gate::default(),}}@@ -137,6 +142,7 @@ pub struct NoulAnswer {answer_confidence: f64,action: ActionInfo,window: Option<WindowInfo>,+ gate: Gate,}impl NoulAnswer {@@ -153,6 +159,7 @@ impl NoulAnswer {answer_confidence,action,window: None,+ gate: Gate::default(),}}@@ -227,6 +234,59 @@ impl Answer {self}+ pub(crate) fn set_gate(&mut self, gate: Gate) {+ match self {+ Answer::Choice(c) => c.gate = gate,+ Answer::Score(s) => s.gate = gate,+ Answer::Noul(n) => n.gate = gate,+ }+ }++ fn gate(&self) -> Gate {+ match self {+ Answer::Choice(c) => c.gate,+ Answer::Score(s) => s.gate,+ Answer::Noul(n) => n.gate,+ }+ }++ /// Whether the `min_confidence` gate flagged this answer: its confidence (`answer_confidence`,+ /// else `confidence`) fell below the threshold. `false` when no gate ran, and `false` for an+ /// answer with no usable confidence (that one reads `"unevaluated"`). Ports Python's+ /// `low_confidence: True`, which is present only when flagged.+ pub fn low_confidence(&self) -> bool {+ self.gate().low_confidence+ }++ /// The gate's state for this answer: one of [`crate::confidence::GATE_STATES`]+ /// (`"passed"`, `"abstained"`, `"unevaluated"`), or `None` when no `min_confidence` was set+ /// for the call that produced it. Ports Python's `abstention`.+ pub fn abstention(&self) -> Option<&'static str> {+ self.gate().abstention+ }++ /// The threshold this answer was gated at: the scalar, or the map entry for its option-count+ /// bucket (else the map's `"default"`, else `0.0`). `None` when no gate ran. Ports Python's+ /// `abstention_threshold`.+ pub fn abstention_threshold(&self) -> Option<f64> {+ self.gate().threshold+ }++ /// The `confidence` Jev would report for this answer's (rounded) probabilities, which is not+ /// Laya's normalized entropy: choice `(max(p) - 1/n) / (1 - 1/n)`, score `max(0, 1 -+ /// sum(p_i * |i - mode|) / D)`. `None` for a noul answer and for probabilities that are not+ /// finite and non-negative. Ports `jev_confidence` from `laya/confidence.py` (#302).+ pub fn jev_confidence(&self) -> Option<f64> {+ confidence::jev_confidence(self)+ }++ /// The `qtype:size` bucket a per-bucket `min_confidence` map gates this answer under, e.g.+ /// `"choice:3-5"` (see [`crate::confidence::bucket_keys`]). `None` for a choice or score+ /// answer with no options.+ pub fn option_bucket(&self) -> Option<String> {+ confidence::option_bucket(self)+ }+/// The action head's output.pub fn action(&self) -> ActionInfo {match self {@@ -394,6 +454,11 @@ impl LayaResult {self}+ /// Applies `f` to every answer, in no particular order (the gate writes each independently).+ pub(crate) fn for_each_answer_mut(&mut self, mut f: impl FnMut(&mut Answer)) {+ self.by_id.values_mut().for_each(&mut f);+ }+/// The question ids, in the order they were asked.pub fn ids(&self) -> &[String] {&self.orderdiff --git a/laya-rust/src/batch_prediction.rs b/laya-rust/src/batch_prediction.rsindex f27e9b99..c9b8e8db 100644--- a/laya-rust/src/batch_prediction.rs+++ b/laya-rust/src/batch_prediction.rs@@ -7,10 +7,11 @@//! and the order restoration are testable without a model: the engine supplies the tokenizing and//! the forward pass.//!-//! Python's per-call hooks, `min_confidence`, `lang`, `max_len` and `head_max_len` are not part of-//! this port yet. [`BatchOptions`] is `#[non_exhaustive]` so they can be added as fields without a-//! signature change.+//! Python's per-call hooks, `lang`, `max_len` and `head_max_len` are not part of this port yet.+//! [`BatchOptions`] is `#[non_exhaustive]` so they can be added as fields without a signature+//! change.+use crate::confidence::MinConfidence;use crate::error::{LayaError, Result};use std::ops::Range;@@ -21,9 +22,9 @@ const SORT_WINDOW_BATCHES: usize = 8;/// Options for `LayaEngine::predict_batch` and `LayaRouter::predict_batch`. Every field is/// optional; `Default` gives Python's defaults. Build one with the chained setters.///-/// Per-call hooks, `min_confidence`, `lang`, `max_len` and `head_max_len`, which Python's-/// `predict_batch` also accepts, are not part of this port yet.-#[derive(Debug, Clone, Default, PartialEq, Eq)]+/// Per-call hooks, `lang`, `max_len` and `head_max_len`, which Python's `predict_batch` also+/// accepts, are not part of this port yet.+#[derive(Debug, Clone, Default, PartialEq)]#[non_exhaustive]pub struct BatchOptions {/// Most states per forward pass. `None` or `0` sends them all in one pass. Python treats a@@ -36,6 +37,10 @@ pub struct BatchOptions {/// than the number of states; otherwise it does nothing. Results keep input order. Changing/// the batch shapes can slightly change floating-point predictions near decision thresholds.pub sort_by_length: bool,+ /// Opt-in abstention threshold on `answer_confidence` (#361), applied batch-wide to every+ /// answer of every result; it plays no part in how states are grouped or batched. See+ /// [`crate::PredictOptions::min_confidence`] for what it reports.+ pub min_confidence: Option<MinConfidence>,}impl BatchOptions {@@ -58,6 +63,14 @@ impl BatchOptions {self}+ /// Set [`Self::min_confidence`]. Build the value with [`MinConfidence::scalar`] or+ /// [`MinConfidence::map`].+ #[must_use]+ pub fn min_confidence(mut self, min_confidence: MinConfidence) -> Self {+ self.min_confidence = Some(min_confidence);+ self+ }+/// States per forward pass for `count` states: `batch_size` when it is positive, else all.fn chunk(&self, count: usize) -> usize {match self.batch_size {@@ -391,6 +404,7 @@ mod tests {.get("sort_by_length").and_then(serde_json::Value::as_bool).unwrap_or(false),+ min_confidence: None,}}diff --git a/laya-rust/src/confidence.rs b/laya-rust/src/confidence.rsnew file mode 100644index 00000000..12acc7c1--- /dev/null+++ b/laya-rust/src/confidence.rs@@ -0,0 +1,575 @@+//! The abstention gate: per-call `min_confidence`, the states it reports, and Jev's `confidence`.+//! Ports `laya/confidence.py` (#361, #394), function for function.+//!+//! Python hands the gate a plain `float` or a `dict` and validates it with+//! `check_min_confidence`. Here a threshold is a [`MinConfidence`], built through+//! [`MinConfidence::scalar`] or [`MinConfidence::map`], which raise Python's own `ValueError`+//! text as [`LayaError::InvalidMinConfidence`]. The variants are public, so an unvalidated value+//! can be written by hand; [`MinConfidence::validate`] therefore runs again where Python runs+//! `check_min_confidence` (the start of `predict_batch`, and the router's `predict` and+//! `predict_batch`), so a hand-built value cannot reach the gate.+//!+//! What cannot be reached from Rust is the type check (`bool`, `str`, `None` in place of a float)+//! and the `isinstance(m, dict)` check: the types already rule them out.++use crate::answers::{Answer, LayaResult};+use crate::error::{LayaError, Result};+use crate::laya_router::python_repr_str;+use crate::python_json::PythonJson;+use indexmap::IndexMap;++/// The qtype half of a bucket key, in `common.temp_bucket`'s spelling.+pub const BUCKET_QTYPES: [&str; 3] = ["choice", "score", "noul"];+/// The option-count half of a bucket key.+pub const BUCKET_SIZES: [&str; 4] = ["2", "3-5", "6-10", "11+"];+/// The one non-bucket key a map may carry: the threshold for every bucket the map does not name.+pub const DEFAULT_BUCKET_KEY: &str = "default";++/// A gate ran and this answer's confidence cleared it.+pub const GATE_PASSED: &str = "passed";+/// A gate ran and this answer's confidence fell below it.+pub const GATE_ABSTAINED: &str = "abstained";+/// A gate ran and this answer carried no usable confidence.+pub const GATE_UNEVALUATED: &str = "unevaluated";++/// The states [`Answer::abstention`] reports, and the only ones. There is deliberately no member+/// for "no gate was configured": with no `min_confidence` an answer reports nothing at all+/// ([`Answer::abstention`] is `None`).+pub const GATE_STATES: [&str; 3] = [GATE_PASSED, GATE_ABSTAINED, GATE_UNEVALUATED];++/// Every option-count bucket a `min_confidence` map can name, e.g. `"choice:3-5"`. Ports+/// `bucket_keys`. Python returns a set; this is the same twelve keys in qtype-major order.+pub fn bucket_keys() -> Vec<String> {+ BUCKET_QTYPES+ .iter()+ .flat_map(|qtype| {+ BUCKET_SIZES+ .iter()+ .map(move |size| format!("{qtype}:{size}"))+ })+ .collect()+}++/// Python's `repr(float)`, which spells the non-finite values `nan`, `inf` and `-inf` (unlike+/// the JSON spelling [`PythonJson::repr`] gives them).+fn repr_float(value: f64) -> String {+ if value.is_nan() {+ "nan".to_string()+ } else if value.is_infinite() {+ if value > 0.0 { "inf" } else { "-inf" }.to_string()+ } else {+ PythonJson::repr(value)+ }+}++/// Python's `repr(dict)` for a `str -> float` map: `{'choice:2': 1.5, 'default': 0.5}`.+fn repr_map(map: &IndexMap<String, f64>) -> String {+ let items: Vec<String> = map+ .iter()+ .map(|(key, value)| format!("{}: {}", python_repr_str(key), repr_float(*value)))+ .collect();+ format!("{{{}}}", items.join(", "))+}++/// Ports `_check_one_threshold`.+fn check_one_threshold(value: f64) -> Result<f64> {+ if !value.is_finite() || !(0.0..=1.0).contains(&value) {+ return Err(LayaError::InvalidMinConfidence(format!(+ "min_confidence must be a float in [0.0, 1.0], got {}",+ repr_float(value)+ )));+ }+ Ok(value)+}++/// An opt-in abstention threshold on `answer_confidence`: one number for every answer, or a+/// per-bucket map so the threshold can differ by option count (#394).+///+/// Build one with [`Self::scalar`] or [`Self::map`], which validate as Python's+/// `check_min_confidence` does.+#[derive(Debug, Clone, PartialEq)]+pub enum MinConfidence {+ /// One threshold for every answer, in `[0.0, 1.0]`.+ Scalar(f64),+ /// A threshold per option-count bucket (every [`bucket_keys`] member) plus the optional+ /// `"default"` for any bucket the map does not name.+ Map(IndexMap<String, f64>),+}++impl MinConfidence {+ /// A validated scalar threshold. Errors with Python's text, e.g.+ /// `min_confidence must be a float in [0.0, 1.0], got 1.5`.+ pub fn scalar(threshold: f64) -> Result<Self> {+ let value = Self::Scalar(threshold);+ value.validate()?;+ Ok(value)+ }++ /// A validated per-bucket map, in the order given (which decides which entry an error names+ /// first). Keys are [`bucket_keys`] members or `"default"`; values are floats in `[0.0, 1.0]`.+ pub fn map<K: Into<String>>(thresholds: impl IntoIterator<Item = (K, f64)>) -> Result<Self> {+ let value = Self::Map(+ thresholds+ .into_iter()+ .map(|(key, threshold)| (key.into(), threshold))+ .collect(),+ );+ value.validate()?;+ Ok(value)+ }++ /// Ports `check_min_confidence` / `check_min_confidence_map`. Run again wherever Python runs+ /// it, because the variants are public.+ pub fn validate(&self) -> Result<()> {+ match self {+ Self::Scalar(threshold) => check_one_threshold(*threshold).map(|_| ()),+ Self::Map(map) => {+ if map.is_empty() {+ return Err(LayaError::InvalidMinConfidence(format!(+ "a min_confidence map must be a non-empty dict of bucket -> float, got {}",+ repr_map(map)+ )));+ }+ let allowed = bucket_keys();+ for (key, threshold) in map {+ if key != DEFAULT_BUCKET_KEY && !allowed.contains(key) {+ return Err(LayaError::InvalidMinConfidence(format!(+ "min_confidence map keys must be a bucket like 'choice:3-5' or \+ 'default', got {}",+ python_repr_str(key)+ )));+ }+ check_one_threshold(*threshold)?;+ }+ Ok(())+ }+ }+ }++ /// Ports `resolve_min_confidence` (a scalar passes straight through): the threshold this+ /// answer is gated at.+ fn resolve(&self, answer: &Answer) -> f64 {+ match self {+ Self::Scalar(threshold) => *threshold,+ Self::Map(map) => answer+ .option_bucket()+ .and_then(|key| map.get(&key))+ .or_else(|| map.get(DEFAULT_BUCKET_KEY))+ .copied()+ .unwrap_or(0.0),+ }+ }+}++/// The gate's verdict on one answer, kept on the answer.+#[derive(Debug, Clone, Copy, PartialEq, Default)]+pub(crate) struct Gate {+ pub(crate) low_confidence: bool,+ pub(crate) abstention: Option<&'static str>,+ pub(crate) threshold: Option<f64>,+}++/// Ports `_gate_confidence`: `answer_confidence` when usable, else `confidence`, else nothing.+fn gate_confidence(answer: &Answer) -> Option<f64> {+ [answer.answer_confidence(), answer.confidence()]+ .into_iter()+ .find(|value| value.is_finite())+}++/// Ports `flag_low_confidence` and `apply_confidence_gate` for one answer. The verdict replaces+/// whatever an earlier gate left, so a reused answer cannot keep a stale flag (#910).+fn gate_answer(answer: &mut Answer, min_confidence: &MinConfidence) {+ let threshold = min_confidence.resolve(answer);+ let confidence = gate_confidence(answer);+ let low_confidence = confidence.is_some_and(|c| c < threshold);+ let abstention = if low_confidence {+ GATE_ABSTAINED+ } else if confidence.is_none() {+ GATE_UNEVALUATED+ } else {+ GATE_PASSED+ };+ answer.set_gate(Gate {+ low_confidence,+ abstention: Some(abstention),+ threshold: Some(threshold),+ });+}++/// Ports `apply_confidence_gate`: with a threshold, reports the gate's state on every answer of+/// every result; with `None`, writes nothing.+pub(crate) fn apply_confidence_gate(+ results: &mut [LayaResult],+ min_confidence: Option<&MinConfidence>,+) {+ let Some(min_confidence) = min_confidence else {+ return;+ };+ for result in results {+ result.for_each_answer_mut(|answer| gate_answer(answer, min_confidence));+ }+}++/// Python's `sum()` over floats. Since CPython 3.12 it is not a plain left-to-right addition: it+/// carries a Neumaier compensation, so `sum([0.7, 0.2, 0.1])` is `1.0` where the naive loop gives+/// `0.9999999999999999`. This is that algorithm (`builtin_sum_impl`, float branch), so the totals+/// and spreads below round as Python 3.12+ rounds them.+fn python_sum(values: impl IntoIterator<Item = f64>) -> f64 {+ let mut values = values.into_iter();+ // The start value is the int 0; adding the first float gives that float.+ let mut sum = match values.next() {+ Some(first) => 0.0 + first,+ None => return 0.0,+ };+ let mut compensation = 0.0_f64;+ for value in values {+ let total = sum + value;+ if sum.abs() >= value.abs() {+ compensation += (sum - total) + value;+ } else {+ compensation += (value - total) + sum;+ }+ sum = total;+ }+ // Do not let the compensation turn an infinite or overflowed sum into NaN.+ if compensation != 0.0 && compensation.is_finite() {+ sum += compensation;+ }+ sum+}++/// Ports `jev_confidence` over an answer's (already rounded) probabilities: the `confidence` Jev+/// would report, or `None` for a noul answer and for a malformed document.+pub(crate) fn jev_confidence(answer: &Answer) -> Option<f64> {+ let probabilities: Vec<f64> = match answer {+ Answer::Choice(choice) => choice.probabilities.values().copied().collect(),+ Answer::Score(score) => score.probabilities.clone(),+ Answer::Noul(_) => return None,+ };+ if probabilities.is_empty() || !probabilities.iter().all(|v| v.is_finite() && *v >= 0.0) {+ return None;+ }+ let n = probabilities.len();+ if n == 1 {+ return Some(1.0);+ }+ let count = n as f64;+ let total = python_sum(probabilities.iter().copied());+ let p: Vec<f64> = if total != 0.0 {+ probabilities.iter().map(|v| v / total).collect()+ } else {+ vec![1.0 / count; n]+ };+ // `max` and `max(range(n), key=...)` keep the first of equal values.+ let mut mode = 0;+ for (i, value) in p.iter().enumerate().skip(1) {+ if *value > p[mode] {+ mode = i;+ }+ }+ if matches!(answer, Answer::Choice(_)) {+ return Some((p[mode] - 1.0 / count) / (1.0 - 1.0 / count));+ }+ let spread = python_sum(+ p.iter()+ .enumerate()+ .map(|(i, v)| v * (i as f64 - mode as f64).abs()),+ );+ let center = (n - 1) as f64 / 2.0;+ let uniform_spread = python_sum((0..n).map(|i| (i as f64 - center).abs())) / count;+ // Python's `max(0.0, x)`: zero unless `x` is greater.+ let jev = 1.0 - spread / uniform_spread;+ Some(if jev > 0.0 { jev } else { 0.0 })+}++/// Ports `_option_bucket`: the `qtype:size` key an answer is gated under.+pub(crate) fn option_bucket(answer: &Answer) -> Option<String> {+ let (qtype, options) = match answer {+ Answer::Choice(choice) => ("choice", choice.probabilities.len()),+ Answer::Score(score) => ("score", score.probabilities.len()),+ // A noul answer carries no probabilities dict, so Python reads it as two options.+ Answer::Noul(_) => ("noul", 2),+ };+ if options == 0 {+ return None;+ }+ let size = match options {+ 0..=2 => "2",+ 3..=5 => "3-5",+ 6..=10 => "6-10",+ _ => "11+",+ };+ Some(format!("{qtype}:{size}"))+}++#[cfg(test)]+mod tests {+ use super::*;+ use crate::answers::{ActionInfo, ChoiceAnswer, NoulAnswer, ScoreAnswer};+ use std::collections::HashMap;++ fn choice(probabilities: &[f64], confidence: f64, answer_confidence: f64) -> Answer {+ let map: IndexMap<String, f64> = probabilities+ .iter()+ .enumerate()+ .map(|(i, p)| (format!("o{i}"), *p))+ .collect();+ Answer::Choice(ChoiceAnswer::new(+ "o0",+ map,+ confidence,+ answer_confidence,+ ActionInfo::default(),+ ))+ }++ fn score(probabilities: &[f64]) -> Answer {+ Answer::Score(ScoreAnswer::new(+ 0.0,+ Vec::new(),+ probabilities.to_vec(),+ 0.5,+ 0.5,+ ActionInfo::default(),+ ))+ }++ fn noul(confidence: f64, answer_confidence: f64) -> Answer {+ Answer::Noul(NoulAnswer::new(+ 0.7,+ confidence,+ answer_confidence,+ ActionInfo::default(),+ ))+ }++ fn result(answers: Vec<(&str, Answer)>) -> LayaResult {+ let order = answers.iter().map(|(id, _)| id.to_string()).collect();+ let by_id: HashMap<String, Answer> = answers+ .into_iter()+ .map(|(id, a)| (id.to_string(), a))+ .collect();+ LayaResult::new("m", order, by_id, crate::answers::Usage::default())+ }++ fn gated(answer: Answer, min_confidence: &MinConfidence) -> Answer {+ let mut results = [result(vec![("q", answer)])];+ apply_confidence_gate(&mut results, Some(min_confidence));+ results[0]["q"].clone()+ }++ #[test]+ fn an_ungated_answer_reports_nothing() {+ let answer = choice(&[0.6, 0.4], 0.3, 0.6);+ assert!(!answer.low_confidence());+ assert_eq!(answer.abstention(), None);+ assert_eq!(answer.abstention_threshold(), None);+ let mut results = [result(vec![("q", answer)])];+ apply_confidence_gate(&mut results, None);+ assert_eq!(results[0]["q"].abstention(), None);+ }++ #[test]+ fn scalar_gate_reports_passed_abstained_and_the_threshold() {+ let threshold = MinConfidence::scalar(0.7).unwrap();+ let low = gated(choice(&[0.6, 0.4], 0.9, 0.6), &threshold);+ assert!(low.low_confidence());+ assert_eq!(low.abstention(), Some("abstained"));+ assert_eq!(low.abstention_threshold(), Some(0.7));+ let high = gated(choice(&[0.8, 0.2], 0.1, 0.8), &threshold);+ assert!(!high.low_confidence());+ assert_eq!(high.abstention(), Some("passed"));+ // Equal to the threshold clears it: the test is strictly below.+ let equal = gated(choice(&[0.7, 0.3], 0.1, 0.7), &threshold);+ assert_eq!(equal.abstention(), Some("passed"));+ }++ #[test]+ fn the_gate_reads_answer_confidence_first_then_confidence() {+ let threshold = MinConfidence::scalar(0.5).unwrap();+ // answer_confidence passes although confidence would not.+ let a = gated(choice(&[0.6, 0.4], 0.1, 0.6), &threshold);+ assert_eq!(a.abstention(), Some("passed"));+ // A NaN answer_confidence falls back to confidence (confidence.py:86).+ let b = gated(choice(&[0.6, 0.4], 0.1, f64::NAN), &threshold);+ assert_eq!(b.abstention(), Some("abstained"));+ let c = gated(choice(&[0.6, 0.4], 0.9, f64::INFINITY), &threshold);+ assert_eq!(c.abstention(), Some("passed"));+ }++ #[test]+ fn an_answer_with_no_usable_confidence_is_unevaluated_not_flagged() {+ let threshold = MinConfidence::scalar(0.9).unwrap();+ let answer = gated(noul(f64::NAN, f64::NAN), &threshold);+ assert!(!answer.low_confidence());+ assert_eq!(answer.abstention(), Some("unevaluated"));+ assert_eq!(answer.abstention_threshold(), Some(0.9));+ }++ #[test]+ fn zero_is_a_gate_that_passes_everything_usable_and_echoes_zero() {+ let threshold = MinConfidence::scalar(0.0).unwrap();+ let answer = gated(choice(&[0.5, 0.5], 0.0, 0.0), &threshold);+ assert_eq!(answer.abstention(), Some("passed"));+ assert_eq!(answer.abstention_threshold(), Some(0.0));+ assert!(!answer.low_confidence());+ let unusable = gated(noul(f64::NAN, f64::NAN), &threshold);+ assert_eq!(unusable.abstention(), Some("unevaluated"));+ }++ #[test]+ fn a_map_gates_each_answer_at_its_own_bucket_then_default_then_zero() {+ let map =+ MinConfidence::map([("choice:2", 0.9), ("score:3-5", 0.8), ("default", 0.55)]).unwrap();+ let two = gated(choice(&[0.6, 0.4], 0.5, 0.6), &map);+ assert_eq!(two.abstention_threshold(), Some(0.9));+ assert_eq!(two.abstention(), Some("abstained"));+ let three_scores = gated(score(&[0.2, 0.7, 0.1]), &map);+ assert_eq!(three_scores.abstention_threshold(), Some(0.8));+ // noul:2 is not named, so it falls to the default.+ let unnamed = gated(noul(0.7, 0.7), &map);+ assert_eq!(unnamed.abstention_threshold(), Some(0.55));+ assert_eq!(unnamed.abstention(), Some("passed"));+ // No default either: gate nothing.+ let no_default = MinConfidence::map([("choice:2", 0.9)]).unwrap();+ let ungated = gated(noul(0.1, 0.1), &no_default);+ assert_eq!(ungated.abstention_threshold(), Some(0.0));+ assert_eq!(ungated.abstention(), Some("passed"));+ }++ #[test]+ fn option_buckets_follow_the_size_bands() {+ let sizes = [+ (1, "2"),+ (2, "2"),+ (3, "3-5"),+ (5, "3-5"),+ (6, "6-10"),+ (10, "6-10"),+ (11, "11+"),+ ];+ for (count, size) in sizes {+ let p = vec![1.0 / count as f64; count];+ assert_eq!(+ choice(&p, 0.5, 0.5).option_bucket(),+ Some(format!("choice:{size}"))+ );+ assert_eq!(score(&p).option_bucket(), Some(format!("score:{size}")));+ }+ assert_eq!(noul(0.5, 0.5).option_bucket().as_deref(), Some("noul:2"));+ assert_eq!(choice(&[], 0.5, 0.5).option_bucket(), None);+ }++ /// #910: a result gated at a strict threshold and then at a lenient one must not keep the+ /// flag. Results are values here, so the verdict is rewritten on every gate.+ #[test]+ fn regating_a_reused_result_clears_a_stale_flag() {+ let mut results = [result(vec![("q", choice(&[0.6, 0.4], 0.5, 0.6))])];+ apply_confidence_gate(&mut results, Some(&MinConfidence::scalar(0.9).unwrap()));+ assert!(results[0]["q"].low_confidence());+ apply_confidence_gate(&mut results, Some(&MinConfidence::scalar(0.0).unwrap()));+ assert!(!results[0]["q"].low_confidence());+ assert_eq!(results[0]["q"].abstention(), Some("passed"));+ assert_eq!(results[0]["q"].abstention_threshold(), Some(0.0));+ // A later call with no threshold writes nothing, so the last verdict is untouched.+ apply_confidence_gate(&mut results, None);+ assert_eq!(results[0]["q"].abstention(), Some("passed"));+ }++ fn approx(a: Option<f64>, b: f64) {+ let a = a.expect("a jev confidence");+ assert!((a - b).abs() < 1e-12, "{a} vs {b}");+ }++ #[test]+ fn jev_confidence_for_choice_and_score() {+ approx(choice(&[0.5, 0.5], 0.0, 0.5).jev_confidence(), 0.0);+ approx(choice(&[1.0, 0.0], 0.0, 1.0).jev_confidence(), 1.0);+ approx(+ choice(&[0.6, 0.3, 0.1], 0.0, 0.6).jev_confidence(),+ (0.6 - 1.0 / 3.0) / (2.0 / 3.0),+ );+ // A single option is 1.0, and a zero total reads as uniform.+ approx(choice(&[0.0], 0.0, 0.0).jev_confidence(), 1.0);+ approx(choice(&[0.0, 0.0], 0.0, 0.0).jev_confidence(), 0.0);+ // Unnormalised input is normalised by its sum.+ approx(choice(&[3.0, 1.0], 0.0, 0.0).jev_confidence(), 0.5);+ // Score: a point mass has zero spread; a uniform distribution is 0.+ approx(score(&[0.0, 1.0, 0.0]).jev_confidence(), 1.0);+ approx(score(&[0.25, 0.25, 0.25, 0.25]).jev_confidence(), 0.0);+ }++ #[test]+ fn jev_confidence_is_none_for_noul_and_malformed_probabilities() {+ assert_eq!(noul(0.5, 0.5).jev_confidence(), None);+ assert_eq!(choice(&[], 0.5, 0.5).jev_confidence(), None);+ assert_eq!(choice(&[0.5, f64::NAN], 0.5, 0.5).jev_confidence(), None);+ assert_eq!(+ choice(&[0.5, f64::INFINITY], 0.5, 0.5).jev_confidence(),+ None+ );+ assert_eq!(choice(&[0.5, -0.1], 0.5, 0.5).jev_confidence(), None);+ assert_eq!(score(&[0.5, -0.1]).jev_confidence(), None);+ assert_eq!(score(&[]).jev_confidence(), None);+ }++ #[test]+ fn scalar_validation_uses_pythons_text_and_repr() {+ let text = |v: f64| MinConfidence::scalar(v).unwrap_err().to_string();+ let head = "min_confidence must be a float in [0.0, 1.0], got ";+ assert_eq!(text(1.5), format!("{head}1.5"));+ assert_eq!(text(2.0), format!("{head}2.0"));+ assert_eq!(text(-0.1), format!("{head}-0.1"));+ assert_eq!(text(f64::NAN), format!("{head}nan"));+ assert_eq!(text(f64::INFINITY), format!("{head}inf"));+ assert_eq!(text(f64::NEG_INFINITY), format!("{head}-inf"));+ assert_eq!(text(1e16), format!("{head}1e+16"));+ assert!(MinConfidence::scalar(0.0).is_ok());+ assert!(MinConfidence::scalar(1.0).is_ok());+ }++ #[test]+ fn map_validation_uses_pythons_text_in_python_order() {+ let err = |m: Vec<(&str, f64)>| MinConfidence::map(m).unwrap_err().to_string();+ let keys = "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got ";+ let value = "min_confidence must be a float in [0.0, 1.0], got 1.5";+ assert_eq!(+ err(vec![]),+ "a min_confidence map must be a non-empty dict of bucket -> float, got {}"+ );+ assert_eq!(err(vec![("choice:1", 0.5)]), format!("{keys}'choice:1'"));+ assert_eq!(err(vec![("it's", 0.5)]), format!("{keys}\"it's\""));+ assert_eq!(err(vec![("choice:2", 1.5)]), value);+ // The first bad entry in insertion order decides which error is raised.+ assert_eq!(err(vec![("choice:2", 1.5), ("nope", 0.5)]), value);+ assert_eq!(+ err(vec![("nope", 0.5), ("choice:2", 1.5)]),+ format!("{keys}'nope'")+ );+ assert!(MinConfidence::map([("default", 0.5), ("noul:11+", 0.0)]).is_ok());+ }++ #[test]+ fn hand_built_variants_are_validated_again() {+ assert!(MinConfidence::Scalar(2.0).validate().is_err());+ assert!(MinConfidence::Map(IndexMap::new()).validate().is_err());+ }++ /// CPython 3.12 compensates `sum()`: the naive loop would give 0.9999999999999999.+ #[test]+ fn python_sum_is_compensated_like_cpython_312() {+ assert_eq!(python_sum([0.7, 0.2, 0.1]), 1.0);+ assert_eq!(python_sum([0.1; 10]), 1.0);+ assert_eq!(python_sum([]), 0.0);+ assert_eq!(python_sum([f64::INFINITY, 1.0]), f64::INFINITY);+ }++ #[test]+ fn bucket_keys_are_the_twelve_buckets() {+ let keys = bucket_keys();+ assert_eq!(keys.len(), 12);+ assert_eq!(keys[0], "choice:2");+ assert_eq!(keys[11], "noul:11+");+ assert_eq!(GATE_STATES, ["passed", "abstained", "unevaluated"]);+ }+}diff --git a/laya-rust/src/error.rs b/laya-rust/src/error.rsindex 93828e3a..29e9eb50 100644--- a/laya-rust/src/error.rs+++ b/laya-rust/src/error.rs@@ -121,6 +121,12 @@ pub enum LayaError {/// router whose predictor has no scan capability).#[error("{0}")]NotSupported(String),++ /// A `min_confidence` threshold or per-bucket map that Python's `check_min_confidence` would+ /// reject with a `ValueError`: out of `[0.0, 1.0]`, NaN, an empty map, or a key that is not a+ /// bucket. The text is Python's, with `%r` rendered as Python's `repr`.+ #[error("{0}")]+ InvalidMinConfidence(String),}#[cfg(test)]diff --git a/laya-rust/src/laya_engine.rs b/laya-rust/src/laya_engine.rsindex f637102f..f6555c02 100644--- a/laya-rust/src/laya_engine.rs+++ b/laya-rust/src/laya_engine.rs@@ -19,11 +19,13 @@ use crate::answers::{use crate::batch_prediction::{self, BatchOptions};use crate::calibration::Calibration;use crate::collator::{CollatedBatch, Collator, SequenceItem};+use crate::confidence::apply_confidence_gate;use crate::error::{LayaError, Result};use crate::laya_config::{Deferred, LayaConfig};use crate::laya_options::{LayaCheckpoint, LayaExecutionProvider, LayaOptions};use crate::long_prediction::{self, LongOptions};use crate::model_artifacts::{ModelArtifacts, ModelPaths};+use crate::predict_options::PredictOptions;use crate::questions::{Question, QuestionSet};use crate::sequence_builder::SequenceBuilder;use crate::tokenization::{HfTokenizer, LayaTokenizer};@@ -195,7 +197,20 @@ impl LayaEngine {/// Returns [`LayaError::NoMarkers`] if a question's options cannot fit within/// `Config::head_max_len`, rather than silently scoring fewer options than it was asked about.pub fn predict(&self, state: impl Into<Value>, questions: &QuestionSet) -> Result<LayaResult> {- self.predict_value(state.into(), questions)+ self.predict_value(state.into(), questions, &PredictOptions::default())+ }++ /// [`Self::predict`] with per-call [`PredictOptions`]. Ports `system_one(min_confidence=...)`:+ /// with `options.min_confidence` set, every answer reports its gate state (see+ /// [`Answer::abstention`]); a threshold Python's `check_min_confidence` rejects is+ /// [`LayaError::InvalidMinConfidence`], checked before the state is, as in `predict_batch`.+ pub fn predict_with_options(+ &self,+ state: impl Into<Value>,+ questions: &QuestionSet,+ options: &PredictOptions,+ ) -> Result<LayaResult> {+ self.predict_value(state.into(), questions, options)}/// [`Self::predict`], for a state that implements [`serde::Serialize`] but not@@ -206,15 +221,25 @@ impl LayaEngine {questions: &QuestionSet,) -> Result<LayaResult> {let value = serde_json::to_value(state)?;- self.predict_value(value, questions)+ self.predict_value(value, questions, &PredictOptions::default())}/// Python's `system_one` is `predict_batch([state], ...)`, so this is a one-state batch: a/// `null` state is refused and no questions give an empty result.- fn predict_value(&self, state: Value, questions: &QuestionSet) -> Result<LayaResult> {+ fn predict_value(+ &self,+ state: Value,+ questions: &QuestionSet,+ options: &PredictOptions,+ ) -> Result<LayaResult> {+ let min_confidence = options.min_confidence.as_ref();+ if let Some(min_confidence) = min_confidence {+ min_confidence.validate()?;+ }let states = [state];refuse_null_states(&states)?;let mut results = self.predict_states(&states, questions)?;+ apply_confidence_gate(&mut results, min_confidence);results.pop().ok_or_else(|| LayaError::Onnx("internal error: no result for the state".to_string()))@@ -238,9 +263,10 @@ impl LayaEngine {/// slightly from the original after re-encoding, exactly as in Python. Windows go through the/// model together unless [`LongOptions::batch_size`] or the automatic cap chunks them.///- /// Per-call hooks, per-language temperatures and the confidence gate are not part of this- /// port. Errors: [`LayaError::UnsupportedAggregate`], [`LayaError::InvalidWindow`] (no room- /// for the state, or a stride that leaves gaps), plus everything [`Self::predict`] returns.+ /// Per-call hooks and per-language temperatures are not part of this port, and, as in Python,+ /// `predict_long` takes no `min_confidence`. Errors: [`LayaError::UnsupportedAggregate`],+ /// [`LayaError::InvalidWindow`] (no room for the state, or a stride that leaves gaps), plus+ /// everything [`Self::predict`] returns.#[track_caller]pub fn predict_long(&self,@@ -338,16 +364,34 @@ impl LayaEngine {/// * No states give an empty list. No questions give one empty result per state, with no/// state usage, as Python does.///- /// A `null` state is refused with [`LayaError::InvalidState`] (Python's `TypeError`: scoring it- /// would answer a decision about the text `"null"`). Everything else [`Self::predict`] returns- /// applies per state, and the first error fails the whole call.+ /// * `min_confidence` gates every answer of every result, batch-wide; see [`BatchOptions`].+ ///+ /// A threshold Python's `check_min_confidence` rejects is [`LayaError::InvalidMinConfidence`],+ /// checked first. A `null` state is refused with [`LayaError::InvalidState`] (Python's+ /// `TypeError`: scoring it would answer a decision about the text `"null"`). Everything else+ /// [`Self::predict`] returns applies per state, and the first error fails the whole call.pub fn predict_batch(&self,states: &[Value],questions: &QuestionSet,options: &BatchOptions,) -> Result<Vec<LayaResult>> {+ let min_confidence = options.min_confidence.as_ref();+ if let Some(min_confidence) = min_confidence {+ min_confidence.validate()?;+ }refuse_null_states(states)?;+ let mut results = self.predict_batch_ungated(states, questions, options)?;+ apply_confidence_gate(&mut results, min_confidence);+ Ok(results)+ }++ fn predict_batch_ungated(+ &self,+ states: &[Value],+ questions: &QuestionSet,+ options: &BatchOptions,+ ) -> Result<Vec<LayaResult>> {if states.is_empty() {return Ok(Vec::new());}diff --git a/laya-rust/src/laya_router.rs b/laya-rust/src/laya_router.rsindex 9180f9d9..e1e8321e 100644--- a/laya-rust/src/laya_router.rs+++ b/laya-rust/src/laya_router.rs@@ -38,11 +38,13 @@use crate::answers::LayaResult;use crate::batch_prediction::BatchOptions;+use crate::confidence::apply_confidence_gate;use crate::error::{LayaError, Result};use crate::language_detection::{LanguageAnalysis, LanguageDetection, prefix_chars, py_strip};use crate::laya_engine::{LayaEngine, refuse_null_states};use crate::laya_options::{LayaCheckpoint, LayaOptions};use crate::long_prediction::LongOptions;+use crate::predict_options::PredictOptions;use crate::questions::QuestionSet;use serde_json::Value;use std::collections::{HashMap, HashSet};@@ -910,16 +912,49 @@ impl LayaRouter {task: Option<&str>,lang: Option<&str>,lang_guess: Option<&LangGuess>,+ ) -> Result<LayaResult> {+ self.predict_with_options(+ state,+ questions,+ model,+ task,+ lang,+ lang_guess,+ &PredictOptions::default(),+ )+ }++ /// [`Self::predict_with_lang_guess`] with per-call [`PredictOptions`]. Ports+ /// `Router.predict(min_confidence=...)`: a `null` state is refused first, then the threshold is+ /// validated ([`LayaError::InvalidMinConfidence`]), both before anything is routed or loaded.+ /// The gate runs on the router's own result, so it also covers an attached predictor that+ /// knows nothing of it.+ #[allow(clippy::too_many_arguments)]+ pub fn predict_with_options(+ &self,+ state: impl Into<Value>,+ questions: &QuestionSet,+ model: Option<&str>,+ task: Option<&str>,+ lang: Option<&str>,+ lang_guess: Option<&LangGuess>,+ options: &PredictOptions,) -> Result<LayaResult> {let state = state.into();// Python's `Router.predict` refuses `None` before it routes or loads anything.refuse_null_states(std::slice::from_ref(&state))?;+ let min_confidence = options.min_confidence.as_ref();+ if let Some(min_confidence) = min_confidence {+ min_confidence.validate()?;+ }let decision =self.route_with_lang_guess(&state, Some(questions), model, task, lang, lang_guess)?;let engine = self.load_checkpoint(decision.model)?;// Inference deliberately runs after the `Arc<dyn LayaPredictor>` clone above and outside// any lock: see the module docs on eviction safety.- let result = engine.predict(state, questions)?;+ let mut results = [engine.predict(state, questions)?];+ apply_confidence_gate(&mut results, min_confidence);+ let [result] = results;Ok(result.with_routing(decision))}@@ -964,11 +999,20 @@ impl LayaRouter {/// `1 < batch_size < group size` gate. No requests give an empty list. The first failing group/// fails the call. Returns [`LayaError::NotSupported`] when a chosen predictor (for example/// one registered with [`Self::attach`]) has no batch capability.+ ///+ /// `options.min_confidence` is validated before anything is routed+ /// ([`LayaError::InvalidMinConfidence`]) and gates every result of the call, as one batch-wide+ /// threshold: it takes no part in the grouping, and each result is gated here, by the router,+ /// so an attached predictor needs no support for it.pub fn predict_batch(&self,requests: &[BatchRequest],options: &BatchOptions,) -> Result<Vec<LayaResult>> {+ let min_confidence = options.min_confidence.as_ref();+ if let Some(min_confidence) = min_confidence {+ min_confidence.validate()?;+ }let decisions = self.route_batch(requests)?;if decisions.is_empty() {return Ok(Vec::new());@@ -987,6 +1031,11 @@ impl LayaRouter {}}+ // The gate is the router's: the predictors score ungated, whatever the caller asked for.+ let ungated = BatchOptions {+ min_confidence: None,+ ..options.clone()+ };let mut results: Vec<Option<LayaResult>> = Vec::with_capacity(requests.len());results.resize_with(requests.len(), || None);for (model, indices) in groups {@@ -1022,7 +1071,7 @@ impl LayaRouter {.map(|&index| requests[index].state.clone()).collect();let batch =- batcher.predict_batch(&states, &requests[items[0]].questions, options)?;+ batcher.predict_batch(&states, &requests[items[0]].questions, &ungated)?;if batch.len() != items.len() {return Err(LayaError::Onnx(format!("internal error: predict_batch returned {} results for {} states",@@ -1036,7 +1085,7 @@ impl LayaRouter {}}- results+ let mut results = results.into_iter().map(|result| {result.ok_or_else(|| {@@ -1045,7 +1094,9 @@ impl LayaRouter {)})})- .collect()+ .collect::<Result<Vec<_>>>()?;+ apply_confidence_gate(&mut results, min_confidence);+ Ok(results)}/// Route once, on the original state, then scan the whole state on the chosen checkpoint withdiff --git a/laya-rust/src/lib.rs b/laya-rust/src/lib.rsindex db35896b..406f3d26 100644--- a/laya-rust/src/lib.rs+++ b/laya-rust/src/lib.rs@@ -44,6 +44,7 @@ pub mod answers;pub mod batch_prediction;pub mod calibration;pub mod collator;+pub mod confidence;pub mod error;pub mod language_detection;pub mod laya_config;@@ -55,6 +56,7 @@ pub mod laya_router;pub mod laya_shortlist;pub mod long_prediction;pub mod model_artifacts;+pub mod predict_options;pub mod python_json;mod python_json_reader;pub mod questions;@@ -68,6 +70,7 @@ pub use answers::{pub use batch_prediction::BatchOptions;pub use calibration::Calibration;pub use collator::{CollatedBatch, Collator, SequenceItem};+pub use confidence::{GATE_STATES, MinConfidence, bucket_keys};pub use error::{LayaError, Result};pub use language_detection::{LanguageAnalysis, LanguageDetection, LatinProfile};pub use laya_config::LayaConfig;@@ -84,6 +87,7 @@ pub use laya_router::{pub use laya_shortlist::{DEFAULT_SHORTLIST_K, LayaShortlist, ShortlistInfo};pub use long_prediction::LongOptions;pub use model_artifacts::{ModelArtifacts, ModelLayout, ModelPaths};+pub use predict_options::PredictOptions;pub use python_json::PythonJson;pub use questions::{ChoiceQuestion, NoulQuestion, Question, QuestionSet, QuestionType, ScoreQuestion,diff --git a/laya-rust/src/predict_options.rs b/laya-rust/src/predict_options.rsnew file mode 100644index 00000000..d5b06b4f--- /dev/null+++ b/laya-rust/src/predict_options.rs@@ -0,0 +1,36 @@+//! Per-call options for `LayaEngine::predict_with_options` and `LayaRouter::predict_with_options`.+//!+//! Python's `system_one` and `Router.predict` take `min_confidence`, `lang`, `max_len`,+//! `head_max_len` and per-call hooks. Only `min_confidence` is ported so far; [`PredictOptions`] is+//! `#[non_exhaustive]` so the others can be added as fields without a signature change.++use crate::confidence::MinConfidence;++/// Options for one `predict` call. `Default` gives Python's defaults (nothing set). Build one with+/// the chained setters.+///+/// `lang`, `max_len`, `head_max_len` and the per-call hooks, which Python's `system_one` also+/// accepts, are not part of this port yet. `predict_long` takes no `min_confidence`, as in Python.+#[derive(Debug, Clone, Default, PartialEq)]+#[non_exhaustive]+pub struct PredictOptions {+ /// Opt-in abstention threshold on `answer_confidence` (#361). With it set, every answer+ /// reports `Answer::abstention` and `Answer::abstention_threshold`, and `Answer::low_confidence`+ /// is true for those below it. `None` writes nothing, so an ungated call is unchanged.+ pub min_confidence: Option<MinConfidence>,+}++impl PredictOptions {+ /// Python's defaults: no gate.+ pub fn new() -> Self {+ Self::default()+ }++ /// Set [`Self::min_confidence`]. Build the value with [`MinConfidence::scalar`] or+ /// [`MinConfidence::map`].+ #[must_use]+ pub fn min_confidence(mut self, min_confidence: MinConfidence) -> Self {+ self.min_confidence = Some(min_confidence);+ self+ }+}diff --git a/laya-rust/tests/common/golden_data.rs b/laya-rust/tests/common/golden_data.rsindex 688f2875..79fd94cd 100644--- a/laya-rust/tests/common/golden_data.rs+++ b/laya-rust/tests/common/golden_data.rs@@ -8,7 +8,10 @@//! parses to an integer `Number`, `3.0` to a float `Number`), so a parsed `Value` can be handed//! straight to [`laya::questions::Question`] constructors.+use indexmap::IndexMap;+use laya::confidence::MinConfidence;use laya::error::Result as LayaResult;+use laya::predict_options::PredictOptions;use laya::questions::{Question, QuestionSet};use serde_json::Value;use std::path::{Path, PathBuf};@@ -22,6 +25,31 @@ pub fn golden_base_dir() -> PathBuf {}}+/// The `min_confidence` a recorded case passed to Python (`options.min_confidence`), written as+/// the variant itself and not through the validating constructors, so an invalid recorded value+/// reaches the engine's own validation. `None` when the case recorded no threshold.+pub fn recorded_min_confidence(options: &Value) -> Option<MinConfidence> {+ match options.get("min_confidence")? {+ Value::Object(map) => Some(MinConfidence::Map(+ map.iter()+ .map(|(key, value)| (key.clone(), value.as_f64().expect("a float threshold")))+ .collect::<IndexMap<_, _>>(),+ )),+ value => Some(MinConfidence::Scalar(+ value.as_f64().expect("a float threshold"),+ )),+ }+}++/// The per-call options of a recorded `system_one` case: its `options`, when it recorded any.+pub fn recorded_predict_options(case: &Value) -> PredictOptions {+ let mut options = PredictOptions::new();+ if let Some(recorded) = case.get("options") {+ options.min_confidence = recorded_min_confidence(recorded);+ }+ options+}+/// The golden directory for one checkpoint subfolder (`"multilingual"`, `"english"`,/// `"typed-decisions"`).pub fn checkpoint_dir(subdir: &str) -> PathBuf {diff --git a/laya-rust/tests/common/result_compare.rs b/laya-rust/tests/common/result_compare.rsindex 5e935893..645f159d 100644--- a/laya-rust/tests/common/result_compare.rs+++ b/laya-rust/tests/common/result_compare.rs@@ -86,6 +86,8 @@ pub fn assert_answers_match(where_case: &str, expected: &Value, result: &LayaRes&format!("{where_q}.action.act_probability"),);+ assert_gate_matches(want, got, &where_q);+match got {Answer::Choice(choice) => assert_choice(want, choice, &where_q),Answer::Score(score) => assert_score(want, score, &where_q),@@ -94,6 +96,32 @@ pub fn assert_answers_match(where_case: &str, expected: &Value, result: &LayaRes}}+/// What the abstention gate wrote on one answer must be what Python wrote: `low_confidence` only+/// when flagged, `abstention` and `abstention_threshold` only when a threshold was set. A golden+/// recorded without a threshold carries none of the three, so this also pins "unset writes+/// nothing".+pub fn assert_gate_matches(want: &Value, got: &Answer, where_q: &str) {+ assert_eq!(+ want.get("low_confidence") == Some(&Value::Bool(true)),+ got.low_confidence(),+ "{where_q}: low_confidence"+ );+ assert!(+ want.get("low_confidence").is_none() || want["low_confidence"] == Value::Bool(true),+ "{where_q}: python only ever writes low_confidence: true"+ );+ assert_eq!(+ want.get("abstention").and_then(Value::as_str),+ got.abstention(),+ "{where_q}: abstention"+ );+ assert_eq!(+ want.get("abstention_threshold").and_then(Value::as_f64),+ got.abstention_threshold(),+ "{where_q}: abstention_threshold"+ );+}+pub fn assert_state_usage_matches(where_case: &str, expected: &Value, result: &LayaResult) {let usage = result.state_usage()diff --git a/laya-rust/tests/golden/english/case_batch_min_confidence_bad_key.json b/laya-rust/tests/golden/english/case_batch_min_confidence_bad_key.jsonnew file mode 100644index 00000000..424420ec--- /dev/null+++ b/laya-rust/tests/golden/english/case_batch_min_confidence_bad_key.json@@ -0,0 +1,38 @@+{+ "name": "batch_min_confidence_bad_key",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": {+ "choice:2": 0.5,+ "choice:3": 0.5+ }+ },+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'"+}diff --git a/laya-rust/tests/golden/english/case_batch_min_confidence_before_none_state.json b/laya-rust/tests/golden/english/case_batch_min_confidence_before_none_state.jsonnew file mode 100644index 00000000..75fd7564--- /dev/null+++ b/laya-rust/tests/golden/english/case_batch_min_confidence_before_none_state.json@@ -0,0 +1,36 @@+{+ "name": "batch_min_confidence_before_none_state",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ null+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": -0.25+ },+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -0.25"+}diff --git a/laya-rust/tests/golden/english/case_batch_min_confidence_empty_questions.json b/laya-rust/tests/golden/english/case_batch_min_confidence_empty_questions.jsonnew file mode 100644index 00000000..255f7bc4--- /dev/null+++ b/laya-rust/tests/golden/english/case_batch_min_confidence_empty_questions.json@@ -0,0 +1,44 @@+{+ "name": "batch_min_confidence_empty_questions",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, the refund arrived and everything works now.",+ {+ "ticket": 4821,+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "+ }+ ],+ "questions": {},+ "options": {+ "min_confidence": 0.5+ },+ "state_widths": [],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {},+ "usage": {+ "input_tokens": 0,+ "output_tokens": 0+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {},+ "usage": {+ "input_tokens": 0,+ "output_tokens": 0+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {},+ "usage": {+ "input_tokens": 0,+ "output_tokens": 0+ }+ }+ ],+ "runs": []+}diff --git a/laya-rust/tests/golden/english/case_batch_min_confidence_empty_states.json b/laya-rust/tests/golden/english/case_batch_min_confidence_empty_states.jsonnew file mode 100644index 00000000..dfb141d9--- /dev/null+++ b/laya-rust/tests/golden/english/case_batch_min_confidence_empty_states.json@@ -0,0 +1,34 @@+{+ "name": "batch_min_confidence_empty_states",+ "states": [],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.5+ },+ "state_widths": [],+ "results": [],+ "runs": []+}diff --git a/laya-rust/tests/golden/english/case_batch_min_confidence_invalid.json b/laya-rust/tests/golden/english/case_batch_min_confidence_invalid.jsonnew file mode 100644index 00000000..72496b52--- /dev/null+++ b/laya-rust/tests/golden/english/case_batch_min_confidence_invalid.json@@ -0,0 +1,35 @@+{+ "name": "batch_min_confidence_invalid",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 1.5+ },+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"+}diff --git a/laya-rust/tests/golden/english/case_batch_min_confidence_map.json b/laya-rust/tests/golden/english/case_batch_min_confidence_map.jsonnew file mode 100644index 00000000..f1c18d0c--- /dev/null+++ b/laya-rust/tests/golden/english/case_batch_min_confidence_map.json@@ -0,0 +1,497 @@+{+ "name": "batch_min_confidence_map",+ "states": [+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "batch_size": 3,+ "sort_by_length": true,+ "min_confidence": {+ "choice:3-5": 0.8,+ "score:3-5": 0.5,+ "noul:2": 0.7+ }+ },+ "state_widths": [+ 199,+ 103,+ 145,+ 91,+ 175,+ 121,+ 157+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.896,+ "technical": 0.0688,+ "other": 0.0352+ },+ "confidence": 0.6355,+ "answer_confidence": 0.896,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.7317,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0621,+ "1": 0.1442,+ "2": 0.7937+ },+ "confidence": 0.4219,+ "answer_confidence": 0.7937,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9824,+ "confidence": 0.9824,+ "answer_confidence": 0.9824,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 589,+ "output_tokens": 0,+ "state_tokens": 165,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8605,+ "technical": 0.0968,+ "other": 0.0428+ },+ "confidence": 0.5539,+ "answer_confidence": 0.8605,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.7395,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0594,+ "1": 0.1416,+ "2": 0.7989+ },+ "confidence": 0.432,+ "answer_confidence": 0.7989,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9368,+ "confidence": 0.9368,+ "answer_confidence": 0.9368,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 301,+ "output_tokens": 0,+ "state_tokens": 69,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9048,+ "technical": 0.064,+ "other": 0.0312+ },+ "confidence": 0.6591,+ "answer_confidence": 0.9048,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.7761,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0368,+ "1": 0.1503,+ "2": 0.8129+ },+ "confidence": 0.4768,+ "answer_confidence": 0.8129,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9441,+ "confidence": 0.9441,+ "answer_confidence": 0.9441,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 427,+ "output_tokens": 0,+ "state_tokens": 111,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8355,+ "technical": 0.1139,+ "other": 0.0506+ },+ "confidence": 0.5008,+ "answer_confidence": 0.8355,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.8578,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0339,+ "1": 0.0745,+ "2": 0.8916+ },+ "confidence": 0.6264,+ "answer_confidence": 0.8916,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.968,+ "confidence": 0.968,+ "answer_confidence": 0.968,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8943,+ "technical": 0.071,+ "other": 0.0347+ },+ "confidence": 0.632,+ "answer_confidence": 0.8943,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.7077,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0712,+ "1": 0.1499,+ "2": 0.7789+ },+ "confidence": 0.3927,+ "answer_confidence": 0.7789,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.973,+ "confidence": 0.973,+ "answer_confidence": 0.973,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 517,+ "output_tokens": 0,+ "state_tokens": 141,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8917,+ "technical": 0.0738,+ "other": 0.0345+ },+ "confidence": 0.6261,+ "answer_confidence": 0.8917,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.7692,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0487,+ "1": 0.1333,+ "2": 0.818+ },+ "confidence": 0.4719,+ "answer_confidence": 0.818,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9284,+ "confidence": 0.9284,+ "answer_confidence": 0.9284,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 355,+ "output_tokens": 0,+ "state_tokens": 87,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8991,+ "technical": 0.0687,+ "other": 0.0322+ },+ "confidence": 0.6449,+ "answer_confidence": 0.8991,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.7492,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0463,+ "1": 0.1583,+ "2": 0.7955+ },+ "confidence": 0.4393,+ "answer_confidence": 0.7955,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9494,+ "confidence": 0.9494,+ "answer_confidence": 0.9494,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 463,+ "output_tokens": 0,+ "state_tokens": 123,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 9,+ 121+ ],+ [+ 9,+ 175+ ],+ [+ 3,+ 199+ ]+ ]+}diff --git a/laya-rust/tests/golden/english/case_batch_min_confidence_map_default.json b/laya-rust/tests/golden/english/case_batch_min_confidence_map_default.jsonnew file mode 100644index 00000000..9e741685--- /dev/null+++ b/laya-rust/tests/golden/english/case_batch_min_confidence_map_default.json@@ -0,0 +1,237 @@+{+ "name": "batch_min_confidence_map_default",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, the refund arrived and everything works now.",+ {+ "ticket": 4821,+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "+ }+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": {+ "default": 0.55+ }+ },+ "state_widths": [+ 91,+ 44,+ 157+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8355,+ "technical": 0.1139,+ "other": 0.0506+ },+ "confidence": 0.5008,+ "answer_confidence": 0.8355,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 1.8578,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0339,+ "1": 0.0745,+ "2": 0.8916+ },+ "confidence": 0.6264,+ "answer_confidence": 0.8916,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "cancel": {+ "type": "noul",+ "noul": 0.968,+ "confidence": 0.968,+ "answer_confidence": 0.968,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8178,+ "technical": 0.1682,+ "other": 0.014+ },+ "confidence": 0.5229,+ "answer_confidence": 0.8178,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 0.7428,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.5839,+ "1": 0.0894,+ "2": 0.3267+ },+ "confidence": 0.1849,+ "answer_confidence": 0.5839,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0144,+ "confidence": 0.9856,+ "answer_confidence": 0.9856,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 124,+ "output_tokens": 0,+ "state_tokens": 10,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7468,+ "technical": 0.1514,+ "other": 0.1017+ },+ "confidence": 0.3297,+ "answer_confidence": 0.7468,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 1.6364,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1029,+ "1": 0.1578,+ "2": 0.7393+ },+ "confidence": 0.3185,+ "answer_confidence": 0.7393,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9238,+ "confidence": 0.9238,+ "answer_confidence": 0.9238,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 463,+ "output_tokens": 0,+ "state_tokens": 123,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 9,+ 157+ ]+ ]+}diff --git a/laya-rust/tests/golden/english/case_batch_min_confidence_scalar.json b/laya-rust/tests/golden/english/case_batch_min_confidence_scalar.jsonnew file mode 100644index 00000000..c111a0d8--- /dev/null+++ b/laya-rust/tests/golden/english/case_batch_min_confidence_scalar.json@@ -0,0 +1,571 @@+{+ "name": "batch_min_confidence_scalar",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, that is all sorted now.",+ {+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",+ "message": "Please cancel my account today."+ },+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.6+ },+ "state_widths": [+ 91,+ 110,+ 144,+ 34,+ 163,+ 42,+ 98,+ 56+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8355,+ "technical": 0.1139,+ "other": 0.0506+ },+ "confidence": 0.5008,+ "answer_confidence": 0.8355,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.8578,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0339,+ "1": 0.0745,+ "2": 0.8916+ },+ "confidence": 0.6264,+ "answer_confidence": 0.8916,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.968,+ "confidence": 0.968,+ "answer_confidence": 0.968,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7826,+ "technical": 0.1583,+ "other": 0.0591+ },+ "confidence": 0.4076,+ "answer_confidence": 0.7826,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.7512,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0438,+ "1": 0.1611,+ "2": 0.795+ },+ "confidence": 0.4415,+ "answer_confidence": 0.795,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9312,+ "confidence": 0.9312,+ "answer_confidence": 0.9312,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 322,+ "output_tokens": 0,+ "state_tokens": 76,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7843,+ "technical": 0.1543,+ "other": 0.0614+ },+ "confidence": 0.4082,+ "answer_confidence": 0.7843,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.7074,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0568,+ "1": 0.179,+ "2": 0.7642+ },+ "confidence": 0.3843,+ "answer_confidence": 0.7642,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.8762,+ "confidence": 0.8762,+ "answer_confidence": 0.8762,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 424,+ "output_tokens": 0,+ "state_tokens": 110,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5445,+ "technical": 0.2427,+ "other": 0.2129+ },+ "confidence": 0.0861,+ "answer_confidence": 0.5445,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 0.6874,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.4819,+ "1": 0.3488,+ "2": 0.1693+ },+ "confidence": 0.0717,+ "answer_confidence": 0.4819,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.3894,+ "confidence": 0.6106,+ "answer_confidence": 0.6106,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 94,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9119,+ "technical": 0.0619,+ "other": 0.0263+ },+ "confidence": 0.6797,+ "answer_confidence": 0.9119,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.8174,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0327,+ "1": 0.1173,+ "2": 0.85+ },+ "confidence": 0.5437,+ "answer_confidence": 0.85,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9744,+ "confidence": 0.9744,+ "answer_confidence": 0.9744,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 481,+ "output_tokens": 0,+ "state_tokens": 129,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "technical",+ "probabilities": {+ "refund": 0.2918,+ "technical": 0.398,+ "other": 0.3102+ },+ "confidence": 0.0086,+ "answer_confidence": 0.398,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 0.7918,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.5403,+ "1": 0.1276,+ "2": 0.3321+ },+ "confidence": 0.1249,+ "answer_confidence": 0.5403,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0548,+ "confidence": 0.9452,+ "answer_confidence": 0.9452,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 118,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.573,+ "technical": 0.1873,+ "other": 0.2397+ },+ "confidence": 0.1124,+ "answer_confidence": 0.573,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.8166,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0486,+ "1": 0.0862,+ "2": 0.8652+ },+ "confidence": 0.5599,+ "answer_confidence": 0.8652,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.8598,+ "confidence": 0.8598,+ "answer_confidence": 0.8598,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 286,+ "output_tokens": 0,+ "state_tokens": 64,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5426,+ "technical": 0.3484,+ "other": 0.109+ },+ "confidence": 0.1438,+ "answer_confidence": 0.5426,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 0.516,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.6281,+ "1": 0.2278,+ "2": 0.1441+ },+ "confidence": 0.1733,+ "answer_confidence": 0.6281,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.3136,+ "confidence": 0.6864,+ "answer_confidence": 0.6864,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 160,+ "output_tokens": 0,+ "state_tokens": 22,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 24,+ 163+ ]+ ]+}diff --git a/laya-rust/tests/golden/english/case_batch_min_confidence_zero.json b/laya-rust/tests/golden/english/case_batch_min_confidence_zero.jsonnew file mode 100644index 00000000..8f9f1a21--- /dev/null+++ b/laya-rust/tests/golden/english/case_batch_min_confidence_zero.json@@ -0,0 +1,565 @@+{+ "name": "batch_min_confidence_zero",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, that is all sorted now.",+ {+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",+ "message": "Please cancel my account today."+ },+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.0+ },+ "state_widths": [+ 91,+ 110,+ 144,+ 34,+ 163,+ 42,+ 98,+ 56+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8355,+ "technical": 0.1139,+ "other": 0.0506+ },+ "confidence": 0.5008,+ "answer_confidence": 0.8355,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.8578,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0339,+ "1": 0.0745,+ "2": 0.8916+ },+ "confidence": 0.6264,+ "answer_confidence": 0.8916,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.968,+ "confidence": 0.968,+ "answer_confidence": 0.968,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7826,+ "technical": 0.1583,+ "other": 0.0591+ },+ "confidence": 0.4076,+ "answer_confidence": 0.7826,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.7512,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0438,+ "1": 0.1611,+ "2": 0.795+ },+ "confidence": 0.4415,+ "answer_confidence": 0.795,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9312,+ "confidence": 0.9312,+ "answer_confidence": 0.9312,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 322,+ "output_tokens": 0,+ "state_tokens": 76,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7843,+ "technical": 0.1543,+ "other": 0.0614+ },+ "confidence": 0.4082,+ "answer_confidence": 0.7843,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.7074,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0568,+ "1": 0.179,+ "2": 0.7642+ },+ "confidence": 0.3843,+ "answer_confidence": 0.7642,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.8762,+ "confidence": 0.8762,+ "answer_confidence": 0.8762,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 424,+ "output_tokens": 0,+ "state_tokens": 110,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5445,+ "technical": 0.2427,+ "other": 0.2129+ },+ "confidence": 0.0861,+ "answer_confidence": 0.5445,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 0.6874,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.4819,+ "1": 0.3488,+ "2": 0.1693+ },+ "confidence": 0.0717,+ "answer_confidence": 0.4819,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.3894,+ "confidence": 0.6106,+ "answer_confidence": 0.6106,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 94,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9119,+ "technical": 0.0619,+ "other": 0.0263+ },+ "confidence": 0.6797,+ "answer_confidence": 0.9119,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.8174,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0327,+ "1": 0.1173,+ "2": 0.85+ },+ "confidence": 0.5437,+ "answer_confidence": 0.85,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9744,+ "confidence": 0.9744,+ "answer_confidence": 0.9744,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 481,+ "output_tokens": 0,+ "state_tokens": 129,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "technical",+ "probabilities": {+ "refund": 0.2918,+ "technical": 0.398,+ "other": 0.3102+ },+ "confidence": 0.0086,+ "answer_confidence": 0.398,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 0.7918,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.5403,+ "1": 0.1276,+ "2": 0.3321+ },+ "confidence": 0.1249,+ "answer_confidence": 0.5403,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0548,+ "confidence": 0.9452,+ "answer_confidence": 0.9452,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 118,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.573,+ "technical": 0.1873,+ "other": 0.2397+ },+ "confidence": 0.1124,+ "answer_confidence": 0.573,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.8166,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0486,+ "1": 0.0862,+ "2": 0.8652+ },+ "confidence": 0.5599,+ "answer_confidence": 0.8652,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.8598,+ "confidence": 0.8598,+ "answer_confidence": 0.8598,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 286,+ "output_tokens": 0,+ "state_tokens": 64,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5426,+ "technical": 0.3484,+ "other": 0.109+ },+ "confidence": 0.1438,+ "answer_confidence": 0.5426,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 0.516,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.6281,+ "1": 0.2278,+ "2": 0.1441+ },+ "confidence": 0.1733,+ "answer_confidence": 0.6281,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.3136,+ "confidence": 0.6864,+ "answer_confidence": 0.6864,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 160,+ "output_tokens": 0,+ "state_tokens": 22,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 24,+ 163+ ]+ ]+}diff --git a/laya-rust/tests/golden/english/case_min_confidence_bad_key.json b/laya-rust/tests/golden/english/case_min_confidence_bad_key.jsonnew file mode 100644index 00000000..7eab72d1--- /dev/null+++ b/laya-rust/tests/golden/english/case_min_confidence_bad_key.json@@ -0,0 +1,45 @@+{+ "name": "min_confidence_bad_key",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'",+ "options": {+ "min_confidence": {+ "default": 0.5,+ "choice:3": 0.5+ }+ }+}diff --git a/laya-rust/tests/golden/english/case_min_confidence_empty_map.json b/laya-rust/tests/golden/english/case_min_confidence_empty_map.jsonnew file mode 100644index 00000000..28fa69e2--- /dev/null+++ b/laya-rust/tests/golden/english/case_min_confidence_empty_map.json@@ -0,0 +1,42 @@+{+ "name": "min_confidence_empty_map",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "a min_confidence map must be a non-empty dict of bucket -> float, got {}",+ "options": {+ "min_confidence": {}+ }+}diff --git a/laya-rust/tests/golden/english/case_min_confidence_invalid.json b/laya-rust/tests/golden/english/case_min_confidence_invalid.jsonnew file mode 100644index 00000000..5ecd652a--- /dev/null+++ b/laya-rust/tests/golden/english/case_min_confidence_invalid.json@@ -0,0 +1,42 @@+{+ "name": "min_confidence_invalid",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5",+ "options": {+ "min_confidence": 1.5+ }+}diff --git a/laya-rust/tests/golden/english/case_min_confidence_map.json b/laya-rust/tests/golden/english/case_min_confidence_map.jsonnew file mode 100644index 00000000..c531def3--- /dev/null+++ b/laya-rust/tests/golden/english/case_min_confidence_map.json@@ -0,0 +1,1524 @@+{+ "name": "min_confidence_map",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 4.488276958465576,+ -2.729666233062744,+ -2.4480319023132324,+ -10000.0+ ],+ [+ -0.8349542617797852,+ 0.8832842707633972,+ 3.6844465732574463,+ 1.2054039239883423+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0+ ],+ [+ 0.7536231279373169,+ 0.41879838705062866,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4074.86572265625,+ -3334.4990234375+ ],+ [+ 4040.2958984375,+ -3316.3017578125+ ],+ [+ 2926.997314453125,+ -2398.78125+ ],+ [+ 4615.162109375,+ -3778.93896484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9653,+ "technical": 0.016,+ "account": 0.0188+ },+ "confidence": 0.8409,+ "answer_confidence": 0.9653,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.9+ },+ "urgency": {+ "type": "score",+ "score": 1.9821,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0212,+ "1": 0.0839,+ "2": 0.7864,+ "3": 0.1085+ },+ "confidence": 0.4809,+ "answer_confidence": 0.7864,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.968,+ "confidence": 0.968,+ "answer_confidence": 0.968,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.4579,+ "confidence": 0.5421,+ "answer_confidence": 0.5421,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": {+ "choice:3-5": 0.9,+ "score:3-5": 0.5,+ "noul:2": 0.7+ }+ }+}diff --git a/laya-rust/tests/golden/english/case_min_confidence_map_default.json b/laya-rust/tests/golden/english/case_min_confidence_map_default.jsonnew file mode 100644index 00000000..65d5dbe1--- /dev/null+++ b/laya-rust/tests/golden/english/case_min_confidence_map_default.json@@ -0,0 +1,935 @@+{+ "name": "min_confidence_map_default",+ "state": "Thanks, that is all sorted now.",+ "serialized_state": "Thanks, that is all sorted now.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 0.7524149417877197,+ 0.5028965473175049,+ -0.09920725226402283,+ -10000.0+ ],+ [+ 2.5853981971740723,+ 1.4106674194335938,+ 2.1949286460876465,+ -0.39206796884536743+ ],+ [+ 5.697894096374512,+ 0.04901987314224243,+ -10000.0,+ -10000.0+ ],+ [+ 0.0804542601108551,+ 0.9528194665908813,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4516.2138671875,+ -3693.35986328125+ ],+ [+ 3973.955078125,+ -3267.04638671875+ ],+ [+ 4118.57275390625,+ -3372.958984375+ ],+ [+ 4505.318359375,+ -3688.783203125+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.4025,+ "technical": 0.3493,+ "account": 0.2481+ },+ "confidence": 0.0174,+ "answer_confidence": 0.4025,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 0.9626,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.4513,+ "1": 0.1765,+ "2": 0.3304,+ "3": 0.0418+ },+ "confidence": 0.1605,+ "answer_confidence": 0.4513,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.55+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0548,+ "confidence": 0.9452,+ "answer_confidence": 0.9452,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.6082,+ "confidence": 0.6082,+ "answer_confidence": 0.6082,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 183,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": {+ "default": 0.55+ }+ }+}diff --git a/laya-rust/tests/golden/english/case_min_confidence_scalar.json b/laya-rust/tests/golden/english/case_min_confidence_scalar.jsonnew file mode 100644index 00000000..8b4fd11f--- /dev/null+++ b/laya-rust/tests/golden/english/case_min_confidence_scalar.json@@ -0,0 +1,1520 @@+{+ "name": "min_confidence_scalar",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 4.488276958465576,+ -2.729666233062744,+ -2.4480319023132324,+ -10000.0+ ],+ [+ -0.8349542617797852,+ 0.8832842707633972,+ 3.6844465732574463,+ 1.2054039239883423+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0+ ],+ [+ 0.7536231279373169,+ 0.41879838705062866,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4074.86572265625,+ -3334.4990234375+ ],+ [+ 4040.2958984375,+ -3316.3017578125+ ],+ [+ 2926.997314453125,+ -2398.78125+ ],+ [+ 4615.162109375,+ -3778.93896484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9653,+ "technical": 0.016,+ "account": 0.0188+ },+ "confidence": 0.8409,+ "answer_confidence": 0.9653,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.9821,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0212,+ "1": 0.0839,+ "2": 0.7864,+ "3": 0.1085+ },+ "confidence": 0.4809,+ "answer_confidence": 0.7864,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.968,+ "confidence": 0.968,+ "answer_confidence": 0.968,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.4579,+ "confidence": 0.5421,+ "answer_confidence": 0.5421,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.6+ }+}diff --git a/laya-rust/tests/golden/english/case_min_confidence_zero.json b/laya-rust/tests/golden/english/case_min_confidence_zero.jsonnew file mode 100644index 00000000..5eb00a71--- /dev/null+++ b/laya-rust/tests/golden/english/case_min_confidence_zero.json@@ -0,0 +1,1519 @@+{+ "name": "min_confidence_zero",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 4.488276958465576,+ -2.729666233062744,+ -2.4480319023132324,+ -10000.0+ ],+ [+ -0.8349542617797852,+ 0.8832842707633972,+ 3.6844465732574463,+ 1.2054039239883423+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0+ ],+ [+ 0.7536231279373169,+ 0.41879838705062866,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4074.86572265625,+ -3334.4990234375+ ],+ [+ 4040.2958984375,+ -3316.3017578125+ ],+ [+ 2926.997314453125,+ -2398.78125+ ],+ [+ 4615.162109375,+ -3778.93896484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9653,+ "technical": 0.016,+ "account": 0.0188+ },+ "confidence": 0.8409,+ "answer_confidence": 0.9653,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.9821,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0212,+ "1": 0.0839,+ "2": 0.7864,+ "3": 0.1085+ },+ "confidence": 0.4809,+ "answer_confidence": 0.7864,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.968,+ "confidence": 0.968,+ "answer_confidence": 0.968,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.4579,+ "confidence": 0.5421,+ "answer_confidence": 0.5421,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.0+ }+}diff --git a/laya-rust/tests/golden/english/index.json b/laya-rust/tests/golden/english/index.jsonindex eb70e281..8b7bcfdf 100644--- a/laya-rust/tests/golden/english/index.json+++ b/laya-rust/tests/golden/english/index.json@@ -337,6 +337,42 @@"expects_error": false,"kind": "batch"},+ {+ "name": "batch_min_confidence_scalar",+ "file": "case_batch_min_confidence_scalar.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_zero",+ "file": "case_batch_min_confidence_zero.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_map",+ "file": "case_batch_min_confidence_map.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_map_default",+ "file": "case_batch_min_confidence_map_default.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_empty_questions",+ "file": "case_batch_min_confidence_empty_questions.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_empty_states",+ "file": "case_batch_min_confidence_empty_states.json",+ "expects_error": false,+ "kind": "batch"+ },{"name": "batch_none_state","file": "case_batch_none_state.json",@@ -348,5 +384,58 @@"file": "case_batch_options_exceed_head.json","expects_error": true,"kind": "batch"+ },+ {+ "name": "batch_min_confidence_invalid",+ "file": "case_batch_min_confidence_invalid.json",+ "expects_error": true,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_before_none_state",+ "file": "case_batch_min_confidence_before_none_state.json",+ "expects_error": true,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_bad_key",+ "file": "case_batch_min_confidence_bad_key.json",+ "expects_error": true,+ "kind": "batch"+ },+ {+ "name": "min_confidence_scalar",+ "file": "case_min_confidence_scalar.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_zero",+ "file": "case_min_confidence_zero.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_map",+ "file": "case_min_confidence_map.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_map_default",+ "file": "case_min_confidence_map_default.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_invalid",+ "file": "case_min_confidence_invalid.json",+ "expects_error": true+ },+ {+ "name": "min_confidence_empty_map",+ "file": "case_min_confidence_empty_map.json",+ "expects_error": true+ },+ {+ "name": "min_confidence_bad_key",+ "file": "case_min_confidence_bad_key.json",+ "expects_error": true}]diff --git a/laya-rust/tests/golden/english/meta.json b/laya-rust/tests/golden/english/meta.jsonindex 250e282f..ffd93825 100644--- a/laya-rust/tests/golden/english/meta.json+++ b/laya-rust/tests/golden/english/meta.json@@ -1,7 +1,7 @@{"generator": "tools/dump_golden.py","onnx_dir": "english",- "laya_version": "0.4.14",+ "laya_version": "0.4.15","onnxruntime_version": "1.30.0","transformers_version": "5.17.0","config": {diff --git a/laya-rust/tests/golden/multilingual/case_batch_min_confidence_bad_key.json b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_bad_key.jsonnew file mode 100644index 00000000..424420ec--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_bad_key.json@@ -0,0 +1,38 @@+{+ "name": "batch_min_confidence_bad_key",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": {+ "choice:2": 0.5,+ "choice:3": 0.5+ }+ },+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'"+}diff --git a/laya-rust/tests/golden/multilingual/case_batch_min_confidence_before_none_state.json b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_before_none_state.jsonnew file mode 100644index 00000000..75fd7564--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_before_none_state.json@@ -0,0 +1,36 @@+{+ "name": "batch_min_confidence_before_none_state",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ null+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": -0.25+ },+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -0.25"+}diff --git a/laya-rust/tests/golden/multilingual/case_batch_min_confidence_empty_questions.json b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_empty_questions.jsonnew file mode 100644index 00000000..255f7bc4--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_empty_questions.json@@ -0,0 +1,44 @@+{+ "name": "batch_min_confidence_empty_questions",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, the refund arrived and everything works now.",+ {+ "ticket": 4821,+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "+ }+ ],+ "questions": {},+ "options": {+ "min_confidence": 0.5+ },+ "state_widths": [],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {},+ "usage": {+ "input_tokens": 0,+ "output_tokens": 0+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {},+ "usage": {+ "input_tokens": 0,+ "output_tokens": 0+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {},+ "usage": {+ "input_tokens": 0,+ "output_tokens": 0+ }+ }+ ],+ "runs": []+}diff --git a/laya-rust/tests/golden/multilingual/case_batch_min_confidence_empty_states.json b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_empty_states.jsonnew file mode 100644index 00000000..dfb141d9--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_empty_states.json@@ -0,0 +1,34 @@+{+ "name": "batch_min_confidence_empty_states",+ "states": [],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.5+ },+ "state_widths": [],+ "results": [],+ "runs": []+}diff --git a/laya-rust/tests/golden/multilingual/case_batch_min_confidence_invalid.json b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_invalid.jsonnew file mode 100644index 00000000..72496b52--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_invalid.json@@ -0,0 +1,35 @@+{+ "name": "batch_min_confidence_invalid",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 1.5+ },+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"+}diff --git a/laya-rust/tests/golden/multilingual/case_batch_min_confidence_map.json b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_map.jsonnew file mode 100644index 00000000..30dee3e1--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_map.json@@ -0,0 +1,497 @@+{+ "name": "batch_min_confidence_map",+ "states": [+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "batch_size": 3,+ "sort_by_length": true,+ "min_confidence": {+ "choice:3-5": 0.8,+ "score:3-5": 0.5,+ "noul:2": 0.7+ }+ },+ "state_widths": [+ 180,+ 100,+ 135,+ 90,+ 160,+ 115,+ 145+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9823,+ "technical": 0.0007,+ "other": 0.017+ },+ "confidence": 0.916,+ "answer_confidence": 0.9823,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.3568,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0709,+ "1": 0.5013,+ "2": 0.4278+ },+ "confidence": 0.1834,+ "answer_confidence": 0.5013,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0867,+ "confidence": 0.9133,+ "answer_confidence": 0.9133,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 534,+ "output_tokens": 0,+ "state_tokens": 146,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9664,+ "technical": 0.0022,+ "other": 0.0314+ },+ "confidence": 0.8589,+ "answer_confidence": 0.9664,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.2262,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0367,+ "1": 0.7004,+ "2": 0.2629+ },+ "confidence": 0.3429,+ "answer_confidence": 0.7004,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0634,+ "confidence": 0.9366,+ "answer_confidence": 0.9366,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 294,+ "output_tokens": 0,+ "state_tokens": 66,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9708,+ "technical": 0.0008,+ "other": 0.0284+ },+ "confidence": 0.8764,+ "answer_confidence": 0.9708,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.6446,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0315,+ "1": 0.2924,+ "2": 0.6761+ },+ "confidence": 0.3327,+ "answer_confidence": 0.6761,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0554,+ "confidence": 0.9446,+ "answer_confidence": 0.9446,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 399,+ "output_tokens": 0,+ "state_tokens": 101,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.836,+ "technical": 0.0114,+ "other": 0.1526+ },+ "confidence": 0.5562,+ "answer_confidence": 0.836,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.3713,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0231,+ "1": 0.5824,+ "2": 0.3945+ },+ "confidence": 0.3001,+ "answer_confidence": 0.5824,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0414,+ "confidence": 0.9586,+ "answer_confidence": 0.9586,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 264,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.972,+ "technical": 0.0011,+ "other": 0.0269+ },+ "confidence": 0.8798,+ "answer_confidence": 0.972,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.5259,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0331,+ "1": 0.4079,+ "2": 0.559+ },+ "confidence": 0.2684,+ "answer_confidence": 0.559,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0972,+ "confidence": 0.9028,+ "answer_confidence": 0.9028,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 474,+ "output_tokens": 0,+ "state_tokens": 126,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9345,+ "technical": 0.0011,+ "other": 0.0644+ },+ "confidence": 0.7748,+ "answer_confidence": 0.9345,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.3357,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0468,+ "1": 0.5707,+ "2": 0.3825+ },+ "confidence": 0.2436,+ "answer_confidence": 0.5707,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.1013,+ "confidence": 0.8987,+ "answer_confidence": 0.8987,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 339,+ "output_tokens": 0,+ "state_tokens": 81,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9765,+ "technical": 0.0006,+ "other": 0.0229+ },+ "confidence": 0.896,+ "answer_confidence": 0.9765,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.6128,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0311,+ "1": 0.325,+ "2": 0.6439+ },+ "confidence": 0.3113,+ "answer_confidence": 0.6439,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0431,+ "confidence": 0.9569,+ "answer_confidence": 0.9569,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 429,+ "output_tokens": 0,+ "state_tokens": 111,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 9,+ 115+ ],+ [+ 9,+ 160+ ],+ [+ 3,+ 180+ ]+ ]+}diff --git a/laya-rust/tests/golden/multilingual/case_batch_min_confidence_map_default.json b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_map_default.jsonnew file mode 100644index 00000000..3f9d2494--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_map_default.json@@ -0,0 +1,238 @@+{+ "name": "batch_min_confidence_map_default",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, the refund arrived and everything works now.",+ {+ "ticket": 4821,+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "+ }+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": {+ "default": 0.55+ }+ },+ "state_widths": [+ 90,+ 44,+ 147+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.836,+ "technical": 0.0114,+ "other": 0.1526+ },+ "confidence": 0.5562,+ "answer_confidence": 0.836,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 1.3713,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0231,+ "1": 0.5824,+ "2": 0.3945+ },+ "confidence": 0.3001,+ "answer_confidence": 0.5824,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0414,+ "confidence": 0.9586,+ "answer_confidence": 0.9586,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 264,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9999,+ "technical": 0.0,+ "other": 0.0+ },+ "confidence": 0.9993,+ "answer_confidence": 0.9999,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 1.6231,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0528,+ "1": 0.2714,+ "2": 0.6759+ },+ "confidence": 0.2956,+ "answer_confidence": 0.6759,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0067,+ "confidence": 0.9933,+ "answer_confidence": 0.9933,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 126,+ "output_tokens": 0,+ "state_tokens": 10,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.2098,+ "technical": 0.0637,+ "other": 0.7266+ },+ "confidence": 0.331,+ "answer_confidence": 0.7266,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 1.3977,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0308,+ "1": 0.5407,+ "2": 0.4285+ },+ "confidence": 0.2692,+ "answer_confidence": 0.5407,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.55+ },+ "cancel": {+ "type": "noul",+ "noul": 0.1873,+ "confidence": 0.8127,+ "answer_confidence": 0.8127,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 435,+ "output_tokens": 0,+ "state_tokens": 113,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 9,+ 147+ ]+ ]+}diff --git a/laya-rust/tests/golden/multilingual/case_batch_min_confidence_scalar.json b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_scalar.jsonnew file mode 100644index 00000000..3362d652--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_scalar.json@@ -0,0 +1,572 @@+{+ "name": "batch_min_confidence_scalar",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, that is all sorted now.",+ {+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",+ "message": "Please cancel my account today."+ },+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.6+ },+ "state_widths": [+ 90,+ 113,+ 143,+ 34,+ 150,+ 42,+ 90,+ 48+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.836,+ "technical": 0.0114,+ "other": 0.1526+ },+ "confidence": 0.5562,+ "answer_confidence": 0.836,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.3713,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0231,+ "1": 0.5824,+ "2": 0.3945+ },+ "confidence": 0.3001,+ "answer_confidence": 0.5824,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0414,+ "confidence": 0.9586,+ "answer_confidence": 0.9586,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 264,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6527,+ "technical": 0.0268,+ "other": 0.3205+ },+ "confidence": 0.3264,+ "answer_confidence": 0.6527,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.4526,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0142,+ "1": 0.519,+ "2": 0.4668+ },+ "confidence": 0.3115,+ "answer_confidence": 0.519,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.3678,+ "confidence": 0.6322,+ "answer_confidence": 0.6322,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 333,+ "output_tokens": 0,+ "state_tokens": 79,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.2035,+ "technical": 0.1274,+ "other": 0.6691+ },+ "confidence": 0.2215,+ "answer_confidence": 0.6691,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.3642,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0252,+ "1": 0.5854,+ "2": 0.3894+ },+ "confidence": 0.2959,+ "answer_confidence": 0.5854,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0571,+ "confidence": 0.9429,+ "answer_confidence": 0.9429,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 423,+ "output_tokens": 0,+ "state_tokens": 109,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.806,+ "technical": 0.1113,+ "other": 0.0827+ },+ "confidence": 0.4317,+ "answer_confidence": 0.806,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.1334,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1501,+ "1": 0.5664,+ "2": 0.2835+ },+ "confidence": 0.1225,+ "answer_confidence": 0.5664,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0063,+ "confidence": 0.9937,+ "answer_confidence": 0.9937,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 96,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9867,+ "technical": 0.0005,+ "other": 0.0129+ },+ "confidence": 0.9336,+ "answer_confidence": 0.9867,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.5707,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0398,+ "1": 0.3498,+ "2": 0.6105+ },+ "confidence": 0.2746,+ "answer_confidence": 0.6105,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0882,+ "confidence": 0.9118,+ "answer_confidence": 0.9118,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 444,+ "output_tokens": 0,+ "state_tokens": 116,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.0595,+ "technical": 0.0895,+ "other": 0.851+ },+ "confidence": 0.5256,+ "answer_confidence": 0.851,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.3418,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1574,+ "1": 0.3433,+ "2": 0.4993+ },+ "confidence": 0.0853,+ "answer_confidence": 0.4993,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0056,+ "confidence": 0.9944,+ "answer_confidence": 0.9944,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 120,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.468,+ "technical": 0.301,+ "other": 0.231+ },+ "confidence": 0.0395,+ "answer_confidence": 0.468,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.8737,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0305,+ "1": 0.0652,+ "2": 0.9042+ },+ "confidence": 0.6581,+ "answer_confidence": 0.9042,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.4844,+ "confidence": 0.5156,+ "answer_confidence": 0.5156,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 264,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9822,+ "technical": 0.0057,+ "other": 0.012+ },+ "confidence": 0.9086,+ "answer_confidence": 0.9822,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.7021,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0303,+ "1": 0.2373,+ "2": 0.7324+ },+ "confidence": 0.3852,+ "answer_confidence": 0.7324,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.1059,+ "confidence": 0.8941,+ "answer_confidence": 0.8941,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 138,+ "output_tokens": 0,+ "state_tokens": 14,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 24,+ 150+ ]+ ]+}diff --git a/laya-rust/tests/golden/multilingual/case_batch_min_confidence_zero.json b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_zero.jsonnew file mode 100644index 00000000..bbea5c84--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_batch_min_confidence_zero.json@@ -0,0 +1,565 @@+{+ "name": "batch_min_confidence_zero",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, that is all sorted now.",+ {+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",+ "message": "Please cancel my account today."+ },+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.0+ },+ "state_widths": [+ 90,+ 113,+ 143,+ 34,+ 150,+ 42,+ 90,+ 48+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.836,+ "technical": 0.0114,+ "other": 0.1526+ },+ "confidence": 0.5562,+ "answer_confidence": 0.836,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.3713,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0231,+ "1": 0.5824,+ "2": 0.3945+ },+ "confidence": 0.3001,+ "answer_confidence": 0.5824,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0414,+ "confidence": 0.9586,+ "answer_confidence": 0.9586,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 264,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6527,+ "technical": 0.0268,+ "other": 0.3205+ },+ "confidence": 0.3264,+ "answer_confidence": 0.6527,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.4526,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0142,+ "1": 0.519,+ "2": 0.4668+ },+ "confidence": 0.3115,+ "answer_confidence": 0.519,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.3678,+ "confidence": 0.6322,+ "answer_confidence": 0.6322,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 333,+ "output_tokens": 0,+ "state_tokens": 79,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.2035,+ "technical": 0.1274,+ "other": 0.6691+ },+ "confidence": 0.2215,+ "answer_confidence": 0.6691,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.3642,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0252,+ "1": 0.5854,+ "2": 0.3894+ },+ "confidence": 0.2959,+ "answer_confidence": 0.5854,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0571,+ "confidence": 0.9429,+ "answer_confidence": 0.9429,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 423,+ "output_tokens": 0,+ "state_tokens": 109,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.806,+ "technical": 0.1113,+ "other": 0.0827+ },+ "confidence": 0.4317,+ "answer_confidence": 0.806,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.1334,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1501,+ "1": 0.5664,+ "2": 0.2835+ },+ "confidence": 0.1225,+ "answer_confidence": 0.5664,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0063,+ "confidence": 0.9937,+ "answer_confidence": 0.9937,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 96,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9867,+ "technical": 0.0005,+ "other": 0.0129+ },+ "confidence": 0.9336,+ "answer_confidence": 0.9867,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.5707,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0398,+ "1": 0.3498,+ "2": 0.6105+ },+ "confidence": 0.2746,+ "answer_confidence": 0.6105,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0882,+ "confidence": 0.9118,+ "answer_confidence": 0.9118,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 444,+ "output_tokens": 0,+ "state_tokens": 116,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.0595,+ "technical": 0.0895,+ "other": 0.851+ },+ "confidence": 0.5256,+ "answer_confidence": 0.851,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.3418,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1574,+ "1": 0.3433,+ "2": 0.4993+ },+ "confidence": 0.0853,+ "answer_confidence": 0.4993,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0056,+ "confidence": 0.9944,+ "answer_confidence": 0.9944,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 120,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.468,+ "technical": 0.301,+ "other": 0.231+ },+ "confidence": 0.0395,+ "answer_confidence": 0.468,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.8737,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0305,+ "1": 0.0652,+ "2": 0.9042+ },+ "confidence": 0.6581,+ "answer_confidence": 0.9042,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.4844,+ "confidence": 0.5156,+ "answer_confidence": 0.5156,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 264,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9822,+ "technical": 0.0057,+ "other": 0.012+ },+ "confidence": 0.9086,+ "answer_confidence": 0.9822,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.7021,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0303,+ "1": 0.2373,+ "2": 0.7324+ },+ "confidence": 0.3852,+ "answer_confidence": 0.7324,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.1059,+ "confidence": 0.8941,+ "answer_confidence": 0.8941,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 138,+ "output_tokens": 0,+ "state_tokens": 14,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 24,+ 150+ ]+ ]+}diff --git a/laya-rust/tests/golden/multilingual/case_min_confidence_bad_key.json b/laya-rust/tests/golden/multilingual/case_min_confidence_bad_key.jsonnew file mode 100644index 00000000..7eab72d1--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_min_confidence_bad_key.json@@ -0,0 +1,45 @@+{+ "name": "min_confidence_bad_key",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'",+ "options": {+ "min_confidence": {+ "default": 0.5,+ "choice:3": 0.5+ }+ }+}diff --git a/laya-rust/tests/golden/multilingual/case_min_confidence_empty_map.json b/laya-rust/tests/golden/multilingual/case_min_confidence_empty_map.jsonnew file mode 100644index 00000000..28fa69e2--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_min_confidence_empty_map.json@@ -0,0 +1,42 @@+{+ "name": "min_confidence_empty_map",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "a min_confidence map must be a non-empty dict of bucket -> float, got {}",+ "options": {+ "min_confidence": {}+ }+}diff --git a/laya-rust/tests/golden/multilingual/case_min_confidence_invalid.json b/laya-rust/tests/golden/multilingual/case_min_confidence_invalid.jsonnew file mode 100644index 00000000..5ecd652a--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_min_confidence_invalid.json@@ -0,0 +1,42 @@+{+ "name": "min_confidence_invalid",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5",+ "options": {+ "min_confidence": 1.5+ }+}diff --git a/laya-rust/tests/golden/multilingual/case_min_confidence_map.json b/laya-rust/tests/golden/multilingual/case_min_confidence_map.jsonnew file mode 100644index 00000000..61b29433--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_min_confidence_map.json@@ -0,0 +1,1488 @@+{+ "name": "min_confidence_map",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 7.727238655090332,+ -4.555615425109863,+ -2.0609042644500732,+ -10000.0+ ],+ [+ -3.1127467155456543,+ 0.14541220664978027,+ -0.22580814361572266,+ -0.7165543437004089+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0+ ],+ [+ 1.247997522354126,+ -0.26682233810424805,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1252.7354736328125,+ -1494.896240234375+ ],+ [+ 1394.948486328125,+ -1645.8040771484375+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ],+ [+ 1266.344970703125,+ -1504.056396484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9999,+ "technical": 0.0,+ "account": 0.0001+ },+ "confidence": 0.9994,+ "answer_confidence": 0.9999,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.9+ },+ "urgency": {+ "type": "score",+ "score": 1.6956,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0179,+ "1": 0.465,+ "2": 0.3208,+ "3": 0.1964+ },+ "confidence": 0.1976,+ "answer_confidence": 0.465,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0414,+ "confidence": 0.9586,+ "answer_confidence": 0.9586,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.1802,+ "confidence": 0.8198,+ "answer_confidence": 0.8198,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 375,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": {+ "choice:3-5": 0.9,+ "score:3-5": 0.5,+ "noul:2": 0.7+ }+ }+}diff --git a/laya-rust/tests/golden/multilingual/case_min_confidence_map_default.json b/laya-rust/tests/golden/multilingual/case_min_confidence_map_default.jsonnew file mode 100644index 00000000..b7dba7b1--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_min_confidence_map_default.json@@ -0,0 +1,911 @@+{+ "name": "min_confidence_map_default",+ "state": "Thanks, that is all sorted now.",+ "serialized_state": "Thanks, that is all sorted now.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 7807,+ 235269,+ 674,+ 603,+ 832,+ 24087,+ 1490,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 7807,+ 235269,+ 674,+ 603,+ 832,+ 24087,+ 1490,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 7807,+ 235269,+ 674,+ 603,+ 832,+ 24087,+ 1490,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 7807,+ 235269,+ 674,+ 603,+ 832,+ 24087,+ 1490,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 7807,+ 235269,+ 674,+ 603,+ 832,+ 24087,+ 1490,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 7807,+ 235269,+ 674,+ 603,+ 832,+ 24087,+ 1490,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 7807,+ 235269,+ 674,+ 603,+ 832,+ 24087,+ 1490,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 7807,+ 235269,+ 674,+ 603,+ 832,+ 24087,+ 1490,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ -1.6179548501968384,+ -1.8165647983551025,+ -1.123018503189087,+ -10000.0+ ],+ [+ -0.4120713472366333,+ 0.4631410241127014,+ -0.00779646635055542,+ 0.054005563259124756+ ],+ [+ 4.764176845550537,+ -0.41596102714538574,+ -10000.0,+ -10000.0+ ],+ [+ 0.335951566696167,+ -0.49896663427352905,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1420.1611328125,+ -1689.47265625+ ],+ [+ 1465.701416015625,+ -1730.4908447265625+ ],+ [+ 1524.2642822265625,+ -1806.6566162109375+ ],+ [+ 1414.691650390625,+ -1679.205322265625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "account",+ "probabilities": {+ "billing": 0.289,+ "technical": 0.2369,+ "account": 0.4741+ },+ "confidence": 0.0408,+ "answer_confidence": 0.4741,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 1.5678,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.1541,+ "1": 0.3696,+ "2": 0.2308,+ "3": 0.2455+ },+ "confidence": 0.034,+ "answer_confidence": 0.3696,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.55+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0056,+ "confidence": 0.9944,+ "answer_confidence": 0.9944,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.3026,+ "confidence": 0.6974,+ "answer_confidence": 0.6974,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 183,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": {+ "default": 0.55+ }+ }+}diff --git a/laya-rust/tests/golden/multilingual/case_min_confidence_scalar.json b/laya-rust/tests/golden/multilingual/case_min_confidence_scalar.jsonnew file mode 100644index 00000000..e73f1194--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_min_confidence_scalar.json@@ -0,0 +1,1484 @@+{+ "name": "min_confidence_scalar",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 7.727238655090332,+ -4.555615425109863,+ -2.0609042644500732,+ -10000.0+ ],+ [+ -3.1127467155456543,+ 0.14541220664978027,+ -0.22580814361572266,+ -0.7165543437004089+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0+ ],+ [+ 1.247997522354126,+ -0.26682233810424805,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1252.7354736328125,+ -1494.896240234375+ ],+ [+ 1394.948486328125,+ -1645.8040771484375+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ],+ [+ 1266.344970703125,+ -1504.056396484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9999,+ "technical": 0.0,+ "account": 0.0001+ },+ "confidence": 0.9994,+ "answer_confidence": 0.9999,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.6956,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0179,+ "1": 0.465,+ "2": 0.3208,+ "3": 0.1964+ },+ "confidence": 0.1976,+ "answer_confidence": 0.465,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0414,+ "confidence": 0.9586,+ "answer_confidence": 0.9586,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.1802,+ "confidence": 0.8198,+ "answer_confidence": 0.8198,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 375,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.6+ }+}diff --git a/laya-rust/tests/golden/multilingual/case_min_confidence_zero.json b/laya-rust/tests/golden/multilingual/case_min_confidence_zero.jsonnew file mode 100644index 00000000..a3687573--- /dev/null+++ b/laya-rust/tests/golden/multilingual/case_min_confidence_zero.json@@ -0,0 +1,1483 @@+{+ "name": "min_confidence_zero",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 7.727238655090332,+ -4.555615425109863,+ -2.0609042644500732,+ -10000.0+ ],+ [+ -3.1127467155456543,+ 0.14541220664978027,+ -0.22580814361572266,+ -0.7165543437004089+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0+ ],+ [+ 1.247997522354126,+ -0.26682233810424805,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1252.7354736328125,+ -1494.896240234375+ ],+ [+ 1394.948486328125,+ -1645.8040771484375+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ],+ [+ 1266.344970703125,+ -1504.056396484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9999,+ "technical": 0.0,+ "account": 0.0001+ },+ "confidence": 0.9994,+ "answer_confidence": 0.9999,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.6956,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0179,+ "1": 0.465,+ "2": 0.3208,+ "3": 0.1964+ },+ "confidence": 0.1976,+ "answer_confidence": 0.465,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0414,+ "confidence": 0.9586,+ "answer_confidence": 0.9586,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.1802,+ "confidence": 0.8198,+ "answer_confidence": 0.8198,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 375,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.0+ }+}diff --git a/laya-rust/tests/golden/multilingual/index.json b/laya-rust/tests/golden/multilingual/index.jsonindex eb70e281..8b7bcfdf 100644--- a/laya-rust/tests/golden/multilingual/index.json+++ b/laya-rust/tests/golden/multilingual/index.json@@ -337,6 +337,42 @@"expects_error": false,"kind": "batch"},+ {+ "name": "batch_min_confidence_scalar",+ "file": "case_batch_min_confidence_scalar.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_zero",+ "file": "case_batch_min_confidence_zero.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_map",+ "file": "case_batch_min_confidence_map.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_map_default",+ "file": "case_batch_min_confidence_map_default.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_empty_questions",+ "file": "case_batch_min_confidence_empty_questions.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_empty_states",+ "file": "case_batch_min_confidence_empty_states.json",+ "expects_error": false,+ "kind": "batch"+ },{"name": "batch_none_state","file": "case_batch_none_state.json",@@ -348,5 +384,58 @@"file": "case_batch_options_exceed_head.json","expects_error": true,"kind": "batch"+ },+ {+ "name": "batch_min_confidence_invalid",+ "file": "case_batch_min_confidence_invalid.json",+ "expects_error": true,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_before_none_state",+ "file": "case_batch_min_confidence_before_none_state.json",+ "expects_error": true,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_bad_key",+ "file": "case_batch_min_confidence_bad_key.json",+ "expects_error": true,+ "kind": "batch"+ },+ {+ "name": "min_confidence_scalar",+ "file": "case_min_confidence_scalar.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_zero",+ "file": "case_min_confidence_zero.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_map",+ "file": "case_min_confidence_map.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_map_default",+ "file": "case_min_confidence_map_default.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_invalid",+ "file": "case_min_confidence_invalid.json",+ "expects_error": true+ },+ {+ "name": "min_confidence_empty_map",+ "file": "case_min_confidence_empty_map.json",+ "expects_error": true+ },+ {+ "name": "min_confidence_bad_key",+ "file": "case_min_confidence_bad_key.json",+ "expects_error": true}]diff --git a/laya-rust/tests/golden/multilingual/meta.json b/laya-rust/tests/golden/multilingual/meta.jsonindex 279b5e0b..add98031 100644--- a/laya-rust/tests/golden/multilingual/meta.json+++ b/laya-rust/tests/golden/multilingual/meta.json@@ -1,7 +1,7 @@{"generator": "tools/dump_golden.py","onnx_dir": "multilingual",- "laya_version": "0.4.14",+ "laya_version": "0.4.15","onnxruntime_version": "1.30.0","transformers_version": "5.17.0","config": {diff --git a/laya-rust/tests/golden/routing/confidence_probe.json b/laya-rust/tests/golden/routing/confidence_probe.jsonnew file mode 100644index 00000000..793a2b14--- /dev/null+++ b/laya-rust/tests/golden/routing/confidence_probe.json@@ -0,0 +1,4507 @@+{+ "bucket_keys": [+ "choice:11+",+ "choice:2",+ "choice:3-5",+ "choice:6-10",+ "noul:11+",+ "noul:2",+ "noul:3-5",+ "noul:6-10",+ "score:11+",+ "score:2",+ "score:3-5",+ "score:6-10"+ ],+ "gate_states": [+ "passed",+ "abstained",+ "unevaluated"+ ],+ "scalars": [+ {+ "value": 0.0,+ "valid": true,+ "result": 0.0+ },+ {+ "value": 1.0,+ "valid": true,+ "result": 1.0+ },+ {+ "value": 0.5,+ "valid": true,+ "result": 0.5+ },+ {+ "value": 0.35,+ "valid": true,+ "result": 0.35+ },+ {+ "value": 1e-09,+ "valid": true,+ "result": 1e-09+ },+ {+ "value": 0.9999999999,+ "valid": true,+ "result": 0.9999999999+ },+ {+ "value": 1e-05,+ "valid": true,+ "result": 1e-05+ },+ {+ "value": -0.1,+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -0.1"+ },+ {+ "value": 1.5,+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"+ },+ {+ "value": -1e-05,+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -1e-05"+ },+ {+ "value": 1.0000000001,+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.0000000001"+ },+ {+ "value": 2.0,+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 2.0"+ },+ {+ "value": 100.0,+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 100.0"+ },+ {+ "value": 1e+16,+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1e+16"+ },+ {+ "value": 1.2345678901234566e-07,+ "valid": true,+ "result": 1.2345678901234566e-07+ },+ {+ "value": "nan",+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got nan"+ },+ {+ "value": "inf",+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got inf"+ },+ {+ "value": "-inf",+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -inf"+ }+ ],+ "maps": [+ {+ "pairs": [],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "a min_confidence map must be a non-empty dict of bucket -> float, got {}"+ },+ {+ "pairs": [+ [+ "default",+ 0.4+ ]+ ],+ "valid": true,+ "result": [+ [+ "default",+ 0.4+ ]+ ]+ },+ {+ "pairs": [+ [+ "choice:2",+ 0.7+ ],+ [+ "default",+ 0.4+ ]+ ],+ "valid": true,+ "result": [+ [+ "choice:2",+ 0.7+ ],+ [+ "default",+ 0.4+ ]+ ]+ },+ {+ "pairs": [+ [+ "choice:3-5",+ 0.5+ ]+ ],+ "valid": true,+ "result": [+ [+ "choice:3-5",+ 0.5+ ]+ ]+ },+ {+ "pairs": [+ [+ "score:11+",+ 0.0+ ],+ [+ "noul:2",+ 1.0+ ]+ ],+ "valid": true,+ "result": [+ [+ "score:11+",+ 0.0+ ],+ [+ "noul:2",+ 1.0+ ]+ ]+ },+ {+ "pairs": [+ [+ "choice:2",+ 0.05+ ],+ [+ "choice:3-5",+ 0.1+ ],+ [+ "choice:6-10",+ 0.15+ ],+ [+ "choice:11+",+ 0.2+ ],+ [+ "score:2",+ 0.25+ ],+ [+ "score:3-5",+ 0.3+ ],+ [+ "score:6-10",+ 0.35+ ],+ [+ "score:11+",+ 0.4+ ],+ [+ "noul:2",+ 0.45+ ],+ [+ "noul:3-5",+ 0.5+ ],+ [+ "noul:6-10",+ 0.55+ ],+ [+ "noul:11+",+ 0.6+ ],+ [+ "default",+ 0.5+ ]+ ],+ "valid": true,+ "result": [+ [+ "choice:2",+ 0.05+ ],+ [+ "choice:3-5",+ 0.1+ ],+ [+ "choice:6-10",+ 0.15+ ],+ [+ "choice:11+",+ 0.2+ ],+ [+ "score:2",+ 0.25+ ],+ [+ "score:3-5",+ 0.3+ ],+ [+ "score:6-10",+ 0.35+ ],+ [+ "score:11+",+ 0.4+ ],+ [+ "noul:2",+ 0.45+ ],+ [+ "noul:3-5",+ 0.5+ ],+ [+ "noul:6-10",+ 0.55+ ],+ [+ "noul:11+",+ 0.6+ ],+ [+ "default",+ 0.5+ ]+ ]+ },+ {+ "pairs": [+ [+ "choice:2",+ 1.5+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"+ },+ {+ "pairs": [+ [+ "choice:2",+ "nan"+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got nan"+ },+ {+ "pairs": [+ [+ "choice:2",+ -0.1+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -0.1"+ },+ {+ "pairs": [+ [+ "choice:2",+ "inf"+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got inf"+ },+ {+ "pairs": [+ [+ "bogus",+ 0.5+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'bogus'"+ },+ {+ "pairs": [+ [+ "Choice:2",+ 0.5+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'Choice:2'"+ },+ {+ "pairs": [+ [+ "choice:3",+ 0.5+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'"+ },+ {+ "pairs": [+ [+ "choice:2",+ 0.5+ ],+ [+ "bogus",+ 0.5+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'bogus'"+ },+ {+ "pairs": [+ [+ "choice:2",+ 1.5+ ],+ [+ "bogus",+ 0.1+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"+ },+ {+ "pairs": [+ [+ "bogus",+ 1.5+ ],+ [+ "choice:2",+ 0.1+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'bogus'"+ },+ {+ "pairs": [+ [+ "choice:2",+ 0.5+ ],+ [+ "choice:3-5",+ 2.0+ ],+ [+ "nope",+ 0.1+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 2.0"+ },+ {+ "pairs": [+ [+ "it's",+ 0.5+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got \"it's\""+ },+ {+ "pairs": [+ [+ "say \"hi\"",+ 0.5+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'say \"hi\"'"+ },+ {+ "pairs": [+ [+ "it's \"x\"",+ 0.5+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'it\\'s \"x\"'"+ },+ {+ "pairs": [+ [+ "tab\there",+ 0.5+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'tab\\there'"+ },+ {+ "pairs": [+ [+ "café",+ 0.5+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'café'"+ },+ {+ "pairs": [+ [+ "",+ 0.5+ ]+ ],+ "valid": false,+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got ''"+ },+ {+ "pairs": [+ [+ "default",+ 0.5+ ],+ [+ "default",+ 0.25+ ]+ ],+ "valid": true,+ "result": [+ [+ "default",+ 0.25+ ]+ ]+ }+ ],+ "buckets": [+ {+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 1.0+ }+ },+ "bucket": "choice:2"+ },+ {+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ }+ },+ "bucket": "choice:2"+ },+ {+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.3333,+ "b": 0.3333,+ "c": 0.3333+ }+ },+ "bucket": "choice:3-5"+ },+ {+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.2,+ "b": 0.2,+ "c": 0.2,+ "d": 0.2,+ "e": 0.2+ }+ },+ "bucket": "choice:3-5"+ },+ {+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.1667,+ "b": 0.1667,+ "c": 0.1667,+ "d": 0.1667,+ "e": 0.1667,+ "f": 0.1667+ }+ },+ "bucket": "choice:6-10"+ },+ {+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.1,+ "b": 0.1,+ "c": 0.1,+ "d": 0.1,+ "e": 0.1,+ "f": 0.1,+ "g": 0.1,+ "h": 0.1,+ "i": 0.1,+ "j": 0.1+ }+ },+ "bucket": "choice:6-10"+ },+ {+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.0909,+ "b": 0.0909,+ "c": 0.0909,+ "d": 0.0909,+ "e": 0.0909,+ "f": 0.0909,+ "g": 0.0909,+ "h": 0.0909,+ "i": 0.0909,+ "j": 0.0909,+ "k": 0.0909+ }+ },+ "bucket": "choice:11+"+ },+ {+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.0769,+ "b": 0.0769,+ "c": 0.0769,+ "d": 0.0769,+ "e": 0.0769,+ "f": 0.0769,+ "g": 0.0769,+ "h": 0.0769,+ "i": 0.0769,+ "j": 0.0769,+ "k": 0.0769,+ "l": 0.0769,+ "m": 0.0769+ }+ },+ "bucket": "choice:11+"+ },+ {+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 1.0+ }+ },+ "bucket": "score:2"+ },+ {+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.5,+ "1": 0.5+ }+ },+ "bucket": "score:2"+ },+ {+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.3333,+ "1": 0.3333,+ "2": 0.3333+ }+ },+ "bucket": "score:3-5"+ },+ {+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.2,+ "1": 0.2,+ "2": 0.2,+ "3": 0.2,+ "4": 0.2+ }+ },+ "bucket": "score:3-5"+ },+ {+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.1667,+ "1": 0.1667,+ "2": 0.1667,+ "3": 0.1667,+ "4": 0.1667,+ "5": 0.1667+ }+ },+ "bucket": "score:6-10"+ },+ {+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.1,+ "1": 0.1,+ "2": 0.1,+ "3": 0.1,+ "4": 0.1,+ "5": 0.1,+ "6": 0.1,+ "7": 0.1,+ "8": 0.1,+ "9": 0.1+ }+ },+ "bucket": "score:6-10"+ },+ {+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0909,+ "1": 0.0909,+ "2": 0.0909,+ "3": 0.0909,+ "4": 0.0909,+ "5": 0.0909,+ "6": 0.0909,+ "7": 0.0909,+ "8": 0.0909,+ "9": 0.0909,+ "10": 0.0909+ }+ },+ "bucket": "score:11+"+ },+ {+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0769,+ "1": 0.0769,+ "2": 0.0769,+ "3": 0.0769,+ "4": 0.0769,+ "5": 0.0769,+ "6": 0.0769,+ "7": 0.0769,+ "8": 0.0769,+ "9": 0.0769,+ "10": 0.0769,+ "11": 0.0769,+ "12": 0.0769+ }+ },+ "bucket": "score:11+"+ },+ {+ "answer": {+ "type": "noul",+ "noul": 0.9+ },+ "bucket": "noul:2"+ },+ {+ "answer": {+ "type": "choice",+ "probabilities": {}+ },+ "bucket": null+ },+ {+ "answer": {+ "type": "score",+ "probabilities": {}+ },+ "bucket": null+ }+ ],+ "resolve": [+ {+ "thresholds": {+ "choice:2": 0.7,+ "default": 0.4+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ }+ },+ "result": 0.7+ },+ {+ "thresholds": {+ "choice:2": 0.7,+ "default": 0.4+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.25,+ "b": 0.25,+ "c": 0.25,+ "d": 0.25+ }+ },+ "result": 0.4+ },+ {+ "thresholds": {+ "choice:2": 0.7,+ "default": 0.4+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.125,+ "b": 0.125,+ "c": 0.125,+ "d": 0.125,+ "e": 0.125,+ "f": 0.125,+ "g": 0.125,+ "h": 0.125+ }+ },+ "result": 0.4+ },+ {+ "thresholds": {+ "choice:2": 0.7,+ "default": 0.4+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.0833,+ "b": 0.0833,+ "c": 0.0833,+ "d": 0.0833,+ "e": 0.0833,+ "f": 0.0833,+ "g": 0.0833,+ "h": 0.0833,+ "i": 0.0833,+ "j": 0.0833,+ "k": 0.0833,+ "l": 0.0833+ }+ },+ "result": 0.4+ },+ {+ "thresholds": {+ "choice:2": 0.7,+ "default": 0.4+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.5,+ "1": 0.5+ }+ },+ "result": 0.4+ },+ {+ "thresholds": {+ "choice:2": 0.7,+ "default": 0.4+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.25,+ "1": 0.25,+ "2": 0.25,+ "3": 0.25+ }+ },+ "result": 0.4+ },+ {+ "thresholds": {+ "choice:2": 0.7,+ "default": 0.4+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.125,+ "1": 0.125,+ "2": 0.125,+ "3": 0.125,+ "4": 0.125,+ "5": 0.125,+ "6": 0.125,+ "7": 0.125+ }+ },+ "result": 0.4+ },+ {+ "thresholds": {+ "choice:2": 0.7,+ "default": 0.4+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0833,+ "1": 0.0833,+ "2": 0.0833,+ "3": 0.0833,+ "4": 0.0833,+ "5": 0.0833,+ "6": 0.0833,+ "7": 0.0833,+ "8": 0.0833,+ "9": 0.0833,+ "10": 0.0833,+ "11": 0.0833+ }+ },+ "result": 0.4+ },+ {+ "thresholds": {+ "choice:2": 0.7,+ "default": 0.4+ },+ "answer": {+ "type": "noul",+ "noul": 0.9+ },+ "result": 0.4+ },+ {+ "thresholds": {+ "choice:2": 0.7,+ "default": 0.4+ },+ "answer": {+ "type": "choice",+ "probabilities": {}+ },+ "result": 0.4+ },+ {+ "thresholds": {+ "choice:3-5": 0.5+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "choice:3-5": 0.5+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.25,+ "b": 0.25,+ "c": 0.25,+ "d": 0.25+ }+ },+ "result": 0.5+ },+ {+ "thresholds": {+ "choice:3-5": 0.5+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.125,+ "b": 0.125,+ "c": 0.125,+ "d": 0.125,+ "e": 0.125,+ "f": 0.125,+ "g": 0.125,+ "h": 0.125+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "choice:3-5": 0.5+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.0833,+ "b": 0.0833,+ "c": 0.0833,+ "d": 0.0833,+ "e": 0.0833,+ "f": 0.0833,+ "g": 0.0833,+ "h": 0.0833,+ "i": 0.0833,+ "j": 0.0833,+ "k": 0.0833,+ "l": 0.0833+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "choice:3-5": 0.5+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.5,+ "1": 0.5+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "choice:3-5": 0.5+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.25,+ "1": 0.25,+ "2": 0.25,+ "3": 0.25+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "choice:3-5": 0.5+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.125,+ "1": 0.125,+ "2": 0.125,+ "3": 0.125,+ "4": 0.125,+ "5": 0.125,+ "6": 0.125,+ "7": 0.125+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "choice:3-5": 0.5+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0833,+ "1": 0.0833,+ "2": 0.0833,+ "3": 0.0833,+ "4": 0.0833,+ "5": 0.0833,+ "6": 0.0833,+ "7": 0.0833,+ "8": 0.0833,+ "9": 0.0833,+ "10": 0.0833,+ "11": 0.0833+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "choice:3-5": 0.5+ },+ "answer": {+ "type": "noul",+ "noul": 0.9+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "choice:3-5": 0.5+ },+ "answer": {+ "type": "choice",+ "probabilities": {}+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "default": 0.25+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ }+ },+ "result": 0.25+ },+ {+ "thresholds": {+ "default": 0.25+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.25,+ "b": 0.25,+ "c": 0.25,+ "d": 0.25+ }+ },+ "result": 0.25+ },+ {+ "thresholds": {+ "default": 0.25+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.125,+ "b": 0.125,+ "c": 0.125,+ "d": 0.125,+ "e": 0.125,+ "f": 0.125,+ "g": 0.125,+ "h": 0.125+ }+ },+ "result": 0.25+ },+ {+ "thresholds": {+ "default": 0.25+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.0833,+ "b": 0.0833,+ "c": 0.0833,+ "d": 0.0833,+ "e": 0.0833,+ "f": 0.0833,+ "g": 0.0833,+ "h": 0.0833,+ "i": 0.0833,+ "j": 0.0833,+ "k": 0.0833,+ "l": 0.0833+ }+ },+ "result": 0.25+ },+ {+ "thresholds": {+ "default": 0.25+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.5,+ "1": 0.5+ }+ },+ "result": 0.25+ },+ {+ "thresholds": {+ "default": 0.25+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.25,+ "1": 0.25,+ "2": 0.25,+ "3": 0.25+ }+ },+ "result": 0.25+ },+ {+ "thresholds": {+ "default": 0.25+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.125,+ "1": 0.125,+ "2": 0.125,+ "3": 0.125,+ "4": 0.125,+ "5": 0.125,+ "6": 0.125,+ "7": 0.125+ }+ },+ "result": 0.25+ },+ {+ "thresholds": {+ "default": 0.25+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0833,+ "1": 0.0833,+ "2": 0.0833,+ "3": 0.0833,+ "4": 0.0833,+ "5": 0.0833,+ "6": 0.0833,+ "7": 0.0833,+ "8": 0.0833,+ "9": 0.0833,+ "10": 0.0833,+ "11": 0.0833+ }+ },+ "result": 0.25+ },+ {+ "thresholds": {+ "default": 0.25+ },+ "answer": {+ "type": "noul",+ "noul": 0.9+ },+ "result": 0.25+ },+ {+ "thresholds": {+ "default": 0.25+ },+ "answer": {+ "type": "choice",+ "probabilities": {}+ },+ "result": 0.25+ },+ {+ "thresholds": {+ "score:6-10": 0.8,+ "noul:2": 0.3,+ "choice:11+": 0.6+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "score:6-10": 0.8,+ "noul:2": 0.3,+ "choice:11+": 0.6+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.25,+ "b": 0.25,+ "c": 0.25,+ "d": 0.25+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "score:6-10": 0.8,+ "noul:2": 0.3,+ "choice:11+": 0.6+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.125,+ "b": 0.125,+ "c": 0.125,+ "d": 0.125,+ "e": 0.125,+ "f": 0.125,+ "g": 0.125,+ "h": 0.125+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "score:6-10": 0.8,+ "noul:2": 0.3,+ "choice:11+": 0.6+ },+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.0833,+ "b": 0.0833,+ "c": 0.0833,+ "d": 0.0833,+ "e": 0.0833,+ "f": 0.0833,+ "g": 0.0833,+ "h": 0.0833,+ "i": 0.0833,+ "j": 0.0833,+ "k": 0.0833,+ "l": 0.0833+ }+ },+ "result": 0.6+ },+ {+ "thresholds": {+ "score:6-10": 0.8,+ "noul:2": 0.3,+ "choice:11+": 0.6+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.5,+ "1": 0.5+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "score:6-10": 0.8,+ "noul:2": 0.3,+ "choice:11+": 0.6+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.25,+ "1": 0.25,+ "2": 0.25,+ "3": 0.25+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "score:6-10": 0.8,+ "noul:2": 0.3,+ "choice:11+": 0.6+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.125,+ "1": 0.125,+ "2": 0.125,+ "3": 0.125,+ "4": 0.125,+ "5": 0.125,+ "6": 0.125,+ "7": 0.125+ }+ },+ "result": 0.8+ },+ {+ "thresholds": {+ "score:6-10": 0.8,+ "noul:2": 0.3,+ "choice:11+": 0.6+ },+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0833,+ "1": 0.0833,+ "2": 0.0833,+ "3": 0.0833,+ "4": 0.0833,+ "5": 0.0833,+ "6": 0.0833,+ "7": 0.0833,+ "8": 0.0833,+ "9": 0.0833,+ "10": 0.0833,+ "11": 0.0833+ }+ },+ "result": 0.0+ },+ {+ "thresholds": {+ "score:6-10": 0.8,+ "noul:2": 0.3,+ "choice:11+": 0.6+ },+ "answer": {+ "type": "noul",+ "noul": 0.9+ },+ "result": 0.3+ },+ {+ "thresholds": {+ "score:6-10": 0.8,+ "noul:2": 0.3,+ "choice:11+": 0.6+ },+ "answer": {+ "type": "choice",+ "probabilities": {}+ },+ "result": 0.0+ }+ ],+ "gate_answers": {+ "choice2": {+ "type": "choice",+ "probabilities": {+ "a": 0.6,+ "b": 0.4+ },+ "answer_confidence": 0.6,+ "confidence": 0.029+ },+ "choice4": {+ "type": "choice",+ "probabilities": {+ "a": 0.45,+ "b": 0.25,+ "c": 0.2,+ "d": 0.1+ },+ "answer_confidence": 0.45,+ "confidence": 0.12+ },+ "choice8": {+ "type": "choice",+ "probabilities": {+ "a": 0.125,+ "b": 0.125,+ "c": 0.125,+ "d": 0.125,+ "e": 0.125,+ "f": 0.125,+ "g": 0.125,+ "h": 0.125+ },+ "answer_confidence": 0.3,+ "confidence": 0.05+ },+ "choice12": {+ "type": "choice",+ "probabilities": {+ "a": 0.0833,+ "b": 0.0833,+ "c": 0.0833,+ "d": 0.0833,+ "e": 0.0833,+ "f": 0.0833,+ "g": 0.0833,+ "h": 0.0833,+ "i": 0.0833,+ "j": 0.0833,+ "k": 0.0833,+ "l": 0.0833+ },+ "answer_confidence": 0.15,+ "confidence": 0.02+ },+ "score2": {+ "type": "score",+ "probabilities": {+ "0": 0.7,+ "1": 0.3+ },+ "answer_confidence": 0.7,+ "confidence": 0.1+ },+ "score4": {+ "type": "score",+ "probabilities": {+ "0": 0.1,+ "1": 0.5,+ "2": 0.3,+ "3": 0.1+ },+ "answer_confidence": 0.5,+ "confidence": 0.2+ },+ "score8": {+ "type": "score",+ "probabilities": {+ "0": 0.125,+ "1": 0.125,+ "2": 0.125,+ "3": 0.125,+ "4": 0.125,+ "5": 0.125,+ "6": 0.125,+ "7": 0.125+ },+ "answer_confidence": 0.125,+ "confidence": 0.01+ },+ "noul": {+ "type": "noul",+ "noul": 0.8,+ "answer_confidence": 0.8,+ "confidence": 0.8+ },+ "at_half": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ },+ "answer_confidence": 0.5,+ "confidence": 0.0+ },+ "fallback_conf": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ },+ "confidence": 0.3+ },+ "nan_answer_conf": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ },+ "answer_confidence": "nan",+ "confidence": 0.7+ },+ "inf_answer_conf": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ },+ "answer_confidence": "inf",+ "confidence": 0.2+ },+ "unusable_both": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ },+ "answer_confidence": "nan",+ "confidence": "inf"+ },+ "no_confidence": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ }+ },+ "empty_choice": {+ "type": "choice",+ "probabilities": {},+ "answer_confidence": 0.4+ }+ },+ "gate": [+ {+ "threshold": null,+ "expected": {+ "choice2": {},+ "choice4": {},+ "choice8": {},+ "choice12": {},+ "score2": {},+ "score4": {},+ "score8": {},+ "noul": {},+ "at_half": {},+ "fallback_conf": {},+ "nan_answer_conf": {},+ "inf_answer_conf": {},+ "unusable_both": {},+ "no_confidence": {},+ "empty_choice": {}+ }+ },+ {+ "threshold": 0.0,+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice4": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice12": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "fallback_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "inf_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ }+ },+ {+ "threshold": 1.0,+ "expected": {+ "choice2": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "choice4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "choice8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "score2": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "score4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "noul": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "at_half": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "fallback_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "nan_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "inf_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 1.0+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 1.0+ },+ "empty_choice": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ }+ }+ },+ {+ "threshold": 0.5,+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "choice4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "choice8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "fallback_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "inf_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.5+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.5+ },+ "empty_choice": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ }+ }+ },+ {+ "threshold": 0.35,+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "choice4": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "choice8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.35+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.35+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.35+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "fallback_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.35+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "inf_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.35+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.35+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.35+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ }+ }+ },+ {+ "threshold": 0.2,+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "choice4": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "choice8": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.2+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.2+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "fallback_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "inf_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.2+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.2+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ }+ }+ },+ {+ "threshold": {+ "default": 0.4+ },+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ },+ "choice4": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ },+ "choice8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.4+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.4+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.4+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ },+ "fallback_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.4+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ },+ "inf_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.4+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.4+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.4+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ }+ }+ },+ {+ "threshold": {+ "choice:2": 0.7,+ "default": 0.4+ },+ "expected": {+ "choice2": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ },+ "choice4": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ },+ "choice8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.4+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.4+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.4+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ },+ "at_half": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ },+ "fallback_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.7+ },+ "inf_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.7+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.7+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.4+ }+ }+ },+ {+ "threshold": {+ "choice:3-5": 0.5+ },+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "choice8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice12": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "fallback_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "inf_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ }+ },+ {+ "threshold": {+ "choice:2": 0.7,+ "choice:3-5": 0.5,+ "choice:6-10": 0.4,+ "choice:11+": 0.1,+ "score:2": 0.8,+ "score:3-5": 0.6,+ "score:6-10": 0.2,+ "noul:2": 0.9+ },+ "expected": {+ "choice2": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ },+ "choice4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "choice8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.4+ },+ "choice12": {+ "abstention": "passed",+ "abstention_threshold": 0.1+ },+ "score2": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.8+ },+ "score4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.2+ },+ "noul": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "at_half": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ },+ "fallback_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.7+ },+ "inf_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.7+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.7+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ }+ },+ {+ "threshold": {+ "noul:2": 0.0,+ "score:3-5": 1.0+ },+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice4": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice12": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "score8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "fallback_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "inf_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ }+ }+ ],+ "reuse": [+ {+ "steps": [+ {+ "threshold": 0.9,+ "expected": {+ "choice2": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "choice4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "choice8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "score2": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "score4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "noul": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "at_half": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "fallback_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "nan_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "inf_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.9+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.9+ },+ "empty_choice": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ }+ }+ },+ {+ "threshold": 0.0,+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice4": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice12": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "fallback_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "inf_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ }+ },+ {+ "threshold": 0.2,+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "choice4": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "choice8": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.2+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.2+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "fallback_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "inf_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.2+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.2+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.2+ }+ }+ },+ {+ "threshold": 1.0,+ "expected": {+ "choice2": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "choice4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "choice8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "score2": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "score4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "noul": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "at_half": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "fallback_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "nan_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "inf_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 1.0+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 1.0+ },+ "empty_choice": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 1.0+ }+ }+ },+ {+ "threshold": 0.5,+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "choice4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "choice8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "fallback_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "inf_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.5+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.5+ },+ "empty_choice": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ }+ }+ }+ ]+ },+ {+ "steps": [+ {+ "threshold": {+ "choice:2": 0.9,+ "default": 0.5+ },+ "expected": {+ "choice2": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "choice4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "choice8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "at_half": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "fallback_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "nan_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "inf_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.9+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.9+ },+ "empty_choice": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ }+ }+ },+ {+ "threshold": 0.0,+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice4": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice12": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "fallback_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "inf_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ }+ },+ {+ "threshold": {+ "choice:3-5": 0.9+ },+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice4": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "choice8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "choice12": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "score8": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "fallback_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "inf_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.0+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ }+ },+ {+ "threshold": 0.35,+ "expected": {+ "choice2": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "choice4": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "choice8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.35+ },+ "choice12": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.35+ },+ "score2": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "score4": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "score8": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.35+ },+ "noul": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "at_half": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "fallback_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.35+ },+ "nan_answer_conf": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ },+ "inf_answer_conf": {+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.35+ },+ "unusable_both": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.35+ },+ "no_confidence": {+ "abstention": "unevaluated",+ "abstention_threshold": 0.35+ },+ "empty_choice": {+ "abstention": "passed",+ "abstention_threshold": 0.35+ }+ }+ }+ ]+ }+ ],+ "jev": [+ {+ "name": "choice-confident",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.9,+ "b": 0.1+ }+ },+ "typed": true,+ "result": 0.8+ },+ {+ "name": "choice-even",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": 0.5+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "choice-three",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.7,+ "b": 0.2,+ "c": 0.1+ }+ },+ "typed": true,+ "result": 0.5499999999999999+ },+ {+ "name": "choice-differs-from-entropy",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.7,+ "b": 0.1,+ "c": 0.1,+ "d": 0.1+ }+ },+ "typed": true,+ "result": 0.6+ },+ {+ "name": "choice-unnormalised",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 2.0,+ "b": 1.0,+ "c": 1.0+ }+ },+ "typed": true,+ "result": 0.25+ },+ {+ "name": "choice-zero-total",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.0,+ "b": 0.0,+ "c": 0.0+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "choice-single",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.3+ }+ },+ "typed": true,+ "result": 1.0+ },+ {+ "name": "choice-single-zero",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.0+ }+ },+ "typed": true,+ "result": 1.0+ },+ {+ "name": "choice-zero-among-others",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.0,+ "b": 1.0+ }+ },+ "typed": true,+ "result": 1.0+ },+ {+ "name": "score-peaked",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0,+ "1": 0.95,+ "2": 0.05+ }+ },+ "typed": true,+ "result": 0.925+ },+ {+ "name": "score-jevcompat-example",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0,+ "1": 0.95,+ "2": 0.05+ }+ },+ "typed": true,+ "result": 0.925+ },+ {+ "name": "score-even",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.25,+ "1": 0.25,+ "2": 0.25,+ "3": 0.25+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "score-edge-mode",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.8,+ "1": 0.1,+ "2": 0.1+ }+ },+ "typed": true,+ "result": 0.5499999999999999+ },+ {+ "name": "score-zero-total",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0,+ "1": 0.0,+ "2": 0.0,+ "3": 0.0+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "score-single",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.6+ }+ },+ "typed": true,+ "result": 1.0+ },+ {+ "name": "score-two",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.3,+ "1": 0.7+ }+ },+ "typed": true,+ "result": 0.4+ },+ {+ "name": "score-bimodal",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.45,+ "1": 0.05,+ "2": 0.05,+ "3": 0.45+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "score-tied-modes",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.4,+ "1": 0.4,+ "2": 0.2+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "score-levels-in-numeric-order",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0,+ "1": 0.0,+ "2": 0.0,+ "3": 0.0,+ "4": 0.0,+ "5": 0.0,+ "6": 0.0,+ "7": 0.0,+ "8": 0.0,+ "9": 0.0,+ "10": 1.0+ }+ },+ "typed": true,+ "result": 1.0+ },+ {+ "name": "noul",+ "answer": {+ "type": "noul",+ "noul": 0.9+ },+ "typed": true,+ "result": null+ },+ {+ "name": "empty-choice-probabilities",+ "answer": {+ "type": "choice",+ "probabilities": {}+ },+ "typed": true,+ "result": null+ },+ {+ "name": "empty-score-probabilities",+ "answer": {+ "type": "score",+ "probabilities": {}+ },+ "typed": true,+ "result": null+ },+ {+ "name": "nan-probability",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": "nan",+ "b": 0.5+ }+ },+ "typed": true,+ "result": null+ },+ {+ "name": "infinite-probability",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": "inf",+ "b": 0.5+ }+ },+ "typed": true,+ "result": null+ },+ {+ "name": "negative-infinite-probability",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": "-inf",+ "b": 0.5+ }+ },+ "typed": true,+ "result": null+ },+ {+ "name": "negative-probability",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 2.0,+ "b": -1.0+ }+ },+ "typed": true,+ "result": null+ },+ {+ "name": "mixed-sign-argmax-flips",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": -1.5+ }+ },+ "typed": true,+ "result": null+ },+ {+ "name": "all-negative-probabilities",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": -0.9,+ "b": -0.1+ }+ },+ "typed": true,+ "result": null+ },+ {+ "name": "negative-total-near-zero",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": -0.61,+ "b": 0.6+ }+ },+ "typed": true,+ "result": null+ },+ {+ "name": "negative-score-level",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 1.5,+ "1": -0.5,+ "2": 0.0+ }+ },+ "typed": true,+ "result": null+ },+ {+ "name": "nan-score-level",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.5,+ "1": "nan"+ }+ },+ "typed": true,+ "result": null+ },+ {+ "name": "negative-single",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": -0.5+ }+ },+ "typed": true,+ "result": null+ },+ {+ "name": "nan-single",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": "nan"+ }+ },+ "typed": true,+ "result": null+ },+ {+ "name": "unknown-type",+ "answer": {+ "type": "weird",+ "probabilities": {+ "a": 1.0+ }+ },+ "typed": false,+ "result": null+ },+ {+ "name": "missing-probabilities",+ "answer": {+ "type": "choice"+ },+ "typed": false,+ "result": null+ },+ {+ "name": "non-numeric-probability",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.5,+ "b": "high"+ }+ },+ "typed": false,+ "result": null+ },+ {+ "name": "bool-probability",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": true,+ "b": false+ }+ },+ "typed": false,+ "result": null+ },+ {+ "name": "bool-among-numbers",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": true,+ "b": 0.4+ }+ },+ "typed": false,+ "result": null+ },+ {+ "name": "non-integer-score-level",+ "answer": {+ "type": "score",+ "probabilities": {+ "low": 0.5,+ "high": 0.5+ }+ },+ "typed": false,+ "result": null+ },+ {+ "name": "sum-differs-choice-tenths",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.1,+ "b": 0.1,+ "c": 0.1,+ "d": 0.1,+ "e": 0.1,+ "f": 0.1,+ "g": 0.1,+ "h": 0.1,+ "i": 0.1,+ "j": 0.1+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "sum-differs-score-tenths",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.1,+ "1": 0.1,+ "2": 0.1,+ "3": 0.1,+ "4": 0.1,+ "5": 0.1,+ "6": 0.1,+ "7": 0.1,+ "8": 0.1,+ "9": 0.1+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "sum-differs-choice-hundredths",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.01,+ "b": 0.01,+ "c": 0.01,+ "d": 0.01,+ "e": 0.01,+ "f": 0.01,+ "g": 0.01,+ "h": 0.01,+ "i": 0.01,+ "j": 0.01,+ "k": 0.01,+ "l": 0.01,+ "m": 0.01,+ "n": 0.01,+ "o": 0.01,+ "p": 0.01,+ "q": 0.01,+ "r": 0.01,+ "s": 0.01,+ "t": 0.01,+ "u": 0.01,+ "v": 0.01,+ "w": 0.01,+ "x": 0.01,+ "y": 0.01,+ "z": 0.01,+ "{": 0.01,+ "|": 0.01,+ "}": 0.01,+ "~": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ " ": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ "": 0.01,+ " ": 0.01,+ "¡": 0.01,+ "¢": 0.01,+ "£": 0.01,+ "¤": 0.01,+ "¥": 0.01,+ "¦": 0.01,+ "§": 0.01,+ "¨": 0.01,+ "©": 0.01,+ "ª": 0.01,+ "«": 0.01,+ "¬": 0.01,+ "": 0.01,+ "®": 0.01,+ "¯": 0.01,+ "°": 0.01,+ "±": 0.01,+ "²": 0.01,+ "³": 0.01,+ "´": 0.01,+ "µ": 0.01,+ "¶": 0.01,+ "·": 0.01,+ "¸": 0.01,+ "¹": 0.01,+ "º": 0.01,+ "»": 0.01,+ "¼": 0.01,+ "½": 0.01,+ "¾": 0.01,+ "¿": 0.01,+ "À": 0.01,+ "Á": 0.01,+ "Â": 0.01,+ "Ã": 0.01,+ "Ä": 0.01+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "sum-differs-choice-7",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.6946,+ "b": 0.03,+ "c": 0.3634,+ "d": 0.67,+ "e": 0.2649,+ "f": 0.72,+ "g": 0.6497+ }+ },+ "typed": true,+ "result": 0.08093104600208292+ },+ {+ "name": "sum-differs-choice-5",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.7133,+ "b": 0.3416,+ "c": 0.7,+ "d": 0.8634,+ "e": 0.6+ }+ },+ "typed": true,+ "result": 0.08534785445732211+ },+ {+ "name": "sum-differs-choice-10",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.6,+ "b": 0.1,+ "c": 0.3,+ "d": 0.7918,+ "e": 0.94,+ "f": 0.7,+ "g": 0.9983,+ "h": 0.5,+ "i": 0.8,+ "j": 0.6+ }+ },+ "typed": true,+ "result": 0.06411869919555421+ },+ {+ "name": "sum-differs-choice-16",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.9,+ "b": 0.11,+ "c": 0.4,+ "d": 0.84,+ "e": 0.1412,+ "f": 0.8523,+ "g": 0.45,+ "h": 0.69,+ "i": 0.48,+ "j": 0.5,+ "k": 0.1126,+ "l": 0.9598,+ "m": 0.94,+ "n": 0.18,+ "o": 0.2723,+ "p": 0.87+ }+ },+ "typed": true,+ "result": 0.051034313612778125+ },+ {+ "name": "sum-differs-score-7",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.4,+ "1": 0.4528,+ "2": 0.2,+ "3": 0.1,+ "4": 0.93,+ "5": 0.34,+ "6": 0.89+ }+ },+ "typed": true,+ "result": 0.017729211945584966+ },+ {+ "name": "sum-differs-score-3",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.453,+ "1": 0.4,+ "2": 0.0333+ }+ },+ "typed": true,+ "result": 0.21031253525894167+ },+ {+ "name": "random-choice-00",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.2566,+ "b": 0.4558+ }+ },+ "typed": true,+ "result": 0.27961819202695115+ },+ {+ "name": "random-score-00",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.2566,+ "1": 0.4558+ }+ },+ "typed": true,+ "result": 0.27961819202695104+ },+ {+ "name": "random-choice-01",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.8042,+ "b": 0.9099,+ "c": 0.6461,+ "d": 0.4657,+ "e": 0.914,+ "f": 0.3627,+ "g": 0.1673,+ "h": 0.8291,+ "i": 0.5506,+ "j": 0.1385,+ "k": 0.9363,+ "l": 0.3846+ }+ },+ "typed": true,+ "result": 0.052770495786391154+ },+ {+ "name": "random-score-01",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.8042,+ "1": 0.9099,+ "2": 0.6461,+ "3": 0.4657,+ "4": 0.914,+ "5": 0.3627,+ "6": 0.1673,+ "7": 0.8291,+ "8": 0.5506,+ "9": 0.1385,+ "10": 0.9363,+ "11": 0.3846+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-02",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.6852,+ "b": 0.4922,+ "c": 0.3347,+ "d": 0.8721,+ "e": 0.9064,+ "f": 0.1556,+ "g": 0.1312,+ "h": 0.155,+ "i": 0.5995,+ "j": 0.7911,+ "k": 0.4026,+ "l": 0.7689+ }+ },+ "typed": true,+ "result": 0.06618043169000355+ },+ {+ "name": "random-score-02",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.6852,+ "1": 0.4922,+ "2": 0.3347,+ "3": 0.8721,+ "4": 0.9064,+ "5": 0.1556,+ "6": 0.1312,+ "7": 0.155,+ "8": 0.5995,+ "9": 0.7911,+ "10": 0.4026,+ "11": 0.7689+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-03",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.8998,+ "b": 0.8419,+ "c": 0.0794,+ "d": 0.321+ }+ },+ "typed": true,+ "result": 0.226740114840577+ },+ {+ "name": "random-score-03",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.8998,+ "1": 0.8419,+ "2": 0.0794,+ "3": 0.321+ }+ },+ "typed": true,+ "result": 0.08328275990850109+ },+ {+ "name": "random-choice-04",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.6957,+ "b": 0.7158,+ "c": 0.3912,+ "d": 0.5873,+ "e": 0.7343,+ "f": 0.2801,+ "g": 0.423,+ "h": 0.2488+ }+ },+ "typed": true,+ "result": 0.06302088079233457+ },+ {+ "name": "random-score-04",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.6957,+ "1": 0.7158,+ "2": 0.3912,+ "3": 0.5873,+ "4": 0.7343,+ "5": 0.2801,+ "6": 0.423,+ "7": 0.2488+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-05",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.6537,+ "b": 0.3714,+ "c": 0.6408,+ "d": 0.9808+ }+ },+ "typed": true,+ "result": 0.16076623720104277+ },+ {+ "name": "random-score-05",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.6537,+ "1": 0.3714,+ "2": 0.6408,+ "3": 0.9808+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-06",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.0542,+ "b": 0.993,+ "c": 0.2939,+ "d": 0.2498,+ "e": 0.8353,+ "f": 0.7231,+ "g": 0.8383,+ "h": 0.6679,+ "i": 0.3134,+ "j": 0.7684,+ "k": 0.3708,+ "l": 0.114+ }+ },+ "typed": true,+ "result": 0.08319173152589524+ },+ {+ "name": "random-score-06",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0542,+ "1": 0.993,+ "2": 0.2939,+ "3": 0.2498,+ "4": 0.8353,+ "5": 0.7231,+ "6": 0.8383,+ "7": 0.6679,+ "8": 0.3134,+ "9": 0.7684,+ "10": 0.3708,+ "11": 0.114+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-07",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.8131,+ "b": 0.9647,+ "c": 0.0528,+ "d": 0.1295,+ "e": 0.2193,+ "f": 0.4647,+ "g": 0.5681,+ "h": 0.8712,+ "i": 0.1082,+ "j": 0.4005,+ "k": 0.7698,+ "l": 0.436+ }+ },+ "typed": true,+ "result": 0.09060490553789853+ },+ {+ "name": "random-score-07",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.8131,+ "1": 0.9647,+ "2": 0.0528,+ "3": 0.1295,+ "4": 0.2193,+ "5": 0.4647,+ "6": 0.5681,+ "7": 0.8712,+ "8": 0.1082,+ "9": 0.4005,+ "10": 0.7698,+ "11": 0.436+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-08",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.3693,+ "b": 0.8736,+ "c": 0.329,+ "d": 0.1395,+ "e": 0.2551+ }+ },+ "typed": true,+ "result": 0.30530129672006096+ },+ {+ "name": "random-score-08",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.3693,+ "1": 0.8736,+ "2": 0.329,+ "3": 0.1395,+ "4": 0.2551+ }+ },+ "typed": true,+ "result": 0.26154758877871+ },+ {+ "name": "random-choice-09",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.686,+ "b": 0.3542,+ "c": 0.2812,+ "d": 0.8562+ }+ },+ "typed": true,+ "result": 0.1909135439627724+ },+ {+ "name": "random-score-09",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.686,+ "1": 0.3542,+ "2": 0.2812,+ "3": 0.8562+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-10",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.6386,+ "b": 0.4663+ }+ },+ "typed": true,+ "result": 0.15594171418227876+ },+ {+ "name": "random-score-10",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.6386,+ "1": 0.4663+ }+ },+ "typed": true,+ "result": 0.15594171418227898+ },+ {+ "name": "random-choice-11",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.2097,+ "b": 0.6566,+ "c": 0.0187,+ "d": 0.9506+ }+ },+ "typed": true,+ "result": 0.35715842231422973+ },+ {+ "name": "random-score-11",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.2097,+ "1": 0.6566,+ "2": 0.0187,+ "3": 0.9506+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-12",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.4693,+ "b": 0.1916+ }+ },+ "typed": true,+ "result": 0.42018459676199105+ },+ {+ "name": "random-score-12",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.4693,+ "1": 0.1916+ }+ },+ "typed": true,+ "result": 0.42018459676199127+ },+ {+ "name": "random-choice-13",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.9691,+ "b": 0.6389,+ "c": 0.7036,+ "d": 0.671,+ "e": 0.5951,+ "f": 0.1533,+ "g": 0.6329,+ "h": 0.495+ }+ },+ "typed": true,+ "result": 0.08508392552106149+ },+ {+ "name": "random-score-13",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.9691,+ "1": 0.6389,+ "2": 0.7036,+ "3": 0.671,+ "4": 0.5951,+ "5": 0.1533,+ "6": 0.6329,+ "7": 0.495+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-14",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.7871,+ "b": 0.069,+ "c": 0.9774+ }+ },+ "typed": true,+ "result": 0.29961821652577036+ },+ {+ "name": "random-score-14",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.7871,+ "1": 0.069,+ "2": 0.9774+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-15",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.9361,+ "b": 0.0913,+ "c": 0.9088,+ "d": 0.2108,+ "e": 0.4095+ }+ },+ "typed": true,+ "result": 0.20770584783884216+ },+ {+ "name": "random-score-15",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.9361,+ "1": 0.0913,+ "2": 0.9088,+ "3": 0.2108,+ "4": 0.4095+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-16",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.141,+ "b": 0.6816,+ "c": 0.801+ }+ },+ "typed": true,+ "result": 0.2400221729490023+ },+ {+ "name": "random-score-16",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.141,+ "1": 0.6816,+ "2": 0.801+ }+ },+ "typed": true,+ "result": 0.1097560975609756+ },+ {+ "name": "random-choice-17",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.735,+ "b": 0.398,+ "c": 0.5032,+ "d": 0.2529,+ "e": 0.9528+ }+ },+ "typed": true,+ "result": 0.1690858228649847+ },+ {+ "name": "random-score-17",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.735,+ "1": 0.398,+ "2": 0.5032,+ "3": 0.2529,+ "4": 0.9528+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-18",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.6034,+ "b": 0.5739,+ "c": 0.8956+ }+ },+ "typed": true,+ "result": 0.14807757248299486+ },+ {+ "name": "random-score-18",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.6034,+ "1": 0.5739,+ "2": 0.8956+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-19",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.0916,+ "b": 0.3621,+ "c": 0.0287,+ "d": 0.537,+ "e": 0.8582,+ "f": 0.8354,+ "g": 0.5424,+ "h": 0.9539+ }+ },+ "typed": true,+ "result": 0.11613400259968573+ },+ {+ "name": "random-score-19",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.0916,+ "1": 0.3621,+ "2": 0.0287,+ "3": 0.537,+ "4": 0.8582,+ "5": 0.8354,+ "6": 0.5424,+ "7": 0.9539+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-20",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.9237,+ "b": 0.4328+ }+ },+ "typed": true,+ "result": 0.36188720973092514+ },+ {+ "name": "random-score-20",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.9237,+ "1": 0.4328+ }+ },+ "typed": true,+ "result": 0.36188720973092514+ },+ {+ "name": "random-choice-21",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.5616,+ "b": 0.0699,+ "c": 0.8875,+ "d": 0.0634,+ "e": 0.6978,+ "f": 0.5905,+ "g": 0.6597,+ "h": 0.1242,+ "i": 0.2992,+ "j": 0.5477,+ "k": 0.8902,+ "l": 0.2786+ }+ },+ "typed": true,+ "result": 0.08035649869415279+ },+ {+ "name": "random-score-21",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.5616,+ "1": 0.0699,+ "2": 0.8875,+ "3": 0.0634,+ "4": 0.6978,+ "5": 0.5905,+ "6": 0.6597,+ "7": 0.1242,+ "8": 0.2992,+ "9": 0.5477,+ "10": 0.8902,+ "11": 0.2786+ }+ },+ "typed": true,+ "result": 0.0+ },+ {+ "name": "random-choice-22",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.3113,+ "b": 0.6847,+ "c": 0.3262+ }+ },+ "typed": true,+ "result": 0.27677355921948266+ },+ {+ "name": "random-score-22",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.3113,+ "1": 0.6847,+ "2": 0.3262+ }+ },+ "typed": true,+ "result": 0.27677355921948266+ },+ {+ "name": "random-choice-23",+ "answer": {+ "type": "choice",+ "probabilities": {+ "a": 0.3983,+ "b": 0.9787,+ "c": 0.9707,+ "d": 0.4993+ }+ },+ "typed": true,+ "result": 0.12502048940405106+ },+ {+ "name": "random-score-23",+ "answer": {+ "type": "score",+ "probabilities": {+ "0": 0.3983,+ "1": 0.9787,+ "2": 0.9707,+ "3": 0.4993+ }+ },+ "typed": true,+ "result": 0.1683877766069546+ }+ ]+}diff --git a/laya-rust/tests/golden/routing/meta.json b/laya-rust/tests/golden/routing/meta.jsonindex 92814b6c..2353210a 100644--- a/laya-rust/tests/golden/routing/meta.json+++ b/laya-rust/tests/golden/routing/meta.json@@ -1,6 +1,6 @@{"generator": "tools/dump_routing_golden.py",- "laya_version": "0.4.14",+ "laya_version": "0.4.15","default_shortlist_k": 20,"notes": {"route_probe": "RouteDecision is serialized as {model, reason, detection, workflow}; 'repo' is intentionally omitted (not part of the ported RouteDecision).",@@ -11,6 +11,13 @@"route_probe": 159,"route_batch_probe": 10,"route_batch_probe_errors": 6,+ "confidence_probe_scalars": 18,+ "confidence_probe_maps": 24,+ "confidence_probe_buckets": 19,+ "confidence_probe_resolve": 40,+ "confidence_probe_gate": 11,+ "confidence_probe_reuse": 2,+ "confidence_probe_jev": 96,"email_probe_clean": 253,"email_probe_state": 22,"shortlist_probe": 22,diff --git a/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_bad_key.json b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_bad_key.jsonnew file mode 100644index 00000000..424420ec--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_bad_key.json@@ -0,0 +1,38 @@+{+ "name": "batch_min_confidence_bad_key",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": {+ "choice:2": 0.5,+ "choice:3": 0.5+ }+ },+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'"+}diff --git a/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_before_none_state.json b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_before_none_state.jsonnew file mode 100644index 00000000..75fd7564--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_before_none_state.json@@ -0,0 +1,36 @@+{+ "name": "batch_min_confidence_before_none_state",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ null+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": -0.25+ },+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -0.25"+}diff --git a/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_empty_questions.json b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_empty_questions.jsonnew file mode 100644index 00000000..255f7bc4--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_empty_questions.json@@ -0,0 +1,44 @@+{+ "name": "batch_min_confidence_empty_questions",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, the refund arrived and everything works now.",+ {+ "ticket": 4821,+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "+ }+ ],+ "questions": {},+ "options": {+ "min_confidence": 0.5+ },+ "state_widths": [],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {},+ "usage": {+ "input_tokens": 0,+ "output_tokens": 0+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {},+ "usage": {+ "input_tokens": 0,+ "output_tokens": 0+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {},+ "usage": {+ "input_tokens": 0,+ "output_tokens": 0+ }+ }+ ],+ "runs": []+}diff --git a/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_empty_states.json b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_empty_states.jsonnew file mode 100644index 00000000..dfb141d9--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_empty_states.json@@ -0,0 +1,34 @@+{+ "name": "batch_min_confidence_empty_states",+ "states": [],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.5+ },+ "state_widths": [],+ "results": [],+ "runs": []+}diff --git a/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_invalid.json b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_invalid.jsonnew file mode 100644index 00000000..72496b52--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_invalid.json@@ -0,0 +1,35 @@+{+ "name": "batch_min_confidence_invalid",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 1.5+ },+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"+}diff --git a/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_map.json b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_map.jsonnew file mode 100644index 00000000..a34cfe8e--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_map.json@@ -0,0 +1,504 @@+{+ "name": "batch_min_confidence_map",+ "states": [+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "batch_size": 3,+ "sort_by_length": true,+ "min_confidence": {+ "choice:3-5": 0.8,+ "score:3-5": 0.5,+ "noul:2": 0.7+ }+ },+ "state_widths": [+ 199,+ 103,+ 145,+ 91,+ 175,+ 121,+ 157+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7688,+ "technical": 0.1498,+ "other": 0.0814+ },+ "confidence": 0.3712,+ "answer_confidence": 0.7688,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.839,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0167,+ "1": 0.1276,+ "2": 0.8557+ },+ "confidence": 0.5773,+ "answer_confidence": 0.8557,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.8118,+ "confidence": 0.8118,+ "answer_confidence": 0.8118,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 589,+ "output_tokens": 0,+ "state_tokens": 165,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7324,+ "technical": 0.1799,+ "other": 0.0877+ },+ "confidence": 0.3172,+ "answer_confidence": 0.7324,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.6583,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0379,+ "1": 0.2659,+ "2": 0.6962+ },+ "confidence": 0.337,+ "answer_confidence": 0.6962,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.7988,+ "confidence": 0.7988,+ "answer_confidence": 0.7988,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 301,+ "output_tokens": 0,+ "state_tokens": 69,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7732,+ "technical": 0.1468,+ "other": 0.08+ },+ "confidence": 0.3786,+ "answer_confidence": 0.7732,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.8334,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0162,+ "1": 0.1342,+ "2": 0.8496+ },+ "confidence": 0.5678,+ "answer_confidence": 0.8496,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.8132,+ "confidence": 0.8132,+ "answer_confidence": 0.8132,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 427,+ "output_tokens": 0,+ "state_tokens": 111,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7054,+ "technical": 0.1834,+ "other": 0.1111+ },+ "confidence": 0.2705,+ "answer_confidence": 0.7054,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.8371,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0193,+ "1": 0.1243,+ "2": 0.8564+ },+ "confidence": 0.574,+ "answer_confidence": 0.8564,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.7914,+ "confidence": 0.7914,+ "answer_confidence": 0.7914,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7458,+ "technical": 0.1647,+ "other": 0.0895+ },+ "confidence": 0.3338,+ "answer_confidence": 0.7458,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.8335,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0167,+ "1": 0.1332,+ "2": 0.8501+ },+ "confidence": 0.5678,+ "answer_confidence": 0.8501,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.827,+ "confidence": 0.827,+ "answer_confidence": 0.827,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 517,+ "output_tokens": 0,+ "state_tokens": 141,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7837,+ "technical": 0.1379,+ "other": 0.0784+ },+ "confidence": 0.3958,+ "answer_confidence": 0.7837,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.8327,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0171,+ "1": 0.1331,+ "2": 0.8498+ },+ "confidence": 0.5665,+ "answer_confidence": 0.8498,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.7977,+ "confidence": 0.7977,+ "answer_confidence": 0.7977,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 355,+ "output_tokens": 0,+ "state_tokens": 87,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7287,+ "technical": 0.1816,+ "other": 0.0897+ },+ "confidence": 0.3111,+ "answer_confidence": 0.7287,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.8+ },+ "urgency": {+ "type": "score",+ "score": 1.8351,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0151,+ "1": 0.1346,+ "2": 0.8502+ },+ "confidence": 0.571,+ "answer_confidence": 0.8502,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "cancel": {+ "type": "noul",+ "noul": 0.8224,+ "confidence": 0.8224,+ "answer_confidence": 0.8224,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 463,+ "output_tokens": 0,+ "state_tokens": 123,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 9,+ 121+ ],+ [+ 9,+ 175+ ],+ [+ 3,+ 199+ ]+ ]+}diff --git a/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_map_default.json b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_map_default.jsonnew file mode 100644index 00000000..b7a18f30--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_map_default.json@@ -0,0 +1,240 @@+{+ "name": "batch_min_confidence_map_default",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, the refund arrived and everything works now.",+ {+ "ticket": 4821,+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "+ }+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": {+ "default": 0.55+ }+ },+ "state_widths": [+ 91,+ 44,+ 157+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7054,+ "technical": 0.1834,+ "other": 0.1111+ },+ "confidence": 0.2705,+ "answer_confidence": 0.7054,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 1.8371,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0193,+ "1": 0.1243,+ "2": 0.8564+ },+ "confidence": 0.574,+ "answer_confidence": 0.8564,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "cancel": {+ "type": "noul",+ "noul": 0.7914,+ "confidence": 0.7914,+ "answer_confidence": 0.7914,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.4925,+ "technical": 0.3304,+ "other": 0.1771+ },+ "confidence": 0.0704,+ "answer_confidence": 0.4925,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 1.2205,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.2895,+ "1": 0.2006,+ "2": 0.51+ },+ "confidence": 0.0675,+ "answer_confidence": 0.51,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.55+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0363,+ "confidence": 0.9637,+ "answer_confidence": 0.9637,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 124,+ "output_tokens": 0,+ "state_tokens": 10,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5409,+ "technical": 0.2757,+ "other": 0.1834+ },+ "confidence": 0.0909,+ "answer_confidence": 0.5409,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 1.8074,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0273,+ "1": 0.138,+ "2": 0.8347+ },+ "confidence": 0.5244,+ "answer_confidence": 0.8347,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "cancel": {+ "type": "noul",+ "noul": 0.7748,+ "confidence": 0.7748,+ "answer_confidence": 0.7748,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 463,+ "output_tokens": 0,+ "state_tokens": 123,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 9,+ 157+ ]+ ]+}diff --git a/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_scalar.json b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_scalar.jsonnew file mode 100644index 00000000..28e53889--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_scalar.json@@ -0,0 +1,572 @@+{+ "name": "batch_min_confidence_scalar",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, that is all sorted now.",+ {+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",+ "message": "Please cancel my account today."+ },+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.6+ },+ "state_widths": [+ 91,+ 110,+ 144,+ 34,+ 163,+ 42,+ 98,+ 56+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7054,+ "technical": 0.1834,+ "other": 0.1111+ },+ "confidence": 0.2705,+ "answer_confidence": 0.7054,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.8371,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0193,+ "1": 0.1243,+ "2": 0.8564+ },+ "confidence": 0.574,+ "answer_confidence": 0.8564,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.7914,+ "confidence": 0.7914,+ "answer_confidence": 0.7914,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6869,+ "technical": 0.2079,+ "other": 0.1052+ },+ "confidence": 0.2523,+ "answer_confidence": 0.6869,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.7918,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0204,+ "1": 0.1675,+ "2": 0.8121+ },+ "confidence": 0.5015,+ "answer_confidence": 0.8121,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.7602,+ "confidence": 0.7602,+ "answer_confidence": 0.7602,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 322,+ "output_tokens": 0,+ "state_tokens": 76,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6718,+ "technical": 0.2244,+ "other": 0.1038+ },+ "confidence": 0.2375,+ "answer_confidence": 0.6718,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.7672,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0194,+ "1": 0.194,+ "2": 0.7866+ },+ "confidence": 0.4689,+ "answer_confidence": 0.7866,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.7234,+ "confidence": 0.7234,+ "answer_confidence": 0.7234,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 424,+ "output_tokens": 0,+ "state_tokens": 110,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.3741,+ "technical": 0.2268,+ "other": 0.3991+ },+ "confidence": 0.0252,+ "answer_confidence": 0.3991,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 0.9031,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.368,+ "1": 0.361,+ "2": 0.2711+ },+ "confidence": 0.0083,+ "answer_confidence": 0.368,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.2943,+ "confidence": 0.7057,+ "answer_confidence": 0.7057,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 94,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7312,+ "technical": 0.1851,+ "other": 0.0838+ },+ "confidence": 0.3184,+ "answer_confidence": 0.7312,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.8478,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0137,+ "1": 0.1248,+ "2": 0.8615+ },+ "confidence": 0.5931,+ "answer_confidence": 0.8615,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.8255,+ "confidence": 0.8255,+ "answer_confidence": 0.8255,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 481,+ "output_tokens": 0,+ "state_tokens": 129,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.142,+ "technical": 0.3006,+ "other": 0.5573+ },+ "confidence": 0.1222,+ "answer_confidence": 0.5573,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.2804,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.2652,+ "1": 0.1891,+ "2": 0.5456+ },+ "confidence": 0.092,+ "answer_confidence": 0.5456,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0403,+ "confidence": 0.9597,+ "answer_confidence": 0.9597,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 118,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.2504,+ "technical": 0.3345,+ "other": 0.4151+ },+ "confidence": 0.0187,+ "answer_confidence": 0.4151,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.766,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0394,+ "1": 0.1552,+ "2": 0.8054+ },+ "confidence": 0.4622,+ "answer_confidence": 0.8054,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.776,+ "confidence": 0.776,+ "answer_confidence": 0.776,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 286,+ "output_tokens": 0,+ "state_tokens": 64,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "technical",+ "probabilities": {+ "refund": 0.1839,+ "technical": 0.4141,+ "other": 0.4019+ },+ "confidence": 0.0507,+ "answer_confidence": 0.4141,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 0.9808,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.3293,+ "1": 0.3606,+ "2": 0.3101+ },+ "confidence": 0.0018,+ "answer_confidence": 0.3606,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.2948,+ "confidence": 0.7052,+ "answer_confidence": 0.7052,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 160,+ "output_tokens": 0,+ "state_tokens": 22,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 24,+ 163+ ]+ ]+}diff --git a/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_zero.json b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_zero.jsonnew file mode 100644index 00000000..9d88f298--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_batch_min_confidence_zero.json@@ -0,0 +1,565 @@+{+ "name": "batch_min_confidence_zero",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, that is all sorted now.",+ {+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",+ "message": "Please cancel my account today."+ },+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.0+ },+ "state_widths": [+ 91,+ 110,+ 144,+ 34,+ 163,+ 42,+ 98,+ 56+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7054,+ "technical": 0.1834,+ "other": 0.1111+ },+ "confidence": 0.2705,+ "answer_confidence": 0.7054,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.8371,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0193,+ "1": 0.1243,+ "2": 0.8564+ },+ "confidence": 0.574,+ "answer_confidence": 0.8564,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.7914,+ "confidence": 0.7914,+ "answer_confidence": 0.7914,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6869,+ "technical": 0.2079,+ "other": 0.1052+ },+ "confidence": 0.2523,+ "answer_confidence": 0.6869,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.7918,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0204,+ "1": 0.1675,+ "2": 0.8121+ },+ "confidence": 0.5015,+ "answer_confidence": 0.8121,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.7602,+ "confidence": 0.7602,+ "answer_confidence": 0.7602,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 322,+ "output_tokens": 0,+ "state_tokens": 76,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6718,+ "technical": 0.2244,+ "other": 0.1038+ },+ "confidence": 0.2375,+ "answer_confidence": 0.6718,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.7672,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0194,+ "1": 0.194,+ "2": 0.7866+ },+ "confidence": 0.4689,+ "answer_confidence": 0.7866,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.7234,+ "confidence": 0.7234,+ "answer_confidence": 0.7234,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 424,+ "output_tokens": 0,+ "state_tokens": 110,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.3741,+ "technical": 0.2268,+ "other": 0.3991+ },+ "confidence": 0.0252,+ "answer_confidence": 0.3991,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 0.9031,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.368,+ "1": 0.361,+ "2": 0.2711+ },+ "confidence": 0.0083,+ "answer_confidence": 0.368,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.2943,+ "confidence": 0.7057,+ "answer_confidence": 0.7057,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 94,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7312,+ "technical": 0.1851,+ "other": 0.0838+ },+ "confidence": 0.3184,+ "answer_confidence": 0.7312,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.8478,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0137,+ "1": 0.1248,+ "2": 0.8615+ },+ "confidence": 0.5931,+ "answer_confidence": 0.8615,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.8255,+ "confidence": 0.8255,+ "answer_confidence": 0.8255,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 481,+ "output_tokens": 0,+ "state_tokens": 129,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.142,+ "technical": 0.3006,+ "other": 0.5573+ },+ "confidence": 0.1222,+ "answer_confidence": 0.5573,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.2804,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.2652,+ "1": 0.1891,+ "2": 0.5456+ },+ "confidence": 0.092,+ "answer_confidence": 0.5456,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0403,+ "confidence": 0.9597,+ "answer_confidence": 0.9597,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 118,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.2504,+ "technical": 0.3345,+ "other": 0.4151+ },+ "confidence": 0.0187,+ "answer_confidence": 0.4151,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 1.766,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0394,+ "1": 0.1552,+ "2": 0.8054+ },+ "confidence": 0.4622,+ "answer_confidence": 0.8054,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.776,+ "confidence": 0.776,+ "answer_confidence": 0.776,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 286,+ "output_tokens": 0,+ "state_tokens": 64,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "technical",+ "probabilities": {+ "refund": 0.1839,+ "technical": 0.4141,+ "other": 0.4019+ },+ "confidence": 0.0507,+ "answer_confidence": 0.4141,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 0.9808,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.3293,+ "1": 0.3606,+ "2": 0.3101+ },+ "confidence": 0.0018,+ "answer_confidence": 0.3606,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "cancel": {+ "type": "noul",+ "noul": 0.2948,+ "confidence": 0.7052,+ "answer_confidence": 0.7052,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 160,+ "output_tokens": 0,+ "state_tokens": 22,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 24,+ 163+ ]+ ]+}diff --git a/laya-rust/tests/golden/typed-decisions/case_min_confidence_bad_key.json b/laya-rust/tests/golden/typed-decisions/case_min_confidence_bad_key.jsonnew file mode 100644index 00000000..7eab72d1--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_min_confidence_bad_key.json@@ -0,0 +1,45 @@+{+ "name": "min_confidence_bad_key",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'",+ "options": {+ "min_confidence": {+ "default": 0.5,+ "choice:3": 0.5+ }+ }+}diff --git a/laya-rust/tests/golden/typed-decisions/case_min_confidence_empty_map.json b/laya-rust/tests/golden/typed-decisions/case_min_confidence_empty_map.jsonnew file mode 100644index 00000000..28fa69e2--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_min_confidence_empty_map.json@@ -0,0 +1,42 @@+{+ "name": "min_confidence_empty_map",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "a min_confidence map must be a non-empty dict of bucket -> float, got {}",+ "options": {+ "min_confidence": {}+ }+}diff --git a/laya-rust/tests/golden/typed-decisions/case_min_confidence_invalid.json b/laya-rust/tests/golden/typed-decisions/case_min_confidence_invalid.jsonnew file mode 100644index 00000000..5ecd652a--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_min_confidence_invalid.json@@ -0,0 +1,42 @@+{+ "name": "min_confidence_invalid",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5",+ "options": {+ "min_confidence": 1.5+ }+}diff --git a/laya-rust/tests/golden/typed-decisions/case_min_confidence_map.json b/laya-rust/tests/golden/typed-decisions/case_min_confidence_map.jsonnew file mode 100644index 00000000..7958ebe3--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_min_confidence_map.json@@ -0,0 +1,1525 @@+{+ "name": "min_confidence_map",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.9911494255065918,+ -1.6601243019104004,+ -1.7693676948547363,+ -10000.0+ ],+ [+ -2.449965715408325,+ 0.2764223515987396,+ 2.5646214485168457,+ 1.5527721643447876+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0+ ],+ [+ -1.2292495965957642,+ 0.15064242482185364,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4736.37255859375,+ -3868.318359375+ ],+ [+ 4346.94384765625,+ -3559.03759765625+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ],+ [+ 4845.4169921875,+ -3968.390625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.8041,+ "technical": 0.101,+ "account": 0.0949+ },+ "confidence": 0.4261,+ "answer_confidence": 0.8041,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.9+ },+ "urgency": {+ "type": "score",+ "score": 2.153,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0112,+ "1": 0.0989,+ "2": 0.6156,+ "3": 0.2743+ },+ "confidence": 0.3273,+ "answer_confidence": 0.6156,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.7914,+ "confidence": 0.7914,+ "answer_confidence": 0.7914,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.6672,+ "confidence": 0.6672,+ "answer_confidence": 0.6672,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": {+ "choice:3-5": 0.9,+ "score:3-5": 0.5,+ "noul:2": 0.7+ }+ }+}diff --git a/laya-rust/tests/golden/typed-decisions/case_min_confidence_map_default.json b/laya-rust/tests/golden/typed-decisions/case_min_confidence_map_default.jsonnew file mode 100644index 00000000..390da6a7--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_min_confidence_map_default.json@@ -0,0 +1,936 @@+{+ "name": "min_confidence_map_default",+ "state": "Thanks, that is all sorted now.",+ "serialized_state": "Thanks, that is all sorted now.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 8061,+ 13,+ 326,+ 310,+ 512,+ 20045,+ 1024,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ -1.0131199359893799,+ -0.6967740058898926,+ -0.08753645420074463,+ -10000.0+ ],+ [+ 1.2697174549102783,+ 0.9968152046203613,+ 1.8926589488983154,+ -0.4043006896972656+ ],+ [+ 5.507878303527832,+ -0.7789955735206604,+ -10000.0,+ -10000.0+ ],+ [+ 0.3395328223705292,+ 0.1490795910358429,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4654.4765625,+ -3802.9765625+ ],+ [+ 4617.75,+ -3776.0673828125+ ],+ [+ 3557.7646484375,+ -2898.31103515625+ ],+ [+ 4780.2001953125,+ -3914.19091796875+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "account",+ "probabilities": {+ "billing": 0.2571,+ "technical": 0.3078,+ "account": 0.4351+ },+ "confidence": 0.0224,+ "answer_confidence": 0.4351,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.55+ },+ "urgency": {+ "type": "score",+ "score": 1.3152,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.2694,+ "1": 0.2166,+ "2": 0.4432,+ "3": 0.0707+ },+ "confidence": 0.1108,+ "answer_confidence": 0.4432,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.55+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0403,+ "confidence": 0.9597,+ "answer_confidence": 0.9597,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.55+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.476,+ "confidence": 0.524,+ "answer_confidence": 0.524,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.55+ }+ },+ "usage": {+ "input_tokens": 183,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": {+ "default": 0.55+ }+ }+}diff --git a/laya-rust/tests/golden/typed-decisions/case_min_confidence_scalar.json b/laya-rust/tests/golden/typed-decisions/case_min_confidence_scalar.jsonnew file mode 100644index 00000000..6b87ebce--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_min_confidence_scalar.json@@ -0,0 +1,1519 @@+{+ "name": "min_confidence_scalar",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.9911494255065918,+ -1.6601243019104004,+ -1.7693676948547363,+ -10000.0+ ],+ [+ -2.449965715408325,+ 0.2764223515987396,+ 2.5646214485168457,+ 1.5527721643447876+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0+ ],+ [+ -1.2292495965957642,+ 0.15064242482185364,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4736.37255859375,+ -3868.318359375+ ],+ [+ 4346.94384765625,+ -3559.03759765625+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ],+ [+ 4845.4169921875,+ -3968.390625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.8041,+ "technical": 0.101,+ "account": 0.0949+ },+ "confidence": 0.4261,+ "answer_confidence": 0.8041,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 2.153,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0112,+ "1": 0.0989,+ "2": 0.6156,+ "3": 0.2743+ },+ "confidence": 0.3273,+ "answer_confidence": 0.6156,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.7914,+ "confidence": 0.7914,+ "answer_confidence": 0.7914,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.6672,+ "confidence": 0.6672,+ "answer_confidence": 0.6672,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.6+ }+}diff --git a/laya-rust/tests/golden/typed-decisions/case_min_confidence_zero.json b/laya-rust/tests/golden/typed-decisions/case_min_confidence_zero.jsonnew file mode 100644index 00000000..ecfc4299--- /dev/null+++ b/laya-rust/tests/golden/typed-decisions/case_min_confidence_zero.json@@ -0,0 +1,1519 @@+{+ "name": "min_confidence_zero",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.9911494255065918,+ -1.6601243019104004,+ -1.7693676948547363,+ -10000.0+ ],+ [+ -2.449965715408325,+ 0.2764223515987396,+ 2.5646214485168457,+ 1.5527721643447876+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0+ ],+ [+ -1.2292495965957642,+ 0.15064242482185364,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4736.37255859375,+ -3868.318359375+ ],+ [+ 4346.94384765625,+ -3559.03759765625+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ],+ [+ 4845.4169921875,+ -3968.390625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.8041,+ "technical": 0.101,+ "account": 0.0949+ },+ "confidence": 0.4261,+ "answer_confidence": 0.8041,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "urgency": {+ "type": "score",+ "score": 2.153,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0112,+ "1": 0.0989,+ "2": 0.6156,+ "3": 0.2743+ },+ "confidence": 0.3273,+ "answer_confidence": 0.6156,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.7914,+ "confidence": 0.7914,+ "answer_confidence": 0.7914,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.6672,+ "confidence": 0.6672,+ "answer_confidence": 0.6672,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.0+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.0+ }+}diff --git a/laya-rust/tests/golden/typed-decisions/index.json b/laya-rust/tests/golden/typed-decisions/index.jsonindex eb70e281..8b7bcfdf 100644--- a/laya-rust/tests/golden/typed-decisions/index.json+++ b/laya-rust/tests/golden/typed-decisions/index.json@@ -337,6 +337,42 @@"expects_error": false,"kind": "batch"},+ {+ "name": "batch_min_confidence_scalar",+ "file": "case_batch_min_confidence_scalar.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_zero",+ "file": "case_batch_min_confidence_zero.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_map",+ "file": "case_batch_min_confidence_map.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_map_default",+ "file": "case_batch_min_confidence_map_default.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_empty_questions",+ "file": "case_batch_min_confidence_empty_questions.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_empty_states",+ "file": "case_batch_min_confidence_empty_states.json",+ "expects_error": false,+ "kind": "batch"+ },{"name": "batch_none_state","file": "case_batch_none_state.json",@@ -348,5 +384,58 @@"file": "case_batch_options_exceed_head.json","expects_error": true,"kind": "batch"+ },+ {+ "name": "batch_min_confidence_invalid",+ "file": "case_batch_min_confidence_invalid.json",+ "expects_error": true,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_before_none_state",+ "file": "case_batch_min_confidence_before_none_state.json",+ "expects_error": true,+ "kind": "batch"+ },+ {+ "name": "batch_min_confidence_bad_key",+ "file": "case_batch_min_confidence_bad_key.json",+ "expects_error": true,+ "kind": "batch"+ },+ {+ "name": "min_confidence_scalar",+ "file": "case_min_confidence_scalar.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_zero",+ "file": "case_min_confidence_zero.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_map",+ "file": "case_min_confidence_map.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_map_default",+ "file": "case_min_confidence_map_default.json",+ "expects_error": false+ },+ {+ "name": "min_confidence_invalid",+ "file": "case_min_confidence_invalid.json",+ "expects_error": true+ },+ {+ "name": "min_confidence_empty_map",+ "file": "case_min_confidence_empty_map.json",+ "expects_error": true+ },+ {+ "name": "min_confidence_bad_key",+ "file": "case_min_confidence_bad_key.json",+ "expects_error": true}]diff --git a/laya-rust/tests/golden/typed-decisions/meta.json b/laya-rust/tests/golden/typed-decisions/meta.jsonindex c261ad9b..f2736b71 100644--- a/laya-rust/tests/golden/typed-decisions/meta.json+++ b/laya-rust/tests/golden/typed-decisions/meta.json@@ -1,7 +1,7 @@{"generator": "tools/dump_golden.py","onnx_dir": "typed-decisions",- "laya_version": "0.4.14",+ "laya_version": "0.4.15","onnxruntime_version": "1.30.0","transformers_version": "5.17.0","config": {diff --git a/laya-rust/tests/min_confidence_tests.rs b/laya-rust/tests/min_confidence_tests.rsnew file mode 100644index 00000000..cd495cca--- /dev/null+++ b/laya-rust/tests/min_confidence_tests.rs@@ -0,0 +1,748 @@+//! The abstention gate (`min_confidence`, #361/#394): the router's side of it with fake predictors+//! (validation order, batch-wide application, nothing loaded on a bad threshold), the exact error+//! texts, and the recorded Python results for `confidence_probe.json` and the model cases.+//!+//! The probe is model-free: it holds the output of Python's own `confidence.py` functions. The+//! model-backed cases skip, with a `skip:` line, until the golden files carry them or without+//! `LAYA_ONNX_ROOT`.++mod common;++use common::routing_golden;+use indexmap::IndexMap;+use laya::answers::{ActionInfo, Answer, ChoiceAnswer, LayaResult, NoulAnswer, ScoreAnswer, Usage};+use laya::batch_prediction::BatchOptions;+use laya::confidence::{GATE_STATES, MinConfidence, bucket_keys};+use laya::error::{LayaError, Result};+use laya::laya_options::LayaCheckpoint;+use laya::laya_router::{+ BatchRequest, LayaBatchPredictor, LayaPredictor, LayaRouter, LayaRouterOptions,+};+use laya::predict_options::PredictOptions;+use laya::questions::{Question, QuestionSet};+use serde_json::Value;+use std::collections::HashMap;+use std::sync::{Arc, Mutex};++// ---------------------------------------------------------------------- model-free++type Builds = Arc<Mutex<Vec<LayaCheckpoint>>>;+type Seen = Arc<Mutex<Vec<BatchOptions>>>;++/// A predictor whose answers carry the confidence its state names: the state `"0.8"` gives a+/// choice answer with `answer_confidence` 0.8 and a noul answer with the same.+struct Fake {+ seen: Seen,+}++fn answers_for(state: &Value) -> LayaResult {+ let confidence: f64 = state.as_str().and_then(|s| s.parse().ok()).unwrap_or(0.5);+ let mut probabilities = IndexMap::new();+ probabilities.insert("a".to_string(), confidence);+ probabilities.insert("b".to_string(), 1.0 - confidence);+ let choice = Answer::Choice(ChoiceAnswer::new(+ "a",+ probabilities,+ 0.1,+ confidence,+ ActionInfo::default(),+ ));+ let noul = Answer::Noul(NoulAnswer::new(+ confidence,+ 0.1,+ confidence,+ ActionInfo::default(),+ ));+ let mut by_id = HashMap::new();+ by_id.insert("pick".to_string(), choice);+ by_id.insert("flag".to_string(), noul);+ LayaResult::new(+ "fake",+ vec!["pick".to_string(), "flag".to_string()],+ by_id,+ Usage::default(),+ )+}++impl LayaPredictor for Fake {+ fn predict(&self, state: Value, _questions: &QuestionSet) -> Result<LayaResult> {+ Ok(answers_for(&state))+ }++ fn as_batch_predictor(&self) -> Option<&dyn LayaBatchPredictor> {+ Some(self)+ }+}++impl LayaBatchPredictor for Fake {+ fn predict_batch(+ &self,+ states: &[Value],+ _questions: &QuestionSet,+ options: &BatchOptions,+ ) -> Result<Vec<LayaResult>> {+ self.seen.lock().unwrap().push(options.clone());+ Ok(states.iter().map(answers_for).collect())+ }+}++fn fixture() -> (LayaRouter, Builds, Seen) {+ let builds: Builds = Arc::default();+ let seen: Seen = Arc::default();+ let (factory_builds, factory_seen) = (Arc::clone(&builds), Arc::clone(&seen));+ let router = LayaRouter::with_factory(LayaRouterOptions::default(), move |checkpoint, _| {+ factory_builds.lock().unwrap().push(checkpoint);+ Ok(Arc::new(Fake {+ seen: Arc::clone(&factory_seen),+ }) as Arc<dyn LayaPredictor>)+ })+ .expect("a fake router never fails to construct");+ (router, builds, seen)+}++fn questions() -> QuestionSet {+ QuestionSet::new()+ .with(+ "pick",+ Question::choice_labels("Which?", ["a", "b"]).unwrap(),+ )+ .unwrap()+}++fn scalar(threshold: f64) -> PredictOptions {+ PredictOptions::new().min_confidence(MinConfidence::scalar(threshold).unwrap())+}++#[test]+fn router_predict_gates_with_the_options_threshold() {+ let (router, _, _) = fixture();+ let result = router+ .predict_with_options(+ "0.8",+ &questions(),+ Some("en"),+ None,+ None,+ None,+ &scalar(0.9),+ )+ .unwrap();+ assert_eq!(result["pick"].abstention(), Some("abstained"));+ assert!(result["pick"].low_confidence());+ assert_eq!(result["pick"].abstention_threshold(), Some(0.9));+ let passed = router+ .predict_with_options(+ "0.8",+ &questions(),+ Some("en"),+ None,+ None,+ None,+ &scalar(0.5),+ )+ .unwrap();+ assert_eq!(passed["pick"].abstention(), Some("passed"));+ assert!(passed.routing().is_some(), "the routing decision is kept");+}++#[test]+fn router_predict_without_a_threshold_reports_no_gate() {+ let (router, _, _) = fixture();+ for result in [+ router+ .predict("0.8", &questions(), Some("en"), None, None)+ .unwrap(),+ router+ .predict_with_options(+ "0.8",+ &questions(),+ Some("en"),+ None,+ None,+ None,+ &PredictOptions::default(),+ )+ .unwrap(),+ ] {+ for (_, answer) in result.iter() {+ assert!(!answer.low_confidence());+ assert_eq!(answer.abstention(), None);+ assert_eq!(answer.abstention_threshold(), None);+ }+ }+}++#[test]+fn router_predict_refuses_a_bad_threshold_before_routing_or_loading() {+ let (router, builds, _) = fixture();+ let bad = PredictOptions::new().min_confidence(MinConfidence::Scalar(1.5));+ let error = router+ .predict_with_options("0.8", &questions(), Some("en"), None, None, None, &bad)+ .unwrap_err();+ assert!(matches!(error, LayaError::InvalidMinConfidence(_)));+ assert_eq!(+ error.to_string(),+ "min_confidence must be a float in [0.0, 1.0], got 1.5"+ );+ assert!(builds.lock().unwrap().is_empty(), "nothing was loaded");++ // Router.predict checks the state first (router.py:1336), then the threshold (:1338).+ let error = router+ .predict_with_options(+ Value::Null,+ &questions(),+ Some("en"),+ None,+ None,+ None,+ &bad,+ )+ .unwrap_err();+ assert!(matches!(error, LayaError::InvalidState(_)), "{error}");+}++#[test]+fn router_predict_batch_gates_every_result_with_one_threshold() {+ let (router, _, seen) = fixture();+ let requests = [+ BatchRequest::new("0.95", questions()).model("en"),+ BatchRequest::new("0.2", questions()).model("multi"),+ BatchRequest::new("0.8", questions()).model("en"),+ ];+ let options = BatchOptions::new().min_confidence(MinConfidence::scalar(0.9).unwrap());+ let results = router.predict_batch(&requests, &options).unwrap();+ let states: Vec<_> = results+ .iter()+ .map(|r| (r["pick"].abstention(), r["flag"].abstention()))+ .collect();+ assert_eq!(+ states,+ [+ (Some("passed"), Some("passed")),+ (Some("abstained"), Some("abstained")),+ (Some("abstained"), Some("abstained")),+ ]+ );+ for result in &results {+ assert_eq!(result["pick"].abstention_threshold(), Some(0.9));+ }+ // The threshold is the router's to apply; it is no part of the grouping or of what the+ // predictors are asked.+ assert!(+ seen.lock()+ .unwrap()+ .iter()+ .all(|o| o.min_confidence.is_none())+ );+ assert_eq!(seen.lock().unwrap().len(), 2, "one call per checkpoint");+}++#[test]+fn router_predict_batch_refuses_a_bad_threshold_before_routing_or_loading() {+ let (router, builds, _) = fixture();+ // Even the routing error is not reached: the request names a checkpoint that does not exist.+ let requests = [BatchRequest::new("0.8", questions()).model("no-such-model")];+ let mut map = IndexMap::new();+ map.insert("choice:1".to_string(), 0.5);+ let options = BatchOptions::new().min_confidence(MinConfidence::Map(map));+ let error = router.predict_batch(&requests, &options).unwrap_err();+ assert!(+ matches!(error, LayaError::InvalidMinConfidence(_)),+ "{error}"+ );+ assert!(builds.lock().unwrap().is_empty());+}++#[test]+fn router_predict_batch_with_no_requests_still_validates() {+ let (router, _, _) = fixture();+ let options = BatchOptions::new().min_confidence(MinConfidence::Scalar(-1.0));+ assert!(router.predict_batch(&[], &options).is_err());+ assert!(+ router+ .predict_batch(&[], &BatchOptions::new())+ .unwrap()+ .is_empty()+ );+}++#[test]+fn the_public_vocabulary_is_python_s() {+ assert_eq!(GATE_STATES, ["passed", "abstained", "unevaluated"]);+ let keys = bucket_keys();+ assert_eq!(keys.len(), 12);+ for qtype in ["choice", "score", "noul"] {+ for size in ["2", "3-5", "6-10", "11+"] {+ assert!(keys.contains(&format!("{qtype}:{size}")));+ }+ }+}++// ---------------------------------------------------------------------- confidence_probe.json++/// Python's own `laya/confidence.py` over the shared synthetic inputs of+/// `tools/confidence_cases.py`. A float field is a JSON number, or "nan"/"inf"/"-inf".+fn probe() -> Option<Value> {+ let probe = routing_golden::load("confidence_probe.json");+ if probe.is_none() {+ eprintln!("skip: routing golden not found (tests/golden/routing/confidence_probe.json)");+ }+ probe+}++fn try_float(value: &Value) -> Option<f64> {+ match value {+ Value::Number(n) => n.as_f64(),+ Value::String(s) => match s.as_str() {+ "nan" => Some(f64::NAN),+ "inf" => Some(f64::INFINITY),+ "-inf" => Some(f64::NEG_INFINITY),+ _ => None,+ },+ _ => None,+ }+}++fn float(value: &Value) -> f64 {+ try_float(value).unwrap_or_else(|| panic!("not a float: {value}"))+}++/// A float field the answer may not have reported: a typed answer reads it as NaN.+fn reported(answer: &Value, key: &str) -> f64 {+ answer.get(key).map_or(f64::NAN, float)+}++/// A typed answer for an entry of the probe, or `None` for the shapes a typed answer cannot hold+/// (`"typed": false`: a boolean or text "probability", a non-integer score level).+fn build_answer(entry: &Value) -> Option<Answer> {+ let confidence = reported(entry, "confidence");+ let answer_confidence = reported(entry, "answer_confidence");+ let action = ActionInfo::default();+ let probabilities = entry.get("probabilities").and_then(Value::as_object);+ match entry["type"].as_str()? {+ "choice" => {+ let mut map = IndexMap::new();+ for (label, value) in probabilities? {+ map.insert(label.clone(), try_float(value)?);+ }+ let first = map.keys().next().cloned().unwrap_or_default();+ Some(Answer::Choice(ChoiceAnswer::new(+ first,+ map,+ confidence,+ answer_confidence,+ action,+ )))+ }+ "score" => {+ let mut levels: Vec<(i64, f64)> = Vec::new();+ for (level, value) in probabilities? {+ levels.push((level.parse().ok()?, try_float(value)?));+ }+ levels.sort_by_key(|(level, _)| *level);+ Some(Answer::Score(ScoreAnswer::new(+ 0.0,+ Vec::new(),+ levels.into_iter().map(|(_, p)| p).collect(),+ confidence,+ answer_confidence,+ action,+ )))+ }+ "noul" => Some(Answer::Noul(NoulAnswer::new(+ float(&entry["noul"]),+ confidence,+ answer_confidence,+ action,+ ))),+ _ => None,+ }+}++fn result_of(answers: Vec<(String, Answer)>) -> LayaResult {+ LayaResult::new(+ "probe",+ answers.iter().map(|(id, _)| id.clone()).collect(),+ answers.into_iter().collect(),+ Usage::default(),+ )+}++/// A threshold as the probe spells it: null (unset), a float, or an ordered bucket map.+fn threshold(value: &Value) -> Option<MinConfidence> {+ match value {+ Value::Null => None,+ Value::Object(map) => Some(+ MinConfidence::map(map.iter().map(|(k, v)| (k.clone(), float(v)))).expect("valid map"),+ ),+ other => Some(MinConfidence::scalar(float(other)).expect("valid scalar")),+ }+}++/// A router whose predictor hands back whatever result is in the slot, so a test can feed a+/// gated result back in, as a reused payload is.+struct Slot(Mutex<LayaResult>);++impl LayaPredictor for Slot {+ fn predict(&self, _state: Value, _questions: &QuestionSet) -> Result<LayaResult> {+ Ok(self.0.lock().unwrap().clone())+ }+}++fn slot_router(result: LayaResult) -> (LayaRouter, Arc<Slot>) {+ let slot = Arc::new(Slot(Mutex::new(result)));+ let shared = Arc::clone(&slot);+ let router = LayaRouter::with_factory(LayaRouterOptions::default(), move |_, _| {+ Ok(Arc::clone(&shared) as Arc<dyn LayaPredictor>)+ })+ .expect("a fake router never fails to construct");+ (router, slot)+}++/// Gate the slot's result at `min_confidence` through `Router::predict_with_options`, and store the+/// gated result back, so the next call gates a reused one.+fn gate_slot(+ router: &LayaRouter,+ slot: &Slot,+ min_confidence: Option<MinConfidence>,+) -> LayaResult {+ let mut options = PredictOptions::new();+ options.min_confidence = min_confidence;+ let result = router+ .predict_with_options(+ "state",+ &questions(),+ Some("en"),+ None,+ None,+ None,+ &options,+ )+ .expect("the gate does not fail on a valid threshold");+ *slot.0.lock().unwrap() = result.clone();+ result+}++fn assert_same_float(got: f64, want: f64, context: &str) {+ assert!(+ got.to_bits() == want.to_bits() || (got.is_nan() && want.is_nan()),+ "{context}: got {got:?}, python gave {want:?}"+ );+}++/// What the gate wrote on each answer must be exactly what Python wrote: `low_confidence` only+/// when flagged, `abstention` and `abstention_threshold` only when a threshold was set.+fn assert_verdicts(result: &LayaResult, expected: &Value, context: &str) {+ let expected = expected.as_object().expect("expected verdicts");+ assert_eq!(result.len(), expected.len(), "{context}: answer count");+ for (id, want) in expected {+ let answer = &result[id.as_str()];+ let where_answer = format!("{context} / {id}");+ assert_eq!(+ want.get("low_confidence").is_some(),+ answer.low_confidence(),+ "{where_answer}: low_confidence"+ );+ if let Some(flag) = want.get("low_confidence") {+ assert_eq!(flag, &Value::Bool(true), "{where_answer}: only ever true");+ }+ assert_eq!(+ want.get("abstention").and_then(Value::as_str),+ answer.abstention(),+ "{where_answer}: abstention"+ );+ match (+ want.get("abstention_threshold"),+ answer.abstention_threshold(),+ ) {+ (None, None) => {}+ (Some(want), Some(got)) => {+ assert_same_float(got, float(want), &format!("{where_answer}: threshold"));+ }+ (want, got) => panic!("{where_answer}: threshold {got:?} vs python {want:?}"),+ }+ }+}++fn gate_answers(probe: &Value) -> Vec<(String, Answer)> {+ probe["gate_answers"]+ .as_object()+ .expect("gate_answers")+ .iter()+ .map(|(id, entry)| {+ (+ id.clone(),+ build_answer(entry).expect("every gate answer is typed"),+ )+ })+ .collect()+}++#[test]+fn probe_vocabulary_matches_python() {+ let Some(probe) = probe() else { return };+ let mut keys = bucket_keys();+ keys.sort();+ let expected: Vec<&str> = probe["bucket_keys"]+ .as_array()+ .unwrap()+ .iter()+ .map(|k| k.as_str().unwrap())+ .collect();+ assert_eq!(keys, expected);+ let states: Vec<&str> = probe["gate_states"]+ .as_array()+ .unwrap()+ .iter()+ .map(|k| k.as_str().unwrap())+ .collect();+ assert_eq!(GATE_STATES.to_vec(), states);+}++#[test]+fn probe_scalar_thresholds_validate_as_python_does() {+ let Some(probe) = probe() else { return };+ let cases = probe["scalars"].as_array().unwrap();+ assert!(!cases.is_empty());+ for case in cases {+ let value = float(&case["value"]);+ let got = MinConfidence::scalar(value);+ if case["valid"].as_bool().unwrap() {+ let want = float(&case["result"]);+ match got {+ Ok(MinConfidence::Scalar(kept)) => assert_same_float(kept, want, "scalar"),+ other => panic!("{value:?}: python accepted it, got {other:?}"),+ }+ } else {+ let error = got.expect_err(&format!("{value:?}: python rejected it"));+ assert!(matches!(error, LayaError::InvalidMinConfidence(_)));+ assert_eq!(+ error.to_string(),+ case["error_message"].as_str().unwrap(),+ "{value:?}"+ );+ }+ }+}++#[test]+fn probe_threshold_maps_validate_as_python_does() {+ let Some(probe) = probe() else { return };+ let cases = probe["maps"].as_array().unwrap();+ assert!(!cases.is_empty());+ for case in cases {+ let pairs: Vec<(String, f64)> = case["pairs"]+ .as_array()+ .unwrap()+ .iter()+ .map(|pair| (pair[0].as_str().unwrap().to_string(), float(&pair[1])))+ .collect();+ let got = MinConfidence::map(pairs.clone());+ if case["valid"].as_bool().unwrap() {+ let Ok(MinConfidence::Map(kept)) = got else {+ panic!("{pairs:?}: python accepted it, got {got:?}");+ };+ // A repeated key keeps its first position and its last value, as a dict does.+ let want: Vec<(String, f64)> = case["result"]+ .as_array()+ .unwrap()+ .iter()+ .map(|pair| (pair[0].as_str().unwrap().to_string(), float(&pair[1])))+ .collect();+ let got: Vec<(String, f64)> = kept.into_iter().collect();+ assert_eq!(got.len(), want.len(), "{pairs:?}");+ for ((got_key, got_value), (want_key, want_value)) in got.iter().zip(&want) {+ assert_eq!(got_key, want_key, "{pairs:?}");+ assert_same_float(*got_value, *want_value, &format!("{pairs:?}"));+ }+ } else {+ let error = got.expect_err(&format!("{pairs:?}: python rejected it"));+ assert!(matches!(error, LayaError::InvalidMinConfidence(_)));+ assert_eq!(+ error.to_string(),+ case["error_message"].as_str().unwrap(),+ "{pairs:?}"+ );+ }+ }+}++#[test]+fn probe_option_buckets_match_python() {+ let Some(probe) = probe() else { return };+ let cases = probe["buckets"].as_array().unwrap();+ assert!(!cases.is_empty());+ for case in cases {+ let answer = build_answer(&case["answer"]).expect("a typed answer");+ assert_eq!(+ answer.option_bucket().as_deref(),+ case["bucket"].as_str(),+ "{}",+ case["answer"]+ );+ }+}++#[test]+fn probe_resolved_thresholds_match_python() {+ let Some(probe) = probe() else { return };+ let cases = probe["resolve"].as_array().unwrap();+ assert!(!cases.is_empty());+ for case in cases {+ let answer = build_answer(&case["answer"]).expect("a typed answer");+ let (router, slot) = slot_router(result_of(vec![("q".to_string(), answer)]));+ let gated = gate_slot(&router, &slot, threshold(&case["thresholds"]));+ assert_same_float(+ gated["q"].abstention_threshold().expect("a gate ran"),+ float(&case["result"]),+ &format!("{} under {}", case["answer"], case["thresholds"]),+ );+ }+}++#[test]+fn probe_gate_verdicts_match_python() {+ let Some(probe) = probe() else { return };+ let cases = probe["gate"].as_array().unwrap();+ assert!(!cases.is_empty());+ for case in cases {+ let (router, slot) = slot_router(result_of(gate_answers(&probe)));+ let gated = gate_slot(&router, &slot, threshold(&case["threshold"]));+ assert_verdicts(+ &gated,+ &case["expected"],+ &format!("threshold {}", case["threshold"]),+ );+ }+}++/// #910: python's reused payload carried a stale flag across thresholds. A result here is a+/// value, but one gated and gated again must still end up as the last threshold alone decides.+#[test]+fn probe_reused_results_never_keep_an_earlier_verdict() {+ let Some(probe) = probe() else { return };+ let sequences = probe["reuse"].as_array().unwrap();+ assert!(!sequences.is_empty());+ for (index, sequence) in sequences.iter().enumerate() {+ let (router, slot) = slot_router(result_of(gate_answers(&probe)));+ for (step, case) in sequence["steps"].as_array().unwrap().iter().enumerate() {+ let gated = gate_slot(&router, &slot, threshold(&case["threshold"]));+ assert_verdicts(+ &gated,+ &case["expected"],+ &format!("reuse {index} step {step} at {}", case["threshold"]),+ );+ }+ }+}++#[test]+fn probe_jev_confidence_matches_python() {+ let Some(probe) = probe() else { return };+ let cases = probe["jev"].as_array().unwrap();+ assert!(!cases.is_empty());+ let (mut typed, mut untyped) = (0, 0);+ for case in cases {+ let name = case["name"].as_str().unwrap();+ let want = match &case["result"] {+ Value::Null => None,+ value => Some(float(value)),+ };+ // The shapes a typed answer cannot hold are the ones python refuses: no result.+ let Some(answer) = build_answer(&case["answer"]) else {+ assert!(+ !case["typed"].as_bool().unwrap(),+ "{name}: should be buildable"+ );+ assert_eq!(+ want, None,+ "{name}: python refuses what a typed answer cannot hold"+ );+ untyped += 1;+ continue;+ };+ assert!(+ case["typed"].as_bool().unwrap(),+ "{name}: typed answer built from an untyped case"+ );+ typed += 1;+ match (answer.jev_confidence(), want) {+ (None, None) => {}+ (Some(got), Some(want)) => assert_same_float(got, want, name),+ (got, want) => panic!("{name}: got {got:?}, python gave {want:?}"),+ }+ }+ assert!(+ typed > 60 && untyped >= 4,+ "typed {typed}, untyped {untyped}"+ );+}++// ---------------------------------------------------------------------- model-backed++/// The `min_confidence` cases of the checkpoint goldens (`min_confidence_*` for `system_one`,+/// `batch_min_confidence_*` for `predict_batch`) against the real model. The parity tests+/// (`predict_parity_tests`, `predict_batch_tests`) run every recorded case with its options; this+/// is what says the cases exist at all, since goldens recorded before the gate hold none and+/// would otherwise pass those tests without ever exercising it.+#[test]+fn the_recorded_min_confidence_cases_reproduce() {+ let mut checked_any = false;+ for checkpoint in common::engines::CHECKPOINTS {+ let Some(engine) = common::engines::load(checkpoint) else {+ continue;+ };+ let golden = common::golden_data::CheckpointGoldenData::for_checkpoint(checkpoint.subdir());+ if !golden.available() {+ eprintln!("skip: no golden data for '{}'", checkpoint.subdir());+ continue;+ }+ let single: Vec<_> = golden+ .success_cases()+ .into_iter()+ .filter(|case| case.name.starts_with("min_confidence_"))+ .collect();+ let batch: Vec<_> = golden+ .batch_cases()+ .into_iter()+ .filter(|case| case.name.starts_with("batch_min_confidence_"))+ .collect();+ if single.is_empty() || batch.is_empty() {+ eprintln!(+ "skip: the '{}' goldens hold no min_confidence cases; regenerate them with \+ tools/regen_golden.py",+ checkpoint.subdir()+ );+ continue;+ }+ checked_any = true;+ let mut gated_answers = 0;+ for case in single {+ let root = golden.load_case(&case);+ let state = root["state"].clone();+ let questions = common::golden_data::build_questions(&root["questions"])+ .unwrap_or_else(|e| panic!("{}/{case}: {e}", checkpoint.subdir()));+ let options = common::golden_data::recorded_predict_options(&root);+ assert!(+ options.min_confidence.is_some(),+ "{case}: records a threshold"+ );+ let result = engine+ .predict_with_options(state, &questions, &options)+ .unwrap_or_else(|e| panic!("{}/{case}: {e}", checkpoint.subdir()));+ common::result_compare::assert_result_matches(+ &format!("{}/{case}", checkpoint.subdir()),+ &root["result"],+ &result,+ );+ gated_answers += result+ .iter()+ .filter(|(_, answer)| answer.abstention().is_some())+ .count();+ }+ assert!(gated_answers > 0, "no recorded case reported a gate state");+ }+ if !checked_any {+ eprintln!("skip: no ONNX artifacts + min_confidence golden cases found for any checkpoint");+ }+}diff --git a/laya-rust/tests/predict_batch_tests.rs b/laya-rust/tests/predict_batch_tests.rsindex efc1adc1..07f49a51 100644--- a/laya-rust/tests/predict_batch_tests.rs+++ b/laya-rust/tests/predict_batch_tests.rs@@ -440,6 +440,7 @@ fn batch_options(options: &Value) -> BatchOptions {.get("sort_by_length").and_then(Value::as_bool).unwrap_or(false);+ built.min_confidence = golden_data::recorded_min_confidence(options);built}@@ -580,6 +581,15 @@ fn rejects_the_same_batch_requests_python_rejects() {);assert_eq!(message, error.to_string(), "{}/{case}", checkpoint.subdir());}+ "ValueError" if root["options"].get("min_confidence").is_some() => {+ // A rejected threshold: Python's text, whole, checked before any state.+ assert!(+ matches!(error, LayaError::InvalidMinConfidence(_)),+ "{}/{case}: wrong error variant: {error:?}",+ checkpoint.subdir()+ );+ assert_eq!(message, error.to_string(), "{}/{case}", checkpoint.subdir());+ }"ValueError" => {assert!(matches!(error, LayaError::NoMarkers { .. }),diff --git a/laya-rust/tests/predict_parity_tests.rs b/laya-rust/tests/predict_parity_tests.rsindex a894c05d..f126b9c7 100644--- a/laya-rust/tests/predict_parity_tests.rs+++ b/laya-rust/tests/predict_parity_tests.rs@@ -69,7 +69,11 @@ fn reproduces_the_recorded_answers() {.unwrap_or_else(|e| panic!("{}/{case}: {e}", checkpoint.subdir()));let result = engine- .predict(state, &questions)+ .predict_with_options(+ state,+ &questions,+ &golden_data::recorded_predict_options(&root),+ ).unwrap_or_else(|e| panic!("{}/{case}: predict failed: {e}", checkpoint.subdir()));let expected = &root["result"];let where_case = format!("{}/{case}", checkpoint.subdir());@@ -132,6 +136,7 @@ fn reproduces_the_recorded_answers() {&format!("{where_q}.action.act_probability"),);+ common::result_compare::assert_gate_matches(want, got, &where_q);match got {Answer::Choice(choice) => assert_choice(want, choice, &where_q),Answer::Score(score) => assert_score(want, score, &where_q),@@ -178,7 +183,11 @@ fn reproduces_the_recorded_answers_for_the_split_layout() {.unwrap_or_else(|e| panic!("{}/{case}: {e}", checkpoint.subdir()));let result = engine- .predict(state, &questions)+ .predict_with_options(+ state,+ &questions,+ &golden_data::recorded_predict_options(&root),+ ).unwrap_or_else(|e| panic!("{}/{case}: predict failed: {e}", checkpoint.subdir()));let expected = &root["result"];let where_case = format!("{}/{case} (split layout)", checkpoint.subdir());@@ -217,6 +226,7 @@ fn reproduces_the_recorded_answers_for_the_split_layout() {PROB_TOLERANCE,&format!("{where_q}.answer_confidence"),);+ common::result_compare::assert_gate_matches(want, got, &where_q);match got {Answer::Choice(choice) => assert_choice(want, choice, &where_q),Answer::Score(score) => assert_score(want, score, &where_q),@@ -254,12 +264,33 @@ fn rejects_the_same_questions_python_rejects() {let questions = golden_data::build_questions(&root["questions"]).unwrap_or_else(|e| panic!("{}/{case}: {e}", checkpoint.subdir()));- let error = engine.predict(state, &questions).expect_err(&format!(- "{}/{case}: expected predict to fail",- checkpoint.subdir()- ));+ let error = engine+ .predict_with_options(+ state,+ &questions,+ &golden_data::recorded_predict_options(&root),+ )+ .expect_err(&format!(+ "{}/{case}: expected predict to fail",+ checkpoint.subdir()+ ));let expected_message = root["error_message"].as_str().unwrap();let actual_message = error.to_string();+ // A rejected threshold is Python's ValueError text, whole.+ if root.get("options").is_some() {+ assert!(+ matches!(error, LayaError::InvalidMinConfidence(_)),+ "{}/{case}: expected LayaError::InvalidMinConfidence, got {error:?}",+ checkpoint.subdir()+ );+ assert_eq!(+ expected_message,+ actual_message,+ "{}/{case}: error message",+ checkpoint.subdir()+ );+ continue;+ }assert!(actual_message.starts_with(expected_message),"{}/{case}: error message mismatch\n python: {expected_message}\n rust: {actual_message}",diff --git a/laya-rust/tools/confidence_cases.py b/laya-rust/tools/confidence_cases.pynew file mode 100644index 00000000..386c57db--- /dev/null+++ b/laya-rust/tools/confidence_cases.py@@ -0,0 +1,238 @@+"""Synthetic inputs for the model-free abstention-gate probe, `routing/confidence_probe.json`.++`dump_routing_golden.build_confidence_probe` feeds each of these to Python's own+`laya/confidence.py` and records what it returned, so a port is checked against the shipping+code on the same inputs without loading a model. This list is SDK-neutral: it is the shared+corpus for the .NET and Rust ports, and each port keeps a verbatim copy.++Everything here is representable by a typed SDK answer (a choice with ordered probabilities, a+score with ordered probabilities, a noul with none), except the jev cases flagged `typed: false`,+which exist to pin Python's refusal of shapes a typed answer cannot hold (a boolean or text+"probability", a non-integer score level). A threshold is always a float: JSON `0.5`, never `1`.++Non-finite floats are written by the dumper as the strings "nan", "inf" and "-inf"; a missing+`answer_confidence` or `confidence` key is a field the answer did not report, and a typed port+reads it the same way as NaN (no usable number).+"""+import random++NAN = float("nan")+INF = float("inf")++# ---------------------------------------------------------------------------------------------+# check_min_confidence: scalar thresholds. Only floats, since a typed SDK takes a float.+# ---------------------------------------------------------------------------------------------+SCALAR_THRESHOLDS = [+ 0.0, 1.0, 0.5, 0.35, 1e-09, 0.9999999999, 1e-05,+ -0.1, 1.5, -1e-05, 1.0000000001, 2.0, 100.0, 1e16, 1.2345678901234567e-07,+ NAN, INF, float("-inf"),+]++# ---------------------------------------------------------------------------------------------+# check_min_confidence_map: ordered (key, value) pairs, so an error names the first offender.+# ---------------------------------------------------------------------------------------------+_ALL_BUCKETS = ["%s:%s" % (qt, size) for qt in ("choice", "score", "noul") for size in ("2", "3-5", "6-10", "11+")]++MAP_THRESHOLDS = [+ [],+ [("default", 0.4)],+ [("choice:2", 0.7), ("default", 0.4)],+ [("choice:3-5", 0.5)],+ [("score:11+", 0.0), ("noul:2", 1.0)],+ [(key, round(0.05 * (i + 1), 2)) for i, key in enumerate(_ALL_BUCKETS)] + [("default", 0.5)],+ [("choice:2", 1.5)],+ [("choice:2", NAN)],+ [("choice:2", -0.1)],+ [("choice:2", INF)],+ [("bogus", 0.5)],+ [("Choice:2", 0.5)],+ [("choice:3", 0.5)],+ [("choice:2", 0.5), ("bogus", 0.5)],+ [("choice:2", 1.5), ("bogus", 0.1)],+ [("bogus", 1.5), ("choice:2", 0.1)],+ [("choice:2", 0.5), ("choice:3-5", 2.0), ("nope", 0.1)],+ [("it's", 0.5)],+ [("say \"hi\"", 0.5)],+ [("it's \"x\"", 0.5)],+ [("tab\there", 0.5)],+ [("café", 0.5)],+ [("", 0.5)],+ [("default", 0.5), ("default", 0.25)],+]+++def _choice(*probs, ac=None, conf=None):+ out = {"type": "choice", "probabilities": {chr(ord("a") + i): p for i, p in enumerate(probs)}}+ _fields(out, ac, conf)+ return out+++def _score(*probs, ac=None, conf=None):+ out = {"type": "score", "probabilities": {str(i): p for i, p in enumerate(probs)}}+ _fields(out, ac, conf)+ return out+++def _noul(ac=None, conf=None, p=0.9):+ out = {"type": "noul", "noul": p}+ _fields(out, ac, conf)+ return out+++def _fields(out, ac, conf):+ if ac is not None:+ out["answer_confidence"] = ac+ if conf is not None:+ out["confidence"] = conf+++def _uniform(kind, k, ac=None, conf=None):+ return (_choice if kind == "choice" else _score)(*([round(1.0 / k, 4)] * k), ac=ac, conf=conf)+++# ---------------------------------------------------------------------------------------------+# _option_bucket: every type at every option count the bands distinguish.+# ---------------------------------------------------------------------------------------------+BUCKET_ANSWERS = (+ [_uniform("choice", k) for k in (1, 2, 3, 5, 6, 10, 11, 13)]+ + [_uniform("score", k) for k in (1, 2, 3, 5, 6, 10, 11, 13)]+ + [_noul()]+ + [{"type": "choice", "probabilities": {}}, {"type": "score", "probabilities": {}}]+)++# ---------------------------------------------------------------------------------------------+# resolve_min_confidence: a validated map against answers of each bucket, and an unnamed one.+# ---------------------------------------------------------------------------------------------+RESOLVE_MAPS = [+ {"choice:2": 0.7, "default": 0.4},+ {"choice:3-5": 0.5},+ {"default": 0.25},+ {"score:6-10": 0.8, "noul:2": 0.3, "choice:11+": 0.6},+]+RESOLVE_ANSWERS = [_uniform("choice", 2), _uniform("choice", 4), _uniform("choice", 8), _uniform("choice", 12),+ _uniform("score", 2), _uniform("score", 4), _uniform("score", 8), _uniform("score", 12),+ _noul(), {"type": "choice", "probabilities": {}}]++# ---------------------------------------------------------------------------------------------+# apply_confidence_gate: a result with these answers, gated at each threshold. A threshold is+# None (unset: nothing is written), a float, or a map.+# ---------------------------------------------------------------------------------------------+GATE_ANSWERS = {+ "choice2": _choice(0.6, 0.4, ac=0.6, conf=0.029),+ "choice4": _choice(0.45, 0.25, 0.2, 0.1, ac=0.45, conf=0.12),+ "choice8": _uniform("choice", 8, ac=0.3, conf=0.05),+ "choice12": _uniform("choice", 12, ac=0.15, conf=0.02),+ "score2": _score(0.7, 0.3, ac=0.7, conf=0.1),+ "score4": _score(0.1, 0.5, 0.3, 0.1, ac=0.5, conf=0.2),+ "score8": _uniform("score", 8, ac=0.125, conf=0.01),+ "noul": _noul(ac=0.8, conf=0.8, p=0.8),+ "at_half": _choice(0.5, 0.5, ac=0.5, conf=0.0),+ "fallback_conf": _choice(0.5, 0.5, conf=0.3),+ "nan_answer_conf": _choice(0.5, 0.5, ac=NAN, conf=0.7),+ "inf_answer_conf": _choice(0.5, 0.5, ac=INF, conf=0.2),+ "unusable_both": _choice(0.5, 0.5, ac=NAN, conf=INF),+ "no_confidence": _choice(0.5, 0.5),+ "empty_choice": {"type": "choice", "probabilities": {}, "answer_confidence": 0.4},+}++GATE_THRESHOLDS = [+ None,+ 0.0,+ 1.0,+ 0.5,+ 0.35,+ 0.2,+ {"default": 0.4},+ {"choice:2": 0.7, "default": 0.4},+ {"choice:3-5": 0.5},+ {"choice:2": 0.7, "choice:3-5": 0.5, "choice:6-10": 0.4, "choice:11+": 0.1,+ "score:2": 0.8, "score:3-5": 0.6, "score:6-10": 0.2, "noul:2": 0.9},+ {"noul:2": 0.0, "score:3-5": 1.0},+]++# The same answers gated again and again, as one reused payload is (#910): each step's verdict must+# depend only on that step's threshold, never on an earlier one.+REUSE_SEQUENCES = [+ [0.9, 0.0, 0.2, 1.0, 0.5],+ [{"choice:2": 0.9, "default": 0.5}, 0.0, {"choice:3-5": 0.9}, 0.35],+]+++# ---------------------------------------------------------------------------------------------+# jev_confidence+# ---------------------------------------------------------------------------------------------+def _jev_corpus():+ cases = [+ ("choice-confident", _choice(0.9, 0.1)),+ ("choice-even", _choice(0.5, 0.5)),+ ("choice-three", _choice(0.7, 0.2, 0.1)),+ ("choice-differs-from-entropy", _choice(0.7, 0.1, 0.1, 0.1)),+ ("choice-unnormalised", _choice(2.0, 1.0, 1.0)),+ ("choice-zero-total", _choice(0.0, 0.0, 0.0)),+ ("choice-single", _choice(0.3)),+ ("choice-single-zero", _choice(0.0)),+ ("choice-zero-among-others", _choice(0.0, 1.0)),+ ("score-peaked", _score(0.0, 0.95, 0.05)),+ ("score-jevcompat-example", _score(0.0, 0.95, 0.05)),+ ("score-even", _score(0.25, 0.25, 0.25, 0.25)),+ ("score-edge-mode", _score(0.8, 0.1, 0.1)),+ ("score-zero-total", _score(0.0, 0.0, 0.0, 0.0)),+ ("score-single", _score(0.6)),+ ("score-two", _score(0.3, 0.7)),+ ("score-bimodal", _score(0.45, 0.05, 0.05, 0.45)),+ ("score-tied-modes", _score(0.4, 0.4, 0.2)),+ ("score-levels-in-numeric-order",+ {"type": "score", "probabilities": {str(i): (1.0 if i == 10 else 0.0) for i in range(11)}}),+ ("noul", {"type": "noul", "noul": 0.9}),+ ("empty-choice-probabilities", {"type": "choice", "probabilities": {}}),+ ("empty-score-probabilities", {"type": "score", "probabilities": {}}),+ ("nan-probability", _choice(NAN, 0.5)),+ ("infinite-probability", _choice(INF, 0.5)),+ ("negative-infinite-probability", _choice(float("-inf"), 0.5)),+ ("negative-probability", _choice(2.0, -1.0)),+ ("mixed-sign-argmax-flips", _choice(0.5, -1.5)),+ ("all-negative-probabilities", _choice(-0.9, -0.1)),+ ("negative-total-near-zero", _choice(-0.61, 0.6)),+ ("negative-score-level", _score(1.5, -0.5, 0.0)),+ ("nan-score-level", _score(0.5, NAN)),+ ("negative-single", _choice(-0.5)),+ ("nan-single", _choice(NAN)),+ ]+ out = [{"name": name, "answer": answer, "typed": True} for name, answer in cases]+ # Shapes a typed answer cannot hold: kept so the corpus records that Python refuses them.+ out += [+ {"name": "unknown-type", "answer": {"type": "weird", "probabilities": {"a": 1.0}}, "typed": False},+ {"name": "missing-probabilities", "answer": {"type": "choice"}, "typed": False},+ {"name": "non-numeric-probability", "answer": _choice(0.5, "high"), "typed": False},+ {"name": "bool-probability", "answer": _choice(True, False), "typed": False},+ {"name": "bool-among-numbers", "answer": _choice(True, 0.4), "typed": False},+ {"name": "non-integer-score-level", "answer": {"type": "score", "probabilities": {"low": 0.5, "high": 0.5}},+ "typed": False},+ ]+ # CPython 3.12's float sum() is compensated (Neumaier), so a plain left-to-right sum differs here+ # in the last bits of the result; the first two are the textbook case, [0.1] * 10.+ sums = [+ ("sum-differs-choice-tenths", "choice", [0.1] * 10),+ ("sum-differs-score-tenths", "score", [0.1] * 10),+ ("sum-differs-choice-hundredths", "choice", [0.01] * 100),+ ("sum-differs-choice-7", "choice", [0.6946, 0.03, 0.3634, 0.67, 0.2649, 0.72, 0.6497]),+ ("sum-differs-choice-5", "choice", [0.7133, 0.3416, 0.7, 0.8634, 0.6]),+ ("sum-differs-choice-10", "choice", [0.6, 0.1, 0.3, 0.7918, 0.94, 0.7, 0.9983, 0.5, 0.8, 0.6]),+ ("sum-differs-choice-16", "choice", [0.9, 0.11, 0.4, 0.84, 0.1412, 0.8523, 0.45, 0.69, 0.48, 0.5, 0.1126,+ 0.9598, 0.94, 0.18, 0.2723, 0.87]),+ ("sum-differs-score-7", "score", [0.4, 0.4528, 0.2, 0.1, 0.93, 0.34, 0.89]),+ ("sum-differs-score-3", "score", [0.453, 0.4, 0.0333]),+ ]+ for name, kind, probs in sums:+ out.append({"name": name, "answer": (_choice if kind == "choice" else _score)(*probs), "typed": True})+ # A deterministic spread of distributions, to pin the arithmetic beyond the hand-picked ones.+ rng = random.Random(1361)+ for i in range(24):+ k = rng.choice([2, 3, 4, 5, 8, 12])+ probs = [round(rng.random(), 4) for _ in range(k)]+ out.append({"name": "random-choice-%02d" % i, "answer": _choice(*probs), "typed": True})+ out.append({"name": "random-score-%02d" % i, "answer": _score(*probs), "typed": True})+ return out+++JEV_CASES = _jev_corpus()diff --git a/laya-rust/tools/dump_golden.py b/laya-rust/tools/dump_golden.pyindex b4625458..cca19b8d 100644--- a/laya-rust/tools/dump_golden.py+++ b/laya-rust/tools/dump_golden.py@@ -47,6 +47,8 @@ from golden_cases import ( # noqa: E402ERROR_CASES,LONG_CASES,LONG_ERROR_CASES,+ MIN_CONFIDENCE_CASES,+ MIN_CONFIDENCE_ERROR_CASES,SHORTLIST_E2E_CASES,)from repr_cases import build_repr_probe # noqa: E402@@ -324,9 +326,10 @@ def describe_questions(agent: Agent, state, questions: dict) -> list:def record_case(agent: Agent, case: dict) -> dict:state, questions = case["state"], case["questions"]per_question = describe_questions(agent, state, questions)- result = agent.system_one(state, questions)+ options = case.get("options")+ result = agent.system_one(state, questions, **(options or {}))last = agent.model.last- return {+ rec = {"name": case["name"],"state": state,"serialized_state": serialize_state(state),@@ -344,6 +347,9 @@ def record_case(agent: Agent, case: dict) -> dict:"act_logits": last["act_logits"].tolist(),"result": result,}+ if options:+ rec["options"] = options+ return recdef record_shortlist_case(agent: Agent, case: dict) -> dict:@@ -397,16 +403,20 @@ def record_shortlist_case(agent: Agent, case: dict) -> dict:def record_error_case(agent: Agent, case: dict) -> dict:+ options = case.get("options")try:- agent.system_one(case["state"], case["questions"])+ agent.system_one(case["state"], case["questions"], **(options or {}))except ValueError as e:- return {+ rec = {"name": case["name"],"state": case["state"],"questions": case["questions"],"error_type": "ValueError","error_message": str(e),}+ if options:+ rec["options"] = options+ return recraise SystemExit("case %r was expected to raise ValueError but did not" % case["name"])@@ -630,6 +640,15 @@ def run_checkpoint(onnx_dir: str, out_dir: str) -> int:"expects_error": expects_error, "kind": "batch",})+ # Appended after the batch cases, so index.json's diff stays a pure append. No `kind`: these are+ # standard system_one cases whose recorded `options` the parity tests forward.+ for cases, expects_error in ((MIN_CONFIDENCE_CASES, False), (MIN_CONFIDENCE_ERROR_CASES, True)):+ for case in cases:+ rec = (record_error_case if expects_error else record_case)(agent, case)+ dump(rec, os.path.join(out_dir, "case_%s.json" % case["name"]))+ index.append({"name": case["name"], "file": "case_%s.json" % case["name"],+ "expects_error": expects_error})+dump(index, os.path.join(out_dir, "index.json"))dump(build_repr_probe(), os.path.join(out_dir, "repr_probe.json"))print("done: %d cases -> %s" % (len(index), out_dir))diff --git a/laya-rust/tools/dump_routing_golden.py b/laya-rust/tools/dump_routing_golden.pyindex 5e3f31fd..ea0505f3 100644--- a/laya-rust/tools/dump_routing_golden.py+++ b/laya-rust/tools/dump_routing_golden.py@@ -21,6 +21,11 @@ corpus from `laya-rust/tools/routing_cases.py` and writes one JSON file per probits ranking/cosine/tie-break logic against this file). NaN/Infvalues in a matrix are written as the strings "nan"/"inf"/"-inf" (see meta.json) because JSON has no literal for them.+ confidence_probe.json laya.confidence's own answers over the synthetic inputs of+ tools/confidence_cases.py: threshold validation (check_min_confidence and+ _map, with the error text), _option_bucket, resolve_min_confidence,+ apply_confidence_gate (incl. reused payloads, #910) and jev_confidence.+ Non-finite floats are written as "nan"/"inf"/"-inf" strings.hashing_embedder_probe.json Exact laya-rust/tools/hashing_embedder.embed() vectors for a fixed probelist of strings, including non-ASCII and astral input.sample_inputs.json The combined routing sample's fixed inputs (detection states,@@ -40,6 +45,7 @@ given -- a content diff, not a mtime check, so re-running with no corpus changessilent no-op."""import argparse+import copyimport jsonimport mathimport os@@ -71,8 +77,10 @@ from routing_cases import ( # noqa: E402SHORTLIST_CASES,)+import confidence_cases as cc # noqa: E402import hashing_embedder # noqa: E402+from laya import confidence # noqa: E402from laya.email import clean_email_body, email_state # noqa: E402from laya.lang import analyse, detect_script, latin_profile, script_profile, state_text # noqa: E402from laya.router import Router # noqa: E402@@ -218,6 +226,78 @@ def build_route_batch_probe():return {"cases": cases, "errors": errors}+# ============================================================================ confidence_probe+def _checked(fn, arg):+ """`{"valid": True, "result": ...}` or `{"valid": False, "error_type", "error_message"}`."""+ try:+ return {"valid": True, "result": fn(arg)}+ except ValueError as e:+ return {"valid": False, "error_type": type(e).__name__, "error_message": str(e)}+++def _gate_verdict(answers):+ """What the gate wrote on each answer: only the keys that are present."""+ return {+ qid: {k: a[k] for k in ("low_confidence", "abstention", "abstention_threshold") if k in a}+ for qid, a in answers.items()+ }+++def _gate(answers, threshold):+ """apply_confidence_gate exactly as ONNXAgent.predict_batch calls it: validate, then gate."""+ mc = confidence.check_min_confidence(threshold) if threshold is not None else None+ confidence.apply_confidence_gate([{"answers": answers}], mc)+++def build_confidence_probe():+ scalars = [dict(value=v, **_checked(confidence.check_min_confidence, v)) for v in cc.SCALAR_THRESHOLDS]++ maps = []+ for pairs in cc.MAP_THRESHOLDS:+ # Python's dict keeps one entry per key, at its first position, holding its last value.+ checked = _checked(confidence.check_min_confidence, dict(pairs))+ if checked["valid"]:+ checked["result"] = [list(item) for item in checked["result"].items()]+ maps.append(dict(pairs=[list(pair) for pair in pairs], **checked))++ buckets = [{"answer": a, "bucket": confidence._option_bucket(a)} for a in cc.BUCKET_ANSWERS]++ resolve = [+ {"thresholds": m, "answer": a, "result": confidence.resolve_min_confidence(a, m)}+ for m in cc.RESOLVE_MAPS for a in cc.RESOLVE_ANSWERS+ ]++ gate_cases = []+ for threshold in cc.GATE_THRESHOLDS:+ answers = copy.deepcopy(cc.GATE_ANSWERS)+ _gate(answers, threshold)+ gate_cases.append({"threshold": threshold, "expected": _gate_verdict(answers)})++ reuse = []+ for steps in cc.REUSE_SEQUENCES:+ answers = copy.deepcopy(cc.GATE_ANSWERS)+ recorded = []+ for threshold in steps:+ _gate(answers, threshold)+ recorded.append({"threshold": threshold, "expected": _gate_verdict(answers)})+ reuse.append({"steps": recorded})++ jev = [dict(case, result=confidence.jev_confidence(case["answer"])) for case in cc.JEV_CASES]++ return {+ "bucket_keys": sorted(confidence.bucket_keys()),+ "gate_states": list(confidence.GATE_STATES),+ "scalars": scalars,+ "maps": maps,+ "buckets": buckets,+ "resolve": resolve,+ "gate_answers": cc.GATE_ANSWERS,+ "gate": gate_cases,+ "reuse": reuse,+ "jev": jev,+ }++# ============================================================================ email_probedef build_email_probe():clean = []@@ -388,6 +468,7 @@ def main() -> int:lang_probe = build_lang_probe()route_probe = build_route_probe()route_batch_probe = build_route_batch_probe()+ confidence_probe = build_confidence_probe()email_probe = build_email_probe()shortlist_probe = build_shortlist_probe()hashing_probe = build_hashing_embedder_probe()@@ -410,6 +491,13 @@ def main() -> int:"route_probe": len(route_probe),"route_batch_probe": len(route_batch_probe["cases"]),"route_batch_probe_errors": len(route_batch_probe["errors"]),+ "confidence_probe_scalars": len(confidence_probe["scalars"]),+ "confidence_probe_maps": len(confidence_probe["maps"]),+ "confidence_probe_buckets": len(confidence_probe["buckets"]),+ "confidence_probe_resolve": len(confidence_probe["resolve"]),+ "confidence_probe_gate": len(confidence_probe["gate"]),+ "confidence_probe_reuse": len(confidence_probe["reuse"]),+ "confidence_probe_jev": len(confidence_probe["jev"]),"email_probe_clean": len(email_probe["clean"]),"email_probe_state": len(email_probe["state"]),"shortlist_probe": len(shortlist_probe),@@ -423,6 +511,9 @@ def main() -> int:results["route_batch_probe.json"] = dump(route_batch_probe, os.path.join(out_dir, "route_batch_probe.json"), args.force)+ results["confidence_probe.json"] = dump(+ _encode_floats(confidence_probe), os.path.join(out_dir, "confidence_probe.json"), args.force+ )results["email_probe.json"] = dump(email_probe, os.path.join(out_dir, "email_probe.json"), args.force)results["shortlist_probe.json"] = dump(shortlist_probe, os.path.join(out_dir, "shortlist_probe.json"), args.force)results["hashing_embedder_probe.json"] = dump(diff --git a/laya-rust/tools/golden_cases.py b/laya-rust/tools/golden_cases.pyindex df3db938..f8a4bfe4 100644--- a/laya-rust/tools/golden_cases.py+++ b/laya-rust/tools/golden_cases.py@@ -503,6 +503,20 @@ BATCH_CASES = ["questions": {"cancel": BATCH_QUESTIONS["cancel"]}, "options": {"batch_size": 2}},{"name": "batch_empty_questions_with_options", "states": BATCH_STATES[:3], "questions": {},"options": {"batch_size": 2, "sort_by_length": True}},+ # The abstention gate (`min_confidence`): batch-wide, applied to every answer of every state.+ {"name": "batch_min_confidence_scalar", "states": BATCH_MIXED_STATES, "questions": BATCH_QUESTIONS,+ "options": {"min_confidence": 0.6}},+ {"name": "batch_min_confidence_zero", "states": BATCH_MIXED_STATES, "questions": BATCH_QUESTIONS,+ "options": {"min_confidence": 0.0}},+ {"name": "batch_min_confidence_map", "states": BATCH_SORT_STATES[:7], "questions": BATCH_QUESTIONS,+ "options": {"batch_size": 3, "sort_by_length": True,+ "min_confidence": {"choice:3-5": 0.8, "score:3-5": 0.5, "noul:2": 0.7}}},+ {"name": "batch_min_confidence_map_default", "states": BATCH_STATES[:3], "questions": BATCH_QUESTIONS,+ "options": {"min_confidence": {"default": 0.55}}},+ {"name": "batch_min_confidence_empty_questions", "states": BATCH_STATES[:3], "questions": {},+ "options": {"min_confidence": 0.5}},+ {"name": "batch_min_confidence_empty_states", "states": [], "questions": BATCH_QUESTIONS,+ "options": {"min_confidence": 0.5}},]BATCH_ERROR_CASES = [@@ -510,4 +524,33 @@ BATCH_ERROR_CASES = ["error_type": "TypeError"},{"name": "batch_options_exceed_head", "states": ["short", QUICKSTART_STATE],"questions": ERROR_CASES[0]["questions"], "error_type": "ValueError"},+ # The threshold is validated before anything else, so it wins over a missing state.+ {"name": "batch_min_confidence_invalid", "states": [QUICKSTART_STATE], "questions": BATCH_QUESTIONS,+ "options": {"min_confidence": 1.5}, "error_type": "ValueError"},+ {"name": "batch_min_confidence_before_none_state", "states": [QUICKSTART_STATE, None],+ "questions": BATCH_QUESTIONS, "options": {"min_confidence": -0.25}, "error_type": "ValueError"},+ {"name": "batch_min_confidence_bad_key", "states": [QUICKSTART_STATE], "questions": BATCH_QUESTIONS,+ "options": {"min_confidence": {"choice:2": 0.5, "choice:3": 0.5}}, "error_type": "ValueError"},+]++# `system_one` with the abstention gate. `options` go to `system_one` verbatim, and are recorded+# only for these cases so every earlier golden stays byte-identical.+MIN_CONFIDENCE_CASES = [+ {"name": "min_confidence_scalar", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],+ "options": {"min_confidence": 0.6}},+ {"name": "min_confidence_zero", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],+ "options": {"min_confidence": 0.0}},+ {"name": "min_confidence_map", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],+ "options": {"min_confidence": {"choice:3-5": 0.9, "score:3-5": 0.5, "noul:2": 0.7}}},+ {"name": "min_confidence_map_default", "state": "Thanks, that is all sorted now.",+ "questions": CASES[0]["questions"], "options": {"min_confidence": {"default": 0.55}}},+]++MIN_CONFIDENCE_ERROR_CASES = [+ {"name": "min_confidence_invalid", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],+ "options": {"min_confidence": 1.5}},+ {"name": "min_confidence_empty_map", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],+ "options": {"min_confidence": {}}},+ {"name": "min_confidence_bad_key", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],+ "options": {"min_confidence": {"default": 0.5, "choice:3": 0.5}}},]