Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(dotnet): port the min_confidence abstention gate and jev_confidence

kevin-gatimu76e2465743e0
diff --git a/laya-dotnet/CHANGELOG.md b/laya-dotnet/CHANGELOG.md
index 19be9c00..b4b50422 100644
--- a/laya-dotnet/CHANGELOG.md
+++ b/laya-dotnet/CHANGELOG.md
@@ -16,8 +16,15 @@ goldens: 0.4.12).
`ONNXAgent.predict_batch` and `Router.predict_batch` / `route_batch`: shared forward passes, `batch_size`
chunking, `sort_by_length` inside windows of eight batches (effective only for `1 < batch_size < len(states)`),
routing grouped by checkpoint then question schema with input order restored, and per-request `lang_guess`.
- Hooks, `min_confidence`, per-call `lang` / `max_len` / `head_max_len` and the router `diagnostics`
+ Hooks, per-call `lang` / `max_len` / `head_max_len` and the router `diagnostics`
dictionary are not ported yet. Existing signatures are unchanged.
+- Abstention gate, ported from Python `laya/confidence.py` (`min_confidence`, `jev_confidence`): `MinConfidence`
+ (`Of` for one threshold, `ForBuckets` for a per-option-count map, `BucketKeys`), `LayaPredictOptions`
+ with new `LayaEngine.Predict(state, questions, options)` and `LayaRouter.Predict(..., langGuess, options)`
+ overloads, `LayaBatchOptions.MinConfidence` for engine and router batches, and on every `Answer`
+ `LowConfidence`, `Abstention` (`GateStates`: `passed`, `abstained`, `unevaluated`; `null` when ungated),
+ `AbstentionThreshold` and `JevConfidence()`. Invalid thresholds throw `ArgumentException` with Python's
+ `ValueError` text. `PredictLong` takes none, as in Python. Existing signatures are unchanged.
- Behaviour change, to match Python (`system_one` is `predict_batch([state])`): `LayaEngine.Predict` and
`LayaRouter.Predict` / `PredictLong` refuse a null state with `ArgumentNullException` carrying Python's
text (`state must not be None; pass a string, dict, or list`) instead of answering the text `"null"`,
@@ -119,6 +126,6 @@ goldens: 0.4.12).
### Known gaps
-- Python-only opt-in features are not yet ported: abstention (`min_confidence`),
- `lang_temperatures`, hooks, question option ordering and digest controls. Batching is ported without
- hooks, `min_confidence` and per-call `lang` / `max_len` / `head_max_len`.
+- Python-only opt-in features are not yet ported: `lang_temperatures`, hooks, question option
+ ordering and digest controls. Batching is ported without hooks and per-call
+ `lang` / `max_len` / `head_max_len`.
diff --git a/laya-dotnet/README.md b/laya-dotnet/README.md
index befb406b..7e47d000 100644
--- a/laya-dotnet/README.md
+++ b/laya-dotnet/README.md
@@ -376,9 +376,44 @@ by routed checkpoint (first-seen order, so each checkpoint loads at most once),
schema (order-sensitive) so same-schema states share a pass, and returns results in request order, each with
`Routing`. `LayaRouter.RouteBatch(requests)` returns just the `RouteDecision`s and loads nothing. A null
state or questions, or an unknown model, fails the whole call before anything loads. Attached predictors must
-implement `ILayaBatchPredictor`, else `NotSupportedException`. Hooks, `min_confidence`, per-call
+implement `ILayaBatchPredictor`, else `NotSupportedException`. Hooks, per-call
`lang` / `max_len` / `head_max_len` and the `diagnostics` dictionary are not ported yet.
+### Abstention: `MinConfidence`
+
+Python's opt-in `min_confidence`, off by default. Pass a `MinConfidence` per call and every answer says
+whether it cleared the gate; leave it unset and nothing is written (`Abstention`, `AbstentionThreshold`
+are `null`, `LowConfidence` is `false`).
+
+```csharp
+var result = engine.Predict(state, questions,
+ new LayaPredictOptions { MinConfidence = MinConfidence.Of(0.85) });
+var answer = result["department"];
+answer.LowConfidence; // true only when AnswerConfidence fell below the threshold
+answer.Abstention; // "passed" | "abstained" | "unevaluated" (GateStates.*)
+answer.AbstentionThreshold; // the threshold this answer was gated at
+
+// One cut per option-count bucket (MinConfidence.BucketKeys, plus "default"):
+var perBucket = MinConfidence.ForBuckets(new Dictionary<string, double>
+{
+ ["choice:2"] = 0.9, ["choice:3-5"] = 0.8, ["default"] = 0.6,
+});
+engine.PredictBatch(states, questions, new LayaBatchOptions { MinConfidence = perBucket });
+router.Predict(state, questions, null, null, null, null, new LayaPredictOptions { MinConfidence = perBucket });
+```
+
+The gate compares `AnswerConfidence`, falling back to `Confidence`; an answer with neither usable (NaN, infinite)
+is `unevaluated`, never `passed`. A map is read per answer bucket (`choice:3-5`, `score:11+`, `noul:2`, ...): a
+bucket the map omits falls to its `"default"`, then to 0.0 (ungated). A threshold outside [0, 1], NaN or infinite,
+an empty map or an unknown key throws `ArgumentException` with Python's `ValueError` text, when the call receives it
+(the engine checks it before anything else, the router after the state and questions and before routing or
+loading). Answers are immutable; the gate returns new answers, so a reused result never carries a stale flag.
+`PredictLong` takes no threshold, as Python's `predict_long` takes none. The router and `LayaBatchOptions.MinConfidence`
+apply one batch-wide threshold; it is not part of the request grouping.
+
+`Answer.JevConfidence()` is Python's `jev_confidence`: the `confidence` Jev would report for the answer's
+(rounded) probabilities, `null` for noul answers and malformed probabilities.
+
---
## Reading results
@@ -984,12 +1019,12 @@ identifies the foreign line or field that overrides a mostly-English state. Unde
Latin text uses the router's configured default. Email cleaning follows current request,
disclaimer, signature and reply-header rules, including Portuguese, Spanish and French markers.
-- **Opt-in features added in Python 0.3.21 are not ported yet:** abstention (`min_confidence`),
- and per-language temperatures (`lang_temperatures`). They are off by default in
- Python, so default predictions are unaffected. They are planned as follow-ups.
+- **Opt-in features added in Python 0.3.21 are not ported yet:** per-language temperatures
+ (`lang_temperatures`). They are off by default in Python, so default predictions are unaffected. They
+ are planned as follow-ups. Abstention (`min_confidence`) is ported (see "Abstention" above).
- **Other Python-only APIs are not ported:** hooks, question option ordering and digest controls.
Batching is ported (`PredictBatch`, `LayaRouter.PredictBatch` / `RouteBatch`) without hooks,
- `min_confidence`, per-call `lang` / `max_len` / `head_max_len` and the router `diagnostics` dictionary.
+ per-call `lang` / `max_len` / `head_max_len` and the router `diagnostics` dictionary.
- **Usage diagnostics:** Python's `state_tokens`, `truncated`, `state_tokens_dropped`, `truncated_questions`
and `options` are reported by `Predict`, `PredictBatch` and `PredictLong`, as Python's `system_one` is
`predict_batch([state])`.
diff --git a/laya-dotnet/src/Laya/Answers.cs b/laya-dotnet/src/Laya/Answers.cs
index 435ed354..e908966d 100644
--- a/laya-dotnet/src/Laya/Answers.cs
+++ b/laya-dotnet/src/Laya/Answers.cs
@@ -131,6 +131,50 @@ public abstract class Answer
return answer;
}
+ /// <summary>
+ /// True only when a <see cref="MinConfidence"/> was set for the call and this answer's confidence fell
+ /// below its threshold (Python's <c>low_confidence: True</c>, which is absent otherwise). False for an
+ /// ungated call and for an answer that cleared the gate or could not be evaluated.
+ /// </summary>
+ public bool LowConfidence { get; private set; }
+
+ /// <summary>
+ /// What the abstention gate decided, one of <see cref="GateStates"/> (<c>passed</c>, <c>abstained</c>,
+ /// <c>unevaluated</c>), or <see langword="null"/> when the call set no <see cref="MinConfidence"/>
+ /// (Python writes no <c>abstention</c> key then).
+ /// </summary>
+ public string? Abstention { get; private set; }
+
+ /// <summary>
+ /// The threshold this answer was gated at: the scalar, or the map entry for its option-count bucket
+ /// (else the map's <c>default</c>, else 0.0). <see langword="null"/> when the call set no
+ /// <see cref="MinConfidence"/> (Python's <c>abstention_threshold</c>).
+ /// </summary>
+ public double? AbstentionThreshold { get; private set; }
+
+ /// <summary>
+ /// The <c>confidence</c> Jev would report for this answer's (rounded) probabilities, which is a different
+ /// formula from <see cref="Confidence"/> (Python's <c>jev_confidence</c>): for choice
+ /// <c>(max(p) - 1/n) / (1 - 1/n)</c>, for score <c>max(0, 1 - sum(p_i * |i - mode|) / D)</c>, both 1.0 for a
+ /// single option. <see langword="null"/> for a noul answer and for malformed probabilities (non-finite or negative).
+ /// </summary>
+ public double? JevConfidence() => this switch
+ {
+ ChoiceAnswer choice => ConfidenceGate.Jev(false, [.. choice.Probabilities.Select(kv => kv.Value)]),
+ ScoreAnswer score => ConfidenceGate.Jev(true, score.Probabilities),
+ _ => null,
+ };
+
+ /// <summary>A copy carrying the gate's verdict; the original stays as it was, so a reused answer never keeps a stale flag.</summary>
+ internal Answer WithGate(bool lowConfidence, string abstention, double threshold)
+ {
+ var answer = (Answer)MemberwiseClone();
+ answer.LowConfidence = lowConfidence;
+ answer.Abstention = abstention;
+ answer.AbstentionThreshold = threshold;
+ return answer;
+ }
+
/// <summary>This answer as a choice answer.</summary>
/// <exception cref="InvalidOperationException">If the question was not a choice question.</exception>
public ChoiceAnswer AsChoice() => this as ChoiceAnswer
@@ -282,6 +326,10 @@ public sealed class LayaResult
internal LayaResult WithShortlist(IReadOnlyDictionary<string, ShortlistInfo> shortlist) =>
new(Model, _order, _byId, Usage, Routing, shortlist);
+ /// <summary>A copy of this result with its answers replaced (same ids). Used internally by the abstention gate.</summary>
+ internal LayaResult WithAnswers(Dictionary<string, Answer> answers) =>
+ new(Model, _order, answers, Usage, Routing, Shortlist);
+
/// <summary>The question ids, in the order they were asked.</summary>
public IReadOnlyList<string> Ids => _order;
diff --git a/laya-dotnet/src/Laya/LayaBatch.cs b/laya-dotnet/src/Laya/LayaBatch.cs
index d2a5ffcd..f93ade53 100644
--- a/laya-dotnet/src/Laya/LayaBatch.cs
+++ b/laya-dotnet/src/Laya/LayaBatch.cs
@@ -4,7 +4,7 @@ namespace Laya;
/// Per-call knobs for <see cref="ILayaBatchPredictor.PredictBatch"/> and
/// <see cref="LayaRouter.PredictBatch"/>. Ports the batching arguments of Python's
/// <c>ONNXAgent.predict_batch</c> (<c>batch_size</c>, <c>sort_by_length</c>). It is a class with
-/// init-only properties so later per-call options (<c>min_confidence</c>, <c>lang</c>,
+/// init-only properties so later per-call options (<c>lang</c>,
/// <c>max_len</c>, <c>head_max_len</c>) can be added without changing any signature.
/// </summary>
public sealed class LayaBatchOptions
@@ -24,6 +24,13 @@ public sealed class LayaBatchOptions
/// decision thresholds.
/// </summary>
public bool SortByLength { get; init; }
+
+ /// <summary>
+ /// Opt-in abstention threshold (Python <c>min_confidence</c>), applied to every answer of every state
+ /// in the call; on a router it is batch-wide, not part of the request grouping. <see langword="null"/>
+ /// writes nothing onto the answers. See <see cref="Laya.MinConfidence"/>.
+ /// </summary>
+ public MinConfidence? MinConfidence { get; init; }
}
/// <summary>
diff --git a/laya-dotnet/src/Laya/LayaEngine.cs b/laya-dotnet/src/Laya/LayaEngine.cs
index 607787cb..87ea083c 100644
--- a/laya-dotnet/src/Laya/LayaEngine.cs
+++ b/laya-dotnet/src/Laya/LayaEngine.cs
@@ -24,7 +24,7 @@ namespace Laya;
/// one <see cref="InferenceSession"/> runs the whole forward pass. For the split layout (the
/// laya-ts exporter's output), two sessions run in sequence: the encoder produces
/// <c>last_hidden_state</c>, which is fed straight into the head. Both cases go through
-/// <see cref="Run"/>, so batching, calibration and <see cref="Predict"/> do not know which
+/// <see cref="Run"/>, so batching, calibration and <see cref="Predict(object?, QuestionSet)"/> do not know which
/// layout is loaded.
/// </para>
/// </remarks>
@@ -204,6 +204,21 @@ public sealed class LayaEngine : IDisposable, ILayaLongPredictor, ILayaBatchPred
return PredictStates([state], questions, diagnostics: true)[0];
}
+ /// <summary>
+ /// <see cref="Predict(object?, QuestionSet)"/> with per-call options. Ports the <c>min_confidence</c>
+ /// argument of Python's <c>system_one</c>: with <see cref="LayaPredictOptions.MinConfidence"/> set, every answer
+ /// reports <see cref="Answer.Abstention"/> and <see cref="Answer.AbstentionThreshold"/>, and carries
+ /// <see cref="Answer.LowConfidence"/> when it falls below the threshold. The threshold is validated before
+ /// anything else, as in Python, where it is the first thing <c>predict_batch</c> checks.
+ /// </summary>
+ /// <exception cref="ArgumentException">The threshold is invalid, with Python's <c>ValueError</c> text.</exception>
+ public LayaResult Predict(object? state, QuestionSet questions, LayaPredictOptions? options)
+ {
+ var gate = options?.MinConfidence?.Check();
+ var result = Predict(state, questions);
+ return gate is { } g ? ConfidenceGate.Apply(result, g) : result;
+ }
+
/// <summary>
/// Scan a long state in overlapping token windows. Ports Python ONNXAgent.predict_long:
/// noul selects the greatest P(true); choice and score select the greatest answer confidence.
@@ -272,9 +287,19 @@ public sealed class LayaEngine : IDisposable, ILayaLongPredictor, ILayaBatchPred
/// <param name="questions">The questions answered about every state. With no questions each state
/// gets an empty answer set and zero usage, as in Python.</param>
/// <param name="options">Batch size and ordering; see <see cref="LayaBatchOptions"/>.</param>
- /// <exception cref="ArgumentException">A null state, or a question whose options cannot fit the head budget.</exception>
+ /// <exception cref="ArgumentException">A null state, an invalid <see cref="LayaBatchOptions.MinConfidence"/>,
+ /// or a question whose options cannot fit the head budget.</exception>
public IReadOnlyList<LayaResult> PredictBatch(
IReadOnlyList<object?> states, QuestionSet questions, LayaBatchOptions? options = null)
+ {
+ // onnx_agent.py:386: the threshold is validated before the states or the questions are looked at.
+ var gate = options?.MinConfidence?.Check();
+ var results = PredictBatchUngated(states, questions, options);
+ return gate is { } g ? [.. results.Select(r => ConfidenceGate.Apply(r, g))] : results;
+ }
+
+ private IReadOnlyList<LayaResult> PredictBatchUngated(
+ IReadOnlyList<object?> states, QuestionSet questions, LayaBatchOptions? options)
{
ArgumentNullException.ThrowIfNull(states);
ArgumentNullException.ThrowIfNull(questions);
diff --git a/laya-dotnet/src/Laya/LayaRouter.cs b/laya-dotnet/src/Laya/LayaRouter.cs
index b5cd6ba3..82b545f9 100644
--- a/laya-dotnet/src/Laya/LayaRouter.cs
+++ b/laya-dotnet/src/Laya/LayaRouter.cs
@@ -501,6 +501,26 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposable
=> PredictRouted(state, questions, model, task, lang, langGuess,
predictor => predictor.Predict(state, questions));
+ /// <summary>
+ /// Routes as <see cref="Predict(object?, QuestionSet)"/> does (no model, task, language or hint overrides), with
+ /// per-call options (a <see langword="null"/> <paramref name="options"/> gates nothing, so a literal <c>Predict(s, q, null)</c> still means what it meant); see <see cref="Predict(object?, QuestionSet, string?, string?, string?, LangGuess?, LayaPredictOptions?)"/>.
+ /// </summary>
+ public LayaResult Predict(object? state, QuestionSet questions, LayaPredictOptions? options)
+ => Predict(state, questions, null, null, null, null, options);
+
+ /// <summary>
+ /// <see cref="Predict(object?, QuestionSet, string?, string?, string?, LangGuess?)"/> with per-call options.
+ /// Ports the <c>min_confidence</c> argument of Python's <c>Router.predict</c>: the threshold is validated after the
+ /// state and questions are checked and before anything is routed or loaded, and the gate is applied to the
+ /// answers whichever checkpoint produced them. <c>PredictLong</c> takes none, as in Python.
+ /// </summary>
+ /// <exception cref="ArgumentException">The threshold is invalid, with Python's <c>ValueError</c> text.</exception>
+ public LayaResult Predict(
+ object? state, QuestionSet questions, string? model, string? task, string? lang, LangGuess? langGuess,
+ LayaPredictOptions? options)
+ => PredictRouted(state, questions, model, task, lang, langGuess,
+ predictor => predictor.Predict(state, questions), options?.MinConfidence);
+
LayaResult ILayaLongPredictor.PredictLong(
object state, QuestionSet questions, int? window, int? stride, string aggregate, int? batchSize) =>
PredictLong(state, questions, window, stride, aggregate, batchSize);
@@ -535,15 +555,18 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposable
private LayaResult PredictRouted(
object? state, QuestionSet questions, string? model, string? task, string? lang, LangGuess? langGuess,
- Func<ILayaPredictor, LayaResult> predict)
+ Func<ILayaPredictor, LayaResult> predict, MinConfidence? minConfidence = null)
{
StateGuard.NotNull(state);
ArgumentNullException.ThrowIfNull(questions);
+ // router.py:1338: validated after the state and questions, before routing or loading anything.
+ var gate = minConfidence?.Check();
var decision = Route(state, questions, model, task, lang, langGuess);
// Inference runs outside the lock, exactly as Python's predict() calls
// agent.system_one after load() has already returned and released the lock — so
// concurrent predictions on a shared checkpoint are not serialised by the router.
- return WithLease(decision.Model, predict).WithRouting(decision);
+ var result = WithLease(decision.Model, predict);
+ return (gate is { } g ? ConfidenceGate.Apply(result, g) : result).WithRouting(decision);
}
// Loads (or finds) the checkpoint, holds a lease so eviction cannot dispose it mid-call, and
@@ -610,11 +633,18 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposable
/// </summary>
/// <param name="requests">The requests; an empty list returns an empty list and loads nothing.</param>
/// <param name="options">Passed to every checkpoint batch; <see cref="LayaBatchOptions.SortByLength"/>
- /// acts per question group, as in Python.</param>
+ /// acts per question group, as in Python. <see cref="LayaBatchOptions.MinConfidence"/> is batch-wide: it is
+ /// applied by the router to every result, is not passed to the checkpoints, and is not part of the grouping.</param>
/// <exception cref="NotSupportedException">A routed checkpoint (an attached predictor) has no batch support.</exception>
public IReadOnlyList<LayaResult> PredictBatch(
IReadOnlyList<LayaBatchRequest> requests, LayaBatchOptions? options = null)
{
+ // router.py:1669: validated before any request is looked at.
+ var gate = options?.MinConfidence?.Check();
+ // router.py:1789 calls the agent without min_confidence and gates the results itself.
+ var inner = options is { MinConfidence: not null }
+ ? new LayaBatchOptions { BatchSize = options.BatchSize, SortByLength = options.SortByLength }
+ : options;
var decisions = RouteBatch(requests);
if (decisions.Count == 0) return [];
@@ -643,12 +673,13 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposable
$"{predictor.GetType().Name} has no PredictBatch, so requests cannot share forward passes");
foreach (var (questions, group) in GroupBySchema(requests, indices))
{
- var batch = batcher.PredictBatch([.. group.Select(i => requests[i].State)], questions, options);
+ var batch = batcher.PredictBatch([.. group.Select(i => requests[i].State)], questions, inner);
if (batch.Count != group.Count)
throw new InvalidOperationException(
$"internal error: Agent.predict_batch returned {batch.Count} results for {group.Count} states");
for (var k = 0; k < batch.Count; k++)
- results[group[k]] = batch[k].WithRouting(decisions[group[k]]);
+ results[group[k]] = (gate is { } g ? ConfidenceGate.Apply(batch[k], g) : batch[k])
+ .WithRouting(decisions[group[k]]);
}
return 0;
});
diff --git a/laya-dotnet/src/Laya/MinConfidence.cs b/laya-dotnet/src/Laya/MinConfidence.cs
new file mode 100644
index 00000000..18b5fde7
--- /dev/null
+++ b/laya-dotnet/src/Laya/MinConfidence.cs
@@ -0,0 +1,224 @@
+using System.Collections.ObjectModel;
+
+namespace Laya;
+
+/// <summary>
+/// The three states of the abstention gate, spelled as Python's <c>GATE_STATES</c>
+/// (<c>laya/confidence.py</c>). <see cref="Answer.Abstention"/> carries one of them whenever a
+/// <see cref="MinConfidence"/> was set for the call, and is <see langword="null"/> otherwise: there is
+/// deliberately no "no gate" member.
+/// </summary>
+public static class GateStates
+{
+ /// <summary>A gate ran and this answer's confidence cleared it.</summary>
+ public const string Passed = "passed";
+
+ /// <summary>A gate ran and this answer's confidence fell below it.</summary>
+ public const string Abstained = "abstained";
+
+ /// <summary>A gate ran and this answer carried no usable confidence.</summary>
+ public const string Unevaluated = "unevaluated";
+
+ /// <summary>Every state, in Python's order (<c>GATE_STATES</c>).</summary>
+ public static IReadOnlyList<string> All { get; } = new ReadOnlyCollection<string>([Passed, Abstained, Unevaluated]);
+}
+
+/// <summary>
+/// An abstention threshold: one number for every answer, or a map from option-count bucket to
+/// number. Ports Python's <c>min_confidence</c> argument (<c>check_min_confidence</c> and
+/// <c>check_min_confidence_map</c> in <c>laya/confidence.py</c>). An answer whose confidence falls below the
+/// threshold is flagged (<see cref="Answer.LowConfidence"/>).
+/// </summary>
+/// <remarks>
+/// Like Python, the value is validated when a prediction call receives it, not when it is built, so
+/// a call that would fail for another reason first (a null state) still fails for that reason when
+/// the order in Python says so. The failure is an <see cref="ArgumentException"/> carrying Python's
+/// <c>ValueError</c> text.
+/// </remarks>
+public sealed class MinConfidence
+{
+ /// <summary>The map key for every bucket the map does not name.</summary>
+ public const string DefaultBucket = "default";
+
+ private static readonly string[] QuestionKinds = ["choice", "score", "noul"];
+ private static readonly string[] Sizes = ["2", "3-5", "6-10", "11+"];
+
+ /// <summary>
+ /// Every option-count bucket a map can name: the answer shapes Python's <c>_option_bucket</c> produces
+ /// (<c>choice:2</c>, <c>choice:3-5</c>, ... <c>noul:11+</c>). <see cref="DefaultBucket"/> is not among them.
+ /// </summary>
+ public static IReadOnlyList<string> BucketKeys { get; } = new ReadOnlyCollection<string>(
+ [.. QuestionKinds.SelectMany(kind => Sizes.Select(size => $"{kind}:{size}"))]);
+
+ private readonly double _scalar;
+ private readonly List<KeyValuePair<string?, double>>? _map;
+
+ private MinConfidence(double scalar, List<KeyValuePair<string?, double>>? map)
+ {
+ _scalar = scalar;
+ _map = map;
+ }
+
+ /// <summary>One threshold for every answer. Must be a finite number in [0.0, 1.0] when the call validates it.</summary>
+ public static MinConfidence Of(double threshold) => new(threshold, null);
+
+ /// <summary>
+ /// A threshold per option-count bucket (<see cref="BucketKeys"/>), plus an optional
+ /// <see cref="DefaultBucket"/> for any bucket the map does not name; a bucket that is neither named
+ /// nor covered by the default is gated at 0.0, that is, not at all. Insertion order is kept for error text.
+ /// </summary>
+ public static MinConfidence ForBuckets(IEnumerable<KeyValuePair<string, double>> thresholds)
+ {
+ ArgumentNullException.ThrowIfNull(thresholds);
+ var map = new List<KeyValuePair<string?, double>>();
+ foreach (var (key, value) in thresholds)
+ {
+ // A Python dict keeps one entry per key, at the key's first position, holding its last value.
+ var at = map.FindIndex(e => string.Equals(e.Key, key, StringComparison.Ordinal));
+ if (at < 0) map.Add(new(key, value));
+ else map[at] = new(key, value);
+ }
+ return new(0.0, map);
+ }
+
+ /// <summary>Whether this is a per-bucket map rather than a single threshold.</summary>
+ public bool IsMap => _map is not null;
+
+ /// <summary>The validated thresholds, or Python's <c>ValueError</c> text as an <see cref="ArgumentException"/>.</summary>
+ internal Resolved Check()
+ {
+ if (_map is null) return new Resolved(CheckOne(_scalar), null);
+ if (_map.Count == 0)
+ throw new ArgumentException("a min_confidence map must be a non-empty dict of bucket -> float, got {}");
+ var allowed = new HashSet<string>(BucketKeys, StringComparer.Ordinal) { DefaultBucket };
+ var checkedMap = new Dictionary<string, double>(StringComparer.Ordinal);
+ foreach (var (key, value) in _map)
+ {
+ if (key is null || !allowed.Contains(key))
+ throw new ArgumentException(
+ "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got "
+ + PythonFloat.Repr(key));
+ checkedMap[key] = CheckOne(value);
+ }
+ return new Resolved(0.0, checkedMap);
+ }
+
+ private static double CheckOne(double value)
+ {
+ if (!double.IsFinite(value) || value < 0.0 || value > 1.0)
+ throw new ArgumentException(
+ "min_confidence must be a float in [0.0, 1.0], got " + PythonFloat.Repr(value));
+ return value;
+ }
+
+ /// <inheritdoc/>
+ public override string ToString() => _map is null
+ ? PythonFloat.Repr(_scalar)
+ : "{" + string.Join(", ", _map.Select(e => PythonFloat.Repr(e.Key) + ": " + PythonFloat.Repr(e.Value))) + "}";
+
+ /// <summary>A validated threshold: the scalar, or the per-bucket map.</summary>
+ internal readonly record struct Resolved(double Scalar, IReadOnlyDictionary<string, double>? Map);
+}
+
+/// <summary>
+/// Per-call knobs for <see cref="LayaEngine.Predict(object?, QuestionSet, LayaPredictOptions?)"/> and
+/// <see cref="LayaRouter"/>'s matching overload. A class with init-only properties so later per-call
+/// options (<c>lang</c>, <c>max_len</c>, <c>head_max_len</c>) can be added without changing a signature.
+/// <c>PredictLong</c> takes none of these, as Python's <c>predict_long</c> takes no <c>min_confidence</c>.
+/// </summary>
+public sealed class LayaPredictOptions
+{
+ /// <summary>
+ /// Opt-in abstention threshold (Python <c>min_confidence</c>). <see langword="null"/> writes nothing onto
+ /// the answers; otherwise every answer reports <see cref="Answer.Abstention"/> and
+ /// <see cref="Answer.AbstentionThreshold"/>, and <see cref="Answer.LowConfidence"/> is set on those below it.
+ /// </summary>
+ public MinConfidence? MinConfidence { get; init; }
+}
+
+/// <summary>
+/// The abstention gate of Python's <c>laya/confidence.py</c> (<c>apply_confidence_gate</c>,
+/// <c>flag_low_confidence</c>, <c>resolve_min_confidence</c>) and <c>jev_confidence</c>, over typed answers.
+/// Answers are immutable, so applying a gate builds new answers instead of editing the old ones.
+/// </summary>
+internal static class ConfidenceGate
+{
+ /// <summary>The number the gate compares: <c>answer_confidence</c> first, then <c>confidence</c>, else none.</summary>
+ internal static double? GateConfidence(Answer answer)
+ {
+ if (double.IsFinite(answer.AnswerConfidence)) return answer.AnswerConfidence;
+ if (double.IsFinite(answer.Confidence)) return answer.Confidence;
+ return null;
+ }
+
+ /// <summary>Python's <c>_option_bucket</c>.</summary>
+ internal static string? OptionBucket(Answer answer)
+ {
+ var count = OptionCount(answer);
+ if (count == 0 && answer.Type != QuestionType.Noul) return null;
+ var k = count > 0 ? count : 2;
+ var size = k <= 2 ? "2" : k <= 5 ? "3-5" : k <= 10 ? "6-10" : "11+";
+ return $"{SequenceBuilder.TypeName(answer.Type)}:{size}";
+ }
+
+ private static int OptionCount(Answer answer) => answer switch
+ {
+ ChoiceAnswer choice => choice.Probabilities.Count,
+ ScoreAnswer score => score.Probabilities.Count,
+ _ => 0,
+ };
+
+ /// <summary>Python's <c>resolve_min_confidence</c>: the bucket's entry, else <c>default</c>, else 0.0.</summary>
+ internal static double Resolve(Answer answer, IReadOnlyDictionary<string, double> thresholds)
+ {
+ var key = OptionBucket(answer);
+ if (key is not null && thresholds.TryGetValue(key, out var own)) return own;
+ return thresholds.TryGetValue(MinConfidence.DefaultBucket, out var fallback) ? fallback : 0.0;
+ }
+
+ /// <summary>One answer, gated: a copy carrying the abstention fields.</summary>
+ internal static Answer Apply(Answer answer, MinConfidence.Resolved gate)
+ {
+ var threshold = gate.Map is null ? gate.Scalar : Resolve(answer, gate.Map);
+ var confidence = GateConfidence(answer);
+ var low = confidence is { } value && value < threshold;
+ var state = low ? GateStates.Abstained : confidence is null ? GateStates.Unevaluated : GateStates.Passed;
+ return answer.WithGate(low, state, threshold);
+ }
+
+ /// <summary>A result whose answers are gated; the original is untouched.</summary>
+ internal static LayaResult Apply(LayaResult result, MinConfidence.Resolved gate)
+ {
+ var answers = new Dictionary<string, Answer>(result.Count, StringComparer.Ordinal);
+ foreach (var id in result.Ids) answers[id] = Apply(result[id], gate);
+ return result.WithAnswers(answers);
+ }
+
+ /// <summary>
+ /// Python's <c>jev_confidence</c> over the probabilities of a choice or score answer, in option order
+ /// (a score's by level index). Null for anything malformed: a non-finite or negative value.
+ /// </summary>
+ internal static double? Jev(bool isScore, IReadOnlyList<double> probabilities)
+ {
+ if (probabilities.Count == 0) return null;
+ foreach (var v in probabilities)
+ if (!double.IsFinite(v) || v < 0.0) return null;
+ var n = probabilities.Count;
+ if (n == 1) return 1.0;
+ var total = PythonSum.Sum(probabilities);
+ var p = new double[n];
+ for (var i = 0; i < n; i++) p[i] = total != 0.0 ? probabilities[i] / total : 1.0 / n;
+ if (!isScore)
+ {
+ var top = p.Max();
+ return (top - 1.0 / n) / (1.0 - 1.0 / n);
+ }
+ var mode = 0;
+ for (var i = 1; i < n; i++)
+ if (p[i] > p[mode]) mode = i;
+ var spread = PythonSum.Sum(p.Select((v, i) => v * Math.Abs(i - mode)));
+ var center = (n - 1) / 2.0;
+ var uniform = PythonSum.Sum(Enumerable.Range(0, n).Select(i => Math.Abs(i - center))) / n;
+ return Math.Max(0.0, 1.0 - spread / uniform);
+ }
+}
diff --git a/laya-dotnet/src/Laya/PythonSum.cs b/laya-dotnet/src/Laya/PythonSum.cs
new file mode 100644
index 00000000..aa020237
--- /dev/null
+++ b/laya-dotnet/src/Laya/PythonSum.cs
@@ -0,0 +1,29 @@
+namespace Laya;
+
+/// <summary>
+/// Python's builtin <c>sum()</c> over floats as CPython 3.12 computes it (<c>builtin_sum_impl</c> in
+/// <c>Python/bltinmodule.c</c>): Neumaier compensated summation, not a plain left-to-right add. The goldens
+/// come from CPython 3.12, so <c>jev_confidence</c> is only bit-identical with this.
+/// </summary>
+internal static class PythonSum
+{
+ /// <summary>
+ /// <c>sum(values)</c> of floats. Start at <c>0.0</c> with compensation <c>c = 0.0</c>; for each <c>x</c>:
+ /// <c>t = f + x</c>; if <c>|f| &gt;= |x|</c> then <c>c += (f - t) + x</c> else <c>c += (x - t) + f</c>; <c>f = t</c>.
+ /// At the end, <c>f + c</c> only when <c>c</c> is nonzero and finite (so an infinite or overflowed sum never
+ /// becomes NaN), else <c>f</c>. (CPython adds the first float to the int start <c>0</c> uncompensated,
+ /// which equals this loop's first step.)
+ /// </summary>
+ internal static double Sum(IEnumerable<double> values)
+ {
+ var f = 0.0;
+ var c = 0.0;
+ foreach (var x in values)
+ {
+ var t = f + x;
+ c += Math.Abs(f) >= Math.Abs(x) ? (f - t) + x : (x - t) + f;
+ f = t;
+ }
+ return c != 0.0 && double.IsFinite(c) ? f + c : f;
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/ConfidenceGateTests.cs b/laya-dotnet/tests/Laya.Tests/ConfidenceGateTests.cs
new file mode 100644
index 00000000..00a37406
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/ConfidenceGateTests.cs
@@ -0,0 +1,473 @@
+using System.Text.Json;
+
+namespace Laya.Tests;
+
+/// <summary>
+/// Model-free tests of the abstention gate (<c>min_confidence</c>, <c>jev_confidence</c>): every case of
+/// <c>golden/routing/confidence_probe.json</c> (Python's own <c>laya/confidence.py</c> over the synthetic
+/// inputs of <c>tools/confidence_cases.py</c>), plus the call-level behaviour over fake predictors.
+/// </summary>
+public sealed class ConfidenceGateTests
+{
+ // ── probe plumbing ────────────────────────────────────────────────────────
+
+ private static readonly Lazy<JsonDocument> Probe = new(() => RoutingGoldenData.Load("confidence_probe.json"));
+ private static JsonElement Root => Probe.Value.RootElement;
+
+ public static IEnumerable<object[]> Indices(string section) =>
+ Enumerable.Range(0, Root.GetProperty(section).GetArrayLength()).Select(i => new object[] { i });
+
+ public static IEnumerable<object[]> ScalarCases() => Indices("scalars");
+ public static IEnumerable<object[]> MapCases() => Indices("maps");
+ public static IEnumerable<object[]> BucketCases() => Indices("buckets");
+ public static IEnumerable<object[]> ResolveCases() => Indices("resolve");
+ public static IEnumerable<object[]> GateCases() => Indices("gate");
+ public static IEnumerable<object[]> ReuseCases() => Indices("reuse");
+ public static IEnumerable<object[]> JevCases() => Indices("jev");
+
+ /// <summary>A number, or one of the strings the dumper uses for the non-finite ones; absent is NaN (no usable number).</summary>
+ private static double Num(JsonElement value) => value.ValueKind switch
+ {
+ JsonValueKind.Number => value.GetDouble(),
+ JsonValueKind.String => value.GetString() switch
+ {
+ "nan" => double.NaN,
+ "inf" => double.PositiveInfinity,
+ "-inf" => double.NegativeInfinity,
+ var other => throw new InvalidOperationException($"unexpected number text {other}"),
+ },
+ _ => throw new InvalidOperationException($"unexpected {value.ValueKind}"),
+ };
+
+ private static double Field(JsonElement answer, string name) =>
+ answer.TryGetProperty(name, out var value) ? Num(value) : double.NaN;
+
+ /// <summary>The typed answer a recorded raw answer stands for.</summary>
+ private static Answer ToAnswer(JsonElement raw)
+ {
+ var confidence = Field(raw, "confidence");
+ var answerConfidence = Field(raw, "answer_confidence");
+ var action = new ActionInfo(0.5);
+ switch (raw.GetProperty("type").GetString())
+ {
+ case "choice":
+ {
+ var probs = raw.GetProperty("probabilities").EnumerateObject()
+ .Select(p => new KeyValuePair<string, double>(p.Name, Num(p.Value))).ToList();
+ return new ChoiceAnswer(probs.Count == 0 ? "" : probs[0].Key, probs, confidence, answerConfidence, action);
+ }
+ case "score":
+ {
+ var probs = raw.GetProperty("probabilities").EnumerateObject()
+ .OrderBy(p => int.Parse(p.Name)).Select(p => Num(p.Value)).ToArray();
+ return new ScoreAnswer(0.0, [.. probs.Select((_, i) => (object?)i.ToString())], probs,
+ confidence, answerConfidence, action);
+ }
+ case "noul":
+ return new NoulAnswer(Field(raw, "noul"), confidence, answerConfidence, action);
+ default:
+ throw new InvalidOperationException("not a typed answer");
+ }
+ }
+
+ private static LayaResult ToResult(JsonElement answers)
+ {
+ var byId = new Dictionary<string, Answer>(StringComparer.Ordinal);
+ var order = new List<string>();
+ foreach (var entry in answers.EnumerateObject())
+ {
+ order.Add(entry.Name);
+ byId[entry.Name] = ToAnswer(entry.Value);
+ }
+ return new LayaResult("m", order, byId, new Usage(0, 0));
+ }
+
+ private static MinConfidence? ToThreshold(JsonElement value) => value.ValueKind switch
+ {
+ JsonValueKind.Null => null,
+ JsonValueKind.Object => MinConfidence.ForBuckets(
+ value.EnumerateObject().Select(p => new KeyValuePair<string, double>(p.Name, Num(p.Value)))),
+ _ => MinConfidence.Of(Num(value)),
+ };
+
+ private static void AssertGated(JsonElement expected, LayaResult got, string where)
+ {
+ Assert.Equal(expected.EnumerateObject().Select(p => p.Name), got.Ids);
+ foreach (var entry in expected.EnumerateObject())
+ AssertGateFields(entry.Value, got[entry.Name], $"{where}/{entry.Name}");
+ }
+
+ /// <summary>
+ /// The three keys Python's gate writes, against an answer: absent in the recording means false/null here.
+ /// Shared with the model-backed parity tests, whose recorded answers carry the same keys.
+ /// </summary>
+ internal static void AssertGateFields(JsonElement want, Answer answer, string at)
+ {
+ Assert.True((want.TryGetProperty("low_confidence", out var low) && low.GetBoolean()) == answer.LowConfidence,
+ at + " low_confidence");
+ Assert.Equal(want.TryGetProperty("abstention", out var state) ? state.GetString() : null, answer.Abstention);
+ Assert.Equal(want.TryGetProperty("abstention_threshold", out var thr) ? thr.GetDouble() : (double?)null,
+ answer.AbstentionThreshold);
+ }
+
+ /// <summary>The <c>min_confidence</c> a recorded case's <c>options</c> carry, or null if it has none.</summary>
+ internal static MinConfidence? MinConfidenceOf(JsonElement root) =>
+ root.TryGetProperty("options", out var options) ? MinConfidenceIn(options) : null;
+
+ /// <summary>The <c>min_confidence</c> in a recorded <c>options</c> object, or null if it has none.</summary>
+ internal static MinConfidence? MinConfidenceIn(JsonElement options) =>
+ options.TryGetProperty("min_confidence", out var value) ? ToThreshold(value) : null;
+
+ // ── check_min_confidence / check_min_confidence_map ───────────────────────
+
+ [Theory]
+ [MemberData(nameof(ScalarCases))]
+ public void ScalarThresholdsAreValidatedLikePython(int index)
+ {
+ var want = Root.GetProperty("scalars")[index];
+ var value = Num(want.GetProperty("value"));
+ if (want.GetProperty("valid").GetBoolean())
+ {
+ Assert.Equal(Num(want.GetProperty("result")), MinConfidence.Of(value).Check().Scalar);
+ return;
+ }
+ var error = Assert.Throws<ArgumentException>(() => MinConfidence.Of(value).Check());
+ Assert.Equal(want.GetProperty("error_message").GetString(), error.Message);
+ }
+
+ [Theory]
+ [MemberData(nameof(MapCases))]
+ public void MapThresholdsAreValidatedLikePython(int index)
+ {
+ var want = Root.GetProperty("maps")[index];
+ var pairs = want.GetProperty("pairs").EnumerateArray()
+ .Select(p => new KeyValuePair<string, double>(p[0].GetString()!, Num(p[1]))).ToArray();
+ var threshold = MinConfidence.ForBuckets(pairs);
+ if (want.GetProperty("valid").GetBoolean())
+ {
+ var got = threshold.Check();
+ var expected = want.GetProperty("result").EnumerateArray().ToDictionary(p => p[0].GetString()!, p => Num(p[1]));
+ Assert.Equal(expected.OrderBy(kv => kv.Key), got.Map!.OrderBy(kv => kv.Key));
+ return;
+ }
+ var error = Assert.Throws<ArgumentException>(() => threshold.Check());
+ Assert.Equal(want.GetProperty("error_message").GetString(), error.Message);
+ }
+
+ [Fact]
+ public void BucketKeysAndGateStatesAreThoseOfPython()
+ {
+ Assert.Equal(Root.GetProperty("bucket_keys").EnumerateArray().Select(k => k.GetString()!).Order(StringComparer.Ordinal),
+ MinConfidence.BucketKeys.Order(StringComparer.Ordinal));
+ Assert.Equal(Root.GetProperty("gate_states").EnumerateArray().Select(k => k.GetString()!), GateStates.All);
+ }
+
+ // ── _option_bucket / resolve_min_confidence ───────────────────────────────
+
+ [Theory]
+ [MemberData(nameof(BucketCases))]
+ public void OptionBucketsAreThoseOfPython(int index)
+ {
+ var want = Root.GetProperty("buckets")[index];
+ var bucket = want.GetProperty("bucket");
+ Assert.Equal(bucket.ValueKind == JsonValueKind.Null ? null : bucket.GetString(),
+ ConfidenceGate.OptionBucket(ToAnswer(want.GetProperty("answer"))));
+ }
+
+ [Theory]
+ [MemberData(nameof(ResolveCases))]
+ public void ThresholdResolutionFollowsPython(int index)
+ {
+ var want = Root.GetProperty("resolve")[index];
+ var map = want.GetProperty("thresholds").EnumerateObject().ToDictionary(p => p.Name, p => Num(p.Value));
+ Assert.Equal(Num(want.GetProperty("result")), ConfidenceGate.Resolve(ToAnswer(want.GetProperty("answer")), map));
+ }
+
+ // ── apply_confidence_gate ─────────────────────────────────────────────────
+
+ [Theory]
+ [MemberData(nameof(GateCases))]
+ public void TheGateWritesWhatPythonWrites(int index)
+ {
+ var want = Root.GetProperty("gate")[index];
+ var result = ToResult(Root.GetProperty("gate_answers"));
+ var threshold = ToThreshold(want.GetProperty("threshold"));
+ var gated = threshold is null ? result : ConfidenceGate.Apply(result, threshold.Check());
+ AssertGated(want.GetProperty("expected"), gated, $"gate[{index}]");
+ }
+
+ // #910: Python re-gates one payload in place; here each step must depend only on its own threshold,
+ // and the answers handed in must not change at all.
+ [Theory]
+ [MemberData(nameof(ReuseCases))]
+ public void AReusedResultCarriesNoStaleVerdict(int index)
+ {
+ var result = ToResult(Root.GetProperty("gate_answers"));
+ var steps = Root.GetProperty("reuse")[index].GetProperty("steps").EnumerateArray().ToArray();
+ for (var s = 0; s < steps.Length; s++)
+ {
+ var gated = ConfidenceGate.Apply(result, ToThreshold(steps[s].GetProperty("threshold"))!.Check());
+ AssertGated(steps[s].GetProperty("expected"), gated, $"reuse[{index}] step {s}");
+ foreach (var id in result.Ids)
+ {
+ Assert.False(result[id].LowConfidence);
+ Assert.Null(result[id].Abstention);
+ Assert.Null(result[id].AbstentionThreshold);
+ }
+ }
+ }
+
+ [Fact]
+ public void GatingTheSameAnswerFromManyThreadsNeverLeaksAVerdictAcrossCalls()
+ {
+ var result = ToResult(Root.GetProperty("gate_answers"));
+ var answer = result["choice2"];
+ Parallel.For(0, 400, i =>
+ {
+ var strict = i % 2 == 0;
+ var gated = ConfidenceGate.Apply(result, MinConfidence.Of(strict ? 0.9 : 0.0).Check())["choice2"];
+ Assert.Equal(strict, gated.LowConfidence);
+ Assert.Equal(strict ? GateStates.Abstained : GateStates.Passed, gated.Abstention);
+ });
+ Assert.Null(answer.Abstention);
+ }
+
+ // ── jev_confidence ────────────────────────────────────────────────────────
+
+ [Theory]
+ [MemberData(nameof(JevCases))]
+ public void JevConfidenceMatchesPython(int index)
+ {
+ var want = Root.GetProperty("jev")[index];
+ if (!want.GetProperty("typed").GetBoolean()) return; // a shape a typed answer cannot hold
+ var result = want.GetProperty("result");
+ var got = ToAnswer(want.GetProperty("answer")).JevConfidence();
+ if (result.ValueKind == JsonValueKind.Null) Assert.Null(got);
+ else
+ {
+ Assert.NotNull(got);
+ // Bit for bit: CPython 3.12's compensated float sum() is reproduced by PythonSum.
+ Assert.Equal(result.GetDouble(), got.Value);
+ }
+ }
+
+ [Fact]
+ public void TheProbeCoversTypedAndUntypedJevCases()
+ {
+ var cases = Root.GetProperty("jev").EnumerateArray().ToArray();
+ Assert.Contains(cases, c => !c.GetProperty("typed").GetBoolean());
+ Assert.True(cases.Count(c => c.GetProperty("typed").GetBoolean()) >= 60);
+ Assert.Equal(9, cases.Count(c => c.GetProperty("name").GetString()!.StartsWith("sum-differs-", StringComparison.Ordinal)));
+ Assert.Contains(cases, c => c.GetProperty("result").ValueKind == JsonValueKind.Null && c.GetProperty("typed").GetBoolean());
+ }
+
+ // ── call level: an ungated call writes nothing ────────────────────────────
+
+ private static ChoiceAnswer Choice(double answerConfidence, double confidence = 0.1) => new(
+ "a", [new("a", answerConfidence), new("b", 1.0 - answerConfidence)], confidence, answerConfidence, new ActionInfo(0.5));
+
+ private sealed class FakeBatchPredictor(string name, double answerConfidence) : ILayaBatchPredictor
+ {
+ public int Calls, BatchCalls;
+ public LayaBatchOptions? LastOptions;
+
+ private LayaResult One() => new(name, ["q"], new Dictionary<string, Answer>(StringComparer.Ordinal)
+ {
+ ["q"] = Choice(answerConfidence),
+ }, new Usage(0, 0));
+
+ public LayaResult Predict(object? state, QuestionSet questions)
+ {
+ Interlocked.Increment(ref Calls);
+ return One();
+ }
+
+ public IReadOnlyList<LayaResult> PredictBatch(IReadOnlyList<object?> states, QuestionSet questions, LayaBatchOptions? options = null)
+ {
+ Interlocked.Increment(ref BatchCalls);
+ LastOptions = options;
+ return [.. states.Select(_ => One())];
+ }
+ }
+
+ private sealed class Rig : IDisposable
+ {
+ public Dictionary<LayaCheckpoint, FakeBatchPredictor> Predictors { get; } = [];
+ public int Builds;
+ public LayaRouter Router { get; }
+
+ public Rig(double answerConfidence = 0.6) =>
+ Router = new LayaRouter(null, checkpoint =>
+ {
+ Interlocked.Increment(ref Builds);
+ return Predictors[checkpoint] = new FakeBatchPredictor(LayaOptions.CheckpointSubfolder(checkpoint), answerConfidence);
+ });
+
+ public void Dispose() => Router.Dispose();
+ }
+
+ private static QuestionSet Questions() => new() { ["q"] = Question.Noul("x") };
+
+
+ [Fact]
+ public void AnUngatedRouterCallWritesNothing()
+ {
+ using var rig = new Rig();
+ var answer = rig.Router.Predict("I was charged twice", Questions())["q"];
+ Assert.False(answer.LowConfidence);
+ Assert.Null(answer.Abstention);
+ Assert.Null(answer.AbstentionThreshold);
+ var viaOptions = rig.Router.Predict("I was charged twice", Questions(), null, null, null, null, new LayaPredictOptions())["q"];
+ Assert.Null(viaOptions.Abstention);
+ }
+
+ [Fact]
+ public void ARouterCallGatesTheAnswersAndKeepsTheRouting()
+ {
+ using var rig = new Rig(answerConfidence: 0.6);
+ var options = new LayaPredictOptions { MinConfidence = MinConfidence.Of(0.7) };
+ var result = rig.Router.Predict("I was charged twice", Questions(), null, null, null, null, options);
+ Assert.NotNull(result.Routing);
+ Assert.True(result["q"].LowConfidence);
+ Assert.Equal(GateStates.Abstained, result["q"].Abstention);
+ Assert.Equal(0.7, result["q"].AbstentionThreshold);
+
+ var passes = rig.Router.Predict("I was charged twice", Questions(), null, null, null, null,
+ new LayaPredictOptions { MinConfidence = MinConfidence.Of(0.0) });
+ Assert.False(passes["q"].LowConfidence);
+ Assert.Equal(GateStates.Passed, passes["q"].Abstention);
+ Assert.Equal(0.0, passes["q"].AbstentionThreshold);
+ }
+
+ [Fact]
+ public void TheThreeArgumentRouterOverloadRoutesLikeThePlainOneAndGates()
+ {
+ using var rig = new Rig(answerConfidence: 0.6);
+ var gated = rig.Router.Predict("I was charged twice", Questions(),
+ new LayaPredictOptions { MinConfidence = MinConfidence.Of(0.7) });
+ Assert.Equal(GateStates.Abstained, gated["q"].Abstention);
+ Assert.Equal(rig.Router.Predict("I was charged twice", Questions()).Routing!.Model, gated.Routing!.Model);
+ Assert.Null(rig.Router.Predict("I was charged twice", Questions(), null)["q"].Abstention);
+ }
+
+ [Fact]
+ public void ARouterMapThresholdIsResolvedPerAnswerBucket()
+ {
+ using var rig = new Rig(answerConfidence: 0.6);
+ var map = MinConfidence.ForBuckets([new("choice:2", 0.65), new("default", 0.1)]);
+ var result = rig.Router.Predict("I was charged twice", Questions(), null, null, null, null,
+ new LayaPredictOptions { MinConfidence = map });
+ Assert.Equal(0.65, result["q"].AbstentionThreshold);
+ Assert.True(result["q"].LowConfidence);
+ }
+
+ [Fact]
+ public void AnInvalidThresholdIsRefusedBeforeAnythingIsRoutedOrLoaded()
+ {
+ using var rig = new Rig();
+ var bad = new LayaPredictOptions { MinConfidence = MinConfidence.Of(1.5) };
+ var error = Assert.Throws<ArgumentException>(
+ () => rig.Router.Predict("text", Questions(), null, null, null, null, bad));
+ Assert.Equal("min_confidence must be a float in [0.0, 1.0], got 1.5", error.Message);
+ Assert.Equal(0, rig.Builds);
+ Assert.Empty(rig.Router.Loaded);
+ }
+
+ [Fact]
+ public void TheRouterChecksStateAndQuestionsBeforeTheThreshold()
+ {
+ // router.py:1335-1338: a missing state is the TypeError even when the threshold is also wrong.
+ using var rig = new Rig();
+ var bad = new LayaPredictOptions { MinConfidence = MinConfidence.Of(double.NaN) };
+ var none = Assert.Throws<ArgumentNullException>(
+ () => rig.Router.Predict(null, Questions(), null, null, null, null, bad));
+ Assert.Equal("state must not be None; pass a string, dict, or list", none.Message);
+ Assert.Throws<ArgumentNullException>(() => rig.Router.Predict("text", null!, null, null, null, null, bad));
+ }
+
+ [Fact]
+ public void ABatchThresholdIsCheckedBeforeTheRequests()
+ {
+ // router.py:1669: the threshold is the first thing predict_batch looks at.
+ using var rig = new Rig();
+ var error = Assert.Throws<ArgumentException>(() => rig.Router.PredictBatch(
+ null!, new LayaBatchOptions { MinConfidence = MinConfidence.ForBuckets([new("bogus", 0.5)]) }));
+ Assert.Equal(typeof(ArgumentException), error.GetType());
+ Assert.Equal("min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'bogus'", error.Message);
+ Assert.Equal(0, rig.Builds);
+ }
+
+ [Fact]
+ public void ABatchThresholdIsBatchWideAndNotPartOfTheGrouping()
+ {
+ using var rig = new Rig(answerConfidence: 0.6);
+ var questions = Questions();
+ LayaBatchRequest[] requests =
+ [
+ new() { State = "I was charged twice", Questions = questions },
+ new() { State = "Refund my invoice", Questions = questions },
+ new() { State = "Please cancel my account", Questions = questions },
+ ];
+ var plain = rig.Router.PredictBatch(requests);
+ var plainCalls = rig.Predictors.Values.Sum(p => p.BatchCalls);
+ var gated = rig.Router.PredictBatch(
+ requests, new LayaBatchOptions { BatchSize = 2, SortByLength = true, MinConfidence = MinConfidence.Of(0.7) });
+ // router.py:1789: the checkpoints get the batching knobs but not the threshold; the router gates.
+ Assert.All(rig.Predictors.Values.Where(p => p.LastOptions is not null), p =>
+ {
+ Assert.Null(p.LastOptions!.MinConfidence);
+ Assert.Equal(2, p.LastOptions.BatchSize);
+ Assert.True(p.LastOptions.SortByLength);
+ });
+ Assert.Equal(plainCalls * 2, rig.Predictors.Values.Sum(p => p.BatchCalls));
+ Assert.All(plain, r => Assert.Null(r["q"].Abstention));
+ Assert.Equal(3, gated.Count);
+ Assert.All(gated, r =>
+ {
+ Assert.NotNull(r.Routing);
+ Assert.Equal(GateStates.Abstained, r["q"].Abstention);
+ Assert.True(r["q"].LowConfidence);
+ Assert.Equal(0.7, r["q"].AbstentionThreshold);
+ });
+ }
+
+ [Fact]
+ public void AnAttachedPredictorWithoutGateSupportIsStillGatedByTheRouter()
+ {
+ using var router = new LayaRouter(null, _ => throw new InvalidOperationException("must not be built"));
+ router.Attach("english", new FakeBatchPredictor("mine", 0.4));
+ var result = router.Predict("I was charged twice", Questions(), "english", null, null, null,
+ new LayaPredictOptions { MinConfidence = MinConfidence.Of(0.5) });
+ Assert.Equal(GateStates.Abstained, result["q"].Abstention);
+ }
+
+ // ── answers keep their gate through the copies the API makes ──────────────
+
+ [Fact]
+ public void AGatedAnswerStaysGatedWhenAWindowIsAttached()
+ {
+ var gated = ConfidenceGate.Apply(Choice(0.2), MinConfidence.Of(0.5).Check());
+ var windowed = gated.WithWindow(new WindowInfo(0, 0, 10, 1));
+ Assert.Equal(GateStates.Abstained, windowed.Abstention);
+ Assert.True(windowed.LowConfidence);
+ Assert.Null(Choice(0.2).Abstention);
+ }
+
+ // ── JevConfidence on an answer ────────────────────────────────────────────
+
+ [Fact]
+ public void PythonSumIsCompensatedLikeCPython312()
+ {
+ // A plain left-to-right sum of ten 0.1s is 0.9999999999999999; CPython 3.12 returns 1.0.
+ Assert.Equal(1.0, PythonSum.Sum(Enumerable.Repeat(0.1, 10)));
+ Assert.Equal(0.0, PythonSum.Sum([]));
+ Assert.True(double.IsPositiveInfinity(PythonSum.Sum([double.MaxValue, double.MaxValue])));
+ Assert.True(double.IsNaN(PythonSum.Sum([double.PositiveInfinity, double.NegativeInfinity])));
+ }
+
+ [Fact]
+ public void JevConfidenceIsNullForNoul()
+ {
+ Assert.Null(new NoulAnswer(0.9, 0.9, 0.9, new ActionInfo(0.5)).JevConfidence());
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs
index f977a1fd..9b798394 100644
--- a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs
+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs
@@ -22,6 +22,7 @@ public sealed partial class PredictParityTests
{
BatchSize = options.TryGetProperty("batch_size", out var size) ? size.GetInt32() : null,
SortByLength = options.TryGetProperty("sort_by_length", out var sort) && sort.GetBoolean(),
+ MinConfidence = ConfidenceGateTests.MinConfidenceIn(options),
};
private static List<object?> BatchStates(JsonElement root) =>
@@ -56,6 +57,7 @@ public sealed partial class PredictParityTests
var at = $"{where}/{entry.Name}";
Assert.Equal(want.GetProperty("type").GetString(), SequenceBuilder.TypeName(answer.Type));
AssertClose(want.GetProperty("confidence").GetDouble(), answer.Confidence, ProbTolerance, at + ".confidence");
+ ConfidenceGateTests.AssertGateFields(want, answer, at);
AssertClose(want.GetProperty("answer_confidence").GetDouble(), answer.AnswerConfidence,
ProbTolerance, at + ".answer_confidence");
AssertClose(want.GetProperty("action").GetProperty("act_probability").GetDouble(),
diff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs
index 8a588a5c..8e8bc7be 100644
--- a/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs
+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs
@@ -78,7 +78,8 @@ public sealed partial class PredictParityTests(AllEnginesFixture fixture)
var state = GoldenData.ToClr(root.GetProperty("state"));
var questions = GoldenData.BuildQuestions(root.GetProperty("questions"));
- var result = engine.Predict(state, questions);
+ var result = engine.Predict(state, questions,
+ new LayaPredictOptions { MinConfidence = ConfidenceGateTests.MinConfidenceOf(root) });
var expected = root.GetProperty("result");
Assert.Equal(expected.GetProperty("model").GetString(), result.Model);
@@ -102,6 +103,7 @@ public sealed partial class PredictParityTests(AllEnginesFixture fixture)
Assert.Equal(want.GetProperty("type").GetString(), SequenceBuilder.TypeName(got.Type));
AssertClose(want.GetProperty("confidence").GetDouble(), got.Confidence,
ProbTolerance, $"{where}.confidence");
+ ConfidenceGateTests.AssertGateFields(want, got, where);
AssertClose(want.GetProperty("action").GetProperty("act_probability").GetDouble(),
got.Action.ActProbability, ProbTolerance, $"{where}.action.act_probability");
@@ -139,7 +141,8 @@ public sealed partial class PredictParityTests(AllEnginesFixture fixture)
var state = GoldenData.ToClr(root.GetProperty("state"));
var questions = GoldenData.BuildQuestions(root.GetProperty("questions"));
- var result = engine.Predict(state, questions);
+ var result = engine.Predict(state, questions,
+ new LayaPredictOptions { MinConfidence = ConfidenceGateTests.MinConfidenceOf(root) });
var expected = root.GetProperty("result");
Assert.Equal(expected.GetProperty("model").GetString(), result.Model);
@@ -158,6 +161,7 @@ public sealed partial class PredictParityTests(AllEnginesFixture fixture)
AssertClose(want.GetProperty("confidence").GetDouble(), got.Confidence,
ProbTolerance, $"{where}.confidence");
+ ConfidenceGateTests.AssertGateFields(want, got, where);
switch (got.Type)
{
@@ -189,7 +193,8 @@ public sealed partial class PredictParityTests(AllEnginesFixture fixture)
var state = GoldenData.ToClr(root.GetProperty("state"));
var questions = GoldenData.BuildQuestions(root.GetProperty("questions"));
- var error = Assert.Throws<ArgumentException>(() => engine.Predict(state, questions));
+ var options = new LayaPredictOptions { MinConfidence = ConfidenceGateTests.MinConfidenceOf(root) };
+ var error = Assert.Throws<ArgumentException>(() => engine.Predict(state, questions, options));
// ArgumentException appends the parameter name, so the recorded text is a prefix of the
// message rather than all of it.
Assert.StartsWith(root.GetProperty("error_message").GetString()!, error.Message, StringComparison.Ordinal);
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_bad_key.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_bad_key.json
new file mode 100644
index 00000000..424420ec
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_bad_key.json
@@ -0,0 +1,38 @@
+{
+ "name": "batch_min_confidence_bad_key",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "choice:2": 0.5,
+ "choice:3": 0.5
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'"
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_before_none_state.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_before_none_state.json
new file mode 100644
index 00000000..75fd7564
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_before_none_state.json
@@ -0,0 +1,36 @@
+{
+ "name": "batch_min_confidence_before_none_state",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ null
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": -0.25
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -0.25"
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_empty_questions.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_empty_questions.json
new file mode 100644
index 00000000..255f7bc4
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_empty_questions.json
@@ -0,0 +1,44 @@
+{
+ "name": "batch_min_confidence_empty_questions",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, the refund arrived and everything works now.",
+ {
+ "ticket": 4821,
+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "
+ }
+ ],
+ "questions": {},
+ "options": {
+ "min_confidence": 0.5
+ },
+ "state_widths": [],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {},
+ "usage": {
+ "input_tokens": 0,
+ "output_tokens": 0
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {},
+ "usage": {
+ "input_tokens": 0,
+ "output_tokens": 0
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {},
+ "usage": {
+ "input_tokens": 0,
+ "output_tokens": 0
+ }
+ }
+ ],
+ "runs": []
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_empty_states.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_empty_states.json
new file mode 100644
index 00000000..dfb141d9
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_empty_states.json
@@ -0,0 +1,34 @@
+{
+ "name": "batch_min_confidence_empty_states",
+ "states": [],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 0.5
+ },
+ "state_widths": [],
+ "results": [],
+ "runs": []
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_invalid.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_invalid.json
new file mode 100644
index 00000000..72496b52
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_invalid.json
@@ -0,0 +1,35 @@
+{
+ "name": "batch_min_confidence_invalid",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 1.5
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_map.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_map.json
new file mode 100644
index 00000000..f1c18d0c
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_map.json
@@ -0,0 +1,497 @@
+{
+ "name": "batch_min_confidence_map",
+ "states": [
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "batch_size": 3,
+ "sort_by_length": true,
+ "min_confidence": {
+ "choice:3-5": 0.8,
+ "score:3-5": 0.5,
+ "noul:2": 0.7
+ }
+ },
+ "state_widths": [
+ 199,
+ 103,
+ 145,
+ 91,
+ 175,
+ 121,
+ 157
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.896,
+ "technical": 0.0688,
+ "other": 0.0352
+ },
+ "confidence": 0.6355,
+ "answer_confidence": 0.896,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7317,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0621,
+ "1": 0.1442,
+ "2": 0.7937
+ },
+ "confidence": 0.4219,
+ "answer_confidence": 0.7937,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.9824,
+ "confidence": 0.9824,
+ "answer_confidence": 0.9824,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 589,
+ "output_tokens": 0,
+ "state_tokens": 165,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.8605,
+ "technical": 0.0968,
+ "other": 0.0428
+ },
+ "confidence": 0.5539,
+ "answer_confidence": 0.8605,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7395,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0594,
+ "1": 0.1416,
+ "2": 0.7989
+ },
+ "confidence": 0.432,
+ "answer_confidence": 0.7989,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.9368,
+ "confidence": 0.9368,
+ "answer_confidence": 0.9368,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 301,
+ "output_tokens": 0,
+ "state_tokens": 69,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9048,
+ "technical": 0.064,
+ "other": 0.0312
+ },
+ "confidence": 0.6591,
+ "answer_confidence": 0.9048,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7761,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0368,
+ "1": 0.1503,
+ "2": 0.8129
+ },
+ "confidence": 0.4768,
+ "answer_confidence": 0.8129,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.9441,
+ "confidence": 0.9441,
+ "answer_confidence": 0.9441,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 427,
+ "output_tokens": 0,
+ "state_tokens": 111,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.8355,
+ "technical": 0.1139,
+ "other": 0.0506
+ },
+ "confidence": 0.5008,
+ "answer_confidence": 0.8355,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8578,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0339,
+ "1": 0.0745,
+ "2": 0.8916
+ },
+ "confidence": 0.6264,
+ "answer_confidence": 0.8916,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.968,
+ "confidence": 0.968,
+ "answer_confidence": 0.968,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 265,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.8943,
+ "technical": 0.071,
+ "other": 0.0347
+ },
+ "confidence": 0.632,
+ "answer_confidence": 0.8943,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7077,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0712,
+ "1": 0.1499,
+ "2": 0.7789
+ },
+ "confidence": 0.3927,
+ "answer_confidence": 0.7789,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.973,
+ "confidence": 0.973,
+ "answer_confidence": 0.973,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 517,
+ "output_tokens": 0,
+ "state_tokens": 141,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.8917,
+ "technical": 0.0738,
+ "other": 0.0345
+ },
+ "confidence": 0.6261,
+ "answer_confidence": 0.8917,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7692,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0487,
+ "1": 0.1333,
+ "2": 0.818
+ },
+ "confidence": 0.4719,
+ "answer_confidence": 0.818,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.9284,
+ "confidence": 0.9284,
+ "answer_confidence": 0.9284,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 355,
+ "output_tokens": 0,
+ "state_tokens": 87,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.8991,
+ "technical": 0.0687,
+ "other": 0.0322
+ },
+ "confidence": 0.6449,
+ "answer_confidence": 0.8991,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7492,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0463,
+ "1": 0.1583,
+ "2": 0.7955
+ },
+ "confidence": 0.4393,
+ "answer_confidence": 0.7955,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.9494,
+ "confidence": 0.9494,
+ "answer_confidence": 0.9494,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 463,
+ "output_tokens": 0,
+ "state_tokens": 123,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 9,
+ 121
+ ],
+ [
+ 9,
+ 175
+ ],
+ [
+ 3,
+ 199
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_map_default.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_map_default.json
new file mode 100644
index 00000000..9e741685
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_map_default.json
@@ -0,0 +1,237 @@
+{
+ "name": "batch_min_confidence_map_default",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, the refund arrived and everything works now.",
+ {
+ "ticket": 4821,
+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "
+ }
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "default": 0.55
+ }
+ },
+ "state_widths": [
+ 91,
+ 44,
+ 157
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.8355,
+ "technical": 0.1139,
+ "other": 0.0506
+ },
+ "confidence": 0.5008,
+ "answer_confidence": 0.8355,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8578,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0339,
+ "1": 0.0745,
+ "2": 0.8916
+ },
+ "confidence": 0.6264,
+ "answer_confidence": 0.8916,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.968,
+ "confidence": 0.968,
+ "answer_confidence": 0.968,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 265,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.8178,
+ "technical": 0.1682,
+ "other": 0.014
+ },
+ "confidence": 0.5229,
+ "answer_confidence": 0.8178,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.7428,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.5839,
+ "1": 0.0894,
+ "2": 0.3267
+ },
+ "confidence": 0.1849,
+ "answer_confidence": 0.5839,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0144,
+ "confidence": 0.9856,
+ "answer_confidence": 0.9856,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 124,
+ "output_tokens": 0,
+ "state_tokens": 10,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7468,
+ "technical": 0.1514,
+ "other": 0.1017
+ },
+ "confidence": 0.3297,
+ "answer_confidence": 0.7468,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.6364,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.1029,
+ "1": 0.1578,
+ "2": 0.7393
+ },
+ "confidence": 0.3185,
+ "answer_confidence": 0.7393,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.9238,
+ "confidence": 0.9238,
+ "answer_confidence": 0.9238,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 463,
+ "output_tokens": 0,
+ "state_tokens": 123,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 9,
+ 157
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_scalar.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_scalar.json
new file mode 100644
index 00000000..c111a0d8
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_scalar.json
@@ -0,0 +1,571 @@
+{
+ "name": "batch_min_confidence_scalar",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ {
+ "subject": "Double charged on invoice 88213",
+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ },
+ [
+ {
+ "role": "user",
+ "content": "Hi, I need help with my invoice."
+ },
+ {
+ "role": "assistant",
+ "content": "Of course, what is wrong?"
+ },
+ {
+ "role": "user",
+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ }
+ ],
+ "",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, that is all sorted now.",
+ {
+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",
+ "message": "Please cancel my account today."
+ },
+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 0.6
+ },
+ "state_widths": [
+ 91,
+ 110,
+ 144,
+ 34,
+ 163,
+ 42,
+ 98,
+ 56
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.8355,
+ "technical": 0.1139,
+ "other": 0.0506
+ },
+ "confidence": 0.5008,
+ "answer_confidence": 0.8355,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8578,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0339,
+ "1": 0.0745,
+ "2": 0.8916
+ },
+ "confidence": 0.6264,
+ "answer_confidence": 0.8916,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.968,
+ "confidence": 0.968,
+ "answer_confidence": 0.968,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 265,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7826,
+ "technical": 0.1583,
+ "other": 0.0591
+ },
+ "confidence": 0.4076,
+ "answer_confidence": 0.7826,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7512,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0438,
+ "1": 0.1611,
+ "2": 0.795
+ },
+ "confidence": 0.4415,
+ "answer_confidence": 0.795,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.9312,
+ "confidence": 0.9312,
+ "answer_confidence": 0.9312,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 322,
+ "output_tokens": 0,
+ "state_tokens": 76,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7843,
+ "technical": 0.1543,
+ "other": 0.0614
+ },
+ "confidence": 0.4082,
+ "answer_confidence": 0.7843,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7074,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0568,
+ "1": 0.179,
+ "2": 0.7642
+ },
+ "confidence": 0.3843,
+ "answer_confidence": 0.7642,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.8762,
+ "confidence": 0.8762,
+ "answer_confidence": 0.8762,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 424,
+ "output_tokens": 0,
+ "state_tokens": 110,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.5445,
+ "technical": 0.2427,
+ "other": 0.2129
+ },
+ "confidence": 0.0861,
+ "answer_confidence": 0.5445,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.6874,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.4819,
+ "1": 0.3488,
+ "2": 0.1693
+ },
+ "confidence": 0.0717,
+ "answer_confidence": 0.4819,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.3894,
+ "confidence": 0.6106,
+ "answer_confidence": 0.6106,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 94,
+ "output_tokens": 0,
+ "state_tokens": 0,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9119,
+ "technical": 0.0619,
+ "other": 0.0263
+ },
+ "confidence": 0.6797,
+ "answer_confidence": 0.9119,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8174,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0327,
+ "1": 0.1173,
+ "2": 0.85
+ },
+ "confidence": 0.5437,
+ "answer_confidence": 0.85,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.9744,
+ "confidence": 0.9744,
+ "answer_confidence": 0.9744,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 481,
+ "output_tokens": 0,
+ "state_tokens": 129,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "technical",
+ "probabilities": {
+ "refund": 0.2918,
+ "technical": 0.398,
+ "other": 0.3102
+ },
+ "confidence": 0.0086,
+ "answer_confidence": 0.398,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.7918,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.5403,
+ "1": 0.1276,
+ "2": 0.3321
+ },
+ "confidence": 0.1249,
+ "answer_confidence": 0.5403,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0548,
+ "confidence": 0.9452,
+ "answer_confidence": 0.9452,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 118,
+ "output_tokens": 0,
+ "state_tokens": 8,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.573,
+ "technical": 0.1873,
+ "other": 0.2397
+ },
+ "confidence": 0.1124,
+ "answer_confidence": 0.573,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8166,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0486,
+ "1": 0.0862,
+ "2": 0.8652
+ },
+ "confidence": 0.5599,
+ "answer_confidence": 0.8652,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.8598,
+ "confidence": 0.8598,
+ "answer_confidence": 0.8598,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 286,
+ "output_tokens": 0,
+ "state_tokens": 64,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.5426,
+ "technical": 0.3484,
+ "other": 0.109
+ },
+ "confidence": 0.1438,
+ "answer_confidence": 0.5426,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.516,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.6281,
+ "1": 0.2278,
+ "2": 0.1441
+ },
+ "confidence": 0.1733,
+ "answer_confidence": 0.6281,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.3136,
+ "confidence": 0.6864,
+ "answer_confidence": 0.6864,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 160,
+ "output_tokens": 0,
+ "state_tokens": 22,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 24,
+ 163
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_zero.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_zero.json
new file mode 100644
index 00000000..8f9f1a21
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_batch_min_confidence_zero.json
@@ -0,0 +1,565 @@
+{
+ "name": "batch_min_confidence_zero",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ {
+ "subject": "Double charged on invoice 88213",
+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ },
+ [
+ {
+ "role": "user",
+ "content": "Hi, I need help with my invoice."
+ },
+ {
+ "role": "assistant",
+ "content": "Of course, what is wrong?"
+ },
+ {
+ "role": "user",
+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ }
+ ],
+ "",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, that is all sorted now.",
+ {
+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",
+ "message": "Please cancel my account today."
+ },
+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 0.0
+ },
+ "state_widths": [
+ 91,
+ 110,
+ 144,
+ 34,
+ 163,
+ 42,
+ 98,
+ 56
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.8355,
+ "technical": 0.1139,
+ "other": 0.0506
+ },
+ "confidence": 0.5008,
+ "answer_confidence": 0.8355,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8578,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0339,
+ "1": 0.0745,
+ "2": 0.8916
+ },
+ "confidence": 0.6264,
+ "answer_confidence": 0.8916,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.968,
+ "confidence": 0.968,
+ "answer_confidence": 0.968,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 265,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7826,
+ "technical": 0.1583,
+ "other": 0.0591
+ },
+ "confidence": 0.4076,
+ "answer_confidence": 0.7826,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7512,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0438,
+ "1": 0.1611,
+ "2": 0.795
+ },
+ "confidence": 0.4415,
+ "answer_confidence": 0.795,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.9312,
+ "confidence": 0.9312,
+ "answer_confidence": 0.9312,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 322,
+ "output_tokens": 0,
+ "state_tokens": 76,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7843,
+ "technical": 0.1543,
+ "other": 0.0614
+ },
+ "confidence": 0.4082,
+ "answer_confidence": 0.7843,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7074,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0568,
+ "1": 0.179,
+ "2": 0.7642
+ },
+ "confidence": 0.3843,
+ "answer_confidence": 0.7642,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.8762,
+ "confidence": 0.8762,
+ "answer_confidence": 0.8762,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 424,
+ "output_tokens": 0,
+ "state_tokens": 110,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.5445,
+ "technical": 0.2427,
+ "other": 0.2129
+ },
+ "confidence": 0.0861,
+ "answer_confidence": 0.5445,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.6874,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.4819,
+ "1": 0.3488,
+ "2": 0.1693
+ },
+ "confidence": 0.0717,
+ "answer_confidence": 0.4819,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.3894,
+ "confidence": 0.6106,
+ "answer_confidence": 0.6106,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 94,
+ "output_tokens": 0,
+ "state_tokens": 0,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9119,
+ "technical": 0.0619,
+ "other": 0.0263
+ },
+ "confidence": 0.6797,
+ "answer_confidence": 0.9119,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8174,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0327,
+ "1": 0.1173,
+ "2": 0.85
+ },
+ "confidence": 0.5437,
+ "answer_confidence": 0.85,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.9744,
+ "confidence": 0.9744,
+ "answer_confidence": 0.9744,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 481,
+ "output_tokens": 0,
+ "state_tokens": 129,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "technical",
+ "probabilities": {
+ "refund": 0.2918,
+ "technical": 0.398,
+ "other": 0.3102
+ },
+ "confidence": 0.0086,
+ "answer_confidence": 0.398,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.7918,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.5403,
+ "1": 0.1276,
+ "2": 0.3321
+ },
+ "confidence": 0.1249,
+ "answer_confidence": 0.5403,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0548,
+ "confidence": 0.9452,
+ "answer_confidence": 0.9452,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 118,
+ "output_tokens": 0,
+ "state_tokens": 8,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.573,
+ "technical": 0.1873,
+ "other": 0.2397
+ },
+ "confidence": 0.1124,
+ "answer_confidence": 0.573,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8166,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0486,
+ "1": 0.0862,
+ "2": 0.8652
+ },
+ "confidence": 0.5599,
+ "answer_confidence": 0.8652,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.8598,
+ "confidence": 0.8598,
+ "answer_confidence": 0.8598,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 286,
+ "output_tokens": 0,
+ "state_tokens": 64,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.5426,
+ "technical": 0.3484,
+ "other": 0.109
+ },
+ "confidence": 0.1438,
+ "answer_confidence": 0.5426,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.516,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.6281,
+ "1": 0.2278,
+ "2": 0.1441
+ },
+ "confidence": 0.1733,
+ "answer_confidence": 0.6281,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.3136,
+ "confidence": 0.6864,
+ "answer_confidence": 0.6864,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 160,
+ "output_tokens": 0,
+ "state_tokens": 22,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 24,
+ 163
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_bad_key.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_bad_key.json
new file mode 100644
index 00000000..7eab72d1
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_bad_key.json
@@ -0,0 +1,45 @@
+{
+ "name": "min_confidence_bad_key",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'",
+ "options": {
+ "min_confidence": {
+ "default": 0.5,
+ "choice:3": 0.5
+ }
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_empty_map.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_empty_map.json
new file mode 100644
index 00000000..28fa69e2
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_empty_map.json
@@ -0,0 +1,42 @@
+{
+ "name": "min_confidence_empty_map",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "a min_confidence map must be a non-empty dict of bucket -> float, got {}",
+ "options": {
+ "min_confidence": {}
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_invalid.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_invalid.json
new file mode 100644
index 00000000..5ecd652a
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_invalid.json
@@ -0,0 +1,42 @@
+{
+ "name": "min_confidence_invalid",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5",
+ "options": {
+ "min_confidence": 1.5
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_map.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_map.json
new file mode 100644
index 00000000..c531def3
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_map.json
@@ -0,0 +1,1524 @@
+{
+ "name": "min_confidence_map",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 24,
+ 34
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 17,
+ 23,
+ 28
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 17,
+ 26
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 24,
+ 34,
+ 0
+ ],
+ [
+ 11,
+ 17,
+ 23,
+ 28
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 26,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ 4.488276958465576,
+ -2.729666233062744,
+ -2.4480319023132324,
+ -10000.0
+ ],
+ [
+ -0.8349542617797852,
+ 0.8832842707633972,
+ 3.6844465732574463,
+ 1.2054039239883423
+ ],
+ [
+ -4.515446186065674,
+ 2.250016689300537,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ 0.7536231279373169,
+ 0.41879838705062866,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 4074.86572265625,
+ -3334.4990234375
+ ],
+ [
+ 4040.2958984375,
+ -3316.3017578125
+ ],
+ [
+ 2926.997314453125,
+ -2398.78125
+ ],
+ [
+ 4615.162109375,
+ -3778.93896484375
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9653,
+ "technical": 0.016,
+ "account": 0.0188
+ },
+ "confidence": 0.8409,
+ "answer_confidence": 0.9653,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.9
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.9821,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.0212,
+ "1": 0.0839,
+ "2": 0.7864,
+ "3": 0.1085
+ },
+ "confidence": 0.4809,
+ "answer_confidence": 0.7864,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.968,
+ "confidence": 0.968,
+ "answer_confidence": 0.968,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.4579,
+ "confidence": 0.5421,
+ "answer_confidence": 0.5421,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 379,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "choice:3-5": 0.9,
+ "score:3-5": 0.5,
+ "noul:2": 0.7
+ }
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_map_default.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_map_default.json
new file mode 100644
index 00000000..65d5dbe1
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_map_default.json
@@ -0,0 +1,935 @@
+{
+ "name": "min_confidence_map_default",
+ "state": "Thanks, that is all sorted now.",
+ "serialized_state": "Thanks, that is all sorted now.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 24,
+ 34
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 17,
+ 23,
+ 28
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 17,
+ 26
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282
+ ],
+ [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 24,
+ 34,
+ 0
+ ],
+ [
+ 11,
+ 17,
+ 23,
+ 28
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 26,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ 0.7524149417877197,
+ 0.5028965473175049,
+ -0.09920725226402283,
+ -10000.0
+ ],
+ [
+ 2.5853981971740723,
+ 1.4106674194335938,
+ 2.1949286460876465,
+ -0.39206796884536743
+ ],
+ [
+ 5.697894096374512,
+ 0.04901987314224243,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ 0.0804542601108551,
+ 0.9528194665908813,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 4516.2138671875,
+ -3693.35986328125
+ ],
+ [
+ 3973.955078125,
+ -3267.04638671875
+ ],
+ [
+ 4118.57275390625,
+ -3372.958984375
+ ],
+ [
+ 4505.318359375,
+ -3688.783203125
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.4025,
+ "technical": 0.3493,
+ "account": 0.2481
+ },
+ "confidence": 0.0174,
+ "answer_confidence": 0.4025,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.9626,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.4513,
+ "1": 0.1765,
+ "2": 0.3304,
+ "3": 0.0418
+ },
+ "confidence": 0.1605,
+ "answer_confidence": 0.4513,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.55
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.0548,
+ "confidence": 0.9452,
+ "answer_confidence": 0.9452,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.6082,
+ "confidence": 0.6082,
+ "answer_confidence": 0.6082,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 183,
+ "output_tokens": 0,
+ "state_tokens": 8,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "default": 0.55
+ }
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_scalar.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_scalar.json
new file mode 100644
index 00000000..8b4fd11f
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_scalar.json
@@ -0,0 +1,1520 @@
+{
+ "name": "min_confidence_scalar",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 24,
+ 34
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 17,
+ 23,
+ 28
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 17,
+ 26
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 24,
+ 34,
+ 0
+ ],
+ [
+ 11,
+ 17,
+ 23,
+ 28
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 26,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ 4.488276958465576,
+ -2.729666233062744,
+ -2.4480319023132324,
+ -10000.0
+ ],
+ [
+ -0.8349542617797852,
+ 0.8832842707633972,
+ 3.6844465732574463,
+ 1.2054039239883423
+ ],
+ [
+ -4.515446186065674,
+ 2.250016689300537,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ 0.7536231279373169,
+ 0.41879838705062866,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 4074.86572265625,
+ -3334.4990234375
+ ],
+ [
+ 4040.2958984375,
+ -3316.3017578125
+ ],
+ [
+ 2926.997314453125,
+ -2398.78125
+ ],
+ [
+ 4615.162109375,
+ -3778.93896484375
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9653,
+ "technical": 0.016,
+ "account": 0.0188
+ },
+ "confidence": 0.8409,
+ "answer_confidence": 0.9653,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.9821,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.0212,
+ "1": 0.0839,
+ "2": 0.7864,
+ "3": 0.1085
+ },
+ "confidence": 0.4809,
+ "answer_confidence": 0.7864,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.968,
+ "confidence": 0.968,
+ "answer_confidence": 0.968,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.4579,
+ "confidence": 0.5421,
+ "answer_confidence": 0.5421,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 379,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": 0.6
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_zero.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_zero.json
new file mode 100644
index 00000000..5eb00a71
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_min_confidence_zero.json
@@ -0,0 +1,1519 @@
+{
+ "name": "min_confidence_zero",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 24,
+ 34
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 17,
+ 23,
+ 28
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 17,
+ 26
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 24,
+ 34,
+ 0
+ ],
+ [
+ 11,
+ 17,
+ 23,
+ 28
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 26,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ 4.488276958465576,
+ -2.729666233062744,
+ -2.4480319023132324,
+ -10000.0
+ ],
+ [
+ -0.8349542617797852,
+ 0.8832842707633972,
+ 3.6844465732574463,
+ 1.2054039239883423
+ ],
+ [
+ -4.515446186065674,
+ 2.250016689300537,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ 0.7536231279373169,
+ 0.41879838705062866,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 4074.86572265625,
+ -3334.4990234375
+ ],
+ [
+ 4040.2958984375,
+ -3316.3017578125
+ ],
+ [
+ 2926.997314453125,
+ -2398.78125
+ ],
+ [
+ 4615.162109375,
+ -3778.93896484375
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9653,
+ "technical": 0.016,
+ "account": 0.0188
+ },
+ "confidence": 0.8409,
+ "answer_confidence": 0.9653,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.9821,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.0212,
+ "1": 0.0839,
+ "2": 0.7864,
+ "3": 0.1085
+ },
+ "confidence": 0.4809,
+ "answer_confidence": 0.7864,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.968,
+ "confidence": 0.968,
+ "answer_confidence": 0.968,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.4579,
+ "confidence": 0.5421,
+ "answer_confidence": 0.5421,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 379,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": 0.0
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/index.json b/laya-dotnet/tests/Laya.Tests/golden/english/index.json
index eb70e281..8b7bcfdf 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/index.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/index.json
@@ -337,6 +337,42 @@
"expects_error": false,
"kind": "batch"
},
+ {
+ "name": "batch_min_confidence_scalar",
+ "file": "case_batch_min_confidence_scalar.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_zero",
+ "file": "case_batch_min_confidence_zero.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_map",
+ "file": "case_batch_min_confidence_map.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_map_default",
+ "file": "case_batch_min_confidence_map_default.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_empty_questions",
+ "file": "case_batch_min_confidence_empty_questions.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_empty_states",
+ "file": "case_batch_min_confidence_empty_states.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
{
"name": "batch_none_state",
"file": "case_batch_none_state.json",
@@ -348,5 +384,58 @@
"file": "case_batch_options_exceed_head.json",
"expects_error": true,
"kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_invalid",
+ "file": "case_batch_min_confidence_invalid.json",
+ "expects_error": true,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_before_none_state",
+ "file": "case_batch_min_confidence_before_none_state.json",
+ "expects_error": true,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_bad_key",
+ "file": "case_batch_min_confidence_bad_key.json",
+ "expects_error": true,
+ "kind": "batch"
+ },
+ {
+ "name": "min_confidence_scalar",
+ "file": "case_min_confidence_scalar.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_zero",
+ "file": "case_min_confidence_zero.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_map",
+ "file": "case_min_confidence_map.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_map_default",
+ "file": "case_min_confidence_map_default.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_invalid",
+ "file": "case_min_confidence_invalid.json",
+ "expects_error": true
+ },
+ {
+ "name": "min_confidence_empty_map",
+ "file": "case_min_confidence_empty_map.json",
+ "expects_error": true
+ },
+ {
+ "name": "min_confidence_bad_key",
+ "file": "case_min_confidence_bad_key.json",
+ "expects_error": true
}
]
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_bad_key.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_bad_key.json
new file mode 100644
index 00000000..424420ec
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_bad_key.json
@@ -0,0 +1,38 @@
+{
+ "name": "batch_min_confidence_bad_key",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "choice:2": 0.5,
+ "choice:3": 0.5
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'"
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_before_none_state.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_before_none_state.json
new file mode 100644
index 00000000..75fd7564
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_before_none_state.json
@@ -0,0 +1,36 @@
+{
+ "name": "batch_min_confidence_before_none_state",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ null
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": -0.25
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -0.25"
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_empty_questions.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_empty_questions.json
new file mode 100644
index 00000000..255f7bc4
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_empty_questions.json
@@ -0,0 +1,44 @@
+{
+ "name": "batch_min_confidence_empty_questions",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, the refund arrived and everything works now.",
+ {
+ "ticket": 4821,
+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "
+ }
+ ],
+ "questions": {},
+ "options": {
+ "min_confidence": 0.5
+ },
+ "state_widths": [],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {},
+ "usage": {
+ "input_tokens": 0,
+ "output_tokens": 0
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {},
+ "usage": {
+ "input_tokens": 0,
+ "output_tokens": 0
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {},
+ "usage": {
+ "input_tokens": 0,
+ "output_tokens": 0
+ }
+ }
+ ],
+ "runs": []
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_empty_states.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_empty_states.json
new file mode 100644
index 00000000..dfb141d9
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_empty_states.json
@@ -0,0 +1,34 @@
+{
+ "name": "batch_min_confidence_empty_states",
+ "states": [],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 0.5
+ },
+ "state_widths": [],
+ "results": [],
+ "runs": []
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_invalid.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_invalid.json
new file mode 100644
index 00000000..72496b52
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_invalid.json
@@ -0,0 +1,35 @@
+{
+ "name": "batch_min_confidence_invalid",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 1.5
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_map.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_map.json
new file mode 100644
index 00000000..30dee3e1
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_map.json
@@ -0,0 +1,497 @@
+{
+ "name": "batch_min_confidence_map",
+ "states": [
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "batch_size": 3,
+ "sort_by_length": true,
+ "min_confidence": {
+ "choice:3-5": 0.8,
+ "score:3-5": 0.5,
+ "noul:2": 0.7
+ }
+ },
+ "state_widths": [
+ 180,
+ 100,
+ 135,
+ 90,
+ 160,
+ 115,
+ 145
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9823,
+ "technical": 0.0007,
+ "other": 0.017
+ },
+ "confidence": 0.916,
+ "answer_confidence": 0.9823,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3568,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0709,
+ "1": 0.5013,
+ "2": 0.4278
+ },
+ "confidence": 0.1834,
+ "answer_confidence": 0.5013,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0867,
+ "confidence": 0.9133,
+ "answer_confidence": 0.9133,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 534,
+ "output_tokens": 0,
+ "state_tokens": 146,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9664,
+ "technical": 0.0022,
+ "other": 0.0314
+ },
+ "confidence": 0.8589,
+ "answer_confidence": 0.9664,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.2262,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0367,
+ "1": 0.7004,
+ "2": 0.2629
+ },
+ "confidence": 0.3429,
+ "answer_confidence": 0.7004,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0634,
+ "confidence": 0.9366,
+ "answer_confidence": 0.9366,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 294,
+ "output_tokens": 0,
+ "state_tokens": 66,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9708,
+ "technical": 0.0008,
+ "other": 0.0284
+ },
+ "confidence": 0.8764,
+ "answer_confidence": 0.9708,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.6446,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0315,
+ "1": 0.2924,
+ "2": 0.6761
+ },
+ "confidence": 0.3327,
+ "answer_confidence": 0.6761,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0554,
+ "confidence": 0.9446,
+ "answer_confidence": 0.9446,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 399,
+ "output_tokens": 0,
+ "state_tokens": 101,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.836,
+ "technical": 0.0114,
+ "other": 0.1526
+ },
+ "confidence": 0.5562,
+ "answer_confidence": 0.836,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3713,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0231,
+ "1": 0.5824,
+ "2": 0.3945
+ },
+ "confidence": 0.3001,
+ "answer_confidence": 0.5824,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0414,
+ "confidence": 0.9586,
+ "answer_confidence": 0.9586,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 264,
+ "output_tokens": 0,
+ "state_tokens": 56,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.972,
+ "technical": 0.0011,
+ "other": 0.0269
+ },
+ "confidence": 0.8798,
+ "answer_confidence": 0.972,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5259,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0331,
+ "1": 0.4079,
+ "2": 0.559
+ },
+ "confidence": 0.2684,
+ "answer_confidence": 0.559,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0972,
+ "confidence": 0.9028,
+ "answer_confidence": 0.9028,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 474,
+ "output_tokens": 0,
+ "state_tokens": 126,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9345,
+ "technical": 0.0011,
+ "other": 0.0644
+ },
+ "confidence": 0.7748,
+ "answer_confidence": 0.9345,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3357,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0468,
+ "1": 0.5707,
+ "2": 0.3825
+ },
+ "confidence": 0.2436,
+ "answer_confidence": 0.5707,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.1013,
+ "confidence": 0.8987,
+ "answer_confidence": 0.8987,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 339,
+ "output_tokens": 0,
+ "state_tokens": 81,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9765,
+ "technical": 0.0006,
+ "other": 0.0229
+ },
+ "confidence": 0.896,
+ "answer_confidence": 0.9765,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.6128,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0311,
+ "1": 0.325,
+ "2": 0.6439
+ },
+ "confidence": 0.3113,
+ "answer_confidence": 0.6439,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0431,
+ "confidence": 0.9569,
+ "answer_confidence": 0.9569,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 429,
+ "output_tokens": 0,
+ "state_tokens": 111,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 9,
+ 115
+ ],
+ [
+ 9,
+ 160
+ ],
+ [
+ 3,
+ 180
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_map_default.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_map_default.json
new file mode 100644
index 00000000..3f9d2494
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_map_default.json
@@ -0,0 +1,238 @@
+{
+ "name": "batch_min_confidence_map_default",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, the refund arrived and everything works now.",
+ {
+ "ticket": 4821,
+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "
+ }
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "default": 0.55
+ }
+ },
+ "state_widths": [
+ 90,
+ 44,
+ 147
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.836,
+ "technical": 0.0114,
+ "other": 0.1526
+ },
+ "confidence": 0.5562,
+ "answer_confidence": 0.836,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3713,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0231,
+ "1": 0.5824,
+ "2": 0.3945
+ },
+ "confidence": 0.3001,
+ "answer_confidence": 0.5824,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0414,
+ "confidence": 0.9586,
+ "answer_confidence": 0.9586,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 264,
+ "output_tokens": 0,
+ "state_tokens": 56,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9999,
+ "technical": 0.0,
+ "other": 0.0
+ },
+ "confidence": 0.9993,
+ "answer_confidence": 0.9999,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.6231,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0528,
+ "1": 0.2714,
+ "2": 0.6759
+ },
+ "confidence": 0.2956,
+ "answer_confidence": 0.6759,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0067,
+ "confidence": 0.9933,
+ "answer_confidence": 0.9933,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 126,
+ "output_tokens": 0,
+ "state_tokens": 10,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "other",
+ "probabilities": {
+ "refund": 0.2098,
+ "technical": 0.0637,
+ "other": 0.7266
+ },
+ "confidence": 0.331,
+ "answer_confidence": 0.7266,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3977,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0308,
+ "1": 0.5407,
+ "2": 0.4285
+ },
+ "confidence": 0.2692,
+ "answer_confidence": 0.5407,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.55
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.1873,
+ "confidence": 0.8127,
+ "answer_confidence": 0.8127,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 435,
+ "output_tokens": 0,
+ "state_tokens": 113,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 9,
+ 147
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_scalar.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_scalar.json
new file mode 100644
index 00000000..3362d652
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_scalar.json
@@ -0,0 +1,572 @@
+{
+ "name": "batch_min_confidence_scalar",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ {
+ "subject": "Double charged on invoice 88213",
+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ },
+ [
+ {
+ "role": "user",
+ "content": "Hi, I need help with my invoice."
+ },
+ {
+ "role": "assistant",
+ "content": "Of course, what is wrong?"
+ },
+ {
+ "role": "user",
+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ }
+ ],
+ "",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, that is all sorted now.",
+ {
+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",
+ "message": "Please cancel my account today."
+ },
+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 0.6
+ },
+ "state_widths": [
+ 90,
+ 113,
+ 143,
+ 34,
+ 150,
+ 42,
+ 90,
+ 48
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.836,
+ "technical": 0.0114,
+ "other": 0.1526
+ },
+ "confidence": 0.5562,
+ "answer_confidence": 0.836,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3713,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0231,
+ "1": 0.5824,
+ "2": 0.3945
+ },
+ "confidence": 0.3001,
+ "answer_confidence": 0.5824,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0414,
+ "confidence": 0.9586,
+ "answer_confidence": 0.9586,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 264,
+ "output_tokens": 0,
+ "state_tokens": 56,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.6527,
+ "technical": 0.0268,
+ "other": 0.3205
+ },
+ "confidence": 0.3264,
+ "answer_confidence": 0.6527,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.4526,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0142,
+ "1": 0.519,
+ "2": 0.4668
+ },
+ "confidence": 0.3115,
+ "answer_confidence": 0.519,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.3678,
+ "confidence": 0.6322,
+ "answer_confidence": 0.6322,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 333,
+ "output_tokens": 0,
+ "state_tokens": 79,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "other",
+ "probabilities": {
+ "refund": 0.2035,
+ "technical": 0.1274,
+ "other": 0.6691
+ },
+ "confidence": 0.2215,
+ "answer_confidence": 0.6691,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3642,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0252,
+ "1": 0.5854,
+ "2": 0.3894
+ },
+ "confidence": 0.2959,
+ "answer_confidence": 0.5854,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0571,
+ "confidence": 0.9429,
+ "answer_confidence": 0.9429,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 423,
+ "output_tokens": 0,
+ "state_tokens": 109,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.806,
+ "technical": 0.1113,
+ "other": 0.0827
+ },
+ "confidence": 0.4317,
+ "answer_confidence": 0.806,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.1334,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.1501,
+ "1": 0.5664,
+ "2": 0.2835
+ },
+ "confidence": 0.1225,
+ "answer_confidence": 0.5664,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0063,
+ "confidence": 0.9937,
+ "answer_confidence": 0.9937,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 96,
+ "output_tokens": 0,
+ "state_tokens": 0,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9867,
+ "technical": 0.0005,
+ "other": 0.0129
+ },
+ "confidence": 0.9336,
+ "answer_confidence": 0.9867,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5707,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0398,
+ "1": 0.3498,
+ "2": 0.6105
+ },
+ "confidence": 0.2746,
+ "answer_confidence": 0.6105,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0882,
+ "confidence": 0.9118,
+ "answer_confidence": 0.9118,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 444,
+ "output_tokens": 0,
+ "state_tokens": 116,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "other",
+ "probabilities": {
+ "refund": 0.0595,
+ "technical": 0.0895,
+ "other": 0.851
+ },
+ "confidence": 0.5256,
+ "answer_confidence": 0.851,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3418,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.1574,
+ "1": 0.3433,
+ "2": 0.4993
+ },
+ "confidence": 0.0853,
+ "answer_confidence": 0.4993,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0056,
+ "confidence": 0.9944,
+ "answer_confidence": 0.9944,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 120,
+ "output_tokens": 0,
+ "state_tokens": 8,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.468,
+ "technical": 0.301,
+ "other": 0.231
+ },
+ "confidence": 0.0395,
+ "answer_confidence": 0.468,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8737,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0305,
+ "1": 0.0652,
+ "2": 0.9042
+ },
+ "confidence": 0.6581,
+ "answer_confidence": 0.9042,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.4844,
+ "confidence": 0.5156,
+ "answer_confidence": 0.5156,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 264,
+ "output_tokens": 0,
+ "state_tokens": 56,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9822,
+ "technical": 0.0057,
+ "other": 0.012
+ },
+ "confidence": 0.9086,
+ "answer_confidence": 0.9822,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7021,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0303,
+ "1": 0.2373,
+ "2": 0.7324
+ },
+ "confidence": 0.3852,
+ "answer_confidence": 0.7324,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.1059,
+ "confidence": 0.8941,
+ "answer_confidence": 0.8941,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 138,
+ "output_tokens": 0,
+ "state_tokens": 14,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 24,
+ 150
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_zero.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_zero.json
new file mode 100644
index 00000000..bbea5c84
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_min_confidence_zero.json
@@ -0,0 +1,565 @@
+{
+ "name": "batch_min_confidence_zero",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ {
+ "subject": "Double charged on invoice 88213",
+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ },
+ [
+ {
+ "role": "user",
+ "content": "Hi, I need help with my invoice."
+ },
+ {
+ "role": "assistant",
+ "content": "Of course, what is wrong?"
+ },
+ {
+ "role": "user",
+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ }
+ ],
+ "",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, that is all sorted now.",
+ {
+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",
+ "message": "Please cancel my account today."
+ },
+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 0.0
+ },
+ "state_widths": [
+ 90,
+ 113,
+ 143,
+ 34,
+ 150,
+ 42,
+ 90,
+ 48
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.836,
+ "technical": 0.0114,
+ "other": 0.1526
+ },
+ "confidence": 0.5562,
+ "answer_confidence": 0.836,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3713,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0231,
+ "1": 0.5824,
+ "2": 0.3945
+ },
+ "confidence": 0.3001,
+ "answer_confidence": 0.5824,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0414,
+ "confidence": 0.9586,
+ "answer_confidence": 0.9586,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 264,
+ "output_tokens": 0,
+ "state_tokens": 56,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.6527,
+ "technical": 0.0268,
+ "other": 0.3205
+ },
+ "confidence": 0.3264,
+ "answer_confidence": 0.6527,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.4526,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0142,
+ "1": 0.519,
+ "2": 0.4668
+ },
+ "confidence": 0.3115,
+ "answer_confidence": 0.519,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.3678,
+ "confidence": 0.6322,
+ "answer_confidence": 0.6322,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 333,
+ "output_tokens": 0,
+ "state_tokens": 79,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "other",
+ "probabilities": {
+ "refund": 0.2035,
+ "technical": 0.1274,
+ "other": 0.6691
+ },
+ "confidence": 0.2215,
+ "answer_confidence": 0.6691,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3642,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0252,
+ "1": 0.5854,
+ "2": 0.3894
+ },
+ "confidence": 0.2959,
+ "answer_confidence": 0.5854,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0571,
+ "confidence": 0.9429,
+ "answer_confidence": 0.9429,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 423,
+ "output_tokens": 0,
+ "state_tokens": 109,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.806,
+ "technical": 0.1113,
+ "other": 0.0827
+ },
+ "confidence": 0.4317,
+ "answer_confidence": 0.806,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.1334,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.1501,
+ "1": 0.5664,
+ "2": 0.2835
+ },
+ "confidence": 0.1225,
+ "answer_confidence": 0.5664,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0063,
+ "confidence": 0.9937,
+ "answer_confidence": 0.9937,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 96,
+ "output_tokens": 0,
+ "state_tokens": 0,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9867,
+ "technical": 0.0005,
+ "other": 0.0129
+ },
+ "confidence": 0.9336,
+ "answer_confidence": 0.9867,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5707,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0398,
+ "1": 0.3498,
+ "2": 0.6105
+ },
+ "confidence": 0.2746,
+ "answer_confidence": 0.6105,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0882,
+ "confidence": 0.9118,
+ "answer_confidence": 0.9118,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 444,
+ "output_tokens": 0,
+ "state_tokens": 116,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "other",
+ "probabilities": {
+ "refund": 0.0595,
+ "technical": 0.0895,
+ "other": 0.851
+ },
+ "confidence": 0.5256,
+ "answer_confidence": 0.851,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3418,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.1574,
+ "1": 0.3433,
+ "2": 0.4993
+ },
+ "confidence": 0.0853,
+ "answer_confidence": 0.4993,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0056,
+ "confidence": 0.9944,
+ "answer_confidence": 0.9944,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 120,
+ "output_tokens": 0,
+ "state_tokens": 8,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.468,
+ "technical": 0.301,
+ "other": 0.231
+ },
+ "confidence": 0.0395,
+ "answer_confidence": 0.468,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8737,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0305,
+ "1": 0.0652,
+ "2": 0.9042
+ },
+ "confidence": 0.6581,
+ "answer_confidence": 0.9042,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.4844,
+ "confidence": 0.5156,
+ "answer_confidence": 0.5156,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 264,
+ "output_tokens": 0,
+ "state_tokens": 56,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.9822,
+ "technical": 0.0057,
+ "other": 0.012
+ },
+ "confidence": 0.9086,
+ "answer_confidence": 0.9822,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7021,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0303,
+ "1": 0.2373,
+ "2": 0.7324
+ },
+ "confidence": 0.3852,
+ "answer_confidence": 0.7324,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.1059,
+ "confidence": 0.8941,
+ "answer_confidence": 0.8941,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 138,
+ "output_tokens": 0,
+ "state_tokens": 14,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 24,
+ 150
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_bad_key.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_bad_key.json
new file mode 100644
index 00000000..7eab72d1
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_bad_key.json
@@ -0,0 +1,45 @@
+{
+ "name": "min_confidence_bad_key",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'",
+ "options": {
+ "min_confidence": {
+ "default": 0.5,
+ "choice:3": 0.5
+ }
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_empty_map.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_empty_map.json
new file mode 100644
index 00000000..28fa69e2
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_empty_map.json
@@ -0,0 +1,42 @@
+{
+ "name": "min_confidence_empty_map",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "a min_confidence map must be a non-empty dict of bucket -> float, got {}",
+ "options": {
+ "min_confidence": {}
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_invalid.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_invalid.json
new file mode 100644
index 00000000..5ecd652a
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_invalid.json
@@ -0,0 +1,42 @@
+{
+ "name": "min_confidence_invalid",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5",
+ "options": {
+ "min_confidence": 1.5
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_map.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_map.json
new file mode 100644
index 00000000..61b29433
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_map.json
@@ -0,0 +1,1488 @@
+{
+ "name": "min_confidence_map",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 2,
+ 6241,
+ 2872,
+ 235292,
+ 12236,
+ 9888,
+ 1412,
+ 6589,
+ 736,
+ 235336,
+ 1,
+ 4,
+ 54972,
+ 235292,
+ 88220,
+ 235269,
+ 15598,
+ 235269,
+ 4199,
+ 12036,
+ 235269,
+ 85869,
+ 4,
+ 9838,
+ 235292,
+ 30608,
+ 235269,
+ 142788,
+ 235269,
+ 16827,
+ 10266,
+ 4,
+ 3185,
+ 235292,
+ 12138,
+ 235256,
+ 235269,
+ 18723,
+ 235269,
+ 1780,
+ 4559,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 11,
+ 22,
+ 31
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 2,
+ 7716,
+ 2872,
+ 235292,
+ 2250,
+ 34722,
+ 603,
+ 736,
+ 3853,
+ 235336,
+ 1,
+ 4,
+ 2403,
+ 235248,
+ 235276,
+ 235292,
+ 780,
+ 34722,
+ 4,
+ 2403,
+ 235248,
+ 235274,
+ 235292,
+ 736,
+ 2788,
+ 4,
+ 2403,
+ 235248,
+ 235284,
+ 235292,
+ 3646,
+ 4,
+ 2403,
+ 235248,
+ 235304,
+ 235292,
+ 7605,
+ 4553,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 11,
+ 18,
+ 25,
+ 31
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 714,
+ 6650,
+ 80670,
+ 577,
+ 12459,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 793,
+ 235269,
+ 573,
+ 6218,
+ 1721,
+ 780,
+ 3385,
+ 4,
+ 1382,
+ 235292,
+ 7778,
+ 235269,
+ 573,
+ 6218,
+ 12723,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 1417,
+ 9286,
+ 476,
+ 3515,
+ 10588,
+ 4644,
+ 1178,
+ 671,
+ 34177,
+ 7260,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 476,
+ 11387,
+ 7260,
+ 116075,
+ 665,
+ 4,
+ 1382,
+ 235292,
+ 476,
+ 1552,
+ 2004,
+ 75103,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 17,
+ 25
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 2,
+ 6241,
+ 2872,
+ 235292,
+ 12236,
+ 9888,
+ 1412,
+ 6589,
+ 736,
+ 235336,
+ 1,
+ 4,
+ 54972,
+ 235292,
+ 88220,
+ 235269,
+ 15598,
+ 235269,
+ 4199,
+ 12036,
+ 235269,
+ 85869,
+ 4,
+ 9838,
+ 235292,
+ 30608,
+ 235269,
+ 142788,
+ 235269,
+ 16827,
+ 10266,
+ 4,
+ 3185,
+ 235292,
+ 12138,
+ 235256,
+ 235269,
+ 18723,
+ 235269,
+ 1780,
+ 4559,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ [
+ 2,
+ 7716,
+ 2872,
+ 235292,
+ 2250,
+ 34722,
+ 603,
+ 736,
+ 3853,
+ 235336,
+ 1,
+ 4,
+ 2403,
+ 235248,
+ 235276,
+ 235292,
+ 780,
+ 34722,
+ 4,
+ 2403,
+ 235248,
+ 235274,
+ 235292,
+ 736,
+ 2788,
+ 4,
+ 2403,
+ 235248,
+ 235284,
+ 235292,
+ 3646,
+ 4,
+ 2403,
+ 235248,
+ 235304,
+ 235292,
+ 7605,
+ 4553,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 714,
+ 6650,
+ 80670,
+ 577,
+ 12459,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 793,
+ 235269,
+ 573,
+ 6218,
+ 1721,
+ 780,
+ 3385,
+ 4,
+ 1382,
+ 235292,
+ 7778,
+ 235269,
+ 573,
+ 6218,
+ 12723,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 1417,
+ 9286,
+ 476,
+ 3515,
+ 10588,
+ 4644,
+ 1178,
+ 671,
+ 34177,
+ 7260,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 476,
+ 11387,
+ 7260,
+ 116075,
+ 665,
+ 4,
+ 1382,
+ 235292,
+ 476,
+ 1552,
+ 2004,
+ 75103,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 22,
+ 31,
+ 0
+ ],
+ [
+ 11,
+ 18,
+ 25,
+ 31
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 25,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ 7.727238655090332,
+ -4.555615425109863,
+ -2.0609042644500732,
+ -10000.0
+ ],
+ [
+ -3.1127467155456543,
+ 0.14541220664978027,
+ -0.22580814361572266,
+ -0.7165543437004089
+ ],
+ [
+ 3.0695688724517822,
+ -0.07253175973892212,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ 1.247997522354126,
+ -0.26682233810424805,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 1252.7354736328125,
+ -1494.896240234375
+ ],
+ [
+ 1394.948486328125,
+ -1645.8040771484375
+ ],
+ [
+ 1234.01416015625,
+ -1468.8492431640625
+ ],
+ [
+ 1266.344970703125,
+ -1504.056396484375
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9999,
+ "technical": 0.0,
+ "account": 0.0001
+ },
+ "confidence": 0.9994,
+ "answer_confidence": 0.9999,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.9
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.6956,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.0179,
+ "1": 0.465,
+ "2": 0.3208,
+ "3": 0.1964
+ },
+ "confidence": 0.1976,
+ "answer_confidence": 0.465,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.0414,
+ "confidence": 0.9586,
+ "answer_confidence": 0.9586,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.1802,
+ "confidence": 0.8198,
+ "answer_confidence": 0.8198,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 375,
+ "output_tokens": 0,
+ "state_tokens": 56,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "choice:3-5": 0.9,
+ "score:3-5": 0.5,
+ "noul:2": 0.7
+ }
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_map_default.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_map_default.json
new file mode 100644
index 00000000..b7dba7b1
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_map_default.json
@@ -0,0 +1,911 @@
+{
+ "name": "min_confidence_map_default",
+ "state": "Thanks, that is all sorted now.",
+ "serialized_state": "Thanks, that is all sorted now.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 2,
+ 6241,
+ 2872,
+ 235292,
+ 12236,
+ 9888,
+ 1412,
+ 6589,
+ 736,
+ 235336,
+ 1,
+ 4,
+ 54972,
+ 235292,
+ 88220,
+ 235269,
+ 15598,
+ 235269,
+ 4199,
+ 12036,
+ 235269,
+ 85869,
+ 4,
+ 9838,
+ 235292,
+ 30608,
+ 235269,
+ 142788,
+ 235269,
+ 16827,
+ 10266,
+ 4,
+ 3185,
+ 235292,
+ 12138,
+ 235256,
+ 235269,
+ 18723,
+ 235269,
+ 1780,
+ 4559,
+ 1,
+ 7807,
+ 235269,
+ 674,
+ 603,
+ 832,
+ 24087,
+ 1490,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 11,
+ 22,
+ 31
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 2,
+ 7716,
+ 2872,
+ 235292,
+ 2250,
+ 34722,
+ 603,
+ 736,
+ 3853,
+ 235336,
+ 1,
+ 4,
+ 2403,
+ 235248,
+ 235276,
+ 235292,
+ 780,
+ 34722,
+ 4,
+ 2403,
+ 235248,
+ 235274,
+ 235292,
+ 736,
+ 2788,
+ 4,
+ 2403,
+ 235248,
+ 235284,
+ 235292,
+ 3646,
+ 4,
+ 2403,
+ 235248,
+ 235304,
+ 235292,
+ 7605,
+ 4553,
+ 1,
+ 7807,
+ 235269,
+ 674,
+ 603,
+ 832,
+ 24087,
+ 1490,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 11,
+ 18,
+ 25,
+ 31
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 714,
+ 6650,
+ 80670,
+ 577,
+ 12459,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 793,
+ 235269,
+ 573,
+ 6218,
+ 1721,
+ 780,
+ 3385,
+ 4,
+ 1382,
+ 235292,
+ 7778,
+ 235269,
+ 573,
+ 6218,
+ 12723,
+ 1,
+ 7807,
+ 235269,
+ 674,
+ 603,
+ 832,
+ 24087,
+ 1490,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 1417,
+ 9286,
+ 476,
+ 3515,
+ 10588,
+ 4644,
+ 1178,
+ 671,
+ 34177,
+ 7260,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 476,
+ 11387,
+ 7260,
+ 116075,
+ 665,
+ 4,
+ 1382,
+ 235292,
+ 476,
+ 1552,
+ 2004,
+ 75103,
+ 1,
+ 7807,
+ 235269,
+ 674,
+ 603,
+ 832,
+ 24087,
+ 1490,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 17,
+ 25
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 2,
+ 6241,
+ 2872,
+ 235292,
+ 12236,
+ 9888,
+ 1412,
+ 6589,
+ 736,
+ 235336,
+ 1,
+ 4,
+ 54972,
+ 235292,
+ 88220,
+ 235269,
+ 15598,
+ 235269,
+ 4199,
+ 12036,
+ 235269,
+ 85869,
+ 4,
+ 9838,
+ 235292,
+ 30608,
+ 235269,
+ 142788,
+ 235269,
+ 16827,
+ 10266,
+ 4,
+ 3185,
+ 235292,
+ 12138,
+ 235256,
+ 235269,
+ 18723,
+ 235269,
+ 1780,
+ 4559,
+ 1,
+ 7807,
+ 235269,
+ 674,
+ 603,
+ 832,
+ 24087,
+ 1490,
+ 235265,
+ 1
+ ],
+ [
+ 2,
+ 7716,
+ 2872,
+ 235292,
+ 2250,
+ 34722,
+ 603,
+ 736,
+ 3853,
+ 235336,
+ 1,
+ 4,
+ 2403,
+ 235248,
+ 235276,
+ 235292,
+ 780,
+ 34722,
+ 4,
+ 2403,
+ 235248,
+ 235274,
+ 235292,
+ 736,
+ 2788,
+ 4,
+ 2403,
+ 235248,
+ 235284,
+ 235292,
+ 3646,
+ 4,
+ 2403,
+ 235248,
+ 235304,
+ 235292,
+ 7605,
+ 4553,
+ 1,
+ 7807,
+ 235269,
+ 674,
+ 603,
+ 832,
+ 24087,
+ 1490,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 714,
+ 6650,
+ 80670,
+ 577,
+ 12459,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 793,
+ 235269,
+ 573,
+ 6218,
+ 1721,
+ 780,
+ 3385,
+ 4,
+ 1382,
+ 235292,
+ 7778,
+ 235269,
+ 573,
+ 6218,
+ 12723,
+ 1,
+ 7807,
+ 235269,
+ 674,
+ 603,
+ 832,
+ 24087,
+ 1490,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 1417,
+ 9286,
+ 476,
+ 3515,
+ 10588,
+ 4644,
+ 1178,
+ 671,
+ 34177,
+ 7260,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 476,
+ 11387,
+ 7260,
+ 116075,
+ 665,
+ 4,
+ 1382,
+ 235292,
+ 476,
+ 1552,
+ 2004,
+ 75103,
+ 1,
+ 7807,
+ 235269,
+ 674,
+ 603,
+ 832,
+ 24087,
+ 1490,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 22,
+ 31,
+ 0
+ ],
+ [
+ 11,
+ 18,
+ 25,
+ 31
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 25,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ -1.6179548501968384,
+ -1.8165647983551025,
+ -1.123018503189087,
+ -10000.0
+ ],
+ [
+ -0.4120713472366333,
+ 0.4631410241127014,
+ -0.00779646635055542,
+ 0.054005563259124756
+ ],
+ [
+ 4.764176845550537,
+ -0.41596102714538574,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ 0.335951566696167,
+ -0.49896663427352905,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 1420.1611328125,
+ -1689.47265625
+ ],
+ [
+ 1465.701416015625,
+ -1730.4908447265625
+ ],
+ [
+ 1524.2642822265625,
+ -1806.6566162109375
+ ],
+ [
+ 1414.691650390625,
+ -1679.205322265625
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "account",
+ "probabilities": {
+ "billing": 0.289,
+ "technical": 0.2369,
+ "account": 0.4741
+ },
+ "confidence": 0.0408,
+ "answer_confidence": 0.4741,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5678,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.1541,
+ "1": 0.3696,
+ "2": 0.2308,
+ "3": 0.2455
+ },
+ "confidence": 0.034,
+ "answer_confidence": 0.3696,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.55
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.0056,
+ "confidence": 0.9944,
+ "answer_confidence": 0.9944,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.3026,
+ "confidence": 0.6974,
+ "answer_confidence": 0.6974,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 183,
+ "output_tokens": 0,
+ "state_tokens": 8,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "default": 0.55
+ }
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_scalar.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_scalar.json
new file mode 100644
index 00000000..e73f1194
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_scalar.json
@@ -0,0 +1,1484 @@
+{
+ "name": "min_confidence_scalar",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 2,
+ 6241,
+ 2872,
+ 235292,
+ 12236,
+ 9888,
+ 1412,
+ 6589,
+ 736,
+ 235336,
+ 1,
+ 4,
+ 54972,
+ 235292,
+ 88220,
+ 235269,
+ 15598,
+ 235269,
+ 4199,
+ 12036,
+ 235269,
+ 85869,
+ 4,
+ 9838,
+ 235292,
+ 30608,
+ 235269,
+ 142788,
+ 235269,
+ 16827,
+ 10266,
+ 4,
+ 3185,
+ 235292,
+ 12138,
+ 235256,
+ 235269,
+ 18723,
+ 235269,
+ 1780,
+ 4559,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 11,
+ 22,
+ 31
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 2,
+ 7716,
+ 2872,
+ 235292,
+ 2250,
+ 34722,
+ 603,
+ 736,
+ 3853,
+ 235336,
+ 1,
+ 4,
+ 2403,
+ 235248,
+ 235276,
+ 235292,
+ 780,
+ 34722,
+ 4,
+ 2403,
+ 235248,
+ 235274,
+ 235292,
+ 736,
+ 2788,
+ 4,
+ 2403,
+ 235248,
+ 235284,
+ 235292,
+ 3646,
+ 4,
+ 2403,
+ 235248,
+ 235304,
+ 235292,
+ 7605,
+ 4553,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 11,
+ 18,
+ 25,
+ 31
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 714,
+ 6650,
+ 80670,
+ 577,
+ 12459,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 793,
+ 235269,
+ 573,
+ 6218,
+ 1721,
+ 780,
+ 3385,
+ 4,
+ 1382,
+ 235292,
+ 7778,
+ 235269,
+ 573,
+ 6218,
+ 12723,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 1417,
+ 9286,
+ 476,
+ 3515,
+ 10588,
+ 4644,
+ 1178,
+ 671,
+ 34177,
+ 7260,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 476,
+ 11387,
+ 7260,
+ 116075,
+ 665,
+ 4,
+ 1382,
+ 235292,
+ 476,
+ 1552,
+ 2004,
+ 75103,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 17,
+ 25
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 2,
+ 6241,
+ 2872,
+ 235292,
+ 12236,
+ 9888,
+ 1412,
+ 6589,
+ 736,
+ 235336,
+ 1,
+ 4,
+ 54972,
+ 235292,
+ 88220,
+ 235269,
+ 15598,
+ 235269,
+ 4199,
+ 12036,
+ 235269,
+ 85869,
+ 4,
+ 9838,
+ 235292,
+ 30608,
+ 235269,
+ 142788,
+ 235269,
+ 16827,
+ 10266,
+ 4,
+ 3185,
+ 235292,
+ 12138,
+ 235256,
+ 235269,
+ 18723,
+ 235269,
+ 1780,
+ 4559,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ [
+ 2,
+ 7716,
+ 2872,
+ 235292,
+ 2250,
+ 34722,
+ 603,
+ 736,
+ 3853,
+ 235336,
+ 1,
+ 4,
+ 2403,
+ 235248,
+ 235276,
+ 235292,
+ 780,
+ 34722,
+ 4,
+ 2403,
+ 235248,
+ 235274,
+ 235292,
+ 736,
+ 2788,
+ 4,
+ 2403,
+ 235248,
+ 235284,
+ 235292,
+ 3646,
+ 4,
+ 2403,
+ 235248,
+ 235304,
+ 235292,
+ 7605,
+ 4553,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 714,
+ 6650,
+ 80670,
+ 577,
+ 12459,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 793,
+ 235269,
+ 573,
+ 6218,
+ 1721,
+ 780,
+ 3385,
+ 4,
+ 1382,
+ 235292,
+ 7778,
+ 235269,
+ 573,
+ 6218,
+ 12723,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 1417,
+ 9286,
+ 476,
+ 3515,
+ 10588,
+ 4644,
+ 1178,
+ 671,
+ 34177,
+ 7260,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 476,
+ 11387,
+ 7260,
+ 116075,
+ 665,
+ 4,
+ 1382,
+ 235292,
+ 476,
+ 1552,
+ 2004,
+ 75103,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 22,
+ 31,
+ 0
+ ],
+ [
+ 11,
+ 18,
+ 25,
+ 31
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 25,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ 7.727238655090332,
+ -4.555615425109863,
+ -2.0609042644500732,
+ -10000.0
+ ],
+ [
+ -3.1127467155456543,
+ 0.14541220664978027,
+ -0.22580814361572266,
+ -0.7165543437004089
+ ],
+ [
+ 3.0695688724517822,
+ -0.07253175973892212,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ 1.247997522354126,
+ -0.26682233810424805,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 1252.7354736328125,
+ -1494.896240234375
+ ],
+ [
+ 1394.948486328125,
+ -1645.8040771484375
+ ],
+ [
+ 1234.01416015625,
+ -1468.8492431640625
+ ],
+ [
+ 1266.344970703125,
+ -1504.056396484375
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9999,
+ "technical": 0.0,
+ "account": 0.0001
+ },
+ "confidence": 0.9994,
+ "answer_confidence": 0.9999,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.6956,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.0179,
+ "1": 0.465,
+ "2": 0.3208,
+ "3": 0.1964
+ },
+ "confidence": 0.1976,
+ "answer_confidence": 0.465,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.0414,
+ "confidence": 0.9586,
+ "answer_confidence": 0.9586,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.1802,
+ "confidence": 0.8198,
+ "answer_confidence": 0.8198,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 375,
+ "output_tokens": 0,
+ "state_tokens": 56,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": 0.6
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_zero.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_zero.json
new file mode 100644
index 00000000..a3687573
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_min_confidence_zero.json
@@ -0,0 +1,1483 @@
+{
+ "name": "min_confidence_zero",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 2,
+ 6241,
+ 2872,
+ 235292,
+ 12236,
+ 9888,
+ 1412,
+ 6589,
+ 736,
+ 235336,
+ 1,
+ 4,
+ 54972,
+ 235292,
+ 88220,
+ 235269,
+ 15598,
+ 235269,
+ 4199,
+ 12036,
+ 235269,
+ 85869,
+ 4,
+ 9838,
+ 235292,
+ 30608,
+ 235269,
+ 142788,
+ 235269,
+ 16827,
+ 10266,
+ 4,
+ 3185,
+ 235292,
+ 12138,
+ 235256,
+ 235269,
+ 18723,
+ 235269,
+ 1780,
+ 4559,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 11,
+ 22,
+ 31
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 2,
+ 7716,
+ 2872,
+ 235292,
+ 2250,
+ 34722,
+ 603,
+ 736,
+ 3853,
+ 235336,
+ 1,
+ 4,
+ 2403,
+ 235248,
+ 235276,
+ 235292,
+ 780,
+ 34722,
+ 4,
+ 2403,
+ 235248,
+ 235274,
+ 235292,
+ 736,
+ 2788,
+ 4,
+ 2403,
+ 235248,
+ 235284,
+ 235292,
+ 3646,
+ 4,
+ 2403,
+ 235248,
+ 235304,
+ 235292,
+ 7605,
+ 4553,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 11,
+ 18,
+ 25,
+ 31
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 714,
+ 6650,
+ 80670,
+ 577,
+ 12459,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 793,
+ 235269,
+ 573,
+ 6218,
+ 1721,
+ 780,
+ 3385,
+ 4,
+ 1382,
+ 235292,
+ 7778,
+ 235269,
+ 573,
+ 6218,
+ 12723,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 1417,
+ 9286,
+ 476,
+ 3515,
+ 10588,
+ 4644,
+ 1178,
+ 671,
+ 34177,
+ 7260,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 476,
+ 11387,
+ 7260,
+ 116075,
+ 665,
+ 4,
+ 1382,
+ 235292,
+ 476,
+ 1552,
+ 2004,
+ 75103,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ "markers": [
+ 17,
+ 25
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 2,
+ 6241,
+ 2872,
+ 235292,
+ 12236,
+ 9888,
+ 1412,
+ 6589,
+ 736,
+ 235336,
+ 1,
+ 4,
+ 54972,
+ 235292,
+ 88220,
+ 235269,
+ 15598,
+ 235269,
+ 4199,
+ 12036,
+ 235269,
+ 85869,
+ 4,
+ 9838,
+ 235292,
+ 30608,
+ 235269,
+ 142788,
+ 235269,
+ 16827,
+ 10266,
+ 4,
+ 3185,
+ 235292,
+ 12138,
+ 235256,
+ 235269,
+ 18723,
+ 235269,
+ 1780,
+ 4559,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1
+ ],
+ [
+ 2,
+ 7716,
+ 2872,
+ 235292,
+ 2250,
+ 34722,
+ 603,
+ 736,
+ 3853,
+ 235336,
+ 1,
+ 4,
+ 2403,
+ 235248,
+ 235276,
+ 235292,
+ 780,
+ 34722,
+ 4,
+ 2403,
+ 235248,
+ 235274,
+ 235292,
+ 736,
+ 2788,
+ 4,
+ 2403,
+ 235248,
+ 235284,
+ 235292,
+ 3646,
+ 4,
+ 2403,
+ 235248,
+ 235304,
+ 235292,
+ 7605,
+ 4553,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 714,
+ 6650,
+ 80670,
+ 577,
+ 12459,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 793,
+ 235269,
+ 573,
+ 6218,
+ 1721,
+ 780,
+ 3385,
+ 4,
+ 1382,
+ 235292,
+ 7778,
+ 235269,
+ 573,
+ 6218,
+ 12723,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 2,
+ 552,
+ 6311,
+ 2872,
+ 235292,
+ 1417,
+ 9286,
+ 476,
+ 3515,
+ 10588,
+ 4644,
+ 1178,
+ 671,
+ 34177,
+ 7260,
+ 235265,
+ 1,
+ 4,
+ 1566,
+ 235292,
+ 476,
+ 11387,
+ 7260,
+ 116075,
+ 665,
+ 4,
+ 1382,
+ 235292,
+ 476,
+ 1552,
+ 2004,
+ 75103,
+ 1,
+ 28437,
+ 235292,
+ 12510,
+ 12497,
+ 611,
+ 42529,
+ 235248,
+ 235321,
+ 235321,
+ 235284,
+ 235274,
+ 235304,
+ 109,
+ 11192,
+ 235269,
+ 590,
+ 729,
+ 100159,
+ 11594,
+ 736,
+ 2514,
+ 604,
+ 573,
+ 1122,
+ 1780,
+ 578,
+ 573,
+ 2257,
+ 6256,
+ 919,
+ 780,
+ 1125,
+ 103147,
+ 235265,
+ 1417,
+ 603,
+ 573,
+ 4906,
+ 1069,
+ 590,
+ 791,
+ 5952,
+ 575,
+ 235265,
+ 1927,
+ 665,
+ 603,
+ 780,
+ 8115,
+ 3646,
+ 590,
+ 1144,
+ 132991,
+ 573,
+ 3185,
+ 235265,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 22,
+ 31,
+ 0
+ ],
+ [
+ 11,
+ 18,
+ 25,
+ 31
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 25,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ 7.727238655090332,
+ -4.555615425109863,
+ -2.0609042644500732,
+ -10000.0
+ ],
+ [
+ -3.1127467155456543,
+ 0.14541220664978027,
+ -0.22580814361572266,
+ -0.7165543437004089
+ ],
+ [
+ 3.0695688724517822,
+ -0.07253175973892212,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ 1.247997522354126,
+ -0.26682233810424805,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 1252.7354736328125,
+ -1494.896240234375
+ ],
+ [
+ 1394.948486328125,
+ -1645.8040771484375
+ ],
+ [
+ 1234.01416015625,
+ -1468.8492431640625
+ ],
+ [
+ 1266.344970703125,
+ -1504.056396484375
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9999,
+ "technical": 0.0,
+ "account": 0.0001
+ },
+ "confidence": 0.9994,
+ "answer_confidence": 0.9999,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.6956,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.0179,
+ "1": 0.465,
+ "2": 0.3208,
+ "3": 0.1964
+ },
+ "confidence": 0.1976,
+ "answer_confidence": 0.465,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.0414,
+ "confidence": 0.9586,
+ "answer_confidence": 0.9586,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.1802,
+ "confidence": 0.8198,
+ "answer_confidence": 0.8198,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 375,
+ "output_tokens": 0,
+ "state_tokens": 56,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": 0.0
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/index.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/index.json
index eb70e281..8b7bcfdf 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/index.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/index.json
@@ -337,6 +337,42 @@
"expects_error": false,
"kind": "batch"
},
+ {
+ "name": "batch_min_confidence_scalar",
+ "file": "case_batch_min_confidence_scalar.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_zero",
+ "file": "case_batch_min_confidence_zero.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_map",
+ "file": "case_batch_min_confidence_map.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_map_default",
+ "file": "case_batch_min_confidence_map_default.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_empty_questions",
+ "file": "case_batch_min_confidence_empty_questions.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_empty_states",
+ "file": "case_batch_min_confidence_empty_states.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
{
"name": "batch_none_state",
"file": "case_batch_none_state.json",
@@ -348,5 +384,58 @@
"file": "case_batch_options_exceed_head.json",
"expects_error": true,
"kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_invalid",
+ "file": "case_batch_min_confidence_invalid.json",
+ "expects_error": true,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_before_none_state",
+ "file": "case_batch_min_confidence_before_none_state.json",
+ "expects_error": true,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_bad_key",
+ "file": "case_batch_min_confidence_bad_key.json",
+ "expects_error": true,
+ "kind": "batch"
+ },
+ {
+ "name": "min_confidence_scalar",
+ "file": "case_min_confidence_scalar.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_zero",
+ "file": "case_min_confidence_zero.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_map",
+ "file": "case_min_confidence_map.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_map_default",
+ "file": "case_min_confidence_map_default.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_invalid",
+ "file": "case_min_confidence_invalid.json",
+ "expects_error": true
+ },
+ {
+ "name": "min_confidence_empty_map",
+ "file": "case_min_confidence_empty_map.json",
+ "expects_error": true
+ },
+ {
+ "name": "min_confidence_bad_key",
+ "file": "case_min_confidence_bad_key.json",
+ "expects_error": true
}
]
diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/confidence_probe.json b/laya-dotnet/tests/Laya.Tests/golden/routing/confidence_probe.json
new file mode 100644
index 00000000..793a2b14
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/confidence_probe.json
@@ -0,0 +1,4507 @@
+{
+ "bucket_keys": [
+ "choice:11+",
+ "choice:2",
+ "choice:3-5",
+ "choice:6-10",
+ "noul:11+",
+ "noul:2",
+ "noul:3-5",
+ "noul:6-10",
+ "score:11+",
+ "score:2",
+ "score:3-5",
+ "score:6-10"
+ ],
+ "gate_states": [
+ "passed",
+ "abstained",
+ "unevaluated"
+ ],
+ "scalars": [
+ {
+ "value": 0.0,
+ "valid": true,
+ "result": 0.0
+ },
+ {
+ "value": 1.0,
+ "valid": true,
+ "result": 1.0
+ },
+ {
+ "value": 0.5,
+ "valid": true,
+ "result": 0.5
+ },
+ {
+ "value": 0.35,
+ "valid": true,
+ "result": 0.35
+ },
+ {
+ "value": 1e-09,
+ "valid": true,
+ "result": 1e-09
+ },
+ {
+ "value": 0.9999999999,
+ "valid": true,
+ "result": 0.9999999999
+ },
+ {
+ "value": 1e-05,
+ "valid": true,
+ "result": 1e-05
+ },
+ {
+ "value": -0.1,
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -0.1"
+ },
+ {
+ "value": 1.5,
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"
+ },
+ {
+ "value": -1e-05,
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -1e-05"
+ },
+ {
+ "value": 1.0000000001,
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.0000000001"
+ },
+ {
+ "value": 2.0,
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 2.0"
+ },
+ {
+ "value": 100.0,
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 100.0"
+ },
+ {
+ "value": 1e+16,
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1e+16"
+ },
+ {
+ "value": 1.2345678901234566e-07,
+ "valid": true,
+ "result": 1.2345678901234566e-07
+ },
+ {
+ "value": "nan",
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got nan"
+ },
+ {
+ "value": "inf",
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got inf"
+ },
+ {
+ "value": "-inf",
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -inf"
+ }
+ ],
+ "maps": [
+ {
+ "pairs": [],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "a min_confidence map must be a non-empty dict of bucket -> float, got {}"
+ },
+ {
+ "pairs": [
+ [
+ "default",
+ 0.4
+ ]
+ ],
+ "valid": true,
+ "result": [
+ [
+ "default",
+ 0.4
+ ]
+ ]
+ },
+ {
+ "pairs": [
+ [
+ "choice:2",
+ 0.7
+ ],
+ [
+ "default",
+ 0.4
+ ]
+ ],
+ "valid": true,
+ "result": [
+ [
+ "choice:2",
+ 0.7
+ ],
+ [
+ "default",
+ 0.4
+ ]
+ ]
+ },
+ {
+ "pairs": [
+ [
+ "choice:3-5",
+ 0.5
+ ]
+ ],
+ "valid": true,
+ "result": [
+ [
+ "choice:3-5",
+ 0.5
+ ]
+ ]
+ },
+ {
+ "pairs": [
+ [
+ "score:11+",
+ 0.0
+ ],
+ [
+ "noul:2",
+ 1.0
+ ]
+ ],
+ "valid": true,
+ "result": [
+ [
+ "score:11+",
+ 0.0
+ ],
+ [
+ "noul:2",
+ 1.0
+ ]
+ ]
+ },
+ {
+ "pairs": [
+ [
+ "choice:2",
+ 0.05
+ ],
+ [
+ "choice:3-5",
+ 0.1
+ ],
+ [
+ "choice:6-10",
+ 0.15
+ ],
+ [
+ "choice:11+",
+ 0.2
+ ],
+ [
+ "score:2",
+ 0.25
+ ],
+ [
+ "score:3-5",
+ 0.3
+ ],
+ [
+ "score:6-10",
+ 0.35
+ ],
+ [
+ "score:11+",
+ 0.4
+ ],
+ [
+ "noul:2",
+ 0.45
+ ],
+ [
+ "noul:3-5",
+ 0.5
+ ],
+ [
+ "noul:6-10",
+ 0.55
+ ],
+ [
+ "noul:11+",
+ 0.6
+ ],
+ [
+ "default",
+ 0.5
+ ]
+ ],
+ "valid": true,
+ "result": [
+ [
+ "choice:2",
+ 0.05
+ ],
+ [
+ "choice:3-5",
+ 0.1
+ ],
+ [
+ "choice:6-10",
+ 0.15
+ ],
+ [
+ "choice:11+",
+ 0.2
+ ],
+ [
+ "score:2",
+ 0.25
+ ],
+ [
+ "score:3-5",
+ 0.3
+ ],
+ [
+ "score:6-10",
+ 0.35
+ ],
+ [
+ "score:11+",
+ 0.4
+ ],
+ [
+ "noul:2",
+ 0.45
+ ],
+ [
+ "noul:3-5",
+ 0.5
+ ],
+ [
+ "noul:6-10",
+ 0.55
+ ],
+ [
+ "noul:11+",
+ 0.6
+ ],
+ [
+ "default",
+ 0.5
+ ]
+ ]
+ },
+ {
+ "pairs": [
+ [
+ "choice:2",
+ 1.5
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"
+ },
+ {
+ "pairs": [
+ [
+ "choice:2",
+ "nan"
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got nan"
+ },
+ {
+ "pairs": [
+ [
+ "choice:2",
+ -0.1
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -0.1"
+ },
+ {
+ "pairs": [
+ [
+ "choice:2",
+ "inf"
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got inf"
+ },
+ {
+ "pairs": [
+ [
+ "bogus",
+ 0.5
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'bogus'"
+ },
+ {
+ "pairs": [
+ [
+ "Choice:2",
+ 0.5
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'Choice:2'"
+ },
+ {
+ "pairs": [
+ [
+ "choice:3",
+ 0.5
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'"
+ },
+ {
+ "pairs": [
+ [
+ "choice:2",
+ 0.5
+ ],
+ [
+ "bogus",
+ 0.5
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'bogus'"
+ },
+ {
+ "pairs": [
+ [
+ "choice:2",
+ 1.5
+ ],
+ [
+ "bogus",
+ 0.1
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"
+ },
+ {
+ "pairs": [
+ [
+ "bogus",
+ 1.5
+ ],
+ [
+ "choice:2",
+ 0.1
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'bogus'"
+ },
+ {
+ "pairs": [
+ [
+ "choice:2",
+ 0.5
+ ],
+ [
+ "choice:3-5",
+ 2.0
+ ],
+ [
+ "nope",
+ 0.1
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 2.0"
+ },
+ {
+ "pairs": [
+ [
+ "it's",
+ 0.5
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got \"it's\""
+ },
+ {
+ "pairs": [
+ [
+ "say \"hi\"",
+ 0.5
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'say \"hi\"'"
+ },
+ {
+ "pairs": [
+ [
+ "it's \"x\"",
+ 0.5
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'it\\'s \"x\"'"
+ },
+ {
+ "pairs": [
+ [
+ "tab\there",
+ 0.5
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'tab\\there'"
+ },
+ {
+ "pairs": [
+ [
+ "café",
+ 0.5
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'café'"
+ },
+ {
+ "pairs": [
+ [
+ "",
+ 0.5
+ ]
+ ],
+ "valid": false,
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got ''"
+ },
+ {
+ "pairs": [
+ [
+ "default",
+ 0.5
+ ],
+ [
+ "default",
+ 0.25
+ ]
+ ],
+ "valid": true,
+ "result": [
+ [
+ "default",
+ 0.25
+ ]
+ ]
+ }
+ ],
+ "buckets": [
+ {
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 1.0
+ }
+ },
+ "bucket": "choice:2"
+ },
+ {
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ }
+ },
+ "bucket": "choice:2"
+ },
+ {
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.3333,
+ "b": 0.3333,
+ "c": 0.3333
+ }
+ },
+ "bucket": "choice:3-5"
+ },
+ {
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.2,
+ "b": 0.2,
+ "c": 0.2,
+ "d": 0.2,
+ "e": 0.2
+ }
+ },
+ "bucket": "choice:3-5"
+ },
+ {
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.1667,
+ "b": 0.1667,
+ "c": 0.1667,
+ "d": 0.1667,
+ "e": 0.1667,
+ "f": 0.1667
+ }
+ },
+ "bucket": "choice:6-10"
+ },
+ {
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.1,
+ "b": 0.1,
+ "c": 0.1,
+ "d": 0.1,
+ "e": 0.1,
+ "f": 0.1,
+ "g": 0.1,
+ "h": 0.1,
+ "i": 0.1,
+ "j": 0.1
+ }
+ },
+ "bucket": "choice:6-10"
+ },
+ {
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0909,
+ "b": 0.0909,
+ "c": 0.0909,
+ "d": 0.0909,
+ "e": 0.0909,
+ "f": 0.0909,
+ "g": 0.0909,
+ "h": 0.0909,
+ "i": 0.0909,
+ "j": 0.0909,
+ "k": 0.0909
+ }
+ },
+ "bucket": "choice:11+"
+ },
+ {
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0769,
+ "b": 0.0769,
+ "c": 0.0769,
+ "d": 0.0769,
+ "e": 0.0769,
+ "f": 0.0769,
+ "g": 0.0769,
+ "h": 0.0769,
+ "i": 0.0769,
+ "j": 0.0769,
+ "k": 0.0769,
+ "l": 0.0769,
+ "m": 0.0769
+ }
+ },
+ "bucket": "choice:11+"
+ },
+ {
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 1.0
+ }
+ },
+ "bucket": "score:2"
+ },
+ {
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ }
+ },
+ "bucket": "score:2"
+ },
+ {
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.3333,
+ "1": 0.3333,
+ "2": 0.3333
+ }
+ },
+ "bucket": "score:3-5"
+ },
+ {
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.2,
+ "1": 0.2,
+ "2": 0.2,
+ "3": 0.2,
+ "4": 0.2
+ }
+ },
+ "bucket": "score:3-5"
+ },
+ {
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.1667,
+ "1": 0.1667,
+ "2": 0.1667,
+ "3": 0.1667,
+ "4": 0.1667,
+ "5": 0.1667
+ }
+ },
+ "bucket": "score:6-10"
+ },
+ {
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.1,
+ "1": 0.1,
+ "2": 0.1,
+ "3": 0.1,
+ "4": 0.1,
+ "5": 0.1,
+ "6": 0.1,
+ "7": 0.1,
+ "8": 0.1,
+ "9": 0.1
+ }
+ },
+ "bucket": "score:6-10"
+ },
+ {
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0909,
+ "1": 0.0909,
+ "2": 0.0909,
+ "3": 0.0909,
+ "4": 0.0909,
+ "5": 0.0909,
+ "6": 0.0909,
+ "7": 0.0909,
+ "8": 0.0909,
+ "9": 0.0909,
+ "10": 0.0909
+ }
+ },
+ "bucket": "score:11+"
+ },
+ {
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0769,
+ "1": 0.0769,
+ "2": 0.0769,
+ "3": 0.0769,
+ "4": 0.0769,
+ "5": 0.0769,
+ "6": 0.0769,
+ "7": 0.0769,
+ "8": 0.0769,
+ "9": 0.0769,
+ "10": 0.0769,
+ "11": 0.0769,
+ "12": 0.0769
+ }
+ },
+ "bucket": "score:11+"
+ },
+ {
+ "answer": {
+ "type": "noul",
+ "noul": 0.9
+ },
+ "bucket": "noul:2"
+ },
+ {
+ "answer": {
+ "type": "choice",
+ "probabilities": {}
+ },
+ "bucket": null
+ },
+ {
+ "answer": {
+ "type": "score",
+ "probabilities": {}
+ },
+ "bucket": null
+ }
+ ],
+ "resolve": [
+ {
+ "thresholds": {
+ "choice:2": 0.7,
+ "default": 0.4
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ }
+ },
+ "result": 0.7
+ },
+ {
+ "thresholds": {
+ "choice:2": 0.7,
+ "default": 0.4
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.25,
+ "b": 0.25,
+ "c": 0.25,
+ "d": 0.25
+ }
+ },
+ "result": 0.4
+ },
+ {
+ "thresholds": {
+ "choice:2": 0.7,
+ "default": 0.4
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.125,
+ "b": 0.125,
+ "c": 0.125,
+ "d": 0.125,
+ "e": 0.125,
+ "f": 0.125,
+ "g": 0.125,
+ "h": 0.125
+ }
+ },
+ "result": 0.4
+ },
+ {
+ "thresholds": {
+ "choice:2": 0.7,
+ "default": 0.4
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0833,
+ "b": 0.0833,
+ "c": 0.0833,
+ "d": 0.0833,
+ "e": 0.0833,
+ "f": 0.0833,
+ "g": 0.0833,
+ "h": 0.0833,
+ "i": 0.0833,
+ "j": 0.0833,
+ "k": 0.0833,
+ "l": 0.0833
+ }
+ },
+ "result": 0.4
+ },
+ {
+ "thresholds": {
+ "choice:2": 0.7,
+ "default": 0.4
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ }
+ },
+ "result": 0.4
+ },
+ {
+ "thresholds": {
+ "choice:2": 0.7,
+ "default": 0.4
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.25,
+ "1": 0.25,
+ "2": 0.25,
+ "3": 0.25
+ }
+ },
+ "result": 0.4
+ },
+ {
+ "thresholds": {
+ "choice:2": 0.7,
+ "default": 0.4
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.125,
+ "1": 0.125,
+ "2": 0.125,
+ "3": 0.125,
+ "4": 0.125,
+ "5": 0.125,
+ "6": 0.125,
+ "7": 0.125
+ }
+ },
+ "result": 0.4
+ },
+ {
+ "thresholds": {
+ "choice:2": 0.7,
+ "default": 0.4
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0833,
+ "1": 0.0833,
+ "2": 0.0833,
+ "3": 0.0833,
+ "4": 0.0833,
+ "5": 0.0833,
+ "6": 0.0833,
+ "7": 0.0833,
+ "8": 0.0833,
+ "9": 0.0833,
+ "10": 0.0833,
+ "11": 0.0833
+ }
+ },
+ "result": 0.4
+ },
+ {
+ "thresholds": {
+ "choice:2": 0.7,
+ "default": 0.4
+ },
+ "answer": {
+ "type": "noul",
+ "noul": 0.9
+ },
+ "result": 0.4
+ },
+ {
+ "thresholds": {
+ "choice:2": 0.7,
+ "default": 0.4
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {}
+ },
+ "result": 0.4
+ },
+ {
+ "thresholds": {
+ "choice:3-5": 0.5
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "choice:3-5": 0.5
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.25,
+ "b": 0.25,
+ "c": 0.25,
+ "d": 0.25
+ }
+ },
+ "result": 0.5
+ },
+ {
+ "thresholds": {
+ "choice:3-5": 0.5
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.125,
+ "b": 0.125,
+ "c": 0.125,
+ "d": 0.125,
+ "e": 0.125,
+ "f": 0.125,
+ "g": 0.125,
+ "h": 0.125
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "choice:3-5": 0.5
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0833,
+ "b": 0.0833,
+ "c": 0.0833,
+ "d": 0.0833,
+ "e": 0.0833,
+ "f": 0.0833,
+ "g": 0.0833,
+ "h": 0.0833,
+ "i": 0.0833,
+ "j": 0.0833,
+ "k": 0.0833,
+ "l": 0.0833
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "choice:3-5": 0.5
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "choice:3-5": 0.5
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.25,
+ "1": 0.25,
+ "2": 0.25,
+ "3": 0.25
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "choice:3-5": 0.5
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.125,
+ "1": 0.125,
+ "2": 0.125,
+ "3": 0.125,
+ "4": 0.125,
+ "5": 0.125,
+ "6": 0.125,
+ "7": 0.125
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "choice:3-5": 0.5
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0833,
+ "1": 0.0833,
+ "2": 0.0833,
+ "3": 0.0833,
+ "4": 0.0833,
+ "5": 0.0833,
+ "6": 0.0833,
+ "7": 0.0833,
+ "8": 0.0833,
+ "9": 0.0833,
+ "10": 0.0833,
+ "11": 0.0833
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "choice:3-5": 0.5
+ },
+ "answer": {
+ "type": "noul",
+ "noul": 0.9
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "choice:3-5": 0.5
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {}
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "default": 0.25
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ }
+ },
+ "result": 0.25
+ },
+ {
+ "thresholds": {
+ "default": 0.25
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.25,
+ "b": 0.25,
+ "c": 0.25,
+ "d": 0.25
+ }
+ },
+ "result": 0.25
+ },
+ {
+ "thresholds": {
+ "default": 0.25
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.125,
+ "b": 0.125,
+ "c": 0.125,
+ "d": 0.125,
+ "e": 0.125,
+ "f": 0.125,
+ "g": 0.125,
+ "h": 0.125
+ }
+ },
+ "result": 0.25
+ },
+ {
+ "thresholds": {
+ "default": 0.25
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0833,
+ "b": 0.0833,
+ "c": 0.0833,
+ "d": 0.0833,
+ "e": 0.0833,
+ "f": 0.0833,
+ "g": 0.0833,
+ "h": 0.0833,
+ "i": 0.0833,
+ "j": 0.0833,
+ "k": 0.0833,
+ "l": 0.0833
+ }
+ },
+ "result": 0.25
+ },
+ {
+ "thresholds": {
+ "default": 0.25
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ }
+ },
+ "result": 0.25
+ },
+ {
+ "thresholds": {
+ "default": 0.25
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.25,
+ "1": 0.25,
+ "2": 0.25,
+ "3": 0.25
+ }
+ },
+ "result": 0.25
+ },
+ {
+ "thresholds": {
+ "default": 0.25
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.125,
+ "1": 0.125,
+ "2": 0.125,
+ "3": 0.125,
+ "4": 0.125,
+ "5": 0.125,
+ "6": 0.125,
+ "7": 0.125
+ }
+ },
+ "result": 0.25
+ },
+ {
+ "thresholds": {
+ "default": 0.25
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0833,
+ "1": 0.0833,
+ "2": 0.0833,
+ "3": 0.0833,
+ "4": 0.0833,
+ "5": 0.0833,
+ "6": 0.0833,
+ "7": 0.0833,
+ "8": 0.0833,
+ "9": 0.0833,
+ "10": 0.0833,
+ "11": 0.0833
+ }
+ },
+ "result": 0.25
+ },
+ {
+ "thresholds": {
+ "default": 0.25
+ },
+ "answer": {
+ "type": "noul",
+ "noul": 0.9
+ },
+ "result": 0.25
+ },
+ {
+ "thresholds": {
+ "default": 0.25
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {}
+ },
+ "result": 0.25
+ },
+ {
+ "thresholds": {
+ "score:6-10": 0.8,
+ "noul:2": 0.3,
+ "choice:11+": 0.6
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "score:6-10": 0.8,
+ "noul:2": 0.3,
+ "choice:11+": 0.6
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.25,
+ "b": 0.25,
+ "c": 0.25,
+ "d": 0.25
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "score:6-10": 0.8,
+ "noul:2": 0.3,
+ "choice:11+": 0.6
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.125,
+ "b": 0.125,
+ "c": 0.125,
+ "d": 0.125,
+ "e": 0.125,
+ "f": 0.125,
+ "g": 0.125,
+ "h": 0.125
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "score:6-10": 0.8,
+ "noul:2": 0.3,
+ "choice:11+": 0.6
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0833,
+ "b": 0.0833,
+ "c": 0.0833,
+ "d": 0.0833,
+ "e": 0.0833,
+ "f": 0.0833,
+ "g": 0.0833,
+ "h": 0.0833,
+ "i": 0.0833,
+ "j": 0.0833,
+ "k": 0.0833,
+ "l": 0.0833
+ }
+ },
+ "result": 0.6
+ },
+ {
+ "thresholds": {
+ "score:6-10": 0.8,
+ "noul:2": 0.3,
+ "choice:11+": 0.6
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "score:6-10": 0.8,
+ "noul:2": 0.3,
+ "choice:11+": 0.6
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.25,
+ "1": 0.25,
+ "2": 0.25,
+ "3": 0.25
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "score:6-10": 0.8,
+ "noul:2": 0.3,
+ "choice:11+": 0.6
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.125,
+ "1": 0.125,
+ "2": 0.125,
+ "3": 0.125,
+ "4": 0.125,
+ "5": 0.125,
+ "6": 0.125,
+ "7": 0.125
+ }
+ },
+ "result": 0.8
+ },
+ {
+ "thresholds": {
+ "score:6-10": 0.8,
+ "noul:2": 0.3,
+ "choice:11+": 0.6
+ },
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0833,
+ "1": 0.0833,
+ "2": 0.0833,
+ "3": 0.0833,
+ "4": 0.0833,
+ "5": 0.0833,
+ "6": 0.0833,
+ "7": 0.0833,
+ "8": 0.0833,
+ "9": 0.0833,
+ "10": 0.0833,
+ "11": 0.0833
+ }
+ },
+ "result": 0.0
+ },
+ {
+ "thresholds": {
+ "score:6-10": 0.8,
+ "noul:2": 0.3,
+ "choice:11+": 0.6
+ },
+ "answer": {
+ "type": "noul",
+ "noul": 0.9
+ },
+ "result": 0.3
+ },
+ {
+ "thresholds": {
+ "score:6-10": 0.8,
+ "noul:2": 0.3,
+ "choice:11+": 0.6
+ },
+ "answer": {
+ "type": "choice",
+ "probabilities": {}
+ },
+ "result": 0.0
+ }
+ ],
+ "gate_answers": {
+ "choice2": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.6,
+ "b": 0.4
+ },
+ "answer_confidence": 0.6,
+ "confidence": 0.029
+ },
+ "choice4": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.45,
+ "b": 0.25,
+ "c": 0.2,
+ "d": 0.1
+ },
+ "answer_confidence": 0.45,
+ "confidence": 0.12
+ },
+ "choice8": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.125,
+ "b": 0.125,
+ "c": 0.125,
+ "d": 0.125,
+ "e": 0.125,
+ "f": 0.125,
+ "g": 0.125,
+ "h": 0.125
+ },
+ "answer_confidence": 0.3,
+ "confidence": 0.05
+ },
+ "choice12": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0833,
+ "b": 0.0833,
+ "c": 0.0833,
+ "d": 0.0833,
+ "e": 0.0833,
+ "f": 0.0833,
+ "g": 0.0833,
+ "h": 0.0833,
+ "i": 0.0833,
+ "j": 0.0833,
+ "k": 0.0833,
+ "l": 0.0833
+ },
+ "answer_confidence": 0.15,
+ "confidence": 0.02
+ },
+ "score2": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.7,
+ "1": 0.3
+ },
+ "answer_confidence": 0.7,
+ "confidence": 0.1
+ },
+ "score4": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.1,
+ "1": 0.5,
+ "2": 0.3,
+ "3": 0.1
+ },
+ "answer_confidence": 0.5,
+ "confidence": 0.2
+ },
+ "score8": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.125,
+ "1": 0.125,
+ "2": 0.125,
+ "3": 0.125,
+ "4": 0.125,
+ "5": 0.125,
+ "6": 0.125,
+ "7": 0.125
+ },
+ "answer_confidence": 0.125,
+ "confidence": 0.01
+ },
+ "noul": {
+ "type": "noul",
+ "noul": 0.8,
+ "answer_confidence": 0.8,
+ "confidence": 0.8
+ },
+ "at_half": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ },
+ "answer_confidence": 0.5,
+ "confidence": 0.0
+ },
+ "fallback_conf": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ },
+ "confidence": 0.3
+ },
+ "nan_answer_conf": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ },
+ "answer_confidence": "nan",
+ "confidence": 0.7
+ },
+ "inf_answer_conf": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ },
+ "answer_confidence": "inf",
+ "confidence": 0.2
+ },
+ "unusable_both": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ },
+ "answer_confidence": "nan",
+ "confidence": "inf"
+ },
+ "no_confidence": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ }
+ },
+ "empty_choice": {
+ "type": "choice",
+ "probabilities": {},
+ "answer_confidence": 0.4
+ }
+ },
+ "gate": [
+ {
+ "threshold": null,
+ "expected": {
+ "choice2": {},
+ "choice4": {},
+ "choice8": {},
+ "choice12": {},
+ "score2": {},
+ "score4": {},
+ "score8": {},
+ "noul": {},
+ "at_half": {},
+ "fallback_conf": {},
+ "nan_answer_conf": {},
+ "inf_answer_conf": {},
+ "unusable_both": {},
+ "no_confidence": {},
+ "empty_choice": {}
+ }
+ },
+ {
+ "threshold": 0.0,
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice12": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "fallback_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "inf_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ }
+ },
+ {
+ "threshold": 1.0,
+ "expected": {
+ "choice2": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "choice4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "choice8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "score2": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "score4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "noul": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "at_half": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "fallback_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "nan_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "inf_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 1.0
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 1.0
+ },
+ "empty_choice": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ }
+ }
+ },
+ {
+ "threshold": 0.5,
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "choice4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "choice8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "fallback_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "inf_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.5
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.5
+ },
+ "empty_choice": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ }
+ }
+ },
+ {
+ "threshold": 0.35,
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "choice4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "choice8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.35
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.35
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.35
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "fallback_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.35
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "inf_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.35
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.35
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.35
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ }
+ }
+ },
+ {
+ "threshold": 0.2,
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "choice4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "choice8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.2
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.2
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "fallback_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "inf_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.2
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.2
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ }
+ }
+ },
+ {
+ "threshold": {
+ "default": 0.4
+ },
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ },
+ "choice4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ },
+ "choice8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.4
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.4
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.4
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ },
+ "fallback_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.4
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ },
+ "inf_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.4
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.4
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.4
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ }
+ }
+ },
+ {
+ "threshold": {
+ "choice:2": 0.7,
+ "default": 0.4
+ },
+ "expected": {
+ "choice2": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.7
+ },
+ "choice4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ },
+ "choice8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.4
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.4
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.4
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ },
+ "at_half": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.7
+ },
+ "fallback_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.7
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ },
+ "inf_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.7
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.7
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.7
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.4
+ }
+ }
+ },
+ {
+ "threshold": {
+ "choice:3-5": 0.5
+ },
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "choice8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice12": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "fallback_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "inf_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ }
+ },
+ {
+ "threshold": {
+ "choice:2": 0.7,
+ "choice:3-5": 0.5,
+ "choice:6-10": 0.4,
+ "choice:11+": 0.1,
+ "score:2": 0.8,
+ "score:3-5": 0.6,
+ "score:6-10": 0.2,
+ "noul:2": 0.9
+ },
+ "expected": {
+ "choice2": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.7
+ },
+ "choice4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "choice8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.4
+ },
+ "choice12": {
+ "abstention": "passed",
+ "abstention_threshold": 0.1
+ },
+ "score2": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.8
+ },
+ "score4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.2
+ },
+ "noul": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "at_half": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.7
+ },
+ "fallback_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.7
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ },
+ "inf_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.7
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.7
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.7
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ }
+ },
+ {
+ "threshold": {
+ "noul:2": 0.0,
+ "score:3-5": 1.0
+ },
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice12": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "score8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "fallback_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "inf_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ }
+ }
+ ],
+ "reuse": [
+ {
+ "steps": [
+ {
+ "threshold": 0.9,
+ "expected": {
+ "choice2": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "choice4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "choice8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "score2": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "score4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "noul": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "at_half": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "fallback_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "nan_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "inf_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.9
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.9
+ },
+ "empty_choice": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ }
+ }
+ },
+ {
+ "threshold": 0.0,
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice12": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "fallback_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "inf_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ }
+ },
+ {
+ "threshold": 0.2,
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "choice4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "choice8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.2
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.2
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "fallback_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "inf_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.2
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.2
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.2
+ }
+ }
+ },
+ {
+ "threshold": 1.0,
+ "expected": {
+ "choice2": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "choice4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "choice8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "score2": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "score4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "noul": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "at_half": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "fallback_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "nan_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "inf_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 1.0
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 1.0
+ },
+ "empty_choice": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ }
+ }
+ },
+ {
+ "threshold": 0.5,
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "choice4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "choice8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "fallback_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "inf_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.5
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.5
+ },
+ "empty_choice": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ }
+ }
+ }
+ ]
+ },
+ {
+ "steps": [
+ {
+ "threshold": {
+ "choice:2": 0.9,
+ "default": 0.5
+ },
+ "expected": {
+ "choice2": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "choice4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "choice8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "at_half": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "fallback_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "nan_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "inf_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.9
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.9
+ },
+ "empty_choice": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ }
+ }
+ },
+ {
+ "threshold": 0.0,
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice12": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "fallback_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "inf_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ }
+ },
+ {
+ "threshold": {
+ "choice:3-5": 0.9
+ },
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice4": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "choice8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "choice12": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "score8": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "fallback_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "inf_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.0
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ }
+ },
+ {
+ "threshold": 0.35,
+ "expected": {
+ "choice2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "choice4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "choice8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.35
+ },
+ "choice12": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.35
+ },
+ "score2": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "score4": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "score8": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.35
+ },
+ "noul": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "at_half": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "fallback_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.35
+ },
+ "nan_answer_conf": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ },
+ "inf_answer_conf": {
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.35
+ },
+ "unusable_both": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.35
+ },
+ "no_confidence": {
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.35
+ },
+ "empty_choice": {
+ "abstention": "passed",
+ "abstention_threshold": 0.35
+ }
+ }
+ }
+ ]
+ }
+ ],
+ "jev": [
+ {
+ "name": "choice-confident",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.9,
+ "b": 0.1
+ }
+ },
+ "typed": true,
+ "result": 0.8
+ },
+ {
+ "name": "choice-even",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": 0.5
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "choice-three",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.7,
+ "b": 0.2,
+ "c": 0.1
+ }
+ },
+ "typed": true,
+ "result": 0.5499999999999999
+ },
+ {
+ "name": "choice-differs-from-entropy",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.7,
+ "b": 0.1,
+ "c": 0.1,
+ "d": 0.1
+ }
+ },
+ "typed": true,
+ "result": 0.6
+ },
+ {
+ "name": "choice-unnormalised",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 2.0,
+ "b": 1.0,
+ "c": 1.0
+ }
+ },
+ "typed": true,
+ "result": 0.25
+ },
+ {
+ "name": "choice-zero-total",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0,
+ "b": 0.0,
+ "c": 0.0
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "choice-single",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.3
+ }
+ },
+ "typed": true,
+ "result": 1.0
+ },
+ {
+ "name": "choice-single-zero",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0
+ }
+ },
+ "typed": true,
+ "result": 1.0
+ },
+ {
+ "name": "choice-zero-among-others",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0,
+ "b": 1.0
+ }
+ },
+ "typed": true,
+ "result": 1.0
+ },
+ {
+ "name": "score-peaked",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0,
+ "1": 0.95,
+ "2": 0.05
+ }
+ },
+ "typed": true,
+ "result": 0.925
+ },
+ {
+ "name": "score-jevcompat-example",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0,
+ "1": 0.95,
+ "2": 0.05
+ }
+ },
+ "typed": true,
+ "result": 0.925
+ },
+ {
+ "name": "score-even",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.25,
+ "1": 0.25,
+ "2": 0.25,
+ "3": 0.25
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "score-edge-mode",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.8,
+ "1": 0.1,
+ "2": 0.1
+ }
+ },
+ "typed": true,
+ "result": 0.5499999999999999
+ },
+ {
+ "name": "score-zero-total",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0,
+ "1": 0.0,
+ "2": 0.0,
+ "3": 0.0
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "score-single",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.6
+ }
+ },
+ "typed": true,
+ "result": 1.0
+ },
+ {
+ "name": "score-two",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.3,
+ "1": 0.7
+ }
+ },
+ "typed": true,
+ "result": 0.4
+ },
+ {
+ "name": "score-bimodal",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.45,
+ "1": 0.05,
+ "2": 0.05,
+ "3": 0.45
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "score-tied-modes",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.4,
+ "1": 0.4,
+ "2": 0.2
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "score-levels-in-numeric-order",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0,
+ "1": 0.0,
+ "2": 0.0,
+ "3": 0.0,
+ "4": 0.0,
+ "5": 0.0,
+ "6": 0.0,
+ "7": 0.0,
+ "8": 0.0,
+ "9": 0.0,
+ "10": 1.0
+ }
+ },
+ "typed": true,
+ "result": 1.0
+ },
+ {
+ "name": "noul",
+ "answer": {
+ "type": "noul",
+ "noul": 0.9
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "empty-choice-probabilities",
+ "answer": {
+ "type": "choice",
+ "probabilities": {}
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "empty-score-probabilities",
+ "answer": {
+ "type": "score",
+ "probabilities": {}
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "nan-probability",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": "nan",
+ "b": 0.5
+ }
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "infinite-probability",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": "inf",
+ "b": 0.5
+ }
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "negative-infinite-probability",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": "-inf",
+ "b": 0.5
+ }
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "negative-probability",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 2.0,
+ "b": -1.0
+ }
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "mixed-sign-argmax-flips",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": -1.5
+ }
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "all-negative-probabilities",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": -0.9,
+ "b": -0.1
+ }
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "negative-total-near-zero",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": -0.61,
+ "b": 0.6
+ }
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "negative-score-level",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 1.5,
+ "1": -0.5,
+ "2": 0.0
+ }
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "nan-score-level",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.5,
+ "1": "nan"
+ }
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "negative-single",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": -0.5
+ }
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "nan-single",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": "nan"
+ }
+ },
+ "typed": true,
+ "result": null
+ },
+ {
+ "name": "unknown-type",
+ "answer": {
+ "type": "weird",
+ "probabilities": {
+ "a": 1.0
+ }
+ },
+ "typed": false,
+ "result": null
+ },
+ {
+ "name": "missing-probabilities",
+ "answer": {
+ "type": "choice"
+ },
+ "typed": false,
+ "result": null
+ },
+ {
+ "name": "non-numeric-probability",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5,
+ "b": "high"
+ }
+ },
+ "typed": false,
+ "result": null
+ },
+ {
+ "name": "bool-probability",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": true,
+ "b": false
+ }
+ },
+ "typed": false,
+ "result": null
+ },
+ {
+ "name": "bool-among-numbers",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": true,
+ "b": 0.4
+ }
+ },
+ "typed": false,
+ "result": null
+ },
+ {
+ "name": "non-integer-score-level",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "low": 0.5,
+ "high": 0.5
+ }
+ },
+ "typed": false,
+ "result": null
+ },
+ {
+ "name": "sum-differs-choice-tenths",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.1,
+ "b": 0.1,
+ "c": 0.1,
+ "d": 0.1,
+ "e": 0.1,
+ "f": 0.1,
+ "g": 0.1,
+ "h": 0.1,
+ "i": 0.1,
+ "j": 0.1
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "sum-differs-score-tenths",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.1,
+ "1": 0.1,
+ "2": 0.1,
+ "3": 0.1,
+ "4": 0.1,
+ "5": 0.1,
+ "6": 0.1,
+ "7": 0.1,
+ "8": 0.1,
+ "9": 0.1
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "sum-differs-choice-hundredths",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.01,
+ "b": 0.01,
+ "c": 0.01,
+ "d": 0.01,
+ "e": 0.01,
+ "f": 0.01,
+ "g": 0.01,
+ "h": 0.01,
+ "i": 0.01,
+ "j": 0.01,
+ "k": 0.01,
+ "l": 0.01,
+ "m": 0.01,
+ "n": 0.01,
+ "o": 0.01,
+ "p": 0.01,
+ "q": 0.01,
+ "r": 0.01,
+ "s": 0.01,
+ "t": 0.01,
+ "u": 0.01,
+ "v": 0.01,
+ "w": 0.01,
+ "x": 0.01,
+ "y": 0.01,
+ "z": 0.01,
+ "{": 0.01,
+ "|": 0.01,
+ "}": 0.01,
+ "~": 0.01,
+ "": 0.01,
+ "€": 0.01,
+ "": 0.01,
+ "‚": 0.01,
+ "ƒ": 0.01,
+ "„": 0.01,
+ "…": 0.01,
+ "†": 0.01,
+ "‡": 0.01,
+ "ˆ": 0.01,
+ "‰": 0.01,
+ "Š": 0.01,
+ "‹": 0.01,
+ "Œ": 0.01,
+ "": 0.01,
+ "Ž": 0.01,
+ "": 0.01,
+ "": 0.01,
+ "‘": 0.01,
+ "’": 0.01,
+ "“": 0.01,
+ "”": 0.01,
+ "•": 0.01,
+ "–": 0.01,
+ "—": 0.01,
+ "˜": 0.01,
+ "™": 0.01,
+ "š": 0.01,
+ "›": 0.01,
+ "œ": 0.01,
+ "": 0.01,
+ "ž": 0.01,
+ "Ÿ": 0.01,
+ " ": 0.01,
+ "¡": 0.01,
+ "¢": 0.01,
+ "£": 0.01,
+ "¤": 0.01,
+ "¥": 0.01,
+ "¦": 0.01,
+ "§": 0.01,
+ "¨": 0.01,
+ "©": 0.01,
+ "ª": 0.01,
+ "«": 0.01,
+ "¬": 0.01,
+ "­": 0.01,
+ "®": 0.01,
+ "¯": 0.01,
+ "°": 0.01,
+ "±": 0.01,
+ "²": 0.01,
+ "³": 0.01,
+ "´": 0.01,
+ "µ": 0.01,
+ "¶": 0.01,
+ "·": 0.01,
+ "¸": 0.01,
+ "¹": 0.01,
+ "º": 0.01,
+ "»": 0.01,
+ "¼": 0.01,
+ "½": 0.01,
+ "¾": 0.01,
+ "¿": 0.01,
+ "À": 0.01,
+ "Á": 0.01,
+ "Â": 0.01,
+ "Ã": 0.01,
+ "Ä": 0.01
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "sum-differs-choice-7",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.6946,
+ "b": 0.03,
+ "c": 0.3634,
+ "d": 0.67,
+ "e": 0.2649,
+ "f": 0.72,
+ "g": 0.6497
+ }
+ },
+ "typed": true,
+ "result": 0.08093104600208292
+ },
+ {
+ "name": "sum-differs-choice-5",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.7133,
+ "b": 0.3416,
+ "c": 0.7,
+ "d": 0.8634,
+ "e": 0.6
+ }
+ },
+ "typed": true,
+ "result": 0.08534785445732211
+ },
+ {
+ "name": "sum-differs-choice-10",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.6,
+ "b": 0.1,
+ "c": 0.3,
+ "d": 0.7918,
+ "e": 0.94,
+ "f": 0.7,
+ "g": 0.9983,
+ "h": 0.5,
+ "i": 0.8,
+ "j": 0.6
+ }
+ },
+ "typed": true,
+ "result": 0.06411869919555421
+ },
+ {
+ "name": "sum-differs-choice-16",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.9,
+ "b": 0.11,
+ "c": 0.4,
+ "d": 0.84,
+ "e": 0.1412,
+ "f": 0.8523,
+ "g": 0.45,
+ "h": 0.69,
+ "i": 0.48,
+ "j": 0.5,
+ "k": 0.1126,
+ "l": 0.9598,
+ "m": 0.94,
+ "n": 0.18,
+ "o": 0.2723,
+ "p": 0.87
+ }
+ },
+ "typed": true,
+ "result": 0.051034313612778125
+ },
+ {
+ "name": "sum-differs-score-7",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.4,
+ "1": 0.4528,
+ "2": 0.2,
+ "3": 0.1,
+ "4": 0.93,
+ "5": 0.34,
+ "6": 0.89
+ }
+ },
+ "typed": true,
+ "result": 0.017729211945584966
+ },
+ {
+ "name": "sum-differs-score-3",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.453,
+ "1": 0.4,
+ "2": 0.0333
+ }
+ },
+ "typed": true,
+ "result": 0.21031253525894167
+ },
+ {
+ "name": "random-choice-00",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.2566,
+ "b": 0.4558
+ }
+ },
+ "typed": true,
+ "result": 0.27961819202695115
+ },
+ {
+ "name": "random-score-00",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.2566,
+ "1": 0.4558
+ }
+ },
+ "typed": true,
+ "result": 0.27961819202695104
+ },
+ {
+ "name": "random-choice-01",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.8042,
+ "b": 0.9099,
+ "c": 0.6461,
+ "d": 0.4657,
+ "e": 0.914,
+ "f": 0.3627,
+ "g": 0.1673,
+ "h": 0.8291,
+ "i": 0.5506,
+ "j": 0.1385,
+ "k": 0.9363,
+ "l": 0.3846
+ }
+ },
+ "typed": true,
+ "result": 0.052770495786391154
+ },
+ {
+ "name": "random-score-01",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.8042,
+ "1": 0.9099,
+ "2": 0.6461,
+ "3": 0.4657,
+ "4": 0.914,
+ "5": 0.3627,
+ "6": 0.1673,
+ "7": 0.8291,
+ "8": 0.5506,
+ "9": 0.1385,
+ "10": 0.9363,
+ "11": 0.3846
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-02",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.6852,
+ "b": 0.4922,
+ "c": 0.3347,
+ "d": 0.8721,
+ "e": 0.9064,
+ "f": 0.1556,
+ "g": 0.1312,
+ "h": 0.155,
+ "i": 0.5995,
+ "j": 0.7911,
+ "k": 0.4026,
+ "l": 0.7689
+ }
+ },
+ "typed": true,
+ "result": 0.06618043169000355
+ },
+ {
+ "name": "random-score-02",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.6852,
+ "1": 0.4922,
+ "2": 0.3347,
+ "3": 0.8721,
+ "4": 0.9064,
+ "5": 0.1556,
+ "6": 0.1312,
+ "7": 0.155,
+ "8": 0.5995,
+ "9": 0.7911,
+ "10": 0.4026,
+ "11": 0.7689
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-03",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.8998,
+ "b": 0.8419,
+ "c": 0.0794,
+ "d": 0.321
+ }
+ },
+ "typed": true,
+ "result": 0.226740114840577
+ },
+ {
+ "name": "random-score-03",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.8998,
+ "1": 0.8419,
+ "2": 0.0794,
+ "3": 0.321
+ }
+ },
+ "typed": true,
+ "result": 0.08328275990850109
+ },
+ {
+ "name": "random-choice-04",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.6957,
+ "b": 0.7158,
+ "c": 0.3912,
+ "d": 0.5873,
+ "e": 0.7343,
+ "f": 0.2801,
+ "g": 0.423,
+ "h": 0.2488
+ }
+ },
+ "typed": true,
+ "result": 0.06302088079233457
+ },
+ {
+ "name": "random-score-04",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.6957,
+ "1": 0.7158,
+ "2": 0.3912,
+ "3": 0.5873,
+ "4": 0.7343,
+ "5": 0.2801,
+ "6": 0.423,
+ "7": 0.2488
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-05",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.6537,
+ "b": 0.3714,
+ "c": 0.6408,
+ "d": 0.9808
+ }
+ },
+ "typed": true,
+ "result": 0.16076623720104277
+ },
+ {
+ "name": "random-score-05",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.6537,
+ "1": 0.3714,
+ "2": 0.6408,
+ "3": 0.9808
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-06",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0542,
+ "b": 0.993,
+ "c": 0.2939,
+ "d": 0.2498,
+ "e": 0.8353,
+ "f": 0.7231,
+ "g": 0.8383,
+ "h": 0.6679,
+ "i": 0.3134,
+ "j": 0.7684,
+ "k": 0.3708,
+ "l": 0.114
+ }
+ },
+ "typed": true,
+ "result": 0.08319173152589524
+ },
+ {
+ "name": "random-score-06",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0542,
+ "1": 0.993,
+ "2": 0.2939,
+ "3": 0.2498,
+ "4": 0.8353,
+ "5": 0.7231,
+ "6": 0.8383,
+ "7": 0.6679,
+ "8": 0.3134,
+ "9": 0.7684,
+ "10": 0.3708,
+ "11": 0.114
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-07",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.8131,
+ "b": 0.9647,
+ "c": 0.0528,
+ "d": 0.1295,
+ "e": 0.2193,
+ "f": 0.4647,
+ "g": 0.5681,
+ "h": 0.8712,
+ "i": 0.1082,
+ "j": 0.4005,
+ "k": 0.7698,
+ "l": 0.436
+ }
+ },
+ "typed": true,
+ "result": 0.09060490553789853
+ },
+ {
+ "name": "random-score-07",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.8131,
+ "1": 0.9647,
+ "2": 0.0528,
+ "3": 0.1295,
+ "4": 0.2193,
+ "5": 0.4647,
+ "6": 0.5681,
+ "7": 0.8712,
+ "8": 0.1082,
+ "9": 0.4005,
+ "10": 0.7698,
+ "11": 0.436
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-08",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.3693,
+ "b": 0.8736,
+ "c": 0.329,
+ "d": 0.1395,
+ "e": 0.2551
+ }
+ },
+ "typed": true,
+ "result": 0.30530129672006096
+ },
+ {
+ "name": "random-score-08",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.3693,
+ "1": 0.8736,
+ "2": 0.329,
+ "3": 0.1395,
+ "4": 0.2551
+ }
+ },
+ "typed": true,
+ "result": 0.26154758877871
+ },
+ {
+ "name": "random-choice-09",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.686,
+ "b": 0.3542,
+ "c": 0.2812,
+ "d": 0.8562
+ }
+ },
+ "typed": true,
+ "result": 0.1909135439627724
+ },
+ {
+ "name": "random-score-09",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.686,
+ "1": 0.3542,
+ "2": 0.2812,
+ "3": 0.8562
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-10",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.6386,
+ "b": 0.4663
+ }
+ },
+ "typed": true,
+ "result": 0.15594171418227876
+ },
+ {
+ "name": "random-score-10",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.6386,
+ "1": 0.4663
+ }
+ },
+ "typed": true,
+ "result": 0.15594171418227898
+ },
+ {
+ "name": "random-choice-11",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.2097,
+ "b": 0.6566,
+ "c": 0.0187,
+ "d": 0.9506
+ }
+ },
+ "typed": true,
+ "result": 0.35715842231422973
+ },
+ {
+ "name": "random-score-11",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.2097,
+ "1": 0.6566,
+ "2": 0.0187,
+ "3": 0.9506
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-12",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.4693,
+ "b": 0.1916
+ }
+ },
+ "typed": true,
+ "result": 0.42018459676199105
+ },
+ {
+ "name": "random-score-12",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.4693,
+ "1": 0.1916
+ }
+ },
+ "typed": true,
+ "result": 0.42018459676199127
+ },
+ {
+ "name": "random-choice-13",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.9691,
+ "b": 0.6389,
+ "c": 0.7036,
+ "d": 0.671,
+ "e": 0.5951,
+ "f": 0.1533,
+ "g": 0.6329,
+ "h": 0.495
+ }
+ },
+ "typed": true,
+ "result": 0.08508392552106149
+ },
+ {
+ "name": "random-score-13",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.9691,
+ "1": 0.6389,
+ "2": 0.7036,
+ "3": 0.671,
+ "4": 0.5951,
+ "5": 0.1533,
+ "6": 0.6329,
+ "7": 0.495
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-14",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.7871,
+ "b": 0.069,
+ "c": 0.9774
+ }
+ },
+ "typed": true,
+ "result": 0.29961821652577036
+ },
+ {
+ "name": "random-score-14",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.7871,
+ "1": 0.069,
+ "2": 0.9774
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-15",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.9361,
+ "b": 0.0913,
+ "c": 0.9088,
+ "d": 0.2108,
+ "e": 0.4095
+ }
+ },
+ "typed": true,
+ "result": 0.20770584783884216
+ },
+ {
+ "name": "random-score-15",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.9361,
+ "1": 0.0913,
+ "2": 0.9088,
+ "3": 0.2108,
+ "4": 0.4095
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-16",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.141,
+ "b": 0.6816,
+ "c": 0.801
+ }
+ },
+ "typed": true,
+ "result": 0.2400221729490023
+ },
+ {
+ "name": "random-score-16",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.141,
+ "1": 0.6816,
+ "2": 0.801
+ }
+ },
+ "typed": true,
+ "result": 0.1097560975609756
+ },
+ {
+ "name": "random-choice-17",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.735,
+ "b": 0.398,
+ "c": 0.5032,
+ "d": 0.2529,
+ "e": 0.9528
+ }
+ },
+ "typed": true,
+ "result": 0.1690858228649847
+ },
+ {
+ "name": "random-score-17",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.735,
+ "1": 0.398,
+ "2": 0.5032,
+ "3": 0.2529,
+ "4": 0.9528
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-18",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.6034,
+ "b": 0.5739,
+ "c": 0.8956
+ }
+ },
+ "typed": true,
+ "result": 0.14807757248299486
+ },
+ {
+ "name": "random-score-18",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.6034,
+ "1": 0.5739,
+ "2": 0.8956
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-19",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.0916,
+ "b": 0.3621,
+ "c": 0.0287,
+ "d": 0.537,
+ "e": 0.8582,
+ "f": 0.8354,
+ "g": 0.5424,
+ "h": 0.9539
+ }
+ },
+ "typed": true,
+ "result": 0.11613400259968573
+ },
+ {
+ "name": "random-score-19",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.0916,
+ "1": 0.3621,
+ "2": 0.0287,
+ "3": 0.537,
+ "4": 0.8582,
+ "5": 0.8354,
+ "6": 0.5424,
+ "7": 0.9539
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-20",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.9237,
+ "b": 0.4328
+ }
+ },
+ "typed": true,
+ "result": 0.36188720973092514
+ },
+ {
+ "name": "random-score-20",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.9237,
+ "1": 0.4328
+ }
+ },
+ "typed": true,
+ "result": 0.36188720973092514
+ },
+ {
+ "name": "random-choice-21",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.5616,
+ "b": 0.0699,
+ "c": 0.8875,
+ "d": 0.0634,
+ "e": 0.6978,
+ "f": 0.5905,
+ "g": 0.6597,
+ "h": 0.1242,
+ "i": 0.2992,
+ "j": 0.5477,
+ "k": 0.8902,
+ "l": 0.2786
+ }
+ },
+ "typed": true,
+ "result": 0.08035649869415279
+ },
+ {
+ "name": "random-score-21",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.5616,
+ "1": 0.0699,
+ "2": 0.8875,
+ "3": 0.0634,
+ "4": 0.6978,
+ "5": 0.5905,
+ "6": 0.6597,
+ "7": 0.1242,
+ "8": 0.2992,
+ "9": 0.5477,
+ "10": 0.8902,
+ "11": 0.2786
+ }
+ },
+ "typed": true,
+ "result": 0.0
+ },
+ {
+ "name": "random-choice-22",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.3113,
+ "b": 0.6847,
+ "c": 0.3262
+ }
+ },
+ "typed": true,
+ "result": 0.27677355921948266
+ },
+ {
+ "name": "random-score-22",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.3113,
+ "1": 0.6847,
+ "2": 0.3262
+ }
+ },
+ "typed": true,
+ "result": 0.27677355921948266
+ },
+ {
+ "name": "random-choice-23",
+ "answer": {
+ "type": "choice",
+ "probabilities": {
+ "a": 0.3983,
+ "b": 0.9787,
+ "c": 0.9707,
+ "d": 0.4993
+ }
+ },
+ "typed": true,
+ "result": 0.12502048940405106
+ },
+ {
+ "name": "random-score-23",
+ "answer": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.3983,
+ "1": 0.9787,
+ "2": 0.9707,
+ "3": 0.4993
+ }
+ },
+ "typed": true,
+ "result": 0.1683877766069546
+ }
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json b/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json
index 444edfcb..2fc01079 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json
@@ -11,6 +11,13 @@
"route_probe": 142,
"route_batch_probe": 10,
"route_batch_probe_errors": 6,
+ "confidence_probe_scalars": 18,
+ "confidence_probe_maps": 24,
+ "confidence_probe_buckets": 19,
+ "confidence_probe_resolve": 40,
+ "confidence_probe_gate": 11,
+ "confidence_probe_reuse": 2,
+ "confidence_probe_jev": 96,
"email_probe_clean": 253,
"email_probe_state": 22,
"shortlist_probe": 22,
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_bad_key.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_bad_key.json
new file mode 100644
index 00000000..424420ec
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_bad_key.json
@@ -0,0 +1,38 @@
+{
+ "name": "batch_min_confidence_bad_key",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "choice:2": 0.5,
+ "choice:3": 0.5
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'"
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_before_none_state.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_before_none_state.json
new file mode 100644
index 00000000..75fd7564
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_before_none_state.json
@@ -0,0 +1,36 @@
+{
+ "name": "batch_min_confidence_before_none_state",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ null
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": -0.25
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got -0.25"
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_empty_questions.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_empty_questions.json
new file mode 100644
index 00000000..255f7bc4
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_empty_questions.json
@@ -0,0 +1,44 @@
+{
+ "name": "batch_min_confidence_empty_questions",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, the refund arrived and everything works now.",
+ {
+ "ticket": 4821,
+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "
+ }
+ ],
+ "questions": {},
+ "options": {
+ "min_confidence": 0.5
+ },
+ "state_widths": [],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {},
+ "usage": {
+ "input_tokens": 0,
+ "output_tokens": 0
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {},
+ "usage": {
+ "input_tokens": 0,
+ "output_tokens": 0
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {},
+ "usage": {
+ "input_tokens": 0,
+ "output_tokens": 0
+ }
+ }
+ ],
+ "runs": []
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_empty_states.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_empty_states.json
new file mode 100644
index 00000000..dfb141d9
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_empty_states.json
@@ -0,0 +1,34 @@
+{
+ "name": "batch_min_confidence_empty_states",
+ "states": [],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 0.5
+ },
+ "state_widths": [],
+ "results": [],
+ "runs": []
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_invalid.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_invalid.json
new file mode 100644
index 00000000..72496b52
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_invalid.json
@@ -0,0 +1,35 @@
+{
+ "name": "batch_min_confidence_invalid",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 1.5
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5"
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_map.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_map.json
new file mode 100644
index 00000000..a34cfe8e
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_map.json
@@ -0,0 +1,504 @@
+{
+ "name": "batch_min_confidence_map",
+ "states": [
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "batch_size": 3,
+ "sort_by_length": true,
+ "min_confidence": {
+ "choice:3-5": 0.8,
+ "score:3-5": 0.5,
+ "noul:2": 0.7
+ }
+ },
+ "state_widths": [
+ 199,
+ 103,
+ 145,
+ 91,
+ 175,
+ 121,
+ 157
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7688,
+ "technical": 0.1498,
+ "other": 0.0814
+ },
+ "confidence": 0.3712,
+ "answer_confidence": 0.7688,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.839,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0167,
+ "1": 0.1276,
+ "2": 0.8557
+ },
+ "confidence": 0.5773,
+ "answer_confidence": 0.8557,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.8118,
+ "confidence": 0.8118,
+ "answer_confidence": 0.8118,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 589,
+ "output_tokens": 0,
+ "state_tokens": 165,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7324,
+ "technical": 0.1799,
+ "other": 0.0877
+ },
+ "confidence": 0.3172,
+ "answer_confidence": 0.7324,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.6583,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0379,
+ "1": 0.2659,
+ "2": 0.6962
+ },
+ "confidence": 0.337,
+ "answer_confidence": 0.6962,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.7988,
+ "confidence": 0.7988,
+ "answer_confidence": 0.7988,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 301,
+ "output_tokens": 0,
+ "state_tokens": 69,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7732,
+ "technical": 0.1468,
+ "other": 0.08
+ },
+ "confidence": 0.3786,
+ "answer_confidence": 0.7732,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8334,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0162,
+ "1": 0.1342,
+ "2": 0.8496
+ },
+ "confidence": 0.5678,
+ "answer_confidence": 0.8496,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.8132,
+ "confidence": 0.8132,
+ "answer_confidence": 0.8132,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 427,
+ "output_tokens": 0,
+ "state_tokens": 111,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7054,
+ "technical": 0.1834,
+ "other": 0.1111
+ },
+ "confidence": 0.2705,
+ "answer_confidence": 0.7054,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8371,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0193,
+ "1": 0.1243,
+ "2": 0.8564
+ },
+ "confidence": 0.574,
+ "answer_confidence": 0.8564,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.7914,
+ "confidence": 0.7914,
+ "answer_confidence": 0.7914,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 265,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7458,
+ "technical": 0.1647,
+ "other": 0.0895
+ },
+ "confidence": 0.3338,
+ "answer_confidence": 0.7458,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8335,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0167,
+ "1": 0.1332,
+ "2": 0.8501
+ },
+ "confidence": 0.5678,
+ "answer_confidence": 0.8501,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.827,
+ "confidence": 0.827,
+ "answer_confidence": 0.827,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 517,
+ "output_tokens": 0,
+ "state_tokens": 141,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7837,
+ "technical": 0.1379,
+ "other": 0.0784
+ },
+ "confidence": 0.3958,
+ "answer_confidence": 0.7837,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8327,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0171,
+ "1": 0.1331,
+ "2": 0.8498
+ },
+ "confidence": 0.5665,
+ "answer_confidence": 0.8498,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.7977,
+ "confidence": 0.7977,
+ "answer_confidence": 0.7977,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 355,
+ "output_tokens": 0,
+ "state_tokens": 87,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7287,
+ "technical": 0.1816,
+ "other": 0.0897
+ },
+ "confidence": 0.3111,
+ "answer_confidence": 0.7287,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.8
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8351,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0151,
+ "1": 0.1346,
+ "2": 0.8502
+ },
+ "confidence": 0.571,
+ "answer_confidence": 0.8502,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.8224,
+ "confidence": 0.8224,
+ "answer_confidence": 0.8224,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 463,
+ "output_tokens": 0,
+ "state_tokens": 123,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 9,
+ 121
+ ],
+ [
+ 9,
+ 175
+ ],
+ [
+ 3,
+ 199
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_map_default.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_map_default.json
new file mode 100644
index 00000000..b7a18f30
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_map_default.json
@@ -0,0 +1,240 @@
+{
+ "name": "batch_min_confidence_map_default",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, the refund arrived and everything works now.",
+ {
+ "ticket": 4821,
+ "message": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "notes": "Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. Routine checks. "
+ }
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "default": 0.55
+ }
+ },
+ "state_widths": [
+ 91,
+ 44,
+ 157
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7054,
+ "technical": 0.1834,
+ "other": 0.1111
+ },
+ "confidence": 0.2705,
+ "answer_confidence": 0.7054,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8371,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0193,
+ "1": 0.1243,
+ "2": 0.8564
+ },
+ "confidence": 0.574,
+ "answer_confidence": 0.8564,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.7914,
+ "confidence": 0.7914,
+ "answer_confidence": 0.7914,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 265,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.4925,
+ "technical": 0.3304,
+ "other": 0.1771
+ },
+ "confidence": 0.0704,
+ "answer_confidence": 0.4925,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.2205,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.2895,
+ "1": 0.2006,
+ "2": 0.51
+ },
+ "confidence": 0.0675,
+ "answer_confidence": 0.51,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.55
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0363,
+ "confidence": 0.9637,
+ "answer_confidence": 0.9637,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 124,
+ "output_tokens": 0,
+ "state_tokens": 10,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.5409,
+ "technical": 0.2757,
+ "other": 0.1834
+ },
+ "confidence": 0.0909,
+ "answer_confidence": 0.5409,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8074,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0273,
+ "1": 0.138,
+ "2": 0.8347
+ },
+ "confidence": 0.5244,
+ "answer_confidence": 0.8347,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.7748,
+ "confidence": 0.7748,
+ "answer_confidence": 0.7748,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 463,
+ "output_tokens": 0,
+ "state_tokens": 123,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 9,
+ 157
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_scalar.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_scalar.json
new file mode 100644
index 00000000..28e53889
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_scalar.json
@@ -0,0 +1,572 @@
+{
+ "name": "batch_min_confidence_scalar",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ {
+ "subject": "Double charged on invoice 88213",
+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ },
+ [
+ {
+ "role": "user",
+ "content": "Hi, I need help with my invoice."
+ },
+ {
+ "role": "assistant",
+ "content": "Of course, what is wrong?"
+ },
+ {
+ "role": "user",
+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ }
+ ],
+ "",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, that is all sorted now.",
+ {
+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",
+ "message": "Please cancel my account today."
+ },
+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 0.6
+ },
+ "state_widths": [
+ 91,
+ 110,
+ 144,
+ 34,
+ 163,
+ 42,
+ 98,
+ 56
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7054,
+ "technical": 0.1834,
+ "other": 0.1111
+ },
+ "confidence": 0.2705,
+ "answer_confidence": 0.7054,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8371,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0193,
+ "1": 0.1243,
+ "2": 0.8564
+ },
+ "confidence": 0.574,
+ "answer_confidence": 0.8564,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.7914,
+ "confidence": 0.7914,
+ "answer_confidence": 0.7914,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 265,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.6869,
+ "technical": 0.2079,
+ "other": 0.1052
+ },
+ "confidence": 0.2523,
+ "answer_confidence": 0.6869,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7918,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0204,
+ "1": 0.1675,
+ "2": 0.8121
+ },
+ "confidence": 0.5015,
+ "answer_confidence": 0.8121,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.7602,
+ "confidence": 0.7602,
+ "answer_confidence": 0.7602,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 322,
+ "output_tokens": 0,
+ "state_tokens": 76,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.6718,
+ "technical": 0.2244,
+ "other": 0.1038
+ },
+ "confidence": 0.2375,
+ "answer_confidence": 0.6718,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7672,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0194,
+ "1": 0.194,
+ "2": 0.7866
+ },
+ "confidence": 0.4689,
+ "answer_confidence": 0.7866,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.7234,
+ "confidence": 0.7234,
+ "answer_confidence": 0.7234,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 424,
+ "output_tokens": 0,
+ "state_tokens": 110,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "other",
+ "probabilities": {
+ "refund": 0.3741,
+ "technical": 0.2268,
+ "other": 0.3991
+ },
+ "confidence": 0.0252,
+ "answer_confidence": 0.3991,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.9031,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.368,
+ "1": 0.361,
+ "2": 0.2711
+ },
+ "confidence": 0.0083,
+ "answer_confidence": 0.368,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.2943,
+ "confidence": 0.7057,
+ "answer_confidence": 0.7057,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 94,
+ "output_tokens": 0,
+ "state_tokens": 0,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7312,
+ "technical": 0.1851,
+ "other": 0.0838
+ },
+ "confidence": 0.3184,
+ "answer_confidence": 0.7312,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8478,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0137,
+ "1": 0.1248,
+ "2": 0.8615
+ },
+ "confidence": 0.5931,
+ "answer_confidence": 0.8615,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.8255,
+ "confidence": 0.8255,
+ "answer_confidence": 0.8255,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 481,
+ "output_tokens": 0,
+ "state_tokens": 129,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "other",
+ "probabilities": {
+ "refund": 0.142,
+ "technical": 0.3006,
+ "other": 0.5573
+ },
+ "confidence": 0.1222,
+ "answer_confidence": 0.5573,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.2804,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.2652,
+ "1": 0.1891,
+ "2": 0.5456
+ },
+ "confidence": 0.092,
+ "answer_confidence": 0.5456,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0403,
+ "confidence": 0.9597,
+ "answer_confidence": 0.9597,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 118,
+ "output_tokens": 0,
+ "state_tokens": 8,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "other",
+ "probabilities": {
+ "refund": 0.2504,
+ "technical": 0.3345,
+ "other": 0.4151
+ },
+ "confidence": 0.0187,
+ "answer_confidence": 0.4151,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.766,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0394,
+ "1": 0.1552,
+ "2": 0.8054
+ },
+ "confidence": 0.4622,
+ "answer_confidence": 0.8054,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.776,
+ "confidence": 0.776,
+ "answer_confidence": 0.776,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 286,
+ "output_tokens": 0,
+ "state_tokens": 64,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "technical",
+ "probabilities": {
+ "refund": 0.1839,
+ "technical": 0.4141,
+ "other": 0.4019
+ },
+ "confidence": 0.0507,
+ "answer_confidence": 0.4141,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.9808,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.3293,
+ "1": 0.3606,
+ "2": 0.3101
+ },
+ "confidence": 0.0018,
+ "answer_confidence": 0.3606,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.6
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.2948,
+ "confidence": 0.7052,
+ "answer_confidence": 0.7052,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 160,
+ "output_tokens": 0,
+ "state_tokens": 22,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 24,
+ 163
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_zero.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_zero.json
new file mode 100644
index 00000000..9d88f298
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_min_confidence_zero.json
@@ -0,0 +1,565 @@
+{
+ "name": "batch_min_confidence_zero",
+ "states": [
+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ {
+ "subject": "Double charged on invoice 88213",
+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ },
+ [
+ {
+ "role": "user",
+ "content": "Hi, I need help with my invoice."
+ },
+ {
+ "role": "assistant",
+ "content": "Of course, what is wrong?"
+ },
+ {
+ "role": "user",
+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."
+ }
+ ],
+ "",
+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "Thanks, that is all sorted now.",
+ {
+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",
+ "message": "Please cancel my account today."
+ },
+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."
+ ],
+ "questions": {
+ "topic": {
+ "type": "choice",
+ "instructions": "What is the request about?",
+ "criteria": {
+ "refund": "money back, duplicate charge",
+ "technical": "bugs and outages",
+ "other": null
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today"
+ ]
+ },
+ "cancel": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ }
+ },
+ "options": {
+ "min_confidence": 0.0
+ },
+ "state_widths": [
+ 91,
+ 110,
+ 144,
+ 34,
+ 163,
+ 42,
+ 98,
+ 56
+ ],
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7054,
+ "technical": 0.1834,
+ "other": 0.1111
+ },
+ "confidence": 0.2705,
+ "answer_confidence": 0.7054,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8371,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0193,
+ "1": 0.1243,
+ "2": 0.8564
+ },
+ "confidence": 0.574,
+ "answer_confidence": 0.8564,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.7914,
+ "confidence": 0.7914,
+ "answer_confidence": 0.7914,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 265,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.6869,
+ "technical": 0.2079,
+ "other": 0.1052
+ },
+ "confidence": 0.2523,
+ "answer_confidence": 0.6869,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7918,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0204,
+ "1": 0.1675,
+ "2": 0.8121
+ },
+ "confidence": 0.5015,
+ "answer_confidence": 0.8121,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.7602,
+ "confidence": 0.7602,
+ "answer_confidence": 0.7602,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 322,
+ "output_tokens": 0,
+ "state_tokens": 76,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.6718,
+ "technical": 0.2244,
+ "other": 0.1038
+ },
+ "confidence": 0.2375,
+ "answer_confidence": 0.6718,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7672,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0194,
+ "1": 0.194,
+ "2": 0.7866
+ },
+ "confidence": 0.4689,
+ "answer_confidence": 0.7866,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.7234,
+ "confidence": 0.7234,
+ "answer_confidence": 0.7234,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 424,
+ "output_tokens": 0,
+ "state_tokens": 110,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "other",
+ "probabilities": {
+ "refund": 0.3741,
+ "technical": 0.2268,
+ "other": 0.3991
+ },
+ "confidence": 0.0252,
+ "answer_confidence": 0.3991,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.9031,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.368,
+ "1": 0.361,
+ "2": 0.2711
+ },
+ "confidence": 0.0083,
+ "answer_confidence": 0.368,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.2943,
+ "confidence": 0.7057,
+ "answer_confidence": 0.7057,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 94,
+ "output_tokens": 0,
+ "state_tokens": 0,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.7312,
+ "technical": 0.1851,
+ "other": 0.0838
+ },
+ "confidence": 0.3184,
+ "answer_confidence": 0.7312,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8478,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0137,
+ "1": 0.1248,
+ "2": 0.8615
+ },
+ "confidence": 0.5931,
+ "answer_confidence": 0.8615,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.8255,
+ "confidence": 0.8255,
+ "answer_confidence": 0.8255,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 481,
+ "output_tokens": 0,
+ "state_tokens": 129,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "other",
+ "probabilities": {
+ "refund": 0.142,
+ "technical": 0.3006,
+ "other": 0.5573
+ },
+ "confidence": 0.1222,
+ "answer_confidence": 0.5573,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.2804,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.2652,
+ "1": 0.1891,
+ "2": 0.5456
+ },
+ "confidence": 0.092,
+ "answer_confidence": 0.5456,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.0403,
+ "confidence": 0.9597,
+ "answer_confidence": 0.9597,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 118,
+ "output_tokens": 0,
+ "state_tokens": 8,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "other",
+ "probabilities": {
+ "refund": 0.2504,
+ "technical": 0.3345,
+ "other": 0.4151
+ },
+ "confidence": 0.0187,
+ "answer_confidence": 0.4151,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.766,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.0394,
+ "1": 0.1552,
+ "2": 0.8054
+ },
+ "confidence": 0.4622,
+ "answer_confidence": 0.8054,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.776,
+ "confidence": 0.776,
+ "answer_confidence": 0.776,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 286,
+ "output_tokens": 0,
+ "state_tokens": 64,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "topic": {
+ "type": "choice",
+ "choice": "technical",
+ "probabilities": {
+ "refund": 0.1839,
+ "technical": 0.4141,
+ "other": 0.4019
+ },
+ "confidence": 0.0507,
+ "answer_confidence": 0.4141,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 0.9808,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today"
+ },
+ "probabilities": {
+ "0": 0.3293,
+ "1": 0.3606,
+ "2": 0.3101
+ },
+ "confidence": 0.0018,
+ "answer_confidence": 0.3606,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "cancel": {
+ "type": "noul",
+ "noul": 0.2948,
+ "confidence": 0.7052,
+ "answer_confidence": 0.7052,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 160,
+ "output_tokens": 0,
+ "state_tokens": 22,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "runs": [
+ [
+ 24,
+ 163
+ ]
+ ]
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_bad_key.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_bad_key.json
new file mode 100644
index 00000000..7eab72d1
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_bad_key.json
@@ -0,0 +1,45 @@
+{
+ "name": "min_confidence_bad_key",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence map keys must be a bucket like 'choice:3-5' or 'default', got 'choice:3'",
+ "options": {
+ "min_confidence": {
+ "default": 0.5,
+ "choice:3": 0.5
+ }
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_empty_map.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_empty_map.json
new file mode 100644
index 00000000..28fa69e2
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_empty_map.json
@@ -0,0 +1,42 @@
+{
+ "name": "min_confidence_empty_map",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "a min_confidence map must be a non-empty dict of bucket -> float, got {}",
+ "options": {
+ "min_confidence": {}
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_invalid.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_invalid.json
new file mode 100644
index 00000000..5ecd652a
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_invalid.json
@@ -0,0 +1,42 @@
+{
+ "name": "min_confidence_invalid",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "error_type": "ValueError",
+ "error_message": "min_confidence must be a float in [0.0, 1.0], got 1.5",
+ "options": {
+ "min_confidence": 1.5
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_map.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_map.json
new file mode 100644
index 00000000..7958ebe3
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_map.json
@@ -0,0 +1,1525 @@
+{
+ "name": "min_confidence_map",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 24,
+ 34
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 17,
+ 23,
+ 28
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 17,
+ 26
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 24,
+ 34,
+ 0
+ ],
+ [
+ 11,
+ 17,
+ 23,
+ 28
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 26,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ 1.9911494255065918,
+ -1.6601243019104004,
+ -1.7693676948547363,
+ -10000.0
+ ],
+ [
+ -2.449965715408325,
+ 0.2764223515987396,
+ 2.5646214485168457,
+ 1.5527721643447876
+ ],
+ [
+ -2.3436622619628906,
+ 0.30052450299263,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ -1.2292495965957642,
+ 0.15064242482185364,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 4736.37255859375,
+ -3868.318359375
+ ],
+ [
+ 4346.94384765625,
+ -3559.03759765625
+ ],
+ [
+ 4573.56103515625,
+ -3744.32958984375
+ ],
+ [
+ 4845.4169921875,
+ -3968.390625
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8041,
+ "technical": 0.101,
+ "account": 0.0949
+ },
+ "confidence": 0.4261,
+ "answer_confidence": 0.8041,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "urgency": {
+ "type": "score",
+ "score": 2.153,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.0112,
+ "1": 0.0989,
+ "2": 0.6156,
+ "3": 0.2743
+ },
+ "confidence": 0.3273,
+ "answer_confidence": 0.6156,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.7914,
+ "confidence": 0.7914,
+ "answer_confidence": 0.7914,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.7
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.6672,
+ "confidence": 0.6672,
+ "answer_confidence": 0.6672,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.7
+ }
+ },
+ "usage": {
+ "input_tokens": 379,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "choice:3-5": 0.9,
+ "score:3-5": 0.5,
+ "noul:2": 0.7
+ }
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_map_default.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_map_default.json
new file mode 100644
index 00000000..390da6a7
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_map_default.json
@@ -0,0 +1,936 @@
+{
+ "name": "min_confidence_map_default",
+ "state": "Thanks, that is all sorted now.",
+ "serialized_state": "Thanks, that is all sorted now.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 24,
+ 34
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 17,
+ 23,
+ 28
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 17,
+ 26
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282
+ ],
+ [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 8061,
+ 13,
+ 326,
+ 310,
+ 512,
+ 20045,
+ 1024,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 24,
+ 34,
+ 0
+ ],
+ [
+ 11,
+ 17,
+ 23,
+ 28
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 26,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ -1.0131199359893799,
+ -0.6967740058898926,
+ -0.08753645420074463,
+ -10000.0
+ ],
+ [
+ 1.2697174549102783,
+ 0.9968152046203613,
+ 1.8926589488983154,
+ -0.4043006896972656
+ ],
+ [
+ 5.507878303527832,
+ -0.7789955735206604,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ 0.3395328223705292,
+ 0.1490795910358429,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 4654.4765625,
+ -3802.9765625
+ ],
+ [
+ 4617.75,
+ -3776.0673828125
+ ],
+ [
+ 3557.7646484375,
+ -2898.31103515625
+ ],
+ [
+ 4780.2001953125,
+ -3914.19091796875
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "account",
+ "probabilities": {
+ "billing": 0.2571,
+ "technical": 0.3078,
+ "account": 0.4351
+ },
+ "confidence": 0.0224,
+ "answer_confidence": 0.4351,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.55
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.3152,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.2694,
+ "1": 0.2166,
+ "2": 0.4432,
+ "3": 0.0707
+ },
+ "confidence": 0.1108,
+ "answer_confidence": 0.4432,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.55
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.0403,
+ "confidence": 0.9597,
+ "answer_confidence": 0.9597,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.55
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.476,
+ "confidence": 0.524,
+ "answer_confidence": 0.524,
+ "action": {
+ "act_probability": 1.0
+ },
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.55
+ }
+ },
+ "usage": {
+ "input_tokens": 183,
+ "output_tokens": 0,
+ "state_tokens": 8,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": {
+ "default": 0.55
+ }
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_scalar.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_scalar.json
new file mode 100644
index 00000000..6b87ebce
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_scalar.json
@@ -0,0 +1,1519 @@
+{
+ "name": "min_confidence_scalar",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 24,
+ 34
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 17,
+ 23,
+ 28
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 17,
+ 26
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 24,
+ 34,
+ 0
+ ],
+ [
+ 11,
+ 17,
+ 23,
+ 28
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 26,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ 1.9911494255065918,
+ -1.6601243019104004,
+ -1.7693676948547363,
+ -10000.0
+ ],
+ [
+ -2.449965715408325,
+ 0.2764223515987396,
+ 2.5646214485168457,
+ 1.5527721643447876
+ ],
+ [
+ -2.3436622619628906,
+ 0.30052450299263,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ -1.2292495965957642,
+ 0.15064242482185364,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 4736.37255859375,
+ -3868.318359375
+ ],
+ [
+ 4346.94384765625,
+ -3559.03759765625
+ ],
+ [
+ 4573.56103515625,
+ -3744.32958984375
+ ],
+ [
+ 4845.4169921875,
+ -3968.390625
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8041,
+ "technical": 0.101,
+ "account": 0.0949
+ },
+ "confidence": 0.4261,
+ "answer_confidence": 0.8041,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "urgency": {
+ "type": "score",
+ "score": 2.153,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.0112,
+ "1": 0.0989,
+ "2": 0.6156,
+ "3": 0.2743
+ },
+ "confidence": 0.3273,
+ "answer_confidence": 0.6156,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.7914,
+ "confidence": 0.7914,
+ "answer_confidence": 0.7914,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.6672,
+ "confidence": 0.6672,
+ "answer_confidence": 0.6672,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.6
+ }
+ },
+ "usage": {
+ "input_tokens": 379,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": 0.6
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_zero.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_zero.json
new file mode 100644
index 00000000..ecfc4299
--- /dev/null
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_min_confidence_zero.json
@@ -0,0 +1,1519 @@
+{
+ "name": "min_confidence_zero",
+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this?",
+ "criteria": {
+ "billing": "invoices, payments, double charges, refunds",
+ "technical": "bugs, outages, integration errors",
+ "account": "logins, seats, plan changes"
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How urgent is this request?",
+ "criteria": [
+ "not urgent",
+ "this week",
+ "today",
+ "drop everything"
+ ]
+ },
+ "churn_risk": {
+ "type": "noul",
+ "instructions": "The customer threatens to cancel the account."
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "criteria": {
+ "false": "a template reply resolves it",
+ "true": "a person must intervene"
+ }
+ }
+ },
+ "per_question": [
+ {
+ "id": "department",
+ "type": "choice",
+ "qtype": 0,
+ "instructions": "Which department should handle this?",
+ "head_text": "choice question: Which department should handle this?",
+ "options": [
+ "billing: invoices, payments, double charges, refunds",
+ "technical: bugs, outages, integration errors",
+ "account: logins, seats, plan changes"
+ ],
+ "input_ids": [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 24,
+ 34
+ ]
+ },
+ {
+ "id": "urgency",
+ "type": "score",
+ "qtype": 1,
+ "instructions": "How urgent is this request?",
+ "head_text": "score question: How urgent is this request?",
+ "options": [
+ "level 0: not urgent",
+ "level 1: this week",
+ "level 2: today",
+ "level 3: drop everything"
+ ],
+ "input_ids": [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 11,
+ 17,
+ 23,
+ 28
+ ]
+ },
+ {
+ "id": "churn_risk",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "The customer threatens to cancel the account.",
+ "head_text": "noul question: The customer threatens to cancel the account.",
+ "options": [
+ "false: no, the statement does not hold",
+ "true: yes, the statement holds"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 14,
+ 24
+ ]
+ },
+ {
+ "id": "needs_human",
+ "type": "noul",
+ "qtype": 2,
+ "instructions": "This requires a human agent rather than an automated reply.",
+ "head_text": "noul question: This requires a human agent rather than an automated reply.",
+ "options": [
+ "false: a template reply resolves it",
+ "true: a person must intervene"
+ ],
+ "input_ids": [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ "markers": [
+ 17,
+ 26
+ ]
+ }
+ ],
+ "batch": {
+ "input_ids": [
+ [
+ 50281,
+ 22122,
+ 1953,
+ 27,
+ 6758,
+ 7811,
+ 943,
+ 6016,
+ 436,
+ 32,
+ 50282,
+ 50284,
+ 33484,
+ 27,
+ 29838,
+ 1271,
+ 13,
+ 10762,
+ 13,
+ 4021,
+ 7260,
+ 13,
+ 1275,
+ 41748,
+ 50284,
+ 7681,
+ 27,
+ 19775,
+ 13,
+ 562,
+ 1131,
+ 13,
+ 9554,
+ 6332,
+ 50284,
+ 2395,
+ 27,
+ 2412,
+ 968,
+ 13,
+ 13512,
+ 13,
+ 2098,
+ 2544,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282
+ ],
+ [
+ 50281,
+ 18891,
+ 1953,
+ 27,
+ 1359,
+ 21007,
+ 310,
+ 436,
+ 2748,
+ 32,
+ 50282,
+ 50284,
+ 1268,
+ 470,
+ 27,
+ 417,
+ 21007,
+ 50284,
+ 1268,
+ 337,
+ 27,
+ 436,
+ 2129,
+ 50284,
+ 1268,
+ 374,
+ 27,
+ 3063,
+ 50284,
+ 1268,
+ 495,
+ 27,
+ 5926,
+ 3253,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 380,
+ 7731,
+ 37654,
+ 281,
+ 14002,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 642,
+ 13,
+ 253,
+ 3908,
+ 1057,
+ 417,
+ 2186,
+ 50284,
+ 2032,
+ 27,
+ 4754,
+ 13,
+ 253,
+ 3908,
+ 6556,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ],
+ [
+ 50281,
+ 79,
+ 3941,
+ 1953,
+ 27,
+ 831,
+ 4419,
+ 247,
+ 1966,
+ 5570,
+ 2581,
+ 685,
+ 271,
+ 16644,
+ 12252,
+ 15,
+ 50282,
+ 50284,
+ 3221,
+ 27,
+ 247,
+ 7646,
+ 12252,
+ 501,
+ 14503,
+ 352,
+ 50284,
+ 2032,
+ 27,
+ 247,
+ 1436,
+ 1364,
+ 32014,
+ 50282,
+ 18961,
+ 27,
+ 18603,
+ 6636,
+ 327,
+ 45156,
+ 854,
+ 3507,
+ 1012,
+ 187,
+ 187,
+ 12764,
+ 13,
+ 309,
+ 369,
+ 47045,
+ 7019,
+ 436,
+ 1770,
+ 323,
+ 253,
+ 1294,
+ 2098,
+ 285,
+ 253,
+ 1273,
+ 4179,
+ 556,
+ 417,
+ 644,
+ 23005,
+ 264,
+ 15,
+ 831,
+ 310,
+ 253,
+ 2626,
+ 673,
+ 309,
+ 452,
+ 3542,
+ 275,
+ 15,
+ 1310,
+ 352,
+ 310,
+ 417,
+ 4229,
+ 3063,
+ 309,
+ 717,
+ 476,
+ 68,
+ 3485,
+ 253,
+ 2395,
+ 15,
+ 50282,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283,
+ 50283
+ ]
+ ],
+ "attention_mask": [
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ]
+ ],
+ "marker_pos": [
+ [
+ 11,
+ 24,
+ 34,
+ 0
+ ],
+ [
+ 11,
+ 17,
+ 23,
+ 28
+ ],
+ [
+ 14,
+ 24,
+ 0,
+ 0
+ ],
+ [
+ 17,
+ 26,
+ 0,
+ 0
+ ]
+ ],
+ "marker_mask": [
+ [
+ true,
+ true,
+ true,
+ false
+ ],
+ [
+ true,
+ true,
+ true,
+ true
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ],
+ [
+ true,
+ true,
+ false,
+ false
+ ]
+ ],
+ "qtype": [
+ 0,
+ 1,
+ 2,
+ 2
+ ]
+ },
+ "marker_axis_padded": false,
+ "logits": [
+ [
+ 1.9911494255065918,
+ -1.6601243019104004,
+ -1.7693676948547363,
+ -10000.0
+ ],
+ [
+ -2.449965715408325,
+ 0.2764223515987396,
+ 2.5646214485168457,
+ 1.5527721643447876
+ ],
+ [
+ -2.3436622619628906,
+ 0.30052450299263,
+ -10000.0,
+ -10000.0
+ ],
+ [
+ -1.2292495965957642,
+ 0.15064242482185364,
+ -10000.0,
+ -10000.0
+ ]
+ ],
+ "act_logits": [
+ [
+ 4736.37255859375,
+ -3868.318359375
+ ],
+ [
+ 4346.94384765625,
+ -3559.03759765625
+ ],
+ [
+ 4573.56103515625,
+ -3744.32958984375
+ ],
+ [
+ 4845.4169921875,
+ -3968.390625
+ ]
+ ],
+ "result": {
+ "model": "laya-rl-agent",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8041,
+ "technical": 0.101,
+ "account": 0.0949
+ },
+ "confidence": 0.4261,
+ "answer_confidence": 0.8041,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "urgency": {
+ "type": "score",
+ "score": 2.153,
+ "legend": {
+ "0": "not urgent",
+ "1": "this week",
+ "2": "today",
+ "3": "drop everything"
+ },
+ "probabilities": {
+ "0": 0.0112,
+ "1": 0.0989,
+ "2": 0.6156,
+ "3": 0.2743
+ },
+ "confidence": 0.3273,
+ "answer_confidence": 0.6156,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "churn_risk": {
+ "type": "noul",
+ "noul": 0.7914,
+ "confidence": 0.7914,
+ "answer_confidence": 0.7914,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.6672,
+ "confidence": 0.6672,
+ "answer_confidence": 0.6672,
+ "action": {
+ "act_probability": 1.0
+ },
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ },
+ "usage": {
+ "input_tokens": 379,
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "options": {
+ "min_confidence": 0.0
+ }
+}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/index.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/index.json
index eb70e281..8b7bcfdf 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/index.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/index.json
@@ -337,6 +337,42 @@
"expects_error": false,
"kind": "batch"
},
+ {
+ "name": "batch_min_confidence_scalar",
+ "file": "case_batch_min_confidence_scalar.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_zero",
+ "file": "case_batch_min_confidence_zero.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_map",
+ "file": "case_batch_min_confidence_map.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_map_default",
+ "file": "case_batch_min_confidence_map_default.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_empty_questions",
+ "file": "case_batch_min_confidence_empty_questions.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_empty_states",
+ "file": "case_batch_min_confidence_empty_states.json",
+ "expects_error": false,
+ "kind": "batch"
+ },
{
"name": "batch_none_state",
"file": "case_batch_none_state.json",
@@ -348,5 +384,58 @@
"file": "case_batch_options_exceed_head.json",
"expects_error": true,
"kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_invalid",
+ "file": "case_batch_min_confidence_invalid.json",
+ "expects_error": true,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_before_none_state",
+ "file": "case_batch_min_confidence_before_none_state.json",
+ "expects_error": true,
+ "kind": "batch"
+ },
+ {
+ "name": "batch_min_confidence_bad_key",
+ "file": "case_batch_min_confidence_bad_key.json",
+ "expects_error": true,
+ "kind": "batch"
+ },
+ {
+ "name": "min_confidence_scalar",
+ "file": "case_min_confidence_scalar.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_zero",
+ "file": "case_min_confidence_zero.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_map",
+ "file": "case_min_confidence_map.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_map_default",
+ "file": "case_min_confidence_map_default.json",
+ "expects_error": false
+ },
+ {
+ "name": "min_confidence_invalid",
+ "file": "case_min_confidence_invalid.json",
+ "expects_error": true
+ },
+ {
+ "name": "min_confidence_empty_map",
+ "file": "case_min_confidence_empty_map.json",
+ "expects_error": true
+ },
+ {
+ "name": "min_confidence_bad_key",
+ "file": "case_min_confidence_bad_key.json",
+ "expects_error": true
}
]
diff --git a/laya-dotnet/tools/confidence_cases.py b/laya-dotnet/tools/confidence_cases.py
new file mode 100644
index 00000000..386c57db
--- /dev/null
+++ b/laya-dotnet/tools/confidence_cases.py
@@ -0,0 +1,238 @@
+"""Synthetic inputs for the model-free abstention-gate probe, `routing/confidence_probe.json`.
+
+`dump_routing_golden.build_confidence_probe` feeds each of these to Python's own
+`laya/confidence.py` and records what it returned, so a port is checked against the shipping
+code on the same inputs without loading a model. This list is SDK-neutral: it is the shared
+corpus for the .NET and Rust ports, and each port keeps a verbatim copy.
+
+Everything here is representable by a typed SDK answer (a choice with ordered probabilities, a
+score with ordered probabilities, a noul with none), except the jev cases flagged `typed: false`,
+which exist to pin Python's refusal of shapes a typed answer cannot hold (a boolean or text
+"probability", a non-integer score level). A threshold is always a float: JSON `0.5`, never `1`.
+
+Non-finite floats are written by the dumper as the strings "nan", "inf" and "-inf"; a missing
+`answer_confidence` or `confidence` key is a field the answer did not report, and a typed port
+reads it the same way as NaN (no usable number).
+"""
+import random
+
+NAN = float("nan")
+INF = float("inf")
+
+# ---------------------------------------------------------------------------------------------
+# check_min_confidence: scalar thresholds. Only floats, since a typed SDK takes a float.
+# ---------------------------------------------------------------------------------------------
+SCALAR_THRESHOLDS = [
+ 0.0, 1.0, 0.5, 0.35, 1e-09, 0.9999999999, 1e-05,
+ -0.1, 1.5, -1e-05, 1.0000000001, 2.0, 100.0, 1e16, 1.2345678901234567e-07,
+ NAN, INF, float("-inf"),
+]
+
+# ---------------------------------------------------------------------------------------------
+# check_min_confidence_map: ordered (key, value) pairs, so an error names the first offender.
+# ---------------------------------------------------------------------------------------------
+_ALL_BUCKETS = ["%s:%s" % (qt, size) for qt in ("choice", "score", "noul") for size in ("2", "3-5", "6-10", "11+")]
+
+MAP_THRESHOLDS = [
+ [],
+ [("default", 0.4)],
+ [("choice:2", 0.7), ("default", 0.4)],
+ [("choice:3-5", 0.5)],
+ [("score:11+", 0.0), ("noul:2", 1.0)],
+ [(key, round(0.05 * (i + 1), 2)) for i, key in enumerate(_ALL_BUCKETS)] + [("default", 0.5)],
+ [("choice:2", 1.5)],
+ [("choice:2", NAN)],
+ [("choice:2", -0.1)],
+ [("choice:2", INF)],
+ [("bogus", 0.5)],
+ [("Choice:2", 0.5)],
+ [("choice:3", 0.5)],
+ [("choice:2", 0.5), ("bogus", 0.5)],
+ [("choice:2", 1.5), ("bogus", 0.1)],
+ [("bogus", 1.5), ("choice:2", 0.1)],
+ [("choice:2", 0.5), ("choice:3-5", 2.0), ("nope", 0.1)],
+ [("it's", 0.5)],
+ [("say \"hi\"", 0.5)],
+ [("it's \"x\"", 0.5)],
+ [("tab\there", 0.5)],
+ [("café", 0.5)],
+ [("", 0.5)],
+ [("default", 0.5), ("default", 0.25)],
+]
+
+
+def _choice(*probs, ac=None, conf=None):
+ out = {"type": "choice", "probabilities": {chr(ord("a") + i): p for i, p in enumerate(probs)}}
+ _fields(out, ac, conf)
+ return out
+
+
+def _score(*probs, ac=None, conf=None):
+ out = {"type": "score", "probabilities": {str(i): p for i, p in enumerate(probs)}}
+ _fields(out, ac, conf)
+ return out
+
+
+def _noul(ac=None, conf=None, p=0.9):
+ out = {"type": "noul", "noul": p}
+ _fields(out, ac, conf)
+ return out
+
+
+def _fields(out, ac, conf):
+ if ac is not None:
+ out["answer_confidence"] = ac
+ if conf is not None:
+ out["confidence"] = conf
+
+
+def _uniform(kind, k, ac=None, conf=None):
+ return (_choice if kind == "choice" else _score)(*([round(1.0 / k, 4)] * k), ac=ac, conf=conf)
+
+
+# ---------------------------------------------------------------------------------------------
+# _option_bucket: every type at every option count the bands distinguish.
+# ---------------------------------------------------------------------------------------------
+BUCKET_ANSWERS = (
+ [_uniform("choice", k) for k in (1, 2, 3, 5, 6, 10, 11, 13)]
+ + [_uniform("score", k) for k in (1, 2, 3, 5, 6, 10, 11, 13)]
+ + [_noul()]
+ + [{"type": "choice", "probabilities": {}}, {"type": "score", "probabilities": {}}]
+)
+
+# ---------------------------------------------------------------------------------------------
+# resolve_min_confidence: a validated map against answers of each bucket, and an unnamed one.
+# ---------------------------------------------------------------------------------------------
+RESOLVE_MAPS = [
+ {"choice:2": 0.7, "default": 0.4},
+ {"choice:3-5": 0.5},
+ {"default": 0.25},
+ {"score:6-10": 0.8, "noul:2": 0.3, "choice:11+": 0.6},
+]
+RESOLVE_ANSWERS = [_uniform("choice", 2), _uniform("choice", 4), _uniform("choice", 8), _uniform("choice", 12),
+ _uniform("score", 2), _uniform("score", 4), _uniform("score", 8), _uniform("score", 12),
+ _noul(), {"type": "choice", "probabilities": {}}]
+
+# ---------------------------------------------------------------------------------------------
+# apply_confidence_gate: a result with these answers, gated at each threshold. A threshold is
+# None (unset: nothing is written), a float, or a map.
+# ---------------------------------------------------------------------------------------------
+GATE_ANSWERS = {
+ "choice2": _choice(0.6, 0.4, ac=0.6, conf=0.029),
+ "choice4": _choice(0.45, 0.25, 0.2, 0.1, ac=0.45, conf=0.12),
+ "choice8": _uniform("choice", 8, ac=0.3, conf=0.05),
+ "choice12": _uniform("choice", 12, ac=0.15, conf=0.02),
+ "score2": _score(0.7, 0.3, ac=0.7, conf=0.1),
+ "score4": _score(0.1, 0.5, 0.3, 0.1, ac=0.5, conf=0.2),
+ "score8": _uniform("score", 8, ac=0.125, conf=0.01),
+ "noul": _noul(ac=0.8, conf=0.8, p=0.8),
+ "at_half": _choice(0.5, 0.5, ac=0.5, conf=0.0),
+ "fallback_conf": _choice(0.5, 0.5, conf=0.3),
+ "nan_answer_conf": _choice(0.5, 0.5, ac=NAN, conf=0.7),
+ "inf_answer_conf": _choice(0.5, 0.5, ac=INF, conf=0.2),
+ "unusable_both": _choice(0.5, 0.5, ac=NAN, conf=INF),
+ "no_confidence": _choice(0.5, 0.5),
+ "empty_choice": {"type": "choice", "probabilities": {}, "answer_confidence": 0.4},
+}
+
+GATE_THRESHOLDS = [
+ None,
+ 0.0,
+ 1.0,
+ 0.5,
+ 0.35,
+ 0.2,
+ {"default": 0.4},
+ {"choice:2": 0.7, "default": 0.4},
+ {"choice:3-5": 0.5},
+ {"choice:2": 0.7, "choice:3-5": 0.5, "choice:6-10": 0.4, "choice:11+": 0.1,
+ "score:2": 0.8, "score:3-5": 0.6, "score:6-10": 0.2, "noul:2": 0.9},
+ {"noul:2": 0.0, "score:3-5": 1.0},
+]
+
+# The same answers gated again and again, as one reused payload is (#910): each step's verdict must
+# depend only on that step's threshold, never on an earlier one.
+REUSE_SEQUENCES = [
+ [0.9, 0.0, 0.2, 1.0, 0.5],
+ [{"choice:2": 0.9, "default": 0.5}, 0.0, {"choice:3-5": 0.9}, 0.35],
+]
+
+
+# ---------------------------------------------------------------------------------------------
+# jev_confidence
+# ---------------------------------------------------------------------------------------------
+def _jev_corpus():
+ cases = [
+ ("choice-confident", _choice(0.9, 0.1)),
+ ("choice-even", _choice(0.5, 0.5)),
+ ("choice-three", _choice(0.7, 0.2, 0.1)),
+ ("choice-differs-from-entropy", _choice(0.7, 0.1, 0.1, 0.1)),
+ ("choice-unnormalised", _choice(2.0, 1.0, 1.0)),
+ ("choice-zero-total", _choice(0.0, 0.0, 0.0)),
+ ("choice-single", _choice(0.3)),
+ ("choice-single-zero", _choice(0.0)),
+ ("choice-zero-among-others", _choice(0.0, 1.0)),
+ ("score-peaked", _score(0.0, 0.95, 0.05)),
+ ("score-jevcompat-example", _score(0.0, 0.95, 0.05)),
+ ("score-even", _score(0.25, 0.25, 0.25, 0.25)),
+ ("score-edge-mode", _score(0.8, 0.1, 0.1)),
+ ("score-zero-total", _score(0.0, 0.0, 0.0, 0.0)),
+ ("score-single", _score(0.6)),
+ ("score-two", _score(0.3, 0.7)),
+ ("score-bimodal", _score(0.45, 0.05, 0.05, 0.45)),
+ ("score-tied-modes", _score(0.4, 0.4, 0.2)),
+ ("score-levels-in-numeric-order",
+ {"type": "score", "probabilities": {str(i): (1.0 if i == 10 else 0.0) for i in range(11)}}),
+ ("noul", {"type": "noul", "noul": 0.9}),
+ ("empty-choice-probabilities", {"type": "choice", "probabilities": {}}),
+ ("empty-score-probabilities", {"type": "score", "probabilities": {}}),
+ ("nan-probability", _choice(NAN, 0.5)),
+ ("infinite-probability", _choice(INF, 0.5)),
+ ("negative-infinite-probability", _choice(float("-inf"), 0.5)),
+ ("negative-probability", _choice(2.0, -1.0)),
+ ("mixed-sign-argmax-flips", _choice(0.5, -1.5)),
+ ("all-negative-probabilities", _choice(-0.9, -0.1)),
+ ("negative-total-near-zero", _choice(-0.61, 0.6)),
+ ("negative-score-level", _score(1.5, -0.5, 0.0)),
+ ("nan-score-level", _score(0.5, NAN)),
+ ("negative-single", _choice(-0.5)),
+ ("nan-single", _choice(NAN)),
+ ]
+ out = [{"name": name, "answer": answer, "typed": True} for name, answer in cases]
+ # Shapes a typed answer cannot hold: kept so the corpus records that Python refuses them.
+ out += [
+ {"name": "unknown-type", "answer": {"type": "weird", "probabilities": {"a": 1.0}}, "typed": False},
+ {"name": "missing-probabilities", "answer": {"type": "choice"}, "typed": False},
+ {"name": "non-numeric-probability", "answer": _choice(0.5, "high"), "typed": False},
+ {"name": "bool-probability", "answer": _choice(True, False), "typed": False},
+ {"name": "bool-among-numbers", "answer": _choice(True, 0.4), "typed": False},
+ {"name": "non-integer-score-level", "answer": {"type": "score", "probabilities": {"low": 0.5, "high": 0.5}},
+ "typed": False},
+ ]
+ # CPython 3.12's float sum() is compensated (Neumaier), so a plain left-to-right sum differs here
+ # in the last bits of the result; the first two are the textbook case, [0.1] * 10.
+ sums = [
+ ("sum-differs-choice-tenths", "choice", [0.1] * 10),
+ ("sum-differs-score-tenths", "score", [0.1] * 10),
+ ("sum-differs-choice-hundredths", "choice", [0.01] * 100),
+ ("sum-differs-choice-7", "choice", [0.6946, 0.03, 0.3634, 0.67, 0.2649, 0.72, 0.6497]),
+ ("sum-differs-choice-5", "choice", [0.7133, 0.3416, 0.7, 0.8634, 0.6]),
+ ("sum-differs-choice-10", "choice", [0.6, 0.1, 0.3, 0.7918, 0.94, 0.7, 0.9983, 0.5, 0.8, 0.6]),
+ ("sum-differs-choice-16", "choice", [0.9, 0.11, 0.4, 0.84, 0.1412, 0.8523, 0.45, 0.69, 0.48, 0.5, 0.1126,
+ 0.9598, 0.94, 0.18, 0.2723, 0.87]),
+ ("sum-differs-score-7", "score", [0.4, 0.4528, 0.2, 0.1, 0.93, 0.34, 0.89]),
+ ("sum-differs-score-3", "score", [0.453, 0.4, 0.0333]),
+ ]
+ for name, kind, probs in sums:
+ out.append({"name": name, "answer": (_choice if kind == "choice" else _score)(*probs), "typed": True})
+ # A deterministic spread of distributions, to pin the arithmetic beyond the hand-picked ones.
+ rng = random.Random(1361)
+ for i in range(24):
+ k = rng.choice([2, 3, 4, 5, 8, 12])
+ probs = [round(rng.random(), 4) for _ in range(k)]
+ out.append({"name": "random-choice-%02d" % i, "answer": _choice(*probs), "typed": True})
+ out.append({"name": "random-score-%02d" % i, "answer": _score(*probs), "typed": True})
+ return out
+
+
+JEV_CASES = _jev_corpus()
diff --git a/laya-dotnet/tools/dump_golden.py b/laya-dotnet/tools/dump_golden.py
index 2862c5f0..bb2e2ddd 100644
--- a/laya-dotnet/tools/dump_golden.py
+++ b/laya-dotnet/tools/dump_golden.py
@@ -47,6 +47,8 @@ from golden_cases import ( # noqa: E402
ERROR_CASES,
LONG_CASES,
LONG_ERROR_CASES,
+ MIN_CONFIDENCE_CASES,
+ MIN_CONFIDENCE_ERROR_CASES,
SHORTLIST_E2E_CASES,
)
@@ -307,9 +309,10 @@ def describe_questions(agent: Agent, state, questions: dict) -> list:
def record_case(agent: Agent, case: dict) -> dict:
state, questions = case["state"], case["questions"]
per_question = describe_questions(agent, state, questions)
- result = agent.system_one(state, questions)
+ options = case.get("options")
+ result = agent.system_one(state, questions, **(options or {}))
last = agent.model.last
- return {
+ rec = {
"name": case["name"],
"state": state,
"serialized_state": serialize_state(state),
@@ -327,6 +330,9 @@ def record_case(agent: Agent, case: dict) -> dict:
"act_logits": last["act_logits"].tolist(),
"result": result,
}
+ if options:
+ rec["options"] = options
+ return rec
def record_shortlist_case(agent: Agent, case: dict) -> dict:
@@ -380,16 +386,20 @@ def record_shortlist_case(agent: Agent, case: dict) -> dict:
def record_error_case(agent: Agent, case: dict) -> dict:
+ options = case.get("options")
try:
- agent.system_one(case["state"], case["questions"])
+ agent.system_one(case["state"], case["questions"], **(options or {}))
except ValueError as e:
- return {
+ rec = {
"name": case["name"],
"state": case["state"],
"questions": case["questions"],
"error_type": "ValueError",
"error_message": str(e),
}
+ if options:
+ rec["options"] = options
+ return rec
raise SystemExit("case %r was expected to raise ValueError but did not" % case["name"])
@@ -613,6 +623,15 @@ def run_checkpoint(onnx_dir: str, out_dir: str) -> int:
"expects_error": expects_error, "kind": "batch",
})
+ # Appended after the batch cases, so index.json's diff stays a pure append. No `kind`: these are
+ # standard system_one cases whose recorded `options` the parity tests forward.
+ for cases, expects_error in ((MIN_CONFIDENCE_CASES, False), (MIN_CONFIDENCE_ERROR_CASES, True)):
+ for case in cases:
+ rec = (record_error_case if expects_error else record_case)(agent, case)
+ dump(rec, os.path.join(out_dir, "case_%s.json" % case["name"]))
+ index.append({"name": case["name"], "file": "case_%s.json" % case["name"],
+ "expects_error": expects_error})
+
dump(index, os.path.join(out_dir, "index.json"))
print("done: %d cases -> %s" % (len(index), out_dir))
return 0
diff --git a/laya-dotnet/tools/dump_routing_golden.py b/laya-dotnet/tools/dump_routing_golden.py
index d2f84f76..d1340f57 100644
--- a/laya-dotnet/tools/dump_routing_golden.py
+++ b/laya-dotnet/tools/dump_routing_golden.py
@@ -22,6 +22,11 @@ corpus from `laya-dotnet/tools/routing_cases.py` and writes one JSON file per pr
its ranking/cosine/tie-break logic against this file). NaN/Inf
values in a matrix are written as the strings "nan"/"inf"/
"-inf" (see meta.json) because JSON has no literal for them.
+ confidence_probe.json laya.confidence's own answers over the synthetic inputs of
+ tools/confidence_cases.py: threshold validation (check_min_confidence and
+ _map, with the error text), _option_bucket, resolve_min_confidence,
+ apply_confidence_gate (incl. reused payloads, #910) and jev_confidence.
+ Non-finite floats are written as "nan"/"inf"/"-inf" strings.
hashing_embedder_probe.json Exact laya-dotnet/tools/hashing_embedder.embed() vectors for a fixed probe
list of strings, including non-ASCII and astral input.
sample_inputs.json The combined routing sample's fixed inputs (detection states,
@@ -41,6 +46,7 @@ given -- a content diff, not a mtime check, so re-running with no corpus changes
silent no-op.
"""
import argparse
+import copy
import json
import math
import os
@@ -72,8 +78,10 @@ from routing_cases import ( # noqa: E402
SHORTLIST_CASES,
)
+import confidence_cases as cc # noqa: E402
import hashing_embedder # noqa: E402
+from laya import confidence # noqa: E402
from laya.email import clean_email_body, email_state # noqa: E402
from laya.lang import analyse, detect_script, latin_profile, script_profile, state_text # noqa: E402
from laya.router import Router # noqa: E402
@@ -220,6 +228,78 @@ def build_route_batch_probe():
return {"cases": cases, "errors": errors}
+# ============================================================================ confidence_probe
+def _checked(fn, arg):
+ """`{"valid": True, "result": ...}` or `{"valid": False, "error_type", "error_message"}`."""
+ try:
+ return {"valid": True, "result": fn(arg)}
+ except ValueError as e:
+ return {"valid": False, "error_type": type(e).__name__, "error_message": str(e)}
+
+
+def _gate_verdict(answers):
+ """What the gate wrote on each answer: only the keys that are present."""
+ return {
+ qid: {k: a[k] for k in ("low_confidence", "abstention", "abstention_threshold") if k in a}
+ for qid, a in answers.items()
+ }
+
+
+def _gate(answers, threshold):
+ """apply_confidence_gate exactly as ONNXAgent.predict_batch calls it: validate, then gate."""
+ mc = confidence.check_min_confidence(threshold) if threshold is not None else None
+ confidence.apply_confidence_gate([{"answers": answers}], mc)
+
+
+def build_confidence_probe():
+ scalars = [dict(value=v, **_checked(confidence.check_min_confidence, v)) for v in cc.SCALAR_THRESHOLDS]
+
+ maps = []
+ for pairs in cc.MAP_THRESHOLDS:
+ # Python's dict keeps one entry per key, at its first position, holding its last value.
+ checked = _checked(confidence.check_min_confidence, dict(pairs))
+ if checked["valid"]:
+ checked["result"] = [list(item) for item in checked["result"].items()]
+ maps.append(dict(pairs=[list(pair) for pair in pairs], **checked))
+
+ buckets = [{"answer": a, "bucket": confidence._option_bucket(a)} for a in cc.BUCKET_ANSWERS]
+
+ resolve = [
+ {"thresholds": m, "answer": a, "result": confidence.resolve_min_confidence(a, m)}
+ for m in cc.RESOLVE_MAPS for a in cc.RESOLVE_ANSWERS
+ ]
+
+ gate_cases = []
+ for threshold in cc.GATE_THRESHOLDS:
+ answers = copy.deepcopy(cc.GATE_ANSWERS)
+ _gate(answers, threshold)
+ gate_cases.append({"threshold": threshold, "expected": _gate_verdict(answers)})
+
+ reuse = []
+ for steps in cc.REUSE_SEQUENCES:
+ answers = copy.deepcopy(cc.GATE_ANSWERS)
+ recorded = []
+ for threshold in steps:
+ _gate(answers, threshold)
+ recorded.append({"threshold": threshold, "expected": _gate_verdict(answers)})
+ reuse.append({"steps": recorded})
+
+ jev = [dict(case, result=confidence.jev_confidence(case["answer"])) for case in cc.JEV_CASES]
+
+ return {
+ "bucket_keys": sorted(confidence.bucket_keys()),
+ "gate_states": list(confidence.GATE_STATES),
+ "scalars": scalars,
+ "maps": maps,
+ "buckets": buckets,
+ "resolve": resolve,
+ "gate_answers": cc.GATE_ANSWERS,
+ "gate": gate_cases,
+ "reuse": reuse,
+ "jev": jev,
+ }
+
+
# ============================================================================ email_probe
def build_email_probe():
clean = []
@@ -390,6 +470,7 @@ def main() -> int:
lang_probe = build_lang_probe()
route_probe = build_route_probe()
route_batch_probe = build_route_batch_probe()
+ confidence_probe = build_confidence_probe()
email_probe = build_email_probe()
shortlist_probe = build_shortlist_probe()
hashing_probe = build_hashing_embedder_probe()
@@ -412,6 +493,13 @@ def main() -> int:
"route_probe": len(route_probe),
"route_batch_probe": len(route_batch_probe["cases"]),
"route_batch_probe_errors": len(route_batch_probe["errors"]),
+ "confidence_probe_scalars": len(confidence_probe["scalars"]),
+ "confidence_probe_maps": len(confidence_probe["maps"]),
+ "confidence_probe_buckets": len(confidence_probe["buckets"]),
+ "confidence_probe_resolve": len(confidence_probe["resolve"]),
+ "confidence_probe_gate": len(confidence_probe["gate"]),
+ "confidence_probe_reuse": len(confidence_probe["reuse"]),
+ "confidence_probe_jev": len(confidence_probe["jev"]),
"email_probe_clean": len(email_probe["clean"]),
"email_probe_state": len(email_probe["state"]),
"shortlist_probe": len(shortlist_probe),
@@ -425,6 +513,9 @@ def main() -> int:
results["route_batch_probe.json"] = dump(
route_batch_probe, os.path.join(out_dir, "route_batch_probe.json"), args.force
)
+ results["confidence_probe.json"] = dump(
+ _encode_floats(confidence_probe), os.path.join(out_dir, "confidence_probe.json"), args.force
+ )
results["email_probe.json"] = dump(email_probe, os.path.join(out_dir, "email_probe.json"), args.force)
results["shortlist_probe.json"] = dump(shortlist_probe, os.path.join(out_dir, "shortlist_probe.json"), args.force)
results["hashing_embedder_probe.json"] = dump(
diff --git a/laya-dotnet/tools/golden_cases.py b/laya-dotnet/tools/golden_cases.py
index 80792ddd..8b08a0b8 100644
--- a/laya-dotnet/tools/golden_cases.py
+++ b/laya-dotnet/tools/golden_cases.py
@@ -503,6 +503,20 @@ BATCH_CASES = [
"questions": {"cancel": BATCH_QUESTIONS["cancel"]}, "options": {"batch_size": 2}},
{"name": "batch_empty_questions_with_options", "states": BATCH_STATES[:3], "questions": {},
"options": {"batch_size": 2, "sort_by_length": True}},
+ # The abstention gate (`min_confidence`): batch-wide, applied to every answer of every state.
+ {"name": "batch_min_confidence_scalar", "states": BATCH_MIXED_STATES, "questions": BATCH_QUESTIONS,
+ "options": {"min_confidence": 0.6}},
+ {"name": "batch_min_confidence_zero", "states": BATCH_MIXED_STATES, "questions": BATCH_QUESTIONS,
+ "options": {"min_confidence": 0.0}},
+ {"name": "batch_min_confidence_map", "states": BATCH_SORT_STATES[:7], "questions": BATCH_QUESTIONS,
+ "options": {"batch_size": 3, "sort_by_length": True,
+ "min_confidence": {"choice:3-5": 0.8, "score:3-5": 0.5, "noul:2": 0.7}}},
+ {"name": "batch_min_confidence_map_default", "states": BATCH_STATES[:3], "questions": BATCH_QUESTIONS,
+ "options": {"min_confidence": {"default": 0.55}}},
+ {"name": "batch_min_confidence_empty_questions", "states": BATCH_STATES[:3], "questions": {},
+ "options": {"min_confidence": 0.5}},
+ {"name": "batch_min_confidence_empty_states", "states": [], "questions": BATCH_QUESTIONS,
+ "options": {"min_confidence": 0.5}},
]
BATCH_ERROR_CASES = [
@@ -510,4 +524,33 @@ BATCH_ERROR_CASES = [
"error_type": "TypeError"},
{"name": "batch_options_exceed_head", "states": ["short", QUICKSTART_STATE],
"questions": ERROR_CASES[0]["questions"], "error_type": "ValueError"},
+ # The threshold is validated before anything else, so it wins over a missing state.
+ {"name": "batch_min_confidence_invalid", "states": [QUICKSTART_STATE], "questions": BATCH_QUESTIONS,
+ "options": {"min_confidence": 1.5}, "error_type": "ValueError"},
+ {"name": "batch_min_confidence_before_none_state", "states": [QUICKSTART_STATE, None],
+ "questions": BATCH_QUESTIONS, "options": {"min_confidence": -0.25}, "error_type": "ValueError"},
+ {"name": "batch_min_confidence_bad_key", "states": [QUICKSTART_STATE], "questions": BATCH_QUESTIONS,
+ "options": {"min_confidence": {"choice:2": 0.5, "choice:3": 0.5}}, "error_type": "ValueError"},
+]
+
+# `system_one` with the abstention gate. `options` go to `system_one` verbatim, and are recorded
+# only for these cases so every earlier golden stays byte-identical.
+MIN_CONFIDENCE_CASES = [
+ {"name": "min_confidence_scalar", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],
+ "options": {"min_confidence": 0.6}},
+ {"name": "min_confidence_zero", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],
+ "options": {"min_confidence": 0.0}},
+ {"name": "min_confidence_map", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],
+ "options": {"min_confidence": {"choice:3-5": 0.9, "score:3-5": 0.5, "noul:2": 0.7}}},
+ {"name": "min_confidence_map_default", "state": "Thanks, that is all sorted now.",
+ "questions": CASES[0]["questions"], "options": {"min_confidence": {"default": 0.55}}},
+]
+
+MIN_CONFIDENCE_ERROR_CASES = [
+ {"name": "min_confidence_invalid", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],
+ "options": {"min_confidence": 1.5}},
+ {"name": "min_confidence_empty_map", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],
+ "options": {"min_confidence": {}}},
+ {"name": "min_confidence_bad_key", "state": QUICKSTART_STATE, "questions": CASES[0]["questions"],
+ "options": {"min_confidence": {"default": 0.5, "choice:3": 0.5}}},
]