Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(dotnet): port calibration files and histogram binning
kevin-gatimub43514558e25
diff --git a/laya-dotnet/CHANGELOG.md b/laya-dotnet/CHANGELOG.mdindex aa341d24..7e5f0a0b 100644--- a/laya-dotnet/CHANGELOG.md+++ b/laya-dotnet/CHANGELOG.md@@ -29,6 +29,16 @@ goldens: 0.4.12).small for the options' markers throws the "option markers fit" `ArgumentException`. New interfaces`ILayaCallPredictor`, `ILayaLongLangPredictor` and `ILayaLangTemperatures` let a router pass these on; existingsignatures are unchanged.+- Calibration files and histogram binning, ported from Python's `calibration=` / `load_calibration`+ (`apply_calibration_payload`, `apply_binning_map`): `LayaOptions.Calibration`, the path of a JSON file read when the engine+ is created, after the checkpoint's temperatures and the `LangTemperatures` table. Its `temperature` and+ `temperature_by_options` replace the checkpoint's (clamped, with Python's warning), and its `binning_map` recalibrates+ `AnswerConfidence` alone (`int(p * bins)` of the unrounded `max(p)`, then rounded to four places; a bucket it lacks passes+ through; skipped for a call whose `Lang` the table holds), so `MinConfidence` and the `PredictLong` window choice see the+ binned value. A file Python refuses throws `InvalidDataException` carrying Python's exception type and text. A version 2 file's+ checkpoint-identity warning is not emitted: the SDK has no equivalent of Python's `model_id_or_path` string; temperatures,+ binning and errors are unaffected. A router hands the path to every engine it builds. Existing+ signatures are unchanged.- Abstention gate, ported from Python `laya/confidence.py` (`min_confidence`, `jev_confidence`): `MinConfidence`(`Of` for one threshold, `ForBuckets` for a per-option-count map, `BucketKeys`), `LayaPredictOptions`with new `LayaEngine.Predict(state, questions, options)` and `LayaRouter.Predict(..., langGuess, options)`diff --git a/laya-dotnet/README.md b/laya-dotnet/README.mdindex 6f77cd1e..8184e488 100644--- a/laya-dotnet/README.md+++ b/laya-dotnet/README.md@@ -422,6 +422,53 @@ engine.PredictLong(document, questions, new LayaLongOptions { Lang = "de", Batchcall throw the same `ArgumentException` a checkpoint with such a budget would (`only N of its M option markers fit inmax_len=...`), while a small or negative `HeadMaxLen` squeezes each option to 4 tokens and the instruction to 8.Booleans are not a .NET `int`: `True`/`False` are `1`/`0`.+### Calibration files and histogram binning++Python's opt-in `calibration=` (`load_calibration`), off by default. A calibration file carries fitted temperatures and,+optionally, a histogram-binning map; this SDK reads and applies one, it does not fit or write them.++```csharp+using var engine = LayaEngine.Create(new LayaOptions { Calibration = "calibration.json" });+```++```json+{+ "version": 1,+ "temperature": [1.8, 2.2, 1.3],+ "temperature_by_options": { "choice:3-5": 2.4, "noul:2": 0.8 },+ "binning_map": {+ "choice:3-5": { "bins": 4, "values": [0.47, 0.31, 0.92, 0.68] },+ "noul:2": { "bins": 2, "values": [0.55, 0.85] }+ }+}+```++- **When it is read.** Once, when the engine is created, after the checkpoint's own temperatures and the+ `LangTemperatures` table (as Python's `ONNXAgent.__init__` does); there is no per-call form. A null or empty path reads+ nothing. A router hands the path to every engine it builds. The file is read as strict UTF-8 and parsed like+ `json.load` (`NaN` and `Infinity` are accepted), as the checkpoint config is.+- **Temperatures.** `temperature` (exactly three) and `temperature_by_options` replace the checkpoint's, both of them: a+ bucket the file does not name falls back to the file's per-type temperature, not the checkpoint's. Values are clamped to+ 0.5..5 and the ones the clamp moved are reported once through `System.Diagnostics.Trace`, with Python's text.+ `temperature_by_options` may be absent or `null`.+- **Binning.** `binning_map` maps an option-count bucket (`MinConfidence.BucketKeys`) to `bins` equal-width bins over+ [0, 1] and one recalibrated value per bin. It recalibrates `AnswerConfidence`, and only that: it is applied to the+ unrounded `max(p)` after the temperature softmax, the bin is `int(p * bins)` clipped to `0..bins-1`, and the value found+ is rounded to four places. `Confidence`, the probabilities, a noul answer's `Probability` and a score's `Score` are+ not binned. A bucket the map does not name keeps the temperature-scaled value. A call that names a language of+ `LangTemperatures` is not binned (that language's temperatures replace the file's). A file with no `binning_map`, or a+ `null` one, installs none.+- **The gate and `PredictLong` see the binned value.** `MinConfidence` compares `AnswerConfidence`, and a long scan keeps+ for a choice or score question the window with the greatest `AnswerConfidence` (a noul question the greatest+ `Probability`, which is not binned).+- **Refusals.** A file Python's `apply_calibration_payload` refuses makes `LayaEngine.Create` throw+ `InvalidDataException`, whose message is Python's exception type and text (`ValueError: calibration JSON must contain+ temperature: [3 floats]`): a payload that is not an object, a `version` that is not an integer (a missing or `null`+ one is 1; `2.0` and `"2"` are 2, `1.9` is refused), a `temperature` that is not a list of three, a+ `temperature_by_options` or `binning_map` that is not an object, and a `binning_map` entry that is not an object with an+ integer `bins >= 1` and `values` of exactly that length, each a finite number in [0, 1] (a bool, a string or a `null` is+ refused). A bad temperature is not refused but clamped; a bad binning value is, because there is nothing to fall back to.+### Abstention: `MinConfidence`Python's opt-in `min_confidence`, off by default. Pass a `MinConfidence` per call and every answer says@@ -1064,7 +1111,8 @@ disclaimer, signature and reply-header rules, including Portuguese, Spanish and- **Opt-in features added in Python 0.3.21:** abstention (`min_confidence`) and per-language temperatures(`lang_temperatures`, with per-call `lang`) are ported (see "Abstention" and "Per-language temperatures" above).- Calibration binning (`load_calibration`) is not, so the binning a language override skips in Python has nothing to skip here.+ Calibration files and histogram binning (`calibration=` / `load_calibration`) are ported too (see "Calibration files and+ histogram binning" above); fitting and saving one (`fit_binning`, `save_calibration`) is not.- **Other Python-only APIs are not ported:** hooks, question option ordering and digest controls.Batching is ported (`PredictBatch`, `LayaRouter.PredictBatch` / `RouteBatch`) without hooks and the router`diagnostics` dictionary.@@ -1091,9 +1139,17 @@ Where this SDK deliberately does not match the Python reference, so nobody has t`LayaRouter.Predict` and `LayaBatchOptions.Lang` / `MaxLen` / `HeadMaxLen` on `LayaRouter.PredictBatch` throw`ArgumentException`: Python's `predict` has the one `lang` argument, and `predict_batch` has no such arguments (it is a`TypeError` there), only the per-request keys.+- **A version 2 calibration file's checkpoint-identity warning is not emitted.** Python warns when the `model_id_or_path`,+ `subfolder` or `config` a file records differs from the agent's, but its `model_id_or_path` is the string the caller+ passed (a PyTorch checkpoint path or Hub id), which this SDK has no equivalent of. Temperatures, binning and errors are+ unaffected, and the file loads as Python's does.+- **A calibration file's refusals are `InvalidDataException`.** Python raises `ValueError` (and an `OverflowError` for an+ integer too large for a float); the message here is the exception's name and text, as for a malformed checkpoint config.+ The clamp warning is a `RuntimeWarning` in Python; it reaches `Trace` here as text, with no category, shown once per call+ site.- **JSON nesting.** Config and tokenizer JSON is read like `json.loads`, with the nesting cut-off set to 2,997 levels,what CPython 3.12 measured from a shallow call. Python's real limit depends on the call depth at the point of loading.-- **Config files are decoded as strict UTF-8.** Python opens them with the locale's encoding (cp1252 on a default Windows+- **Config and calibration files are decoded as strict UTF-8.** Python opens them with the locale's encoding (cp1252 on a default Windowsinstall), which only differs for non-ASCII bytes; a BOM is rejected, as `json.loads` does.---diff --git a/laya-dotnet/src/Laya/Calibration.cs b/laya-dotnet/src/Laya/Calibration.csindex 99dcd252..09115493 100644--- a/laya-dotnet/src/Laya/Calibration.cs+++ b/laya-dotnet/src/Laya/Calibration.cs@@ -63,13 +63,20 @@ public static class Calibrationpublic static double ResolveTemperature(LayaConfig config, QuestionType type, int optionCount){ArgumentNullException.ThrowIfNull(config);+ return ResolveTemperature(config.Temperature, config.TemperatureByOptions, type, optionCount);+ }++ /// <summary><see cref="ResolveTemperature(LayaConfig, QuestionType, int)"/> over temperatures an engine holds, a calibration file's among them.</summary>+ internal static double ResolveTemperature(+ IReadOnlyList<double> temperatures, IReadOnlyDictionary<string, double> byOptions, QuestionType type, int optionCount)+ {// `self.temperature_by_options.get(bucket, self.temperature[qt])` evaluates the default first, so a// list too short for the question's type fails even when the bucket is present (onnx_agent.py:727).- if ((int)type >= config.Temperature.Count)+ if ((int)type >= temperatures.Count)throw new InvalidOperationException("IndexError: list index out of range");- return config.TemperatureByOptions.TryGetValue(TempBucket(type, optionCount), out var t)+ return byOptions.TryGetValue(TempBucket(type, optionCount), out var t)? t- : config.Temperature[(int)type];+ : temperatures[(int)type];}/// <summary>diff --git a/laya-dotnet/src/Laya/CalibrationFile.cs b/laya-dotnet/src/Laya/CalibrationFile.csnew file mode 100644index 00000000..c25b49a9--- /dev/null+++ b/laya-dotnet/src/Laya/CalibrationFile.cs@@ -0,0 +1,210 @@+using System.Globalization;+using System.Numerics;+using System.Text;+using System.Text.RegularExpressions;++namespace Laya;++/// <summary>One bucket of a histogram-binning map: <c>bins</c> equal-width bins over [0, 1] and one recalibrated confidence per bin.</summary>+internal sealed record BinningEntry(int Bins, IReadOnlyList<double> Values);++/// <summary>What <c>apply_calibration_payload</c> installs on an agent: the clamped temperatures and the binning map.</summary>+internal sealed record CalibrationPayload(+ IReadOnlyList<double> Temperature,+ IReadOnlyDictionary<string, double> TemperatureByOptions,+ IReadOnlyDictionary<string, BinningEntry>? BinningMap);++/// <summary>+/// Python's <c>ONNXAgent.load_calibration</c> (<c>onnx_agent.py:266</c>), <c>apply_calibration_payload</c>+/// (<c>calibrate.py:754</c>) and <c>apply_binning_map</c> (<c>calibrate.py:567</c>). Only the reading and applying side+/// is ported: the fitters and <c>save_calibration</c> are not.+/// </summary>+/// <remarks>+/// Every refusal is an <see cref="InvalidDataException"/> whose message is the Python exception's type and text+/// (<c>ValueError: ...</c>), as <see cref="LayaConfig"/> reports a malformed checkpoint config.+/// </remarks>+internal static partial class CalibrationFile+{+ // int(str): [sign] digits, underscores only between digits. The input is already reduced to ASCII.+ [GeneratedRegex(@"^[+-]?[0-9]+(?:_[0-9]+)*$")]+ private static partial Regex IntShape();++ /// <summary><c>open(path)</c>, <c>json.load</c>, then <see cref="Apply"/>.</summary>+ internal static CalibrationPayload Load(string path) =>+ Apply(PythonJsonReader.Parse(PythonJsonReader.ReadAllText(path)));++ /// <summary>+ /// <c>apply_calibration_payload(agent, payload)</c>: validate the shape of <paramref name="payload"/>, and return the temperatures (clamped, with the clamp warning)+ /// and the binning map (<see langword="null"/> when the payload holds none) to install. Warnings go to <paramref name="warn"/>,+ /// by default <see cref="WarnOnce"/>.+ /// </summary>+ internal static CalibrationPayload Apply(object? payload, Action<string>? warn = null)+ {+ warn ??= WarnOnce.Warn;+ if (payload is not IReadOnlyDictionary<string, object?> root)+ throw Refused($"calibration JSON must be an object, got {PythonFloat.TypeName(payload)}");++ // The version is validated, but a version 2 file's recorded checkpoint (`_warn_if_identity_mismatch`) is not compared:+ // Python's `model_id_or_path` is the string its caller passed, which this SDK has no equivalent of.+ _ = ReadVersion(root);++ if (!root.TryGetValue("temperature", out var temps) || temps is not List<object?> { Count: 3 } temperature)+ throw Refused("calibration JSON must contain temperature: [3 floats]");++ // An absent key and an explicit null are the same: a file from before bucket temperatures.+ IReadOnlyDictionary<string, object?> byOptions = new Dictionary<string, object?>();+ if (root.TryGetValue("temperature_by_options", out var rawByOptions) && rawByOptions is not null)+ {+ if (rawByOptions is not IReadOnlyDictionary<string, object?> map)+ throw Refused(+ "calibration JSON temperature_by_options must be an object of bucket -> float, "+ + $"got {PythonFloat.TypeName(rawByOptions)}");+ byOptions = map;+ }++ var binning = ReadBinningMap(root);+++ var (applied, appliedByOptions) = InstallTemperatures(temperature, byOptions, warn);+ return new CalibrationPayload(applied, appliedByOptions, binning);+ }++ /// <summary>+ /// <c>apply_binning_map(confidence, bucket, binning_map)</c>: the recalibrated confidence for the bin <paramref name="confidence"/>+ /// falls in, or <paramref name="confidence"/> itself when the map holds no entry for <paramref name="bucket"/>.+ /// </summary>+ internal static double ApplyBinning(double confidence, string bucket, IReadOnlyDictionary<string, BinningEntry>? map)+ {+ if (map is null || !map.TryGetValue(bucket, out var entry)) return confidence;+ // `int(float(confidence) * bins)` truncates toward zero, and refuses a NaN or an infinity.+ var scaled = confidence * entry.Bins;+ if (double.IsNaN(scaled)) throw new InvalidDataException("ValueError: cannot convert float NaN to integer");+ if (double.IsInfinity(scaled)) throw new InvalidDataException("OverflowError: cannot convert float infinity to integer");+ var truncated = Math.Truncate(scaled);+ var bin = truncated <= 0 ? 0 : truncated >= entry.Bins - 1 ? entry.Bins - 1 : (int)truncated;+ return entry.Values[bin];+ }++ // ── version ──────────────────────────────────────────────────────────────++ private static BigInteger ReadVersion(IReadOnlyDictionary<string, object?> root)+ {+ // payload.get("version", 1), and an explicit null is 1 as well.+ var raw = root.TryGetValue("version", out var found) && found is not null ? found : 1L;+ var message = $"calibration JSON version must be an integer, got {PythonFloat.Repr(raw)}";+ switch (raw)+ {+ case long l: return l;+ case BigInteger big: return big;+ // `int()` truncates, which once read a 1.9 as version 1; a whole number written `2.0` still loads.+ case double d: return double.IsFinite(d) && d == Math.Floor(d) ? new BigInteger(d) : throw Refused(message);+ case string s: return TryParseInt(s, out var parsed) ? parsed : throw Refused(message);+ default: throw Refused(message); // a bool, list or object+ }+ }++ /// <summary><c>int(text)</c> for a string: Unicode whitespace stripped, Unicode decimal digits read as their value.</summary>+ private static bool TryParseInt(string text, out BigInteger value)+ {+ value = BigInteger.Zero;+ var ascii = new StringBuilder(text.Length);+ foreach (var rune in text.EnumerateRunes())+ {+ if (rune.Value < 128) ascii.Append((char)rune.Value);+ else if (PythonStr.IsSpace(rune.Value)) ascii.Append(' ');+ else+ {+ var digit = CharUnicodeInfo.GetDecimalDigitValue(rune.ToString(), 0);+ if (digit < 0 || Rune.GetUnicodeCategory(rune) != UnicodeCategory.DecimalDigitNumber) return false;+ ascii.Append((char)('0' + digit));+ }+ }+ var s = ascii.ToString().Trim(' ', '\t', '\n', '\v', '\f', '\r');+ if (!IntShape().IsMatch(s)) return false;+ var digits = s.TrimStart('+', '-').Replace("_", "");+ // Python 3.11+ refuses to turn more digits than sys.int_info.default_max_str_digits into an int.+ if (digits.Length > 4300) return false;+ value = BigInteger.Parse(digits, CultureInfo.InvariantCulture);+ if (s[0] == '-') value = -value;+ return true;+ }++ // ── binning map ──────────────────────────────────────────────────────────++ private static Dictionary<string, BinningEntry>? ReadBinningMap(IReadOnlyDictionary<string, object?> root)+ {+ if (!root.TryGetValue("binning_map", out var raw) || raw is null) return null;+ if (raw is not IReadOnlyDictionary<string, object?> map)+ throw Refused(+ "calibration JSON binning_map must be an object of bucket -> {bins, values}, "+ + $"got {PythonFloat.TypeName(raw)}");++ var result = new Dictionary<string, BinningEntry>(StringComparer.Ordinal);+ foreach (var (name, entryRaw) in map)+ {+ var at = $"calibration JSON binning_map[{PythonStr.Repr(name)}]";+ if (entryRaw is not IReadOnlyDictionary<string, object?> entry)+ throw Refused($"{at} must be an object with \"bins\" and \"values\", got {PythonFloat.TypeName(entryRaw)}");++ entry.TryGetValue("bins", out var binsRaw);+ entry.TryGetValue("values", out var valuesRaw);+ BigInteger bins = binsRaw switch { long l => l, BigInteger b => b, _ => BigInteger.Zero }; // a bool is not an int here+ if (binsRaw is not (long or BigInteger) || bins < 1)+ throw Refused($"{at} must have an integer \"bins\" >= 1, got {PythonFloat.Repr(binsRaw)}");+ if (valuesRaw is not List<object?> values || values.Count != bins)+ throw Refused($"{at} must have \"values\" of length \"bins\" ({bins.ToString(CultureInfo.InvariantCulture)})");++ var numbers = $"{at} values must be numbers, got {PythonFloat.Repr(values)}";+ // `[float(v) for v in values]` runs over every value before any type check: a value float() refuses is the+ // ValueError, and an integer too large for a float is an OverflowError that nothing catches.+ var parsed = new double[values.Count];+ for (var i = 0; i < values.Count; i++)+ {+ if (values[i] is BigInteger big && double.IsInfinity((double)big))+ throw new InvalidDataException("OverflowError: int too large to convert to float");+ if (!PythonFloat.TryConvert(values[i], out parsed[i])) throw Refused(numbers);+ }+ foreach (var v in values)+ if (v is bool || v is not (long or BigInteger or double)) throw Refused(numbers);+ foreach (var v in parsed)+ {+ if (!double.IsFinite(v)) throw Refused($"{at} values must be finite, got {PythonFloat.Repr(values)}");+ if (!(0.0 <= v && v <= 1.0)) throw Refused($"{at} values must be in [0, 1], got {PythonFloat.Repr(values)}");+ }+ result[name] = new BinningEntry((int)bins, parsed);+ }+ return result;+ }++ // ── temperatures ─────────────────────────────────────────────────────────++ /// <summary><c>_install_temperatures</c>: clamp every entry, and name the ones the clamp had to replace.</summary>+ private static (double[] Temperature, Dictionary<string, double> ByOptions) InstallTemperatures(+ List<object?> raw, IReadOnlyDictionary<string, object?> rawByOptions, Action<string> warn)+ {+ // `clamp_temperature(<int too large for a float>)` raises an OverflowError that nothing catches.+ foreach (var value in raw.Concat(rawByOptions.Values))+ if (value is BigInteger big && double.IsInfinity((double)big))+ throw new InvalidDataException("OverflowError: int too large to convert to float");++ var applied = raw.Select(Calibration.ClampTemperature).ToArray();+ var appliedByOptions = new Dictionary<string, double>(StringComparer.Ordinal);+ foreach (var (key, value) in rawByOptions) appliedByOptions[key] = Calibration.ClampTemperature(value);++ var rejected = new List<string>();+ for (var i = 0; i < raw.Count; i++)+ if (Calibration.TemperatureRejected(raw[i], applied[i]))+ rejected.Add($"temperature[{i}]={PythonFloat.Repr(raw[i])} -> {PythonFloat.FormatG(applied[i])}");+ foreach (var (key, value) in rawByOptions)+ if (Calibration.TemperatureRejected(value, appliedByOptions[key]))+ rejected.Add($"{key}={PythonFloat.Repr(value)} -> {PythonFloat.FormatG(appliedByOptions[key])}");+ if (rejected.Count > 0)+ warn(+ $"laya: calibration temperatures are invalid or outside [{PythonFloat.FormatG(Calibration.TempMin)}, "+ + $"{PythonFloat.FormatG(Calibration.TempMax)}]; using {string.Join(", ", rejected)}. "+ + "Treat confidence from the affected entries as uncalibrated.");+ return (applied, appliedByOptions);+ }++ private static InvalidDataException Refused(string text) => new($"ValueError: {text}");+}diff --git a/laya-dotnet/src/Laya/LayaEngine.cs b/laya-dotnet/src/Laya/LayaEngine.csindex 3c381223..3fad299d 100644--- a/laya-dotnet/src/Laya/LayaEngine.cs+++ b/laya-dotnet/src/Laya/LayaEngine.cs@@ -41,13 +41,19 @@ public sealed class LayaEngine : IDisposable, ILayaLongLangPredictor, ILayaBatchprivate readonly bool _ownsTokenizer;private readonly Lock _tokenizerGate = new();private readonly IReadOnlyDictionary<string, ResolvedLangTemperature> _langTemperatures;+ // The temperatures the checkpoint answers with: its own, or a calibration file's (which replaces both), and the file's+ // binning map (null: none, so answer_confidence is the temperature-scaled one).+ private readonly IReadOnlyList<double> _temperature;+ private readonly IReadOnlyDictionary<string, double> _temperatureByOptions;+ private readonly IReadOnlyDictionary<string, BinningEntry>? _binningMap;// 0 = live, 1 = disposed; int so Interlocked.Exchange can operate on it.private int _disposed;private LayaEngine(InferenceSession? fusedSession, InferenceSession? encoderSession,InferenceSession? headSession, HfTokenizer tokenizer, bool ownsTokenizer,LayaConfig config, ModelArtifacts artifacts,- IReadOnlyDictionary<string, ResolvedLangTemperature> langTemperatures)+ IReadOnlyDictionary<string, ResolvedLangTemperature> langTemperatures,+ CalibrationPayload? calibration){_fusedSession = fusedSession;_encoderSession = encoderSession;@@ -57,6 +63,10 @@ public sealed class LayaEngine : IDisposable, ILayaLongLangPredictor, ILayaBatchConfig = config;Artifacts = artifacts;_langTemperatures = langTemperatures;+ _temperature = calibration?.Temperature ?? config.Temperature;+ _temperatureByOptions = calibration?.TemperatureByOptions ?? config.TemperatureByOptions;+ // Python tests the map's truth, so an empty one is no map at all.+ _binningMap = calibration?.BinningMap is { Count: > 0 } map ? map : null;}/// <summary>The loaded checkpoint's configuration.</summary>@@ -90,6 +100,9 @@ public sealed class LayaEngine : IDisposable, ILayaLongLangPredictor, ILayaBatch/// When <see cref="LayaOptions.LangTemperatures"/> holds a language whose temperature is not three values, with/// Python's <c>ValueError</c> text./// </exception>+ /// <exception cref="InvalidDataException">+ /// When <see cref="LayaOptions.Calibration"/> names a file Python's <c>load_calibration</c> refuses, with its exception type and text.+ /// </exception>[WarnSite, MethodImpl(MethodImplOptions.NoInlining)]public static LayaEngine Create(LayaOptions? options = null){@@ -130,8 +143,15 @@ public sealed class LayaEngine : IDisposable, ILayaLongLangPredictor, ILayaBatch}tokenizer = new HfTokenizer(artifacts.TokenizerPath);++ // onnx_agent.py:263: last, after the tokenizer, the session, the checkpoint's temperatures and+ // the language table, so a broken checkpoint reports its own error before a bad calibration file.+ CalibrationPayload? calibration = null;+ if (!string.IsNullOrEmpty(options.Calibration))+ calibration = CalibrationFile.Load(options.Calibration);+return new LayaEngine(fusedSession, encoderSession, headSession,- tokenizer, ownsTokenizer: true, config, artifacts, langTemperatures);+ tokenizer, ownsTokenizer: true, config, artifacts, langTemperatures, calibration);}catch{@@ -440,20 +460,11 @@ public sealed class LayaEngine : IDisposable, ILayaLongLangPredictor, ILayaBatchvar row = offset + q;var question = questions[ids[q]];var k = batch.MarkerCounts[row];- var temperature = Calibration.ResolveTemperature(Config, question.Type, k);- // onnx_agent.py:728-731: a language the table holds replaces both the checkpoint's bucket map and its- // per-type temperature, with no fallback to either. The binning skip of onnx_agent.py:743 has nothing- // to skip here: `load_calibration` and its binning map are not ported.- if (LangTemperatures.Find(_langTemperatures, lang) is { } langTemperature)- temperature = langTemperature.TemperatureByOptions.TryGetValue(Calibration.TempBucket(question.Type, k), out var byBucket)- ? byBucket- : langTemperature.Temperature[(int)question.Type];var rowLogits = logits.AsSpan(row * batch.MarkerCount, batch.MarkerCount);- var p = Calibration.Probabilities(rowLogits, k, temperature);-var action = new ActionInfo(Calibration.Round4(Calibration.Softmax(actLogits.AsSpan(row * nAct, nAct))[0]));- answers[ids[q]] = BuildAnswer(question, p, k, action);+ answers[ids[q]] = DecodeRow(question, rowLogits, k, _temperature, _temperatureByOptions,+ LangTemperatures.Find(_langTemperatures, lang), _binningMap, action);}var stateDetails = details.GetRange(offset, ids.Count);var usage = new Usage(stateDetails.Sum(d => d.Ids.Length), 0);@@ -478,10 +489,33 @@ public sealed class LayaEngine : IDisposable, ILayaLongLangPredictor, ILayaBatchreturn results;}- private static Answer BuildAnswer(Question question, double[] p, int k, ActionInfo action)+ /// <summary>+ /// One row of <c>_decode_answers</c> (<c>onnx_agent.py:725-756</c>): the temperature (<paramref name="langTemperature"/>, when+ /// the call's language is one the table holds, replaces both the bucket map and the per-type temperature, with no fallback to+ /// either), the softmax, and the answer. <paramref name="binningMap"/> recalibrates <c>answer_confidence</c> alone, on the+ /// unrounded <c>max(p)</c> and before the rounding, and is skipped under a language override.+ /// </summary>+ internal static Answer DecodeRow(+ Question question, ReadOnlySpan<float> rowLogits, int k,+ IReadOnlyList<double> temperature, IReadOnlyDictionary<string, double> temperatureByOptions,+ ResolvedLangTemperature? langTemperature, IReadOnlyDictionary<string, BinningEntry>? binningMap, ActionInfo action)+ {+ var bucket = Calibration.TempBucket(question.Type, k);+ var scale = Calibration.ResolveTemperature(temperature, temperatureByOptions, question.Type, k);+ if (langTemperature is not null)+ scale = langTemperature.TemperatureByOptions.TryGetValue(bucket, out var byBucket)+ ? byBucket+ : langTemperature.Temperature[(int)question.Type];+ var p = Calibration.Probabilities(rowLogits, k, scale);+ var answerConfidence = Calibration.AnswerConfidence(p, k);+ if (binningMap is not null && langTemperature is null)+ answerConfidence = CalibrationFile.ApplyBinning(answerConfidence, bucket, binningMap);+ return BuildAnswer(question, p, k, action, Calibration.Round4(answerConfidence));+ }++ private static Answer BuildAnswer(Question question, double[] p, int k, ActionInfo action, double answerConfidence){var confidence = Calibration.Round4(Calibration.ConfidenceFromProbs(p, k));- var answerConfidence = Calibration.Round4(Calibration.AnswerConfidence(p, k));switch (question){case ChoiceQuestion choice:diff --git a/laya-dotnet/src/Laya/LayaOptions.cs b/laya-dotnet/src/Laya/LayaOptions.csindex d28954e1..15535a12 100644--- a/laya-dotnet/src/Laya/LayaOptions.cs+++ b/laya-dotnet/src/Laya/LayaOptions.cs@@ -160,6 +160,28 @@ public sealed class LayaOptions/// </remarks>public IEnumerable<KeyValuePair<string, LayaLangTemperature?>>? LangTemperatures { get; set; }+ /// <summary>+ /// Path of a calibration file, Python's <c>calibration</c> constructor option (<c>load_calibration</c>): a JSON object+ /// with <c>temperature</c> (three numbers), <c>temperature_by_options</c> (bucket to number), an optional+ /// <c>version</c> and an optional <c>binning_map</c> (bucket to <c>{"bins": n, "values": [n numbers in [0, 1]]}</c>).+ /// It is read when the engine is created, after the checkpoint's own temperatures and the+ /// <see cref="LangTemperatures"/> table, and replaces the checkpoint's temperature and its per-bucket map. The+ /// binning map recalibrates <c>answer_confidence</c> (and only that) of every answer whose option-count bucket the+ /// map names, after the temperature softmax; a call that names a language of <see cref="LangTemperatures"/> is not+ /// binned. A file with no <c>binning_map</c> installs none. Default <see langword="null"/> or empty: no file.+ /// </summary>+ /// <remarks>+ /// The file is read as UTF-8 and parsed as Python's <c>json.load</c> parses it (<c>NaN</c> and <c>Infinity</c>+ /// accepted), like <c>rl_agent_config.json</c>. A malformed file makes <see cref="LayaEngine.Create"/> throw an+ /// <see cref="InvalidDataException"/> whose message is Python's exception type and text; a temperature outside+ /// <see cref="Laya.Calibration.TempMin"/>..<see cref="Laya.Calibration.TempMax"/> is clamped and reported once through+ /// <see cref="System.Diagnostics.Trace"/>. A version 2 file's recorded checkpoint (<c>model_id_or_path</c>, <c>subfolder</c>,+ /// <c>config</c>) is not compared with the engine's: Python's <c>model_id_or_path</c> is the string its caller passed, which this+ /// SDK has no equivalent of, so that warning is not emitted. A+ /// <see cref="LayaRouter"/> hands this path to every engine it builds. There is no per-call form.+ /// </remarks>+ public string? Calibration { get; set; }+// ── internal helpers ──────────────────────────────────────────────────────/// <summary>diff --git a/laya-dotnet/src/Laya/LayaRouter.cs b/laya-dotnet/src/Laya/LayaRouter.csindex 61bc48be..29d26916 100644--- a/laya-dotnet/src/Laya/LayaRouter.cs+++ b/laya-dotnet/src/Laya/LayaRouter.cs@@ -28,7 +28,7 @@ public sealed class LayaRouterOptions/// <summary>/// Template used to build each checkpoint's <see cref="LayaEngine"/>: execution provider,/// thread counts, <see cref="LayaOptions.AllowDownload"/>, cache directory, HuggingFace- /// token/repo and <see cref="LayaOptions.LangTemperatures"/> are all taken from this template. <see cref="LayaOptions.Checkpoint"/> and+ /// token/repo, <see cref="LayaOptions.LangTemperatures"/> and <see cref="LayaOptions.Calibration"/> are all taken from this template. <see cref="LayaOptions.Checkpoint"/> and/// <see cref="LayaOptions.ModelDirectory"/> on the template are ignored — the router always/// resolves each checkpoint through its own <see cref="LayaOptions.Checkpoint"/>-aware path/// (<c>LAYA_ONNX_ROOT</c>, cache, or download), the same way <see cref="LayaEngine.Create"/> does.@@ -185,7 +185,11 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposable}private static Func<LayaCheckpoint, ILayaPredictor> BuildDefaultFactory(LayaOptions template) => checkpoint =>- LayaEngine.Create(new LayaOptions+ LayaEngine.Create(EngineOptions(template, checkpoint));++ /// <summary>The options of the engine the router builds for <paramref name="checkpoint"/> from its <paramref name="template"/>.</summary>+ internal static LayaOptions EngineOptions(LayaOptions template, LayaCheckpoint checkpoint) =>+ new(){Checkpoint = checkpoint,ExecutionProvider = template.ExecutionProvider,@@ -198,7 +202,8 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposableDownloadProgress = template.DownloadProgress,// agent_kwargs reaches every checkpoint the router builds (router.py:994); each engine resolves it itself.LangTemperatures = template.LangTemperatures,- });+ Calibration = template.Calibration,+ };// ── name resolution ──────────────────────────────────────────────────────diff --git a/laya-dotnet/src/Laya/WarnOnce.cs b/laya-dotnet/src/Laya/WarnOnce.csindex 7feaefd8..8202d18b 100644--- a/laya-dotnet/src/Laya/WarnOnce.cs+++ b/laya-dotnet/src/Laya/WarnOnce.cs@@ -42,7 +42,7 @@ internal static class WarnOnce// or a window budgeted directly) the first frame outside it is the site.private static bool IsPlumbing(StackFrame frame) =>frame.GetMethod()?.DeclaringType is { } type- && (type == typeof(WarnOnce) || type == typeof(LayaConfig) || type == typeof(LongPrediction) || type == typeof(LangTemperatures));+ && (type == typeof(WarnOnce) || type == typeof(LayaConfig) || type == typeof(LongPrediction) || type == typeof(LangTemperatures) || type == typeof(CalibrationFile));private static string Site(){diff --git a/laya-dotnet/tests/Laya.Tests/CalibrationFileTests.cs b/laya-dotnet/tests/Laya.Tests/CalibrationFileTests.csnew file mode 100644index 00000000..aa0a5fa2--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/CalibrationFileTests.cs@@ -0,0 +1,341 @@+using System.Text.Json;++namespace Laya.Tests;++/// <summary>+/// Model-free tests of calibration files and histogram binning: <c>apply_calibration_payload</c> and+/// <c>apply_binning_map</c> against <c>golden/routing/calibration_probe.json</c> (Python's own functions over the+/// synthetic inputs of <c>tools/calibration_cases.py</c>), the loader's error texts, and the decode of one row over synthetic logits. The decode over the real+/// weights is pinned by the <c>calibration_*</c> cases of <see cref="PredictParityTests"/>.+/// </summary>+public sealed class CalibrationFileTests+{+ // ── apply_calibration_payload ─────────────────────────────────────────────++ private static readonly Lazy<JsonDocument> Probe = new(() => RoutingGoldenData.Load("calibration_probe.json"));++ private static JsonElement Payloads => Probe.Value.RootElement.GetProperty("payloads");++ private static JsonElement Lookups => Probe.Value.RootElement.GetProperty("lookups");++ public static IEnumerable<object[]> PayloadCases() => Enumerable.Range(0, Payloads.GetArrayLength()).Select(i => new object[] { i });++ public static IEnumerable<object[]> LookupCases() => Enumerable.Range(0, Lookups.GetArrayLength()).Select(i => new object[] { i });++ [Fact]+ public void TheProbeCoversAcceptedPayloadsRefusalsAndWarnings()+ {+ Assert.True(Payloads.GetArrayLength() > 140);+ Assert.Contains(PayloadCases(), c => !Payloads[(int)c[0]].TryGetProperty("error_message", out _));+ Assert.Contains(PayloadCases(), c => Payloads[(int)c[0]].TryGetProperty("error_message", out _));+ Assert.Contains(PayloadCases(), c => Payloads[(int)c[0]].GetProperty("warnings").GetArrayLength() > 0);+ Assert.Equal(2, Probe.Value.RootElement.GetProperty("version").GetInt32());+ Assert.Equal(Calibration.TempMin, Probe.Value.RootElement.GetProperty("temp_min").GetDouble());+ Assert.Equal(Calibration.TempMax, Probe.Value.RootElement.GetProperty("temp_max").GetDouble());+ }++ private const string IdentityWarningPrefix = "calibration was fitted for model_id_or_path=";++ [Theory]+ [MemberData(nameof(PayloadCases))]+ public void AppliesAPayloadLikePython(int index)+ {+ var expected = Payloads[index];+ var label = expected.GetProperty("label").GetString();+ var payload = PythonJsonReader.Parse(expected.GetProperty("json").GetString()!);++ CalibrationPayload? got = null;+ InvalidDataException? refused = null;+ List<string> warnings = [];+ try { got = CalibrationFile.Apply(payload, warnings.Add); }+ catch (InvalidDataException e) { refused = e; }++ // The probe also records Python's checkpoint-identity warning (`_warn_if_identity_mismatch`, a version 2 file whose+ // recorded model_id_or_path/subfolder/config differs from the agent's). This SDK has no equivalent of Python's+ // model_id_or_path (the string its caller passed), so it does not emit it; every other warning text is asserted.+ Assert.Equal(+ [.. expected.GetProperty("warnings").EnumerateArray().Select(w => w.GetProperty("message").GetString()!)+ .Where(m => !m.StartsWith(IdentityWarningPrefix, StringComparison.Ordinal))],+ warnings);++ if (expected.TryGetProperty("error_message", out var message))+ {+ Assert.True(refused is not null, $"{label}: expected {expected.GetProperty("error_type").GetString()}");+ Assert.Equal($"{expected.GetProperty("error_type").GetString()}: {message.GetString()}", refused!.Message);+ return;+ }+ Assert.True(refused is null, $"{label}: {refused?.Message}");++ var result = expected.GetProperty("result");+ Assert.Equal([.. result.GetProperty("temperature").EnumerateArray().Select(CallOptions.Number)], got!.Temperature);+ Assert.Equal(+ result.GetProperty("temperature_by_options").EnumerateObject().Select(p => (p.Name, CallOptions.Number(p.Value))),+ got.TemperatureByOptions.Select(kv => (kv.Key, kv.Value)));+ var binning = result.GetProperty("binning_map");+ if (binning.ValueKind == JsonValueKind.Null)+ {+ Assert.Null(got.BinningMap);+ return;+ }+ Assert.NotNull(got.BinningMap);+ Assert.Equal(binning.EnumerateObject().Select(p => p.Name), got.BinningMap!.Keys);+ foreach (var entry in binning.EnumerateObject())+ {+ Assert.Equal(entry.Value.GetProperty("bins").GetInt32(), got.BinningMap[entry.Name].Bins);+ Assert.Equal(+ [.. entry.Value.GetProperty("values").EnumerateArray().Select(CallOptions.Number)],+ got.BinningMap[entry.Name].Values);+ }+ }++ // ── apply_binning_map ─────────────────────────────────────────────────────++ private static IReadOnlyDictionary<string, BinningEntry> Map(string name)+ {+ var map = new Dictionary<string, BinningEntry>(StringComparer.Ordinal);+ foreach (var entry in Probe.Value.RootElement.GetProperty("maps").GetProperty(name).EnumerateObject())+ map[entry.Name] = new BinningEntry(+ entry.Value.GetProperty("bins").GetInt32(),+ [.. entry.Value.GetProperty("values").EnumerateArray().Select(CallOptions.Number)]);+ return map;+ }++ [Fact]+ public void TheLookupProbeCoversEdgesMissesAndRefusals()+ {+ Assert.True(Lookups.GetArrayLength() > 60);+ var labels = Enumerable.Range(0, Lookups.GetArrayLength()).Select(i => Lookups[i].GetProperty("label").GetString()!).ToList();+ Assert.Contains("choice:3-5 at 0.0", labels);+ Assert.Contains("choice:3-5 at 1.0", labels);+ Assert.Contains(labels, l => l.StartsWith("miss at", StringComparison.Ordinal));+ Assert.Contains(Enumerable.Range(0, Lookups.GetArrayLength()), i => Lookups[i].TryGetProperty("error_message", out _));+ }++ [Theory]+ [MemberData(nameof(LookupCases))]+ public void BinsAConfidenceLikePython(int index)+ {+ var expected = Lookups[index];+ var map = Map(expected.GetProperty("map").GetString()!);+ var confidence = CallOptions.Number(expected.GetProperty("confidence"));+ var bucket = expected.GetProperty("bucket").GetString()!;++ if (expected.TryGetProperty("error_message", out var message))+ {+ var error = Assert.Throws<InvalidDataException>(() => CalibrationFile.ApplyBinning(confidence, bucket, map));+ Assert.Equal($"{expected.GetProperty("error_type").GetString()}: {message.GetString()}", error.Message);+ return;+ }+ Assert.Equal(CallOptions.Number(expected.GetProperty("result")), CalibrationFile.ApplyBinning(confidence, bucket, map));+ }++ [Fact]+ public void NoMapIsNoBinning()+ {+ Assert.Equal(0.37, CalibrationFile.ApplyBinning(0.37, "choice:3-5", null));+ }++ // ── the shared fixture ────────────────────────────────────────────────────++ [Fact]+ public void TheFixtureLoadsAndHoldsSeveralBuckets()+ {+ var loaded = CalibrationFile.Load(CalibrationFixture.Path);+ Assert.Equal([1.8, 2.2, 1.3], loaded.Temperature);+ Assert.Equal(2.4, loaded.TemperatureByOptions["choice:3-5"]);+ Assert.Equal(0.6, loaded.TemperatureByOptions["choice:6-10"]);+ Assert.NotNull(loaded.BinningMap);+ Assert.Equal(["choice:3-5", "noul:2", "score:6-10", "choice:11+", "score:2"], loaded.BinningMap!.Keys);+ Assert.Equal(4, loaded.BinningMap["choice:3-5"].Bins);+ }++ // ── the loader and its error texts, without a model ───────────────────────+ // Python loads the file last, after the session and the tokenizer (onnx_agent.py:263), so `LayaEngine.Create` reaches it only+ // over real artifacts: the order against the checkpoint's own errors and warnings is pinned by `PredictParityTests`. The file's+ // own texts are tested here on the loader.++ private static string Write(string text)+ {+ var path = Path.Combine(Path.GetTempPath(), "laya-calib-" + Guid.NewGuid().ToString("N") + ".json");+ File.WriteAllText(path, text);+ return path;+ }++ private static T WithFile<T>(string text, Func<string, T> body)+ {+ var path = Write(text);+ try { return body(path); }+ finally { File.Delete(path); }+ }++ [Fact]+ public void AMalformedFileIsRefusedWithPythonsText()+ {+ var error = Assert.Throws<InvalidDataException>(() => WithFile("[1, 2]", CalibrationFile.Load));+ Assert.Equal("ValueError: calibration JSON must be an object, got list", error.Message);++ error = Assert.Throws<InvalidDataException>(() => WithFile(+ "{\"temperature\": [1.0, 1.0, 1.0], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [0.5, 1.5]}}}", CalibrationFile.Load));+ Assert.Equal(+ "ValueError: calibration JSON binning_map['choice:3-5'] values must be in [0, 1], got [0.5, 1.5]", error.Message);+ }++ [Fact]+ public void AFileThatIsNotJsonOrNotUtf8IsRefusedAsTheConfigIs()+ {+ Assert.ThrowsAny<JsonException>(() => WithFile("{\"temperature\": [1, 2, 3],}", CalibrationFile.Load));+ var path = Write("");+ try+ {+ File.WriteAllBytes(path, [0x7B, 0xFF, 0x7D]);+ Assert.ThrowsAny<JsonException>(() => CalibrationFile.Load(path));+ }+ finally { File.Delete(path); }+ }++ [Fact]+ public void AMissingFileIsAFileNotFound() =>+ Assert.Throws<FileNotFoundException>(() => CalibrationFile.Load(Path.Combine(Path.GetTempPath(), "absent-" + Guid.NewGuid().ToString("N") + ".json")));++ [Fact]+ public void AVersion2FileFittedForAnotherCheckpointLoadsWithoutAnIdentityWarning()+ {+ // Python's checkpoint-identity warning is not emitted: this SDK has no equivalent of its model_id_or_path string.+ List<string> warnings = [];+ var loaded = CalibrationFile.Apply(PythonJsonReader.Parse(+ "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"model_id_or_path\": \"somewhere/else\", \"subfolder\": \"x\", \"config\": {\"max_len\": 1}}"),+ warnings.Add);+ Assert.Empty(warnings);+ Assert.Equal([1.5, 2.0, 1.25], loaded.Temperature);+ }++ // ── a router hands the file to every engine it builds ─────────────────────++ [Fact]+ public void ARouterPassesTheFileAndTheLanguageTableToEveryEngine()+ {+ var template = new LayaOptions+ {+ Calibration = "calibration.json",+ LangTemperatures = new Dictionary<string, LayaLangTemperature?> { ["de"] = null },+ ModelDirectory = "ignored",+ };+ foreach (var checkpoint in Enum.GetValues<LayaCheckpoint>())+ {+ var built = LayaRouter.EngineOptions(template, checkpoint);+ Assert.Equal("calibration.json", built.Calibration);+ Assert.Same(template.LangTemperatures, built.LangTemperatures);+ Assert.Equal(checkpoint, built.Checkpoint);+ Assert.Null(built.ModelDirectory);+ }+ }++ // ── the decode of one row ─────────────────────────────────────────────────++ private static readonly IReadOnlyList<double> One = [1.0, 1.0, 1.0];+ private static readonly IReadOnlyDictionary<string, double> NoBuckets = new Dictionary<string, double>();++ private static readonly IReadOnlyDictionary<string, BinningEntry> Binning = new Dictionary<string, BinningEntry>+ {+ ["noul:2"] = new(4, [0.1, 0.4, 0.6, 0.9]),+ ["choice:3-5"] = new(2, [0.25, 0.75]),+ };++ private static Answer Decode(Question question, float[] logits, IReadOnlyDictionary<string, BinningEntry>? binning,+ ResolvedLangTemperature? lang = null, IReadOnlyList<double>? temperature = null,+ IReadOnlyDictionary<string, double>? buckets = null) =>+ LayaEngine.DecodeRow(question, logits, logits.Length, temperature ?? One, buckets ?? NoBuckets, lang, binning, new ActionInfo(0.5));++ private static float[] Logits(double pTrue) => [0f, (float)Math.Log(pTrue / (1 - pTrue))];++ [Fact]+ public void BinsAnswerConfidenceAndNothingElse()+ {+ var question = Question.Noul("Is it?");+ var plain = Decode(question, Logits(0.8), null).AsNoul();+ var binned = Decode(question, Logits(0.8), Binning).AsNoul();++ Assert.Equal(0.8, plain.AnswerConfidence, 4);+ Assert.Equal(0.9, binned.AnswerConfidence); // 0.8 * 4 = 3.2, bin 3+ Assert.Equal(plain.Probability, binned.Probability);+ Assert.Equal(plain.Confidence, binned.Confidence);+ Assert.Equal(plain.Action, binned.Action);+ }++ [Fact]+ public void ChoiceAndScoreKeepTheirProbabilitiesAndEntropyConfidence()+ {+ float[] logits = [2f, 0.5f, 0f];+ var choice = Question.Choice("Which?", "a", "b", "c");+ var plain = Decode(choice, logits, null).AsChoice();+ var binned = Decode(choice, logits, Binning).AsChoice();+ Assert.Equal(plain.Probabilities, binned.Probabilities);+ Assert.Equal(plain.Confidence, binned.Confidence);+ Assert.Equal(plain.Choice, binned.Choice);+ Assert.Equal(0.75, binned.AnswerConfidence); // max(p) is about 0.73, bin 1 of 2+ Assert.NotEqual(plain.AnswerConfidence, binned.AnswerConfidence);++ var score = Question.Score("How much?", "low", "mid", "high");+ var plainScore = Decode(score, logits, null).AsScore();+ var binnedScore = Decode(score, logits, Binning).AsScore(); // score:3-5 has no entry: a miss+ Assert.Equal(plainScore.AnswerConfidence, binnedScore.AnswerConfidence);+ Assert.Equal(plainScore.Probabilities, binnedScore.Probabilities);+ }++ [Fact]+ public void BinsTheUnroundedConfidenceAndRoundsTheBinnedValue()+ {+ // max(p) = 0.74996 is in bin 2 of 4; rounded to four places first it would be 0.75, in bin 3.+ var question = Question.Noul("Is it?");+ var answer = Decode(question, Logits(0.74996), Binning).AsNoul();+ Assert.Equal(0.6, answer.AnswerConfidence);+ Assert.Equal(0.75, Calibration.Round4(0.74996));++ // The value read from the map is rounded like any other: 1/3 stays at four places.+ var thirds = new Dictionary<string, BinningEntry> { ["noul:2"] = new(1, [1.0 / 3.0]) };+ Assert.Equal(0.3333, Decode(question, Logits(0.8), thirds).AsNoul().AnswerConfidence);+ }++ [Fact]+ public void AMissingBucketKeepsTheTemperatureScaledValue()+ {+ var question = Question.Noul("Is it?");+ var only = new Dictionary<string, BinningEntry> { ["choice:11+"] = new(2, [0.2, 0.8]) };+ Assert.Equal(Decode(question, Logits(0.8), null).AnswerConfidence, Decode(question, Logits(0.8), only).AnswerConfidence);+ }++ [Fact]+ public void ALanguageTheTableHoldsReplacesTheTemperatureAndSkipsTheBinning()+ {+ var question = Question.Noul("Is it?");+ var lang = new ResolvedLangTemperature([2.0, 2.0, 2.0], new Dictionary<string, double>());+ var plain = Decode(question, Logits(0.8), null, lang);+ var binned = Decode(question, Logits(0.8), Binning, lang);+ Assert.Equal(plain.AnswerConfidence, binned.AnswerConfidence);+ Assert.NotEqual(0.9, binned.AnswerConfidence);+ // The language's temperature (2.0) softened it relative to the file's.+ Assert.True(binned.AnswerConfidence < Decode(question, Logits(0.8), null).AnswerConfidence);+ }++ [Fact]+ public void TheTemperaturesAreTheFilesWhenOneWasLoaded()+ {+ var question = Question.Noul("Is it?");+ var sharp = Decode(question, Logits(0.8), null, temperature: [1.0, 1.0, 0.5]);+ var soft = Decode(question, Logits(0.8), null, temperature: [1.0, 1.0, 2.0]);+ var bucket = Decode(question, Logits(0.8), null, temperature: [1.0, 1.0, 2.0], buckets: new Dictionary<string, double> { ["noul:2"] = 0.5 });+ Assert.True(sharp.AnswerConfidence > soft.AnswerConfidence);+ Assert.Equal(sharp.AnswerConfidence, bucket.AnswerConfidence);+ }++ [Fact]+ public void TheGateReadsTheBinnedValue()+ {+ var question = Question.Noul("Is it?");+ var gate = MinConfidence.Of(0.5).Check();+ // 0.8 passes a 0.5 gate unbinned, but its bin holds 0.4.+ var lowBin = new Dictionary<string, BinningEntry> { ["noul:2"] = new(1, [0.4]) };+ Assert.Equal(GateStates.Passed, ConfidenceGate.Apply(Decode(question, Logits(0.8), null), gate).Abstention);+ Assert.Equal(GateStates.Abstained, ConfidenceGate.Apply(Decode(question, Logits(0.8), lowBin), gate).Abstention);+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/CalibrationFixture.cs b/laya-dotnet/tests/Laya.Tests/CalibrationFixture.csnew file mode 100644index 00000000..56eb1704--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/CalibrationFixture.cs@@ -0,0 +1,46 @@+using System.Text.Json;+using System.Text.Json.Nodes;++namespace Laya.Tests;++/// <summary>+/// The hand-made calibration file the <c>calibration_*</c> golden cases were recorded with+/// (<c>laya-dotnet/tools/calibration_fixture.json</c>), which an engine under test loads through+/// <see cref="LayaOptions.Calibration"/>.+/// </summary>+internal static class CalibrationFixture+{+ internal const string FileName = "calibration_fixture.json";++ private static readonly Lazy<string> PathLazy = new(Locate);++ /// <summary>The fixture file, found by walking up from the test binary to the SDK's <c>tools</c> directory.</summary>+ internal static string Path => PathLazy.Value;++ /// <summary>Whether a recorded case was made with the calibration file.</summary>+ internal static bool Recorded(JsonElement root) => root.TryGetProperty("calibration", out var calibration) && calibration.ValueKind == JsonValueKind.Object;++ /// <summary>+ /// For a case recorded with a calibration file: the file it names is the fixture, and what it held then is what it holds now,+ /// so an engine loaded from the fixture is the engine that was recorded.+ /// </summary>+ internal static void AssertIsTheRecordedFile(JsonElement root)+ {+ if (!Recorded(root)) return;+ var recorded = root.GetProperty("calibration");+ Assert.Equal(FileName, recorded.GetProperty("file").GetString());+ Assert.True(+ JsonNode.DeepEquals(JsonNode.Parse(recorded.GetProperty("payload").GetRawText()), JsonNode.Parse(File.ReadAllText(Path))),+ $"{FileName} has changed since the golden was recorded; run tools/regen_golden.py");+ }++ private static string Locate()+ {+ for (var dir = new DirectoryInfo(AppContext.BaseDirectory); dir is not null; dir = dir.Parent)+ {+ var candidate = System.IO.Path.Combine(dir.FullName, "tools", FileName);+ if (File.Exists(candidate)) return candidate;+ }+ throw new FileNotFoundException($"{FileName} not found in a tools directory above {AppContext.BaseDirectory}");+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.csindex 5b140ab8..851011ff 100644--- a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs@@ -35,9 +35,9 @@ public sealed partial class PredictParityTests[MemberData(nameof(BatchSuccessCases))]public void ReproducesBatchPredictions(LayaCheckpoint checkpoint, GoldenCaseInfo info, bool split){- var engine = split ? fixture.RequireSplit(checkpoint) : fixture.Require(checkpoint);using var doc = GoldenData.For(checkpoint).Load(info);var root = doc.RootElement;+ var engine = fixture.RequireFor(checkpoint, root, split);var states = BatchStates(root);var questions = GoldenData.BuildQuestions(root.GetProperty("questions"));diff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Calibration.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Calibration.csnew file mode 100644index 00000000..491ec85b--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Calibration.cs@@ -0,0 +1,116 @@+using System.Diagnostics;++namespace Laya.Tests;++// Where `LayaOptions.Calibration` is read in `LayaEngine.Create`: Python's ONNXAgent.__init__ loads the file last+// (onnx_agent.py:263), after the checkpoint and language-table warnings, the ONNX session and the tokenizer. That needs real+// artifacts, so it is pinned here, with the parity lanes; the file's own error texts are tested model-free in+// CalibrationFileTests.+public sealed partial class PredictParityTests+{+ private sealed class TraceCapture : TraceListener+ {+ public List<string> Messages { get; } = [];+ public override void Write(string? message) => Messages.Add(message ?? "");+ public override void WriteLine(string? message) => Messages.Add(message ?? "");+ public override void TraceEvent(TraceEventCache? eventCache, string source, TraceEventType eventType, int id, string? format, params object?[]? args) =>+ Messages.Add(args is null ? format ?? "" : string.Format(format ?? "", args));+ }++ public static IEnumerable<object[]> ParityCheckpointCases() => TestArtifacts.ParityCheckpoints.Select(c => new object[] { c });++ private string ArtifactDirectory(LayaCheckpoint checkpoint)+ {+ if (!fixture.Available(checkpoint)) fixture.Require(checkpoint); // skips, with the usual reason+ return TestArtifacts.For(checkpoint).Directory;+ }++ [Theory]+ [MemberData(nameof(ParityCheckpointCases))]+ public void TheCalibrationClampWarningComesAfterTheCheckpointAndLanguageWarnings(LayaCheckpoint checkpoint)+ {+ var dir = ArtifactDirectory(checkpoint);+ // Names of this run's own, so the texts are not shared with a test that warned earlier or in parallel.+ var token = "calib" + Guid.NewGuid().ToString("N");+ var file = Path.Combine(Path.GetTempPath(), token + ".json");+ File.WriteAllText(file, $"{{\"temperature\": [9.0, 1.0, 1.0], \"temperature_by_options\": {{\"{token}\": 0.1}}}}");+ var capture = new TraceCapture();+ Trace.Listeners.Add(capture);+ try+ {+ using var engine = LayaEngine.Create(new LayaOptions+ {+ ModelDirectory = dir,+ LangTemperatures = new Dictionary<string, LayaLangTemperature?> { [token] = new() { Temperature = [0.1, 1.0, 1.0] } },+ Calibration = file,+ });+ }+ finally+ {+ Trace.Listeners.Remove(capture);+ File.Delete(file);+ }++ var messages = capture.Messages;+ var ofLanguage = messages.FindIndex(m => m.StartsWith($"laya: language override '{token}'", StringComparison.Ordinal));+ var ofCalibration = messages.FindIndex(m => m.StartsWith(+ $"laya: calibration temperatures are invalid or outside [0.5, 5]; using temperature[0]=9.0 -> 5, {token}=0.1 -> 0.5.", StringComparison.Ordinal));+ Assert.True(ofLanguage >= 0 && ofCalibration > ofLanguage, string.Join("\n", messages));+ // A checkpoint that ships temperatures the clamp moves reports them after the language table and before the file.+ var ofCheckpoint = messages.FindIndex(m => m.StartsWith("laya ONNX: this checkpoint ships temperatures outside", StringComparison.Ordinal));+ if (ofCheckpoint >= 0)+ Assert.True(ofLanguage < ofCheckpoint && ofCheckpoint < ofCalibration, string.Join("\n", messages));+ }++ [Theory]+ [MemberData(nameof(ParityCheckpointCases))]+ public void ABrokenCheckpointFailsBeforeABadCalibrationFileIsRead(LayaCheckpoint checkpoint)+ {+ var dir = ArtifactDirectory(checkpoint);+ // The real graph with a tokenizer that cannot be read: Python builds the session, fails on the tokenizer, and never+ // reaches the calibration file. Linking the graph and its weights needs symbolic links, which Windows may refuse.+ var broken = Path.Combine(Path.GetTempPath(), "laya-broken-" + Guid.NewGuid().ToString("N"));+ var bad = Path.Combine(broken, "calibration.json");+ try+ {+ Directory.CreateDirectory(Path.Combine(broken, "tokenizer"));+ try+ {+ foreach (var name in new[] { "model.onnx", "model.onnx.data" })+ File.CreateSymbolicLink(Path.Combine(broken, name), Path.Combine(dir, name));+ }+ catch (Exception e) when (e is UnauthorizedAccessException or IOException or PlatformNotSupportedException)+ {+ Assert.Skip($"cannot link the real graph into a scratch directory: {e.Message}");+ }+ File.Copy(Path.Combine(dir, "rl_agent_config.json"), Path.Combine(broken, "rl_agent_config.json"));+ File.WriteAllText(Path.Combine(broken, "tokenizer", "tokenizer.json"), "not a tokenizer");+ File.WriteAllText(bad, "[1, 2]");++ var alone = Assert.ThrowsAny<Exception>(() => LayaEngine.Create(new LayaOptions { ModelDirectory = broken }));+ var withFile = Assert.ThrowsAny<Exception>(() => LayaEngine.Create(new LayaOptions { ModelDirectory = broken, Calibration = bad }));+ Assert.IsNotType<InvalidDataException>(withFile);+ Assert.Equal(alone.GetType(), withFile.GetType());+ Assert.Equal(alone.Message, withFile.Message);+ }+ finally { Directory.Delete(broken, recursive: true); }+ }++ [Theory]+ [MemberData(nameof(ParityCheckpointCases))]+ public void ABadCalibrationFileRefusesTheLoadOfAGoodCheckpoint(LayaCheckpoint checkpoint)+ {+ var dir = ArtifactDirectory(checkpoint);+ var file = Path.Combine(Path.GetTempPath(), "calib" + Guid.NewGuid().ToString("N") + ".json");+ File.WriteAllText(file, "[1, 2]");+ try+ {+ var error = Assert.Throws<InvalidDataException>(() => LayaEngine.Create(new LayaOptions { ModelDirectory = dir, Calibration = file }));+ Assert.Equal("ValueError: calibration JSON must be an object, got list", error.Message);+ Assert.Throws<FileNotFoundException>(() => LayaEngine.Create(new LayaOptions { ModelDirectory = dir, Calibration = file + ".absent" }));+ // `if calibration:` is false for "": no file is read.+ LayaEngine.Create(new LayaOptions { ModelDirectory = dir, Calibration = "" }).Dispose();+ }+ finally { File.Delete(file); }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.csindex 62f2d1a9..5ca2ea25 100644--- a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.cs+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.cs@@ -22,9 +22,9 @@ public sealed partial class PredictParityTests[MemberData(nameof(LongSuccessCases))]public void ReproducesLongPredictions(LayaCheckpoint checkpoint, GoldenCaseInfo info, bool split){- var engine = split ? fixture.RequireSplit(checkpoint) : fixture.Require(checkpoint);using var doc = GoldenData.For(checkpoint).Load(info);var root = doc.RootElement;+ var engine = fixture.RequireFor(checkpoint, root, split);var state = GoldenData.ToClr(root.GetProperty("state"))!;var questions = GoldenData.BuildQuestions(root.GetProperty("questions"));var options = root.GetProperty("options");diff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.csindex 2b560837..c0c0135b 100644--- a/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs@@ -71,10 +71,10 @@ public sealed partial class PredictParityTests(AllEnginesFixture fixture)[MemberData(nameof(AllSuccessCases))]public void ReproducesTheRecordedAnswers(LayaCheckpoint checkpoint, GoldenCaseInfo info){- var engine = fixture.Require(checkpoint);var golden = GoldenData.For(checkpoint);using var doc = golden.Load(info);var root = doc.RootElement;+ var engine = fixture.RequireFor(checkpoint, root);var state = GoldenData.ToClr(root.GetProperty("state"));var questions = GoldenData.BuildQuestions(root.GetProperty("questions"));@@ -133,10 +133,10 @@ public sealed partial class PredictParityTests(AllEnginesFixture fixture)[MemberData(nameof(AllSuccessCasesSplit))]public void ReproducesTheRecordedAnswersSplit(LayaCheckpoint checkpoint, GoldenCaseInfo info){- var engine = fixture.RequireSplit(checkpoint);var golden = GoldenData.For(checkpoint);using var doc = golden.Load(info);var root = doc.RootElement;+ var engine = fixture.RequireFor(checkpoint, root, split: true);var state = GoldenData.ToClr(root.GetProperty("state"));var questions = GoldenData.BuildQuestions(root.GetProperty("questions"));diff --git a/laya-dotnet/tests/Laya.Tests/TestArtifacts.cs b/laya-dotnet/tests/Laya.Tests/TestArtifacts.csindex 16781a54..ba7eaddb 100644--- a/laya-dotnet/tests/Laya.Tests/TestArtifacts.cs+++ b/laya-dotnet/tests/Laya.Tests/TestArtifacts.cs@@ -1,3 +1,5 @@+using System.Text.Json;+namespace Laya.Tests;/// <summary>@@ -214,12 +216,19 @@ public sealed class AllEnginesFixture : IDisposable// class that wants both layouts does not need two [Collection] attributes.private readonly Lazy<LayaEngine>[] _splitEngines;+ // The same two layouts loaded with the calibration file the `calibration_*` golden cases were recorded with.+ // A file replaces the checkpoint's temperatures for every question, so those cases cannot share the engines above.+ private readonly Lazy<LayaEngine>[] _calibratedEngines;+ private readonly Lazy<LayaEngine>[] _calibratedSplitEngines;+/// <summary>Create the fixture. Engines are not loaded until first use.</summary>public AllEnginesFixture(){var values = Enum.GetValues<LayaCheckpoint>();_engines = new Lazy<LayaEngine>[values.Length];_splitEngines = new Lazy<LayaEngine>[values.Length];+ _calibratedEngines = new Lazy<LayaEngine>[values.Length];+ _calibratedSplitEngines = new Lazy<LayaEngine>[values.Length];for (var i = 0; i < values.Length; i++){var checkpoint = values[i]; // captured correctly: new variable each iteration@@ -235,6 +244,18 @@ public sealed class AllEnginesFixture : IDisposableModelDirectory = TestArtifacts.For(checkpoint).SplitDirectoryOrNull!,LangTemperatures = GoldenData.For(checkpoint).LangTemperatures,}));+ _calibratedEngines[(int)checkpoint] = new Lazy<LayaEngine>(() => LayaEngine.Create(new LayaOptions+ {+ ModelDirectory = TestArtifacts.For(checkpoint).Directory,+ LangTemperatures = GoldenData.For(checkpoint).LangTemperatures,+ Calibration = CalibrationFixture.Path,+ }));+ _calibratedSplitEngines[(int)checkpoint] = new Lazy<LayaEngine>(() => LayaEngine.Create(new LayaOptions+ {+ ModelDirectory = TestArtifacts.For(checkpoint).SplitDirectoryOrNull!,+ LangTemperatures = GoldenData.For(checkpoint).LangTemperatures,+ Calibration = CalibrationFixture.Path,+ }));}}@@ -269,9 +290,30 @@ public sealed class AllEnginesFixture : IDisposablereturn _splitEngines[(int)checkpoint].Value;}+ /// <summary>+ /// The engine a recorded case was answered by: the one loaded with the calibration file when the case was recorded with it (after+ /// checking that the file is still the one recorded), else the shared engine of the layout.+ /// </summary>+ internal LayaEngine RequireFor(LayaCheckpoint checkpoint, JsonElement root, bool split = false)+ {+ if (!CalibrationFixture.Recorded(root)) return split ? RequireSplit(checkpoint) : Require(checkpoint);+ CalibrationFixture.AssertIsTheRecordedFile(root);+ if (split)+ {+ RequireSplit(checkpoint);+ return _calibratedSplitEngines[(int)checkpoint].Value;+ }+ Require(checkpoint);+ return _calibratedEngines[(int)checkpoint].Value;+ }+/// <inheritdoc/>public void Dispose(){+ foreach (var lazy in _calibratedEngines)+ if (lazy.IsValueCreated) lazy.Value.Dispose();+ foreach (var lazy in _calibratedSplitEngines)+ if (lazy.IsValueCreated) lazy.Value.Dispose();foreach (var lazy in _engines)if (lazy.IsValueCreated) lazy.Value.Dispose();foreach (var lazy in _splitEngines)diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_batch.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_batch.jsonnew file mode 100644index 00000000..23a139d6--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_batch.json@@ -0,0 +1,584 @@+{+ "name": "calibration_batch",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, that is all sorted now.",+ {+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",+ "message": "Please cancel my account today."+ },+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "batch_size": 3,+ "sort_by_length": true+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_widths": [+ 91,+ 110,+ 144,+ 34,+ 163,+ 42,+ 98,+ 56+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7354,+ "technical": 0.1705,+ "other": 0.094+ },+ "confidence": 0.3174,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6034,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1112,+ "1": 0.1741,+ "2": 0.7147+ },+ "confidence": 0.2821,+ "answer_confidence": 0.7147,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9998,+ "confidence": 0.9998,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6849,+ "technical": 0.2122,+ "other": 0.103+ },+ "confidence": 0.2515,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.5061,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1206,+ "1": 0.2528,+ "2": 0.6267+ },+ "confidence": 0.1848,+ "answer_confidence": 0.6267,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9984,+ "confidence": 0.9984,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 322,+ "output_tokens": 0,+ "state_tokens": 76,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.686,+ "technical": 0.2082,+ "other": 0.1058+ },+ "confidence": 0.2509,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.4634,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1369,+ "1": 0.2629,+ "2": 0.6003+ },+ "confidence": 0.1537,+ "answer_confidence": 0.6003,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9922,+ "confidence": 0.9922,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 424,+ "output_tokens": 0,+ "state_tokens": 110,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.4866,+ "technical": 0.269,+ "other": 0.2444+ },+ "confidence": 0.046,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 0.8119,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.4195,+ "1": 0.3491,+ "2": 0.2314+ },+ "confidence": 0.0256,+ "answer_confidence": 0.4195,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.2469,+ "confidence": 0.7531,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 94,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8243,+ "technical": 0.1146,+ "other": 0.0612+ },+ "confidence": 0.4735,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.5695,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1058,+ "1": 0.2189,+ "2": 0.6753+ },+ "confidence": 0.2397,+ "answer_confidence": 0.6753,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9999,+ "confidence": 0.9999,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 481,+ "output_tokens": 0,+ "state_tokens": 129,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "technical",+ "probabilities": {+ "refund": 0.3029,+ "technical": 0.3803,+ "other": 0.3168+ },+ "confidence": 0.0046,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 0.89,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.4549,+ "1": 0.2002,+ "2": 0.3449+ },+ "confidence": 0.0466,+ "answer_confidence": 0.4549,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0009,+ "confidence": 0.9991,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 118,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5081,+ "technical": 0.2237,+ "other": 0.2681+ },+ "confidence": 0.0607,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.5505,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1328,+ "1": 0.184,+ "2": 0.6832+ },+ "confidence": 0.2356,+ "answer_confidence": 0.6832,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.989,+ "confidence": 0.989,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 286,+ "output_tokens": 0,+ "state_tokens": 64,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.4924,+ "technical": 0.3558,+ "other": 0.1517+ },+ "confidence": 0.0874,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 0.7156,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.5014,+ "1": 0.2816,+ "2": 0.217+ },+ "confidence": 0.0583,+ "answer_confidence": 0.5014,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.1254,+ "confidence": 0.8746,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 160,+ "output_tokens": 0,+ "state_tokens": 22,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 9,+ 56+ ],+ [+ 9,+ 110+ ],+ [+ 6,+ 163+ ]+ ]+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_batch_lang_override.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_batch_lang_override.jsonnew file mode 100644index 00000000..20e55c25--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_batch_lang_override.json@@ -0,0 +1,457 @@+{+ "name": "calibration_batch_lang_override",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "lang": "pt"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_widths": [+ 91,+ 110,+ 144,+ 34,+ 163+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9345,+ "technical": 0.0503,+ "other": 0.0153+ },+ "confidence": 0.7474,+ "answer_confidence": 0.9345,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6495,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0964,+ "1": 0.1578,+ "2": 0.7458+ },+ "confidence": 0.3305,+ "answer_confidence": 0.7458,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9998,+ "confidence": 0.9998,+ "answer_confidence": 0.9998,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.894,+ "technical": 0.0858,+ "other": 0.0202+ },+ "confidence": 0.6453,+ "answer_confidence": 0.894,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.5464,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1065,+ "1": 0.2405,+ "2": 0.653+ },+ "confidence": 0.2176,+ "answer_confidence": 0.653,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9984,+ "confidence": 0.9984,+ "answer_confidence": 0.9984,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 322,+ "output_tokens": 0,+ "state_tokens": 76,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8961,+ "technical": 0.0826,+ "other": 0.0213+ },+ "confidence": 0.6484,+ "answer_confidence": 0.8961,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.5021,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1229,+ "1": 0.252,+ "2": 0.6251+ },+ "confidence": 0.1819,+ "answer_confidence": 0.6251,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9922,+ "confidence": 0.9922,+ "answer_confidence": 0.9922,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 424,+ "output_tokens": 0,+ "state_tokens": 110,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6419,+ "technical": 0.1962,+ "other": 0.1619+ },+ "confidence": 0.1818,+ "answer_confidence": 0.6419,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 0.7944,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.428,+ "1": 0.3496,+ "2": 0.2224+ },+ "confidence": 0.0306,+ "answer_confidence": 0.428,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.2469,+ "confidence": 0.7531,+ "answer_confidence": 0.7531,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 94,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9758,+ "technical": 0.0189,+ "other": 0.0054+ },+ "confidence": 0.8845,+ "answer_confidence": 0.9758,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6127,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0917,+ "1": 0.204,+ "2": 0.7043+ },+ "confidence": 0.2807,+ "answer_confidence": 0.7043,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9999,+ "confidence": 0.9999,+ "answer_confidence": 0.9999,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 481,+ "output_tokens": 0,+ "state_tokens": 129,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 15,+ 163+ ]+ ]+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_batch_min_confidence.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_batch_min_confidence.jsonnew file mode 100644index 00000000..136ab4e4--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_batch_min_confidence.json@@ -0,0 +1,433 @@+{+ "name": "calibration_batch_min_confidence",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.6+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_widths": [+ 91,+ 110,+ 144,+ 34,+ 163+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7354,+ "technical": 0.1705,+ "other": 0.094+ },+ "confidence": 0.3174,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.6034,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1112,+ "1": 0.1741,+ "2": 0.7147+ },+ "confidence": 0.2821,+ "answer_confidence": 0.7147,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9998,+ "confidence": 0.9998,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6849,+ "technical": 0.2122,+ "other": 0.103+ },+ "confidence": 0.2515,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.5061,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1206,+ "1": 0.2528,+ "2": 0.6267+ },+ "confidence": 0.1848,+ "answer_confidence": 0.6267,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9984,+ "confidence": 0.9984,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 322,+ "output_tokens": 0,+ "state_tokens": 76,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.686,+ "technical": 0.2082,+ "other": 0.1058+ },+ "confidence": 0.2509,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.4634,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1369,+ "1": 0.2629,+ "2": 0.6003+ },+ "confidence": 0.1537,+ "answer_confidence": 0.6003,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9922,+ "confidence": 0.9922,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 424,+ "output_tokens": 0,+ "state_tokens": 110,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.4866,+ "technical": 0.269,+ "other": 0.2444+ },+ "confidence": 0.046,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 0.8119,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.4195,+ "1": 0.3491,+ "2": 0.2314+ },+ "confidence": 0.0256,+ "answer_confidence": 0.4195,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.2469,+ "confidence": 0.7531,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 94,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8243,+ "technical": 0.1146,+ "other": 0.0612+ },+ "confidence": 0.4735,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.5695,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1058,+ "1": 0.2189,+ "2": 0.6753+ },+ "confidence": 0.2397,+ "answer_confidence": 0.6753,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9999,+ "confidence": 0.9999,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 481,+ "output_tokens": 0,+ "state_tokens": 129,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 15,+ 163+ ]+ ]+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_buckets.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_buckets.jsonnew file mode 100644index 00000000..0be494f3--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_buckets.json@@ -0,0 +1,1727 @@+{+ "name": "calibration_buckets",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "pair": {+ "type": "choice",+ "instructions": "Which of the two?",+ "criteria": {+ "first": "the first option",+ "second": "the second option"+ }+ },+ "wide": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices and payments",+ "technical": "bugs and outages",+ "account": "logins and seats",+ "sales": "quotes and upgrades",+ "legal": "contracts and privacy",+ "shipping": "delivery and returns",+ "other": "anything else"+ }+ },+ "scale": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "none",+ "low",+ "mild",+ "high",+ "severe",+ "critical"+ ]+ },+ "flag": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "per_question": [+ {+ "id": "pair",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which of the two?",+ "head_text": "choice question: Which of the two?",+ "options": [+ "first: the first option",+ "second: the second option"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 10,+ 16+ ]+ },+ {+ "id": "wide",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices and payments",+ "technical: bugs and outages",+ "account: logins and seats",+ "sales: quotes and upgrades",+ "legal: contracts and privacy",+ "shipping: delivery and returns",+ "other: anything else"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ]+ },+ {+ "id": "scale",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: none",+ "level 1: low",+ "level 2: mild",+ "level 3: high",+ "level 4: severe",+ "level 5: critical"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36+ ]+ },+ {+ "id": "flag",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 10,+ 16,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ],+ [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36,+ 0+ ],+ [+ 14,+ 24,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 0,+ 1,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.2316139936447144,+ 0.3455628752708435,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ],+ [+ 6.5726318359375,+ -2.5407862663269043,+ -2.38329815864563,+ -3.2941930294036865,+ -1.8061529397964478,+ -3.400134801864624,+ -3.246126651763916+ ],+ [+ -2.260613203048706,+ -0.7889672517776489,+ 0.15310415625572205,+ 0.48803675174713135,+ 2.8179776668548584,+ -2.6931307315826416,+ -10000.0+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4353.82666015625,+ -3561.62744140625+ ],+ [+ 3583.10205078125,+ -2931.8779296875+ ],+ [+ 3553.88330078125,+ -2925.64794921875+ ],+ [+ 2926.997314453125,+ -2398.78125+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "pair": {+ "type": "choice",+ "choice": "first",+ "probabilities": {+ "first": 0.6206,+ "second": 0.3794+ },+ "confidence": 0.0424,+ "answer_confidence": 0.6206,+ "action": {+ "act_probability": 1.0+ }+ },+ "wide": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 1.0,+ "technical": 0.0,+ "account": 0.0,+ "sales": 0.0,+ "legal": 0.0,+ "shipping": 0.0,+ "other": 0.0+ },+ "confidence": 1.0,+ "answer_confidence": 1.0,+ "action": {+ "act_probability": 1.0+ }+ },+ "scale": {+ "type": "score",+ "score": 2.9023,+ "legend": {+ "0": "none",+ "1": "low",+ "2": "mild",+ "3": "high",+ "4": "severe",+ "5": "critical"+ },+ "probabilities": {+ "0": 0.0732,+ "1": 0.1195,+ "2": 0.1636,+ "3": 0.1829,+ "4": 0.3976,+ "5": 0.0633+ },+ "confidence": 0.1107,+ "answer_confidence": 0.18,+ "action": {+ "act_probability": 1.0+ }+ },+ "flag": {+ "type": "noul",+ "noul": 0.9998,+ "confidence": 0.9998,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 386,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_buckets_skip_binning.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_buckets_skip_binning.jsonnew file mode 100644index 00000000..72d52f81--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_buckets_skip_binning.json@@ -0,0 +1,1786 @@+{+ "name": "calibration_lang_buckets_skip_binning",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "pair": {+ "type": "choice",+ "instructions": "Which of the two?",+ "criteria": {+ "first": "the first option",+ "second": "the second option"+ }+ },+ "wide": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices and payments",+ "technical": "bugs and outages",+ "account": "logins and seats",+ "sales": "quotes and upgrades",+ "legal": "contracts and privacy",+ "shipping": "delivery and returns",+ "other": "anything else"+ }+ },+ "scale": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "none",+ "low",+ "mild",+ "high",+ "severe",+ "critical"+ ]+ },+ "flag": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "per_question": [+ {+ "id": "pair",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which of the two?",+ "head_text": "choice question: Which of the two?",+ "options": [+ "first: the first option",+ "second: the second option"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 10,+ 16+ ]+ },+ {+ "id": "wide",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices and payments",+ "technical: bugs and outages",+ "account: logins and seats",+ "sales: quotes and upgrades",+ "legal: contracts and privacy",+ "shipping: delivery and returns",+ "other: anything else"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ]+ },+ {+ "id": "scale",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: none",+ "level 1: low",+ "level 2: mild",+ "level 3: high",+ "level 4: severe",+ "level 5: critical"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36+ ]+ },+ {+ "id": "flag",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 10,+ 16,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ],+ [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36,+ 0+ ],+ [+ 14,+ 24,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 0,+ 1,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.2316139936447144,+ 0.3455628752708435,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ],+ [+ 6.5726318359375,+ -2.5407862663269043,+ -2.38329815864563,+ -3.2941930294036865,+ -1.8061529397964478,+ -3.400134801864624,+ -3.246126651763916+ ],+ [+ -2.260613203048706,+ -0.7889672517776489,+ 0.15310415625572205,+ 0.48803675174713135,+ 2.8179776668548584,+ -2.6931307315826416,+ -10000.0+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4353.82666015625,+ -3561.62744140625+ ],+ [+ 3583.10205078125,+ -2931.8779296875+ ],+ [+ 3553.88330078125,+ -2925.64794921875+ ],+ [+ 2926.997314453125,+ -2398.78125+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "pair": {+ "type": "choice",+ "choice": "first",+ "probabilities": {+ "first": 0.5733,+ "second": 0.4267+ },+ "confidence": 0.0156,+ "answer_confidence": 0.5733,+ "action": {+ "act_probability": 1.0+ }+ },+ "wide": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 1.0,+ "technical": 0.0,+ "account": 0.0,+ "sales": 0.0,+ "legal": 0.0,+ "shipping": 0.0,+ "other": 0.0+ },+ "confidence": 1.0,+ "answer_confidence": 1.0,+ "action": {+ "act_probability": 1.0+ }+ },+ "scale": {+ "type": "score",+ "score": 3.1578,+ "legend": {+ "0": "none",+ "1": "low",+ "2": "mild",+ "3": "high",+ "4": "severe",+ "5": "critical"+ },+ "probabilities": {+ "0": 0.0419,+ "1": 0.0875,+ "2": 0.1401,+ "3": 0.1657,+ "4": 0.5311,+ "5": 0.0338+ },+ "confidence": 0.2355,+ "answer_confidence": 0.5311,+ "action": {+ "act_probability": 1.0+ }+ },+ "flag": {+ "type": "noul",+ "noul": 0.968,+ "confidence": 0.968,+ "answer_confidence": 0.968,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 386,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "it"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_miss_bins.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_miss_bins.jsonnew file mode 100644index 00000000..ed8ec41f--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_miss_bins.json@@ -0,0 +1,1625 @@+{+ "name": "calibration_lang_miss_bins",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 4.488276958465576,+ -2.729666233062744,+ -2.4480319023132324,+ -10000.0+ ],+ [+ -0.8349542617797852,+ 0.8832842707633972,+ 3.6844465732574463,+ 1.2054039239883423+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0+ ],+ [+ 0.7536231279373169,+ 0.41879838705062866,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4074.86572265625,+ -3334.4990234375+ ],+ [+ 4040.2958984375,+ -3316.3017578125+ ],+ [+ 2926.997314453125,+ -2398.78125+ ],+ [+ 4615.162109375,+ -3778.93896484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.905,+ "technical": 0.0447,+ "account": 0.0503+ },+ "confidence": 0.6544,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.8775,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.074,+ "1": 0.1616,+ "2": 0.5773,+ "3": 0.1871+ },+ "confidence": 0.1936,+ "answer_confidence": 0.5773,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9998,+ "confidence": 0.9998,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.3969,+ "confidence": 0.6031,+ "answer_confidence": 0.57,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "ja"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_null_entry_inherits_checkpoint.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_null_entry_inherits_checkpoint.jsonnew file mode 100644index 00000000..720a01d8--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_null_entry_inherits_checkpoint.json@@ -0,0 +1,1625 @@+{+ "name": "calibration_lang_null_entry_inherits_checkpoint",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 4.488276958465576,+ -2.729666233062744,+ -2.4480319023132324,+ -10000.0+ ],+ [+ -0.8349542617797852,+ 0.8832842707633972,+ 3.6844465732574463,+ 1.2054039239883423+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0+ ],+ [+ 0.7536231279373169,+ 0.41879838705062866,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4074.86572265625,+ -3334.4990234375+ ],+ [+ 4040.2958984375,+ -3316.3017578125+ ],+ [+ 2926.997314453125,+ -2398.78125+ ],+ [+ 4615.162109375,+ -3778.93896484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9741,+ "technical": 0.0118,+ "account": 0.0141+ },+ "confidence": 0.8743,+ "answer_confidence": 0.9741,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.9821,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0212,+ "1": 0.0839,+ "2": 0.7864,+ "3": 0.1085+ },+ "confidence": 0.4809,+ "answer_confidence": 0.7864,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.968,+ "confidence": 0.968,+ "answer_confidence": 0.968,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.4579,+ "confidence": 0.5421,+ "answer_confidence": 0.5421,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "fr"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_override_skips_binning.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_override_skips_binning.jsonnew file mode 100644index 00000000..bd88f663--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_lang_override_skips_binning.json@@ -0,0 +1,1625 @@+{+ "name": "calibration_lang_override_skips_binning",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 4.488276958465576,+ -2.729666233062744,+ -2.4480319023132324,+ -10000.0+ ],+ [+ -0.8349542617797852,+ 0.8832842707633972,+ 3.6844465732574463,+ 1.2054039239883423+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0+ ],+ [+ 0.7536231279373169,+ 0.41879838705062866,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4074.86572265625,+ -3334.4990234375+ ],+ [+ 4040.2958984375,+ -3316.3017578125+ ],+ [+ 2926.997314453125,+ -2398.78125+ ],+ [+ 4615.162109375,+ -3778.93896484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.945,+ "technical": 0.0256,+ "account": 0.0295+ },+ "confidence": 0.7714,+ "answer_confidence": 0.945,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.8057,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.108,+ "1": 0.1915,+ "2": 0.4872,+ "3": 0.2132+ },+ "confidence": 0.1079,+ "answer_confidence": 0.4872,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9891,+ "confidence": 0.9891,+ "answer_confidence": 0.9891,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.4444,+ "confidence": 0.5556,+ "answer_confidence": 0.5556,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "de"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_long_lang_override.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_long_lang_override.jsonnew file mode 100644index 00000000..7a3a6005--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_long_lang_override.json@@ -0,0 +1,437 @@+{+ "name": "calibration_long_lang_override",+ "state": "The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "window": 96,+ "batch_size": 3,+ "lang": "de"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_tokens": 2037,+ "effective_window": 96,+ "effective_stride": 48,+ "effective_room": 478,+ "windows": [+ {+ "token_start": 0,+ "token_end": 96,+ "text": "The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 48,+ "token_end": 144,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 96,+ "token_end": 192,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 144,+ "token_end": 240,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 192,+ "token_end": 288,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 240,+ "token_end": 336,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 288,+ "token_end": 384,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 336,+ "token_end": 432,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 384,+ "token_end": 480,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 432,+ "token_end": 528,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 480,+ "token_end": 576,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 528,+ "token_end": 624,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 576,+ "token_end": 672,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 624,+ "token_end": 720,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 672,+ "token_end": 768,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 720,+ "token_end": 816,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 768,+ "token_end": 864,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 816,+ "token_end": 912,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 864,+ "token_end": 960,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 912,+ "token_end": 1008,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 960,+ "token_end": 1056,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1008,+ "token_end": 1104,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1056,+ "token_end": 1152,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1104,+ "token_end": 1200,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1152,+ "token_end": 1248,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1200,+ "token_end": 1296,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1248,+ "token_end": 1344,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1296,+ "token_end": 1392,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1344,+ "token_end": 1440,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1392,+ "token_end": 1488,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 1440,+ "token_end": 1536,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 1488,+ "token_end": 1584,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1536,+ "token_end": 1632,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1584,+ "token_end": 1680,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1632,+ "token_end": 1728,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1680,+ "token_end": 1776,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1728,+ "token_end": 1824,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1776,+ "token_end": 1872,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1824,+ "token_end": 1920,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1872,+ "token_end": 1968,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1920,+ "token_end": 2016,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the"+ },+ {+ "token_start": 1968,+ "token_end": 2037,+ "text": ". The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "result": {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8359,+ "technical": 0.099,+ "other": 0.0651+ },+ "confidence": 0.4933,+ "answer_confidence": 0.8359,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 40,+ "token_start": 1920,+ "token_end": 2016,+ "count": 42+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.386,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1776,+ "1": 0.2589,+ "2": 0.5636+ },+ "confidence": 0.108,+ "answer_confidence": 0.5636,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 41,+ "token_start": 1968,+ "token_end": 2037,+ "count": 42+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9702,+ "confidence": 0.9702,+ "answer_confidence": 0.9702,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 41,+ "token_start": 1968,+ "token_end": 2037,+ "count": 42+ }+ }+ },+ "usage": {+ "input_tokens": 15963,+ "output_tokens": 0,+ "state_tokens": 4005,+ "state_tokens_dropped": 0,+ "truncated": 0,+ "truncated_questions": [],+ "windows": 42+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_long_scan.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_long_scan.jsonnew file mode 100644index 00000000..77b77f40--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_long_scan.json@@ -0,0 +1,380 @@+{+ "name": "calibration_long_scan",+ "state": "The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "window": 96,+ "batch_size": 3+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_tokens": 2037,+ "effective_window": 96,+ "effective_stride": 48,+ "effective_room": 478,+ "windows": [+ {+ "token_start": 0,+ "token_end": 96,+ "text": "The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 48,+ "token_end": 144,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 96,+ "token_end": 192,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 144,+ "token_end": 240,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 192,+ "token_end": 288,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 240,+ "token_end": 336,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 288,+ "token_end": 384,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 336,+ "token_end": 432,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 384,+ "token_end": 480,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 432,+ "token_end": 528,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 480,+ "token_end": 576,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 528,+ "token_end": 624,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 576,+ "token_end": 672,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 624,+ "token_end": 720,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 672,+ "token_end": 768,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 720,+ "token_end": 816,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 768,+ "token_end": 864,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 816,+ "token_end": 912,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 864,+ "token_end": 960,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 912,+ "token_end": 1008,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 960,+ "token_end": 1056,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1008,+ "token_end": 1104,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1056,+ "token_end": 1152,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1104,+ "token_end": 1200,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1152,+ "token_end": 1248,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1200,+ "token_end": 1296,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1248,+ "token_end": 1344,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1296,+ "token_end": 1392,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1344,+ "token_end": 1440,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1392,+ "token_end": 1488,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 1440,+ "token_end": 1536,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 1488,+ "token_end": 1584,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1536,+ "token_end": 1632,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1584,+ "token_end": 1680,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1632,+ "token_end": 1728,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1680,+ "token_end": 1776,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1728,+ "token_end": 1824,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1776,+ "token_end": 1872,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1824,+ "token_end": 1920,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1872,+ "token_end": 1968,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1920,+ "token_end": 2016,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the"+ },+ {+ "token_start": 1968,+ "token_end": 2037,+ "text": ". The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "result": {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7481,+ "technical": 0.1173,+ "other": 0.1346+ },+ "confidence": 0.3279,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 0,+ "token_start": 0,+ "token_end": 96,+ "count": 42+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.5106,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1333,+ "1": 0.2229,+ "2": 0.6438+ },+ "confidence": 0.1929,+ "answer_confidence": 0.6438,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 41,+ "token_start": 1968,+ "token_end": 2037,+ "count": 42+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9985,+ "confidence": 0.9985,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 41,+ "token_start": 1968,+ "token_end": 2037,+ "count": 42+ }+ }+ },+ "usage": {+ "input_tokens": 15963,+ "output_tokens": 0,+ "state_tokens": 4005,+ "state_tokens_dropped": 0,+ "truncated": 0,+ "truncated_questions": [],+ "windows": 42+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_long_single_window.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_long_single_window.jsonnew file mode 100644index 00000000..ab592f32--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_long_single_window.json@@ -0,0 +1,148 @@+{+ "name": "calibration_long_single_window",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {},+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_tokens": 57,+ "effective_window": 312,+ "effective_stride": 156,+ "effective_room": 478,+ "windows": [],+ "result": {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7354,+ "technical": 0.1705,+ "other": 0.094+ },+ "confidence": 0.3174,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6034,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1112,+ "1": 0.1741,+ "2": 0.7147+ },+ "confidence": 0.2821,+ "answer_confidence": 0.7147,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9998,+ "confidence": 0.9998,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": [],+ "windows": 1+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence.jsonnew file mode 100644index 00000000..864edef8--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence.json@@ -0,0 +1,1579 @@+{+ "name": "calibration_min_confidence",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 4.488276958465576,+ -2.729666233062744,+ -2.4480319023132324,+ -10000.0+ ],+ [+ -0.8349542617797852,+ 0.8832842707633972,+ 3.6844465732574463,+ 1.2054039239883423+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0+ ],+ [+ 0.7536231279373169,+ 0.41879838705062866,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4074.86572265625,+ -3334.4990234375+ ],+ [+ 4040.2958984375,+ -3316.3017578125+ ],+ [+ 2926.997314453125,+ -2398.78125+ ],+ [+ 4615.162109375,+ -3778.93896484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.905,+ "technical": 0.0447,+ "account": 0.0503+ },+ "confidence": 0.6544,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.8775,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.074,+ "1": 0.1616,+ "2": 0.5773,+ "3": 0.1871+ },+ "confidence": 0.1936,+ "answer_confidence": 0.5773,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9998,+ "confidence": 0.9998,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.3969,+ "confidence": 0.6031,+ "answer_confidence": 0.57,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.6+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence_buckets.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence_buckets.jsonnew file mode 100644index 00000000..e7a838a8--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence_buckets.json@@ -0,0 +1,1739 @@+{+ "name": "calibration_min_confidence_buckets",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "pair": {+ "type": "choice",+ "instructions": "Which of the two?",+ "criteria": {+ "first": "the first option",+ "second": "the second option"+ }+ },+ "wide": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices and payments",+ "technical": "bugs and outages",+ "account": "logins and seats",+ "sales": "quotes and upgrades",+ "legal": "contracts and privacy",+ "shipping": "delivery and returns",+ "other": "anything else"+ }+ },+ "scale": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "none",+ "low",+ "mild",+ "high",+ "severe",+ "critical"+ ]+ },+ "flag": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "per_question": [+ {+ "id": "pair",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which of the two?",+ "head_text": "choice question: Which of the two?",+ "options": [+ "first: the first option",+ "second: the second option"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 10,+ 16+ ]+ },+ {+ "id": "wide",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices and payments",+ "technical: bugs and outages",+ "account: logins and seats",+ "sales: quotes and upgrades",+ "legal: contracts and privacy",+ "shipping: delivery and returns",+ "other: anything else"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ]+ },+ {+ "id": "scale",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: none",+ "level 1: low",+ "level 2: mild",+ "level 3: high",+ "level 4: severe",+ "level 5: critical"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36+ ]+ },+ {+ "id": "flag",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 10,+ 16,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ],+ [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36,+ 0+ ],+ [+ 14,+ 24,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 0,+ 1,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.2316139936447144,+ 0.3455628752708435,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ],+ [+ 6.5726318359375,+ -2.5407862663269043,+ -2.38329815864563,+ -3.2941930294036865,+ -1.8061529397964478,+ -3.400134801864624,+ -3.246126651763916+ ],+ [+ -2.260613203048706,+ -0.7889672517776489,+ 0.15310415625572205,+ 0.48803675174713135,+ 2.8179776668548584,+ -2.6931307315826416,+ -10000.0+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4353.82666015625,+ -3561.62744140625+ ],+ [+ 3583.10205078125,+ -2931.8779296875+ ],+ [+ 3553.88330078125,+ -2925.64794921875+ ],+ [+ 2926.997314453125,+ -2398.78125+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "pair": {+ "type": "choice",+ "choice": "first",+ "probabilities": {+ "first": 0.6206,+ "second": 0.3794+ },+ "confidence": 0.0424,+ "answer_confidence": 0.6206,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "wide": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 1.0,+ "technical": 0.0,+ "account": 0.0,+ "sales": 0.0,+ "legal": 0.0,+ "shipping": 0.0,+ "other": 0.0+ },+ "confidence": 1.0,+ "answer_confidence": 1.0,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "scale": {+ "type": "score",+ "score": 2.9023,+ "legend": {+ "0": "none",+ "1": "low",+ "2": "mild",+ "3": "high",+ "4": "severe",+ "5": "critical"+ },+ "probabilities": {+ "0": 0.0732,+ "1": 0.1195,+ "2": 0.1636,+ "3": 0.1829,+ "4": 0.3976,+ "5": 0.0633+ },+ "confidence": 0.1107,+ "answer_confidence": 0.18,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "flag": {+ "type": "noul",+ "noul": 0.9998,+ "confidence": 0.9998,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ }+ },+ "usage": {+ "input_tokens": 386,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.5+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence_lang_override.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence_lang_override.jsonnew file mode 100644index 00000000..de3a9a14--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence_lang_override.json@@ -0,0 +1,1636 @@+{+ "name": "calibration_min_confidence_lang_override",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 4.488276958465576,+ -2.729666233062744,+ -2.4480319023132324,+ -10000.0+ ],+ [+ -0.8349542617797852,+ 0.8832842707633972,+ 3.6844465732574463,+ 1.2054039239883423+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0+ ],+ [+ 0.7536231279373169,+ 0.41879838705062866,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4074.86572265625,+ -3334.4990234375+ ],+ [+ 4040.2958984375,+ -3316.3017578125+ ],+ [+ 2926.997314453125,+ -2398.78125+ ],+ [+ 4615.162109375,+ -3778.93896484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.945,+ "technical": 0.0256,+ "account": 0.0295+ },+ "confidence": 0.7714,+ "answer_confidence": 0.945,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.8057,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.108,+ "1": 0.1915,+ "2": 0.4872,+ "3": 0.2132+ },+ "confidence": 0.1079,+ "answer_confidence": 0.4872,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9891,+ "confidence": 0.9891,+ "answer_confidence": 0.9891,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.4444,+ "confidence": 0.5556,+ "answer_confidence": 0.5556,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "de",+ "min_confidence": 0.6+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence_map.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence_map.jsonnew file mode 100644index 00000000..e8e7eaaa--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_min_confidence_map.json@@ -0,0 +1,1583 @@+{+ "name": "calibration_min_confidence_map",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 4.488276958465576,+ -2.729666233062744,+ -2.4480319023132324,+ -10000.0+ ],+ [+ -0.8349542617797852,+ 0.8832842707633972,+ 3.6844465732574463,+ 1.2054039239883423+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0+ ],+ [+ 0.7536231279373169,+ 0.41879838705062866,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4074.86572265625,+ -3334.4990234375+ ],+ [+ 4040.2958984375,+ -3316.3017578125+ ],+ [+ 2926.997314453125,+ -2398.78125+ ],+ [+ 4615.162109375,+ -3778.93896484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.905,+ "technical": 0.0447,+ "account": 0.0503+ },+ "confidence": 0.6544,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ },+ "urgency": {+ "type": "score",+ "score": 1.8775,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.074,+ "1": 0.1616,+ "2": 0.5773,+ "3": 0.1871+ },+ "confidence": 0.1936,+ "answer_confidence": 0.5773,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9998,+ "confidence": 0.9998,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.3969,+ "confidence": 0.6031,+ "answer_confidence": 0.57,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": {+ "choice:3-5": 0.7,+ "score:3-5": 0.5,+ "noul:2": 0.6+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_plain.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_plain.jsonnew file mode 100644index 00000000..5b4abaaa--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_plain.json@@ -0,0 +1,1566 @@+{+ "name": "calibration_plain",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 4.488276958465576,+ -2.729666233062744,+ -2.4480319023132324,+ -10000.0+ ],+ [+ -0.8349542617797852,+ 0.8832842707633972,+ 3.6844465732574463,+ 1.2054039239883423+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0+ ],+ [+ 0.7536231279373169,+ 0.41879838705062866,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4074.86572265625,+ -3334.4990234375+ ],+ [+ 4040.2958984375,+ -3316.3017578125+ ],+ [+ 2926.997314453125,+ -2398.78125+ ],+ [+ 4615.162109375,+ -3778.93896484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.905,+ "technical": 0.0447,+ "account": 0.0503+ },+ "confidence": 0.6544,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.8775,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.074,+ "1": 0.1616,+ "2": 0.5773,+ "3": 0.1871+ },+ "confidence": 0.1936,+ "answer_confidence": 0.5773,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9998,+ "confidence": 0.9998,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.3969,+ "confidence": 0.6031,+ "answer_confidence": 0.57,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_table_without_lang.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_table_without_lang.jsonnew file mode 100644index 00000000..b6c0211e--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_calibration_table_without_lang.json@@ -0,0 +1,1622 @@+{+ "name": "calibration_table_without_lang",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 4.488276958465576,+ -2.729666233062744,+ -2.4480319023132324,+ -10000.0+ ],+ [+ -0.8349542617797852,+ 0.8832842707633972,+ 3.6844465732574463,+ 1.2054039239883423+ ],+ [+ -4.515446186065674,+ 2.250016689300537,+ -10000.0,+ -10000.0+ ],+ [+ 0.7536231279373169,+ 0.41879838705062866,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4074.86572265625,+ -3334.4990234375+ ],+ [+ 4040.2958984375,+ -3316.3017578125+ ],+ [+ 2926.997314453125,+ -2398.78125+ ],+ [+ 4615.162109375,+ -3778.93896484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.905,+ "technical": 0.0447,+ "account": 0.0503+ },+ "confidence": 0.6544,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.8775,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.074,+ "1": 0.1616,+ "2": 0.5773,+ "3": 0.1871+ },+ "confidence": 0.1936,+ "answer_confidence": 0.5773,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9998,+ "confidence": 0.9998,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.3969,+ "confidence": 0.6031,+ "answer_confidence": 0.57,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/index.json b/laya-dotnet/tests/Laya.Tests/golden/english/index.jsonindex 263d884f..db8b9283 100644--- a/laya-dotnet/tests/Laya.Tests/golden/english/index.json+++ b/laya-dotnet/tests/Laya.Tests/golden/english/index.json@@ -653,5 +653,96 @@"file": "case_long_lang_miss.json","expects_error": false,"kind": "long"+ },+ {+ "name": "calibration_plain",+ "file": "case_calibration_plain.json",+ "expects_error": false+ },+ {+ "name": "calibration_buckets",+ "file": "case_calibration_buckets.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_override_skips_binning",+ "file": "case_calibration_lang_override_skips_binning.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_miss_bins",+ "file": "case_calibration_lang_miss_bins.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_null_entry_inherits_checkpoint",+ "file": "case_calibration_lang_null_entry_inherits_checkpoint.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_buckets_skip_binning",+ "file": "case_calibration_lang_buckets_skip_binning.json",+ "expects_error": false+ },+ {+ "name": "calibration_table_without_lang",+ "file": "case_calibration_table_without_lang.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence",+ "file": "case_calibration_min_confidence.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence_map",+ "file": "case_calibration_min_confidence_map.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence_buckets",+ "file": "case_calibration_min_confidence_buckets.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence_lang_override",+ "file": "case_calibration_min_confidence_lang_override.json",+ "expects_error": false+ },+ {+ "name": "calibration_batch",+ "file": "case_calibration_batch.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "calibration_batch_lang_override",+ "file": "case_calibration_batch_lang_override.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "calibration_batch_min_confidence",+ "file": "case_calibration_batch_min_confidence.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "calibration_long_scan",+ "file": "case_calibration_long_scan.json",+ "expects_error": false,+ "kind": "long"+ },+ {+ "name": "calibration_long_single_window",+ "file": "case_calibration_long_single_window.json",+ "expects_error": false,+ "kind": "long"+ },+ {+ "name": "calibration_long_lang_override",+ "file": "case_calibration_long_lang_override.json",+ "expects_error": false,+ "kind": "long"}]diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_batch.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_batch.jsonnew file mode 100644index 00000000..60539cb2--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_batch.json@@ -0,0 +1,584 @@+{+ "name": "calibration_batch",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, that is all sorted now.",+ {+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",+ "message": "Please cancel my account today."+ },+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "batch_size": 3,+ "sort_by_length": true+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_widths": [+ 90,+ 113,+ 143,+ 34,+ 150,+ 42,+ 90,+ 48+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6027,+ "technical": 0.1007,+ "other": 0.2967+ },+ "confidence": 0.1837,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.2934,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1116,+ "1": 0.4835,+ "2": 0.405+ },+ "confidence": 0.1243,+ "answer_confidence": 0.4835,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0193,+ "confidence": 0.9807,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 264,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.4981,+ "technical": 0.1316,+ "other": 0.3703+ },+ "confidence": 0.1062,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.353,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0907,+ "1": 0.4656,+ "2": 0.4437+ },+ "confidence": 0.1497,+ "answer_confidence": 0.4656,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.3369,+ "confidence": 0.6631,+ "answer_confidence": 0.57,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 333,+ "output_tokens": 0,+ "state_tokens": 79,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.2886,+ "technical": 0.2375,+ "other": 0.4739+ },+ "confidence": 0.0407,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.2856,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1157,+ "1": 0.483,+ "2": 0.4013+ },+ "confidence": 0.1194,+ "answer_confidence": 0.483,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0291,+ "confidence": 0.9709,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 423,+ "output_tokens": 0,+ "state_tokens": 109,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5478,+ "technical": 0.2401,+ "other": 0.2121+ },+ "confidence": 0.0887,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.0805,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.2401,+ "1": 0.4392,+ "2": 0.3207+ },+ "confidence": 0.0273,+ "answer_confidence": 0.4392,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0018,+ "confidence": 0.9982,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 96,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8291,+ "technical": 0.035,+ "other": 0.1359+ },+ "confidence": 0.5049,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.3443,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1399,+ "1": 0.3759,+ "2": 0.4842+ },+ "confidence": 0.0951,+ "answer_confidence": 0.4842,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0512,+ "confidence": 0.9488,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 444,+ "output_tokens": 0,+ "state_tokens": 116,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.1917,+ "technical": 0.2273,+ "other": 0.581+ },+ "confidence": 0.1181,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.1676,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.243,+ "1": 0.3464,+ "2": 0.4106+ },+ "confidence": 0.0201,+ "answer_confidence": 0.4106,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0015,+ "confidence": 0.9985,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 120,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.388,+ "technical": 0.3228,+ "other": 0.2891+ },+ "confidence": 0.0068,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.5179,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1413,+ "1": 0.1995,+ "2": 0.6592+ },+ "confidence": 0.2055,+ "answer_confidence": 0.6592,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.4805,+ "confidence": 0.5195,+ "answer_confidence": 0.57,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 264,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.783,+ "technical": 0.0918,+ "other": 0.1251+ },+ "confidence": 0.3894,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.417,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1282,+ "1": 0.3266,+ "2": 0.5452+ },+ "confidence": 0.1266,+ "answer_confidence": 0.5452,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.065,+ "confidence": 0.935,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 138,+ "output_tokens": 0,+ "state_tokens": 14,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 9,+ 48+ ],+ [+ 9,+ 113+ ],+ [+ 6,+ 150+ ]+ ]+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_batch_lang_override.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_batch_lang_override.jsonnew file mode 100644index 00000000..a46c5b83--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_batch_lang_override.json@@ -0,0 +1,457 @@+{+ "name": "calibration_batch_lang_override",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "lang": "pt"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_widths": [+ 90,+ 113,+ 143,+ 34,+ 150+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7873,+ "technical": 0.022,+ "other": 0.1908+ },+ "confidence": 0.4646,+ "answer_confidence": 0.7873,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.3084,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0985,+ "1": 0.4945,+ "2": 0.407+ },+ "confidence": 0.1421,+ "answer_confidence": 0.4945,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0193,+ "confidence": 0.9807,+ "answer_confidence": 0.9807,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 264,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6163,+ "technical": 0.043,+ "other": 0.3407+ },+ "confidence": 0.2713,+ "answer_confidence": 0.6163,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.3704,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0783,+ "1": 0.4731,+ "2": 0.4486+ },+ "confidence": 0.1689,+ "answer_confidence": 0.4731,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.3369,+ "confidence": 0.6631,+ "answer_confidence": 0.6631,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 333,+ "output_tokens": 0,+ "state_tokens": 79,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.2286,+ "technical": 0.1548,+ "other": 0.6166+ },+ "confidence": 0.1587,+ "answer_confidence": 0.6166,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.3005,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1026,+ "1": 0.4943,+ "2": 0.4031+ },+ "confidence": 0.1369,+ "answer_confidence": 0.4943,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0291,+ "confidence": 0.9709,+ "answer_confidence": 0.9709,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 423,+ "output_tokens": 0,+ "state_tokens": 109,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7451,+ "technical": 0.1431,+ "other": 0.1117+ },+ "confidence": 0.3243,+ "answer_confidence": 0.7451,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.0867,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.2316,+ "1": 0.45,+ "2": 0.3184+ },+ "confidence": 0.0329,+ "answer_confidence": 0.45,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0018,+ "confidence": 0.9982,+ "answer_confidence": 0.9982,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 96,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9722,+ "technical": 0.0017,+ "other": 0.0261+ },+ "confidence": 0.8783,+ "answer_confidence": 0.9722,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.3702,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1268,+ "1": 0.3762,+ "2": 0.497+ },+ "confidence": 0.1105,+ "answer_confidence": 0.497,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0512,+ "confidence": 0.9488,+ "answer_confidence": 0.9488,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 444,+ "output_tokens": 0,+ "state_tokens": 116,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 15,+ 150+ ]+ ]+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_batch_min_confidence.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_batch_min_confidence.jsonnew file mode 100644index 00000000..fc7f22e6--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_batch_min_confidence.json@@ -0,0 +1,439 @@+{+ "name": "calibration_batch_min_confidence",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.6+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_widths": [+ 90,+ 113,+ 143,+ 34,+ 150+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6027,+ "technical": 0.1007,+ "other": 0.2967+ },+ "confidence": 0.1837,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.2934,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1116,+ "1": 0.4835,+ "2": 0.405+ },+ "confidence": 0.1243,+ "answer_confidence": 0.4835,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0193,+ "confidence": 0.9807,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 264,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.4981,+ "technical": 0.1316,+ "other": 0.3703+ },+ "confidence": 0.1062,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.353,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0907,+ "1": 0.4656,+ "2": 0.4437+ },+ "confidence": 0.1497,+ "answer_confidence": 0.4656,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.3369,+ "confidence": 0.6631,+ "answer_confidence": 0.57,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 333,+ "output_tokens": 0,+ "state_tokens": 79,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.2886,+ "technical": 0.2375,+ "other": 0.4739+ },+ "confidence": 0.0407,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.2856,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1157,+ "1": 0.483,+ "2": 0.4013+ },+ "confidence": 0.1194,+ "answer_confidence": 0.483,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0291,+ "confidence": 0.9709,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 423,+ "output_tokens": 0,+ "state_tokens": 109,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5478,+ "technical": 0.2401,+ "other": 0.2121+ },+ "confidence": 0.0887,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.0805,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.2401,+ "1": 0.4392,+ "2": 0.3207+ },+ "confidence": 0.0273,+ "answer_confidence": 0.4392,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0018,+ "confidence": 0.9982,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 96,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8291,+ "technical": 0.035,+ "other": 0.1359+ },+ "confidence": 0.5049,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.3443,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1399,+ "1": 0.3759,+ "2": 0.4842+ },+ "confidence": 0.0951,+ "answer_confidence": 0.4842,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0512,+ "confidence": 0.9488,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 444,+ "output_tokens": 0,+ "state_tokens": 116,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 15,+ 150+ ]+ ]+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_buckets.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_buckets.jsonnew file mode 100644index 00000000..4fa47de0--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_buckets.json@@ -0,0 +1,1703 @@+{+ "name": "calibration_buckets",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "pair": {+ "type": "choice",+ "instructions": "Which of the two?",+ "criteria": {+ "first": "the first option",+ "second": "the second option"+ }+ },+ "wide": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices and payments",+ "technical": "bugs and outages",+ "account": "logins and seats",+ "sales": "quotes and upgrades",+ "legal": "contracts and privacy",+ "shipping": "delivery and returns",+ "other": "anything else"+ }+ },+ "scale": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "none",+ "low",+ "mild",+ "high",+ "severe",+ "critical"+ ]+ },+ "flag": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "per_question": [+ {+ "id": "pair",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which of the two?",+ "head_text": "choice question: Which of the two?",+ "options": [+ "first: the first option",+ "second: the second option"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 576,+ 573,+ 1378,+ 235336,+ 1,+ 4,+ 1370,+ 235292,+ 573,+ 1370,+ 4984,+ 4,+ 2257,+ 235292,+ 573,+ 2257,+ 4984,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 10,+ 16+ ]+ },+ {+ "id": "wide",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices and payments",+ "technical: bugs and outages",+ "account: logins and seats",+ "sales: quotes and upgrades",+ "legal: contracts and privacy",+ "shipping: delivery and returns",+ "other: anything else"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 578,+ 15598,+ 4,+ 9838,+ 235292,+ 30608,+ 578,+ 142788,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 578,+ 18723,+ 4,+ 7108,+ 235292,+ 11442,+ 578,+ 47800,+ 4,+ 7035,+ 235292,+ 20078,+ 578,+ 16961,+ 4,+ 10928,+ 235292,+ 8335,+ 578,+ 8753,+ 4,+ 1156,+ 235292,+ 4341,+ 1354,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 17,+ 23,+ 30,+ 36,+ 42,+ 48+ ]+ },+ {+ "id": "scale",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: none",+ "level 1: low",+ "level 2: mild",+ "level 3: high",+ "level 4: severe",+ "level 5: critical"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 8031,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 3178,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 21870,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 1536,+ 4,+ 2403,+ 235248,+ 235310,+ 235292,+ 14586,+ 4,+ 2403,+ 235248,+ 235308,+ 235292,+ 9523,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 17,+ 23,+ 29,+ 35,+ 41+ ]+ },+ {+ "id": "flag",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 576,+ 573,+ 1378,+ 235336,+ 1,+ 4,+ 1370,+ 235292,+ 573,+ 1370,+ 4984,+ 4,+ 2257,+ 235292,+ 573,+ 2257,+ 4984,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 578,+ 15598,+ 4,+ 9838,+ 235292,+ 30608,+ 578,+ 142788,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 578,+ 18723,+ 4,+ 7108,+ 235292,+ 11442,+ 578,+ 47800,+ 4,+ 7035,+ 235292,+ 20078,+ 578,+ 16961,+ 4,+ 10928,+ 235292,+ 8335,+ 578,+ 8753,+ 4,+ 1156,+ 235292,+ 4341,+ 1354,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 8031,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 3178,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 21870,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 1536,+ 4,+ 2403,+ 235248,+ 235310,+ 235292,+ 14586,+ 4,+ 2403,+ 235248,+ 235308,+ 235292,+ 9523,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 10,+ 16,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 30,+ 36,+ 42,+ 48+ ],+ [+ 11,+ 17,+ 23,+ 29,+ 35,+ 41,+ 0+ ],+ [+ 14,+ 24,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 0,+ 1,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ -4.589033603668213,+ -1.712645173072815,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ],+ [+ 9.534648895263672,+ -4.572531223297119,+ -1.807616114616394,+ -4.547646522521973,+ -4.577828884124756,+ -3.587461233139038,+ -2.5973122119903564+ ],+ [+ -4.291886806488037,+ -1.200973629951477,+ -2.5810704231262207,+ -0.7046122550964355,+ -2.3639867305755615,+ -1.3314738273620605,+ -10000.0+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1125.61669921875,+ -1344.60546875+ ],+ [+ 1235.69873046875,+ -1476.586669921875+ ],+ [+ 1509.5008544921875,+ -1785.146728515625+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "pair": {+ "type": "choice",+ "choice": "second",+ "probabilities": {+ "first": 0.1683,+ "second": 0.8317+ },+ "confidence": 0.3463,+ "answer_confidence": 0.8317,+ "action": {+ "act_probability": 1.0+ }+ },+ "wide": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 1.0,+ "technical": 0.0,+ "account": 0.0,+ "sales": 0.0,+ "legal": 0.0,+ "shipping": 0.0,+ "other": 0.0+ },+ "confidence": 1.0,+ "answer_confidence": 1.0,+ "action": {+ "act_probability": 1.0+ }+ },+ "scale": {+ "type": "score",+ "score": 2.7693,+ "legend": {+ "0": "none",+ "1": "low",+ "2": "mild",+ "3": "high",+ "4": "severe",+ "5": "critical"+ },+ "probabilities": {+ "0": 0.0743,+ "1": 0.2082,+ "2": 0.1314,+ "3": 0.2456,+ "4": 0.1413,+ "5": 0.1993+ },+ "confidence": 0.0349,+ "answer_confidence": 0.44,+ "action": {+ "act_probability": 1.0+ }+ },+ "flag": {+ "type": "noul",+ "noul": 0.0193,+ "confidence": 0.9807,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 386,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_buckets_skip_binning.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_buckets_skip_binning.jsonnew file mode 100644index 00000000..36bea8da--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_buckets_skip_binning.json@@ -0,0 +1,1762 @@+{+ "name": "calibration_lang_buckets_skip_binning",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "pair": {+ "type": "choice",+ "instructions": "Which of the two?",+ "criteria": {+ "first": "the first option",+ "second": "the second option"+ }+ },+ "wide": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices and payments",+ "technical": "bugs and outages",+ "account": "logins and seats",+ "sales": "quotes and upgrades",+ "legal": "contracts and privacy",+ "shipping": "delivery and returns",+ "other": "anything else"+ }+ },+ "scale": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "none",+ "low",+ "mild",+ "high",+ "severe",+ "critical"+ ]+ },+ "flag": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "per_question": [+ {+ "id": "pair",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which of the two?",+ "head_text": "choice question: Which of the two?",+ "options": [+ "first: the first option",+ "second: the second option"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 576,+ 573,+ 1378,+ 235336,+ 1,+ 4,+ 1370,+ 235292,+ 573,+ 1370,+ 4984,+ 4,+ 2257,+ 235292,+ 573,+ 2257,+ 4984,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 10,+ 16+ ]+ },+ {+ "id": "wide",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices and payments",+ "technical: bugs and outages",+ "account: logins and seats",+ "sales: quotes and upgrades",+ "legal: contracts and privacy",+ "shipping: delivery and returns",+ "other: anything else"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 578,+ 15598,+ 4,+ 9838,+ 235292,+ 30608,+ 578,+ 142788,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 578,+ 18723,+ 4,+ 7108,+ 235292,+ 11442,+ 578,+ 47800,+ 4,+ 7035,+ 235292,+ 20078,+ 578,+ 16961,+ 4,+ 10928,+ 235292,+ 8335,+ 578,+ 8753,+ 4,+ 1156,+ 235292,+ 4341,+ 1354,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 17,+ 23,+ 30,+ 36,+ 42,+ 48+ ]+ },+ {+ "id": "scale",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: none",+ "level 1: low",+ "level 2: mild",+ "level 3: high",+ "level 4: severe",+ "level 5: critical"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 8031,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 3178,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 21870,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 1536,+ 4,+ 2403,+ 235248,+ 235310,+ 235292,+ 14586,+ 4,+ 2403,+ 235248,+ 235308,+ 235292,+ 9523,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 17,+ 23,+ 29,+ 35,+ 41+ ]+ },+ {+ "id": "flag",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 576,+ 573,+ 1378,+ 235336,+ 1,+ 4,+ 1370,+ 235292,+ 573,+ 1370,+ 4984,+ 4,+ 2257,+ 235292,+ 573,+ 2257,+ 4984,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 578,+ 15598,+ 4,+ 9838,+ 235292,+ 30608,+ 578,+ 142788,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 578,+ 18723,+ 4,+ 7108,+ 235292,+ 11442,+ 578,+ 47800,+ 4,+ 7035,+ 235292,+ 20078,+ 578,+ 16961,+ 4,+ 10928,+ 235292,+ 8335,+ 578,+ 8753,+ 4,+ 1156,+ 235292,+ 4341,+ 1354,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 8031,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 3178,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 21870,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 1536,+ 4,+ 2403,+ 235248,+ 235310,+ 235292,+ 14586,+ 4,+ 2403,+ 235248,+ 235308,+ 235292,+ 9523,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 10,+ 16,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 30,+ 36,+ 42,+ 48+ ],+ [+ 11,+ 17,+ 23,+ 29,+ 35,+ 41,+ 0+ ],+ [+ 14,+ 24,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 0,+ 1,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ -4.589033603668213,+ -1.712645173072815,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ],+ [+ 9.534648895263672,+ -4.572531223297119,+ -1.807616114616394,+ -4.547646522521973,+ -4.577828884124756,+ -3.587461233139038,+ -2.5973122119903564+ ],+ [+ -4.291886806488037,+ -1.200973629951477,+ -2.5810704231262207,+ -0.7046122550964355,+ -2.3639867305755615,+ -1.3314738273620605,+ -10000.0+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1125.61669921875,+ -1344.60546875+ ],+ [+ 1235.69873046875,+ -1476.586669921875+ ],+ [+ 1509.5008544921875,+ -1785.146728515625+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "pair": {+ "type": "choice",+ "choice": "second",+ "probabilities": {+ "first": 0.2771,+ "second": 0.7229+ },+ "confidence": 0.1485,+ "answer_confidence": 0.7229,+ "action": {+ "act_probability": 1.0+ }+ },+ "wide": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 1.0,+ "technical": 0.0,+ "account": 0.0,+ "sales": 0.0,+ "legal": 0.0,+ "shipping": 0.0,+ "other": 0.0+ },+ "confidence": 1.0,+ "answer_confidence": 1.0,+ "action": {+ "act_probability": 1.0+ }+ },+ "scale": {+ "type": "score",+ "score": 2.8423,+ "legend": {+ "0": "none",+ "1": "low",+ "2": "mild",+ "3": "high",+ "4": "severe",+ "5": "critical"+ },+ "probabilities": {+ "0": 0.0475,+ "1": 0.2226,+ "2": 0.1116,+ "3": 0.2853,+ "4": 0.1244,+ "5": 0.2085+ },+ "confidence": 0.0691,+ "answer_confidence": 0.2853,+ "action": {+ "act_probability": 1.0+ }+ },+ "flag": {+ "type": "noul",+ "noul": 0.0414,+ "confidence": 0.9586,+ "answer_confidence": 0.9586,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 386,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "it"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_miss_bins.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_miss_bins.jsonnew file mode 100644index 00000000..c841d0fe--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_miss_bins.json@@ -0,0 +1,1589 @@+{+ "name": "calibration_lang_miss_bins",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 7.727238655090332,+ -4.555615425109863,+ -2.0609042644500732,+ -10000.0+ ],+ [+ -3.1127467155456543,+ 0.14541220664978027,+ -0.22580814361572266,+ -0.7165543437004089+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0+ ],+ [+ 1.247997522354126,+ -0.26682233810424805,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1252.7354736328125,+ -1494.896240234375+ ],+ [+ 1394.948486328125,+ -1645.8040771484375+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ],+ [+ 1266.344970703125,+ -1504.056396484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9776,+ "technical": 0.0059,+ "account": 0.0166+ },+ "confidence": 0.8906,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.7165,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0828,+ "1": 0.3639,+ "2": 0.3074,+ "3": 0.2459+ },+ "confidence": 0.0755,+ "answer_confidence": 0.3639,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0193,+ "confidence": 0.9807,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.1308,+ "confidence": 0.8692,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 375,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "ja"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_null_entry_inherits_checkpoint.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_null_entry_inherits_checkpoint.jsonnew file mode 100644index 00000000..7f06c6bf--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_null_entry_inherits_checkpoint.json@@ -0,0 +1,1589 @@+{+ "name": "calibration_lang_null_entry_inherits_checkpoint",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 7.727238655090332,+ -4.555615425109863,+ -2.0609042644500732,+ -10000.0+ ],+ [+ -3.1127467155456543,+ 0.14541220664978027,+ -0.22580814361572266,+ -0.7165543437004089+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0+ ],+ [+ 1.247997522354126,+ -0.26682233810424805,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1252.7354736328125,+ -1494.896240234375+ ],+ [+ 1394.948486328125,+ -1645.8040771484375+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ],+ [+ 1266.344970703125,+ -1504.056396484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9999,+ "technical": 0.0,+ "account": 0.0001+ },+ "confidence": 0.9994,+ "answer_confidence": 0.9999,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6956,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0179,+ "1": 0.465,+ "2": 0.3208,+ "3": 0.1964+ },+ "confidence": 0.1976,+ "answer_confidence": 0.465,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0414,+ "confidence": 0.9586,+ "answer_confidence": 0.9586,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.1802,+ "confidence": 0.8198,+ "answer_confidence": 0.8198,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 375,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "fr"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_override_skips_binning.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_override_skips_binning.jsonnew file mode 100644index 00000000..a0337ec6--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_lang_override_skips_binning.json@@ -0,0 +1,1589 @@+{+ "name": "calibration_lang_override_skips_binning",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 7.727238655090332,+ -4.555615425109863,+ -2.0609042644500732,+ -10000.0+ ],+ [+ -3.1127467155456543,+ 0.14541220664978027,+ -0.22580814361572266,+ -0.7165543437004089+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0+ ],+ [+ 1.247997522354126,+ -0.26682233810424805,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1252.7354736328125,+ -1494.896240234375+ ],+ [+ 1394.948486328125,+ -1645.8040771484375+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ],+ [+ 1266.344970703125,+ -1504.056396484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9904,+ "technical": 0.0021,+ "account": 0.0074+ },+ "confidence": 0.9463,+ "answer_confidence": 0.9904,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6888,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.1136,+ "1": 0.3365,+ "2": 0.2974,+ "3": 0.2525+ },+ "confidence": 0.0466,+ "answer_confidence": 0.3365,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.1096,+ "confidence": 0.8904,+ "answer_confidence": 0.8904,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.267,+ "confidence": 0.733,+ "answer_confidence": 0.733,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 375,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "de"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_long_lang_override.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_long_lang_override.jsonnew file mode 100644index 00000000..287028c7--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_long_lang_override.json@@ -0,0 +1,437 @@+{+ "name": "calibration_long_lang_override",+ "state": "The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "window": 96,+ "batch_size": 3,+ "lang": "de"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_tokens": 2036,+ "effective_window": 96,+ "effective_stride": 48,+ "effective_room": 990,+ "windows": [+ {+ "token_start": 0,+ "token_end": 96,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 48,+ "token_end": 144,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 96,+ "token_end": 192,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 144,+ "token_end": 240,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 192,+ "token_end": 288,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 240,+ "token_end": 336,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 288,+ "token_end": 384,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 336,+ "token_end": 432,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 384,+ "token_end": 480,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 432,+ "token_end": 528,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 480,+ "token_end": 576,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 528,+ "token_end": 624,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 576,+ "token_end": 672,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 624,+ "token_end": 720,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 672,+ "token_end": 768,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 720,+ "token_end": 816,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 768,+ "token_end": 864,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 816,+ "token_end": 912,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 864,+ "token_end": 960,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 912,+ "token_end": 1008,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 960,+ "token_end": 1056,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1008,+ "token_end": 1104,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1056,+ "token_end": 1152,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1104,+ "token_end": 1200,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1152,+ "token_end": 1248,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1200,+ "token_end": 1296,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1248,+ "token_end": 1344,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1296,+ "token_end": 1392,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1344,+ "token_end": 1440,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1392,+ "token_end": 1488,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 1440,+ "token_end": 1536,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 1488,+ "token_end": 1584,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1536,+ "token_end": 1632,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1584,+ "token_end": 1680,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1632,+ "token_end": 1728,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1680,+ "token_end": 1776,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1728,+ "token_end": 1824,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1776,+ "token_end": 1872,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1824,+ "token_end": 1920,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1872,+ "token_end": 1968,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1920,+ "token_end": 2016,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\n Hi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is"+ },+ {+ "token_start": 1968,+ "token_end": 2036,+ "text": ". The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\n Hi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "result": {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.9181,+ "technical": 0.037,+ "other": 0.0449+ },+ "confidence": 0.6907,+ "answer_confidence": 0.9181,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 40,+ "token_start": 1920,+ "token_end": 2016,+ "count": 42+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.0023,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.2612,+ "1": 0.4754,+ "2": 0.2635+ },+ "confidence": 0.0392,+ "answer_confidence": 0.4754,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 40,+ "token_start": 1920,+ "token_end": 2016,+ "count": 42+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.1049,+ "confidence": 0.8951,+ "answer_confidence": 0.8951,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 41,+ "token_start": 1968,+ "token_end": 2036,+ "count": 42+ }+ }+ },+ "usage": {+ "input_tokens": 16044,+ "output_tokens": 0,+ "state_tokens": 4004,+ "state_tokens_dropped": 0,+ "truncated": 0,+ "truncated_questions": [],+ "windows": 42+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_long_scan.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_long_scan.jsonnew file mode 100644index 00000000..cd61771e--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_long_scan.json@@ -0,0 +1,380 @@+{+ "name": "calibration_long_scan",+ "state": "The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "window": 96,+ "batch_size": 3+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_tokens": 2036,+ "effective_window": 96,+ "effective_stride": 48,+ "effective_room": 990,+ "windows": [+ {+ "token_start": 0,+ "token_end": 96,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 48,+ "token_end": 144,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 96,+ "token_end": 192,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 144,+ "token_end": 240,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 192,+ "token_end": 288,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 240,+ "token_end": 336,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 288,+ "token_end": 384,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 336,+ "token_end": 432,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 384,+ "token_end": 480,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 432,+ "token_end": 528,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 480,+ "token_end": 576,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 528,+ "token_end": 624,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 576,+ "token_end": 672,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 624,+ "token_end": 720,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 672,+ "token_end": 768,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 720,+ "token_end": 816,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 768,+ "token_end": 864,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 816,+ "token_end": 912,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 864,+ "token_end": 960,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 912,+ "token_end": 1008,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 960,+ "token_end": 1056,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1008,+ "token_end": 1104,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1056,+ "token_end": 1152,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1104,+ "token_end": 1200,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1152,+ "token_end": 1248,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1200,+ "token_end": 1296,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1248,+ "token_end": 1344,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1296,+ "token_end": 1392,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1344,+ "token_end": 1440,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1392,+ "token_end": 1488,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 1440,+ "token_end": 1536,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 1488,+ "token_end": 1584,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1536,+ "token_end": 1632,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1584,+ "token_end": 1680,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1632,+ "token_end": 1728,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1680,+ "token_end": 1776,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1728,+ "token_end": 1824,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1776,+ "token_end": 1872,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1824,+ "token_end": 1920,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1872,+ "token_end": 1968,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1920,+ "token_end": 2016,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\n Hi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is"+ },+ {+ "token_start": 1968,+ "token_end": 2036,+ "text": ". The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\n Hi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "result": {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7109,+ "technical": 0.0676,+ "other": 0.2214+ },+ "confidence": 0.3095,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 41,+ "token_start": 1968,+ "token_end": 2036,+ "count": 42+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.0028,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.2339,+ "1": 0.5294,+ "2": 0.2367+ },+ "confidence": 0.0737,+ "answer_confidence": 0.5294,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 40,+ "token_start": 1920,+ "token_end": 2016,+ "count": 42+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0176,+ "confidence": 0.9824,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 41,+ "token_start": 1968,+ "token_end": 2036,+ "count": 42+ }+ }+ },+ "usage": {+ "input_tokens": 16044,+ "output_tokens": 0,+ "state_tokens": 4004,+ "state_tokens_dropped": 0,+ "truncated": 0,+ "truncated_questions": [],+ "windows": 42+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_long_single_window.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_long_single_window.jsonnew file mode 100644index 00000000..67b3fe7b--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_long_single_window.json@@ -0,0 +1,148 @@+{+ "name": "calibration_long_single_window",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {},+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_tokens": 56,+ "effective_window": 760,+ "effective_stride": 380,+ "effective_room": 990,+ "windows": [],+ "result": {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6027,+ "technical": 0.1007,+ "other": 0.2967+ },+ "confidence": 0.1837,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.2934,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1116,+ "1": 0.4835,+ "2": 0.405+ },+ "confidence": 0.1243,+ "answer_confidence": 0.4835,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0193,+ "confidence": 0.9807,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 264,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": [],+ "windows": 1+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence.jsonnew file mode 100644index 00000000..368c7f9b--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence.json@@ -0,0 +1,1542 @@+{+ "name": "calibration_min_confidence",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 7.727238655090332,+ -4.555615425109863,+ -2.0609042644500732,+ -10000.0+ ],+ [+ -3.1127467155456543,+ 0.14541220664978027,+ -0.22580814361572266,+ -0.7165543437004089+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0+ ],+ [+ 1.247997522354126,+ -0.26682233810424805,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1252.7354736328125,+ -1494.896240234375+ ],+ [+ 1394.948486328125,+ -1645.8040771484375+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ],+ [+ 1266.344970703125,+ -1504.056396484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9776,+ "technical": 0.0059,+ "account": 0.0166+ },+ "confidence": 0.8906,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.7165,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0828,+ "1": 0.3639,+ "2": 0.3074,+ "3": 0.2459+ },+ "confidence": 0.0755,+ "answer_confidence": 0.3639,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0193,+ "confidence": 0.9807,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.1308,+ "confidence": 0.8692,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 375,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.6+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence_buckets.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence_buckets.jsonnew file mode 100644index 00000000..32019370--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence_buckets.json@@ -0,0 +1,1715 @@+{+ "name": "calibration_min_confidence_buckets",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "pair": {+ "type": "choice",+ "instructions": "Which of the two?",+ "criteria": {+ "first": "the first option",+ "second": "the second option"+ }+ },+ "wide": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices and payments",+ "technical": "bugs and outages",+ "account": "logins and seats",+ "sales": "quotes and upgrades",+ "legal": "contracts and privacy",+ "shipping": "delivery and returns",+ "other": "anything else"+ }+ },+ "scale": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "none",+ "low",+ "mild",+ "high",+ "severe",+ "critical"+ ]+ },+ "flag": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "per_question": [+ {+ "id": "pair",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which of the two?",+ "head_text": "choice question: Which of the two?",+ "options": [+ "first: the first option",+ "second: the second option"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 576,+ 573,+ 1378,+ 235336,+ 1,+ 4,+ 1370,+ 235292,+ 573,+ 1370,+ 4984,+ 4,+ 2257,+ 235292,+ 573,+ 2257,+ 4984,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 10,+ 16+ ]+ },+ {+ "id": "wide",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices and payments",+ "technical: bugs and outages",+ "account: logins and seats",+ "sales: quotes and upgrades",+ "legal: contracts and privacy",+ "shipping: delivery and returns",+ "other: anything else"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 578,+ 15598,+ 4,+ 9838,+ 235292,+ 30608,+ 578,+ 142788,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 578,+ 18723,+ 4,+ 7108,+ 235292,+ 11442,+ 578,+ 47800,+ 4,+ 7035,+ 235292,+ 20078,+ 578,+ 16961,+ 4,+ 10928,+ 235292,+ 8335,+ 578,+ 8753,+ 4,+ 1156,+ 235292,+ 4341,+ 1354,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 17,+ 23,+ 30,+ 36,+ 42,+ 48+ ]+ },+ {+ "id": "scale",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: none",+ "level 1: low",+ "level 2: mild",+ "level 3: high",+ "level 4: severe",+ "level 5: critical"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 8031,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 3178,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 21870,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 1536,+ 4,+ 2403,+ 235248,+ 235310,+ 235292,+ 14586,+ 4,+ 2403,+ 235248,+ 235308,+ 235292,+ 9523,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 17,+ 23,+ 29,+ 35,+ 41+ ]+ },+ {+ "id": "flag",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 576,+ 573,+ 1378,+ 235336,+ 1,+ 4,+ 1370,+ 235292,+ 573,+ 1370,+ 4984,+ 4,+ 2257,+ 235292,+ 573,+ 2257,+ 4984,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 578,+ 15598,+ 4,+ 9838,+ 235292,+ 30608,+ 578,+ 142788,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 578,+ 18723,+ 4,+ 7108,+ 235292,+ 11442,+ 578,+ 47800,+ 4,+ 7035,+ 235292,+ 20078,+ 578,+ 16961,+ 4,+ 10928,+ 235292,+ 8335,+ 578,+ 8753,+ 4,+ 1156,+ 235292,+ 4341,+ 1354,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 8031,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 3178,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 21870,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 1536,+ 4,+ 2403,+ 235248,+ 235310,+ 235292,+ 14586,+ 4,+ 2403,+ 235248,+ 235308,+ 235292,+ 9523,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 10,+ 16,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 30,+ 36,+ 42,+ 48+ ],+ [+ 11,+ 17,+ 23,+ 29,+ 35,+ 41,+ 0+ ],+ [+ 14,+ 24,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 0,+ 1,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ -4.589033603668213,+ -1.712645173072815,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ],+ [+ 9.534648895263672,+ -4.572531223297119,+ -1.807616114616394,+ -4.547646522521973,+ -4.577828884124756,+ -3.587461233139038,+ -2.5973122119903564+ ],+ [+ -4.291886806488037,+ -1.200973629951477,+ -2.5810704231262207,+ -0.7046122550964355,+ -2.3639867305755615,+ -1.3314738273620605,+ -10000.0+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1125.61669921875,+ -1344.60546875+ ],+ [+ 1235.69873046875,+ -1476.586669921875+ ],+ [+ 1509.5008544921875,+ -1785.146728515625+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "pair": {+ "type": "choice",+ "choice": "second",+ "probabilities": {+ "first": 0.1683,+ "second": 0.8317+ },+ "confidence": 0.3463,+ "answer_confidence": 0.8317,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "wide": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 1.0,+ "technical": 0.0,+ "account": 0.0,+ "sales": 0.0,+ "legal": 0.0,+ "shipping": 0.0,+ "other": 0.0+ },+ "confidence": 1.0,+ "answer_confidence": 1.0,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "scale": {+ "type": "score",+ "score": 2.7693,+ "legend": {+ "0": "none",+ "1": "low",+ "2": "mild",+ "3": "high",+ "4": "severe",+ "5": "critical"+ },+ "probabilities": {+ "0": 0.0743,+ "1": 0.2082,+ "2": 0.1314,+ "3": 0.2456,+ "4": 0.1413,+ "5": 0.1993+ },+ "confidence": 0.0349,+ "answer_confidence": 0.44,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "flag": {+ "type": "noul",+ "noul": 0.0193,+ "confidence": 0.9807,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ }+ },+ "usage": {+ "input_tokens": 386,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.5+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence_lang_override.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence_lang_override.jsonnew file mode 100644index 00000000..128c2637--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence_lang_override.json@@ -0,0 +1,1599 @@+{+ "name": "calibration_min_confidence_lang_override",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 7.727238655090332,+ -4.555615425109863,+ -2.0609042644500732,+ -10000.0+ ],+ [+ -3.1127467155456543,+ 0.14541220664978027,+ -0.22580814361572266,+ -0.7165543437004089+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0+ ],+ [+ 1.247997522354126,+ -0.26682233810424805,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1252.7354736328125,+ -1494.896240234375+ ],+ [+ 1394.948486328125,+ -1645.8040771484375+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ],+ [+ 1266.344970703125,+ -1504.056396484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9904,+ "technical": 0.0021,+ "account": 0.0074+ },+ "confidence": 0.9463,+ "answer_confidence": 0.9904,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.6888,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.1136,+ "1": 0.3365,+ "2": 0.2974,+ "3": 0.2525+ },+ "confidence": 0.0466,+ "answer_confidence": 0.3365,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.1096,+ "confidence": 0.8904,+ "answer_confidence": 0.8904,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.267,+ "confidence": 0.733,+ "answer_confidence": 0.733,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 375,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "de",+ "min_confidence": 0.6+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence_map.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence_map.jsonnew file mode 100644index 00000000..269185c0--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_min_confidence_map.json@@ -0,0 +1,1547 @@+{+ "name": "calibration_min_confidence_map",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 7.727238655090332,+ -4.555615425109863,+ -2.0609042644500732,+ -10000.0+ ],+ [+ -3.1127467155456543,+ 0.14541220664978027,+ -0.22580814361572266,+ -0.7165543437004089+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0+ ],+ [+ 1.247997522354126,+ -0.26682233810424805,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1252.7354736328125,+ -1494.896240234375+ ],+ [+ 1394.948486328125,+ -1645.8040771484375+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ],+ [+ 1266.344970703125,+ -1504.056396484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9776,+ "technical": 0.0059,+ "account": 0.0166+ },+ "confidence": 0.8906,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.7+ },+ "urgency": {+ "type": "score",+ "score": 1.7165,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0828,+ "1": 0.3639,+ "2": 0.3074,+ "3": 0.2459+ },+ "confidence": 0.0755,+ "answer_confidence": 0.3639,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0193,+ "confidence": 0.9807,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.1308,+ "confidence": 0.8692,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 375,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": {+ "choice:3-5": 0.7,+ "score:3-5": 0.5,+ "noul:2": 0.6+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_plain.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_plain.jsonnew file mode 100644index 00000000..0cd148ad--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_plain.json@@ -0,0 +1,1530 @@+{+ "name": "calibration_plain",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 7.727238655090332,+ -4.555615425109863,+ -2.0609042644500732,+ -10000.0+ ],+ [+ -3.1127467155456543,+ 0.14541220664978027,+ -0.22580814361572266,+ -0.7165543437004089+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0+ ],+ [+ 1.247997522354126,+ -0.26682233810424805,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1252.7354736328125,+ -1494.896240234375+ ],+ [+ 1394.948486328125,+ -1645.8040771484375+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ],+ [+ 1266.344970703125,+ -1504.056396484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9776,+ "technical": 0.0059,+ "account": 0.0166+ },+ "confidence": 0.8906,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.7165,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0828,+ "1": 0.3639,+ "2": 0.3074,+ "3": 0.2459+ },+ "confidence": 0.0755,+ "answer_confidence": 0.3639,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0193,+ "confidence": 0.9807,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.1308,+ "confidence": 0.8692,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 375,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_table_without_lang.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_table_without_lang.jsonnew file mode 100644index 00000000..2b1662e8--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_calibration_table_without_lang.json@@ -0,0 +1,1586 @@+{+ "name": "calibration_table_without_lang",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 22,+ 31+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 11,+ 18,+ 25,+ 31+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ "markers": [+ 17,+ 25+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 2,+ 6241,+ 2872,+ 235292,+ 12236,+ 9888,+ 1412,+ 6589,+ 736,+ 235336,+ 1,+ 4,+ 54972,+ 235292,+ 88220,+ 235269,+ 15598,+ 235269,+ 4199,+ 12036,+ 235269,+ 85869,+ 4,+ 9838,+ 235292,+ 30608,+ 235269,+ 142788,+ 235269,+ 16827,+ 10266,+ 4,+ 3185,+ 235292,+ 12138,+ 235256,+ 235269,+ 18723,+ 235269,+ 1780,+ 4559,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1+ ],+ [+ 2,+ 7716,+ 2872,+ 235292,+ 2250,+ 34722,+ 603,+ 736,+ 3853,+ 235336,+ 1,+ 4,+ 2403,+ 235248,+ 235276,+ 235292,+ 780,+ 34722,+ 4,+ 2403,+ 235248,+ 235274,+ 235292,+ 736,+ 2788,+ 4,+ 2403,+ 235248,+ 235284,+ 235292,+ 3646,+ 4,+ 2403,+ 235248,+ 235304,+ 235292,+ 7605,+ 4553,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 714,+ 6650,+ 80670,+ 577,+ 12459,+ 573,+ 3185,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 793,+ 235269,+ 573,+ 6218,+ 1721,+ 780,+ 3385,+ 4,+ 1382,+ 235292,+ 7778,+ 235269,+ 573,+ 6218,+ 12723,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 2,+ 552,+ 6311,+ 2872,+ 235292,+ 1417,+ 9286,+ 476,+ 3515,+ 10588,+ 4644,+ 1178,+ 671,+ 34177,+ 7260,+ 235265,+ 1,+ 4,+ 1566,+ 235292,+ 476,+ 11387,+ 7260,+ 116075,+ 665,+ 4,+ 1382,+ 235292,+ 476,+ 1552,+ 2004,+ 75103,+ 1,+ 28437,+ 235292,+ 12510,+ 12497,+ 611,+ 42529,+ 235248,+ 235321,+ 235321,+ 235284,+ 235274,+ 235304,+ 109,+ 11192,+ 235269,+ 590,+ 729,+ 100159,+ 11594,+ 736,+ 2514,+ 604,+ 573,+ 1122,+ 1780,+ 578,+ 573,+ 2257,+ 6256,+ 919,+ 780,+ 1125,+ 103147,+ 235265,+ 1417,+ 603,+ 573,+ 4906,+ 1069,+ 590,+ 791,+ 5952,+ 575,+ 235265,+ 1927,+ 665,+ 603,+ 780,+ 8115,+ 3646,+ 590,+ 1144,+ 132991,+ 573,+ 3185,+ 235265,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 22,+ 31,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 31+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 25,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 7.727238655090332,+ -4.555615425109863,+ -2.0609042644500732,+ -10000.0+ ],+ [+ -3.1127467155456543,+ 0.14541220664978027,+ -0.22580814361572266,+ -0.7165543437004089+ ],+ [+ 3.0695688724517822,+ -0.07253175973892212,+ -10000.0,+ -10000.0+ ],+ [+ 1.247997522354126,+ -0.26682233810424805,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 1252.7354736328125,+ -1494.896240234375+ ],+ [+ 1394.948486328125,+ -1645.8040771484375+ ],+ [+ 1234.01416015625,+ -1468.8492431640625+ ],+ [+ 1266.344970703125,+ -1504.056396484375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9776,+ "technical": 0.0059,+ "account": 0.0166+ },+ "confidence": 0.8906,+ "answer_confidence": 0.68,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.7165,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0828,+ "1": 0.3639,+ "2": 0.3074,+ "3": 0.2459+ },+ "confidence": 0.0755,+ "answer_confidence": 0.3639,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.0193,+ "confidence": 0.9807,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.1308,+ "confidence": 0.8692,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 375,+ "output_tokens": 0,+ "state_tokens": 56,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/index.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/index.jsonindex 263d884f..db8b9283 100644--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/index.json+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/index.json@@ -653,5 +653,96 @@"file": "case_long_lang_miss.json","expects_error": false,"kind": "long"+ },+ {+ "name": "calibration_plain",+ "file": "case_calibration_plain.json",+ "expects_error": false+ },+ {+ "name": "calibration_buckets",+ "file": "case_calibration_buckets.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_override_skips_binning",+ "file": "case_calibration_lang_override_skips_binning.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_miss_bins",+ "file": "case_calibration_lang_miss_bins.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_null_entry_inherits_checkpoint",+ "file": "case_calibration_lang_null_entry_inherits_checkpoint.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_buckets_skip_binning",+ "file": "case_calibration_lang_buckets_skip_binning.json",+ "expects_error": false+ },+ {+ "name": "calibration_table_without_lang",+ "file": "case_calibration_table_without_lang.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence",+ "file": "case_calibration_min_confidence.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence_map",+ "file": "case_calibration_min_confidence_map.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence_buckets",+ "file": "case_calibration_min_confidence_buckets.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence_lang_override",+ "file": "case_calibration_min_confidence_lang_override.json",+ "expects_error": false+ },+ {+ "name": "calibration_batch",+ "file": "case_calibration_batch.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "calibration_batch_lang_override",+ "file": "case_calibration_batch_lang_override.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "calibration_batch_min_confidence",+ "file": "case_calibration_batch_min_confidence.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "calibration_long_scan",+ "file": "case_calibration_long_scan.json",+ "expects_error": false,+ "kind": "long"+ },+ {+ "name": "calibration_long_single_window",+ "file": "case_calibration_long_single_window.json",+ "expects_error": false,+ "kind": "long"+ },+ {+ "name": "calibration_long_lang_override",+ "file": "case_calibration_long_lang_override.json",+ "expects_error": false,+ "kind": "long"}]diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/calibration_probe.json b/laya-dotnet/tests/Laya.Tests/golden/routing/calibration_probe.jsonnew file mode 100644index 00000000..89531bf5--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/calibration_probe.json@@ -0,0 +1,5116 @@+{+ "temp_min": 0.5,+ "temp_max": 5.0,+ "version": 2,+ "maps": {+ "main": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.1,+ 0.4,+ 0.6,+ 0.9+ ]+ },+ "noul:2": {+ "bins": 2,+ "values": [+ 0.25,+ 0.75+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ },+ "choice:6-10": {+ "bins": 10,+ "values": [+ 0.05,+ 0.15,+ 0.25,+ 0.35,+ 0.45,+ 0.55,+ 0.65,+ 0.75,+ 0.85,+ 0.95+ ]+ },+ "choice:11+": {+ "bins": 100,+ "values": [+ 0.0,+ 0.37,+ 0.74,+ 0.11,+ 0.48,+ 0.85,+ 0.22,+ 0.59,+ 0.96,+ 0.33,+ 0.7,+ 0.07,+ 0.44,+ 0.81,+ 0.18,+ 0.55,+ 0.92,+ 0.29,+ 0.66,+ 0.03,+ 0.4,+ 0.77,+ 0.14,+ 0.51,+ 0.88,+ 0.25,+ 0.62,+ 0.99,+ 0.36,+ 0.73,+ 0.1,+ 0.47,+ 0.84,+ 0.21,+ 0.58,+ 0.95,+ 0.32,+ 0.69,+ 0.06,+ 0.43,+ 0.8,+ 0.17,+ 0.54,+ 0.91,+ 0.28,+ 0.65,+ 0.02,+ 0.39,+ 0.76,+ 0.13,+ 0.5,+ 0.87,+ 0.24,+ 0.61,+ 0.98,+ 0.35,+ 0.72,+ 0.09,+ 0.46,+ 0.83,+ 0.2,+ 0.57,+ 0.94,+ 0.31,+ 0.68,+ 0.05,+ 0.42,+ 0.79,+ 0.16,+ 0.53,+ 0.9,+ 0.27,+ 0.64,+ 0.01,+ 0.38,+ 0.75,+ 0.12,+ 0.49,+ 0.86,+ 0.23,+ 0.6,+ 0.97,+ 0.34,+ 0.71,+ 0.08,+ 0.45,+ 0.82,+ 0.19,+ 0.56,+ 0.93,+ 0.3,+ 0.67,+ 0.04,+ 0.41,+ 0.78,+ 0.15,+ 0.52,+ 0.89,+ 0.26,+ 0.63+ ]+ }+ },+ "empty": {}+ },+ "payloads": [+ {+ "label": "temperatures only",+ "json": "{\"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "temperatures and buckets",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5, \"noul:2\": 0.8}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.8+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "null buckets",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": null}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "empty buckets",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "bucket names are not validated",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"bogus\": 2.0, \"\": 3.0}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "bogus": 2.0,+ "": 3.0+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "integers are numbers",+ "json": "{\"temperature\": [2, 3, 1], \"temperature_by_options\": {\"noul:2\": 2}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 2.0,+ 3.0,+ 1.0+ ],+ "temperature_by_options": {+ "noul:2": 2.0+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 1",+ "json": "{\"version\": 1, \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "null version is 1",+ "json": "{\"version\": null, \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "a binning map",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 4, \"values\": [0.1, 0.4, 0.6, 0.9]}, \"noul:2\": {\"bins\": 2, \"values\": [0.2, 0.8]}, \"score:2\": {\"bins\": 1, \"values\": [0.5]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.1,+ 0.4,+ 0.6,+ 0.9+ ]+ },+ "noul:2": {+ "bins": 2,+ "values": [+ 0.2,+ 0.8+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ },+ "warnings": []+ },+ {+ "label": "null binning map clears",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": null}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "empty binning map is kept",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": {}+ },+ "warnings": []+ },+ {+ "label": "values at the bounds",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 3, \"values\": [0.0, 1.0, 0.5]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": {+ "choice:3-5": {+ "bins": 3,+ "values": [+ 0.0,+ 1.0,+ 0.5+ ]+ }+ }+ },+ "warnings": []+ },+ {+ "label": "integer values",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [0, 1]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": {+ "choice:3-5": {+ "bins": 2,+ "values": [+ 0,+ 1+ ]+ }+ }+ },+ "warnings": []+ },+ {+ "label": "a bucket name is not validated",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"nonsense\": {\"bins\": 4, \"values\": [0.1, 0.4, 0.6, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": {+ "nonsense": {+ "bins": 4,+ "values": [+ 0.1,+ 0.4,+ 0.6,+ 0.9+ ]+ }+ }+ },+ "warnings": []+ },+ {+ "label": "a quote in a bucket name",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"it's\": {\"bins\": 4, \"values\": [0.1, 0.4, 0.6, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": {+ "it's": {+ "bins": 4,+ "values": [+ 0.1,+ 0.4,+ 0.6,+ 0.9+ ]+ }+ }+ },+ "warnings": []+ },+ {+ "label": "clamp high and low",+ "json": "{\"temperature\": [9.0, 0.1, 1.0]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 5.0,+ 0.5,+ 1.0+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": [+ {+ "category": "RuntimeWarning",+ "message": "laya: calibration temperatures are invalid or outside [0.5, 5]; using temperature[0]=9.0 -> 5, temperature[1]=0.1 -> 0.5. Treat confidence from the affected entries as uncalibrated."+ }+ ]+ },+ {+ "label": "clamp boundaries are accepted",+ "json": "{\"temperature\": [0.5, 5.0, 1.0]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 0.5,+ 5.0,+ 1.0+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "clamp a bucket",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 99.0, \"noul:2\": 0.01, \"score:3-5\": 2.0}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 5.0,+ "noul:2": 0.5,+ "score:3-5": 2.0+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "RuntimeWarning",+ "message": "laya: calibration temperatures are invalid or outside [0.5, 5]; using choice:3-5=99.0 -> 5, noul:2=0.01 -> 0.5. Treat confidence from the affected entries as uncalibrated."+ }+ ]+ },+ {+ "label": "clamp temperatures and buckets in one warning",+ "json": "{\"temperature\": [9.0, 1.0, 0.0], \"temperature_by_options\": {\"choice:2\": -1.0, \"noul:2\": 7.5}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 5.0,+ 1.0,+ 0.5+ ],+ "temperature_by_options": {+ "choice:2": 0.5,+ "noul:2": 5.0+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "RuntimeWarning",+ "message": "laya: calibration temperatures are invalid or outside [0.5, 5]; using temperature[0]=9.0 -> 5, temperature[2]=0.0 -> 0.5, choice:2=-1.0 -> 0.5, noul:2=7.5 -> 5. Treat confidence from the affected entries as uncalibrated."+ }+ ]+ },+ {+ "label": "negative zero",+ "json": "{\"temperature\": [-0.0, 1.0, 1.0]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 0.5,+ 1.0,+ 1.0+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": [+ {+ "category": "RuntimeWarning",+ "message": "laya: calibration temperatures are invalid or outside [0.5, 5]; using temperature[0]=-0.0 -> 0.5. Treat confidence from the affected entries as uncalibrated."+ }+ ]+ },+ {+ "label": "non-finite temperatures",+ "json": "{\"temperature\": [NaN, 1.0, Infinity], \"temperature_by_options\": {\"noul:2\": -Infinity}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.0,+ 1.0,+ 1.0+ ],+ "temperature_by_options": {+ "noul:2": 1.0+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "RuntimeWarning",+ "message": "laya: calibration temperatures are invalid or outside [0.5, 5]; using temperature[0]=nan -> 1, temperature[2]=inf -> 1, noul:2=-inf -> 1. Treat confidence from the affected entries as uncalibrated."+ }+ ]+ },+ {+ "label": "a bool is not a temperature",+ "json": "{\"temperature\": [true, 1.5, false]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.0,+ 1.5,+ 1.0+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": [+ {+ "category": "RuntimeWarning",+ "message": "laya: calibration temperatures are invalid or outside [0.5, 5]; using temperature[0]=True -> 1, temperature[2]=False -> 1. Treat confidence from the affected entries as uncalibrated."+ }+ ]+ },+ {+ "label": "a numeric string is read as float() reads it",+ "json": "{\"temperature\": [\"2.5\", \" 1e1 \", \"abc\"]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 2.5,+ 5.0,+ 1.0+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": [+ {+ "category": "RuntimeWarning",+ "message": "laya: calibration temperatures are invalid or outside [0.5, 5]; using temperature[1]=' 1e1 ' -> 5, temperature[2]='abc' -> 1. Treat confidence from the affected entries as uncalibrated."+ }+ ]+ },+ {+ "label": "null and nested values are 1.0",+ "json": "{\"temperature\": [null, [1.0], {\"a\": 1.0}]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.0,+ 1.0,+ 1.0+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": [+ {+ "category": "RuntimeWarning",+ "message": "laya: calibration temperatures are invalid or outside [0.5, 5]; using temperature[0]=None -> 1, temperature[1]=[1.0] -> 1, temperature[2]={'a': 1.0} -> 1. Treat confidence from the affected entries as uncalibrated."+ }+ ]+ },+ {+ "label": "bucket values that are not numbers",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:2\": true, \"noul:2\": null, \"score:2\": \"2.5\", \"choice:6-10\": 7}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:2": 1.0,+ "noul:2": 1.0,+ "score:2": 2.5,+ "choice:6-10": 5.0+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "RuntimeWarning",+ "message": "laya: calibration temperatures are invalid or outside [0.5, 5]; using choice:2=True -> 1, noul:2=None -> 1, choice:6-10=7 -> 5. Treat confidence from the affected entries as uncalibrated."+ }+ ]+ },+ {+ "label": "a quote in a clamped bucket name",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"it's\": 99.0}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "it's": 5.0+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "RuntimeWarning",+ "message": "laya: calibration temperatures are invalid or outside [0.5, 5]; using it's=99.0 -> 5. Treat confidence from the affected entries as uncalibrated."+ }+ ]+ },+ {+ "label": "version 2 for this checkpoint",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 2 written as a float",+ "json": "{\"version\": 2.0, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 2 as a string",+ "json": "{\"version\": \"2\", \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 2 as a padded string",+ "json": "{\"version\": \" 2 \", \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 2 as a signed string",+ "json": "{\"version\": \"+2\", \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 2 with an underscore",+ "json": "{\"version\": \"0_2\", \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 2 in fullwidth digits",+ "json": "{\"version\": \"\\uff12\", \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 3 is checked too",+ "json": "{\"version\": 3, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "a huge version is checked",+ "json": "{\"version\": 10000000000000000000000000000000000000000, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 1 is never checked",+ "json": "{\"version\": 1, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"elsewhere\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 0 is never checked",+ "json": "{\"version\": 0, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"elsewhere\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "negative version is never checked",+ "json": "{\"version\": -3, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"elsewhere\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 2 for another model",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya-multilingual\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='convaiinnovations/laya-multilingual' subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 for another subfolder",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"typed-decisions\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='convaiinnovations/laya' subfolder='typed-decisions' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 recorded without a subfolder",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": null, \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='convaiinnovations/laya' subfolder=None but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 with another config",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 512, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='convaiinnovations/laya' subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Checkpoint config identity differs. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 with an extra config key",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\", \"extra\": 1}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='convaiinnovations/laya' subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Checkpoint config identity differs. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 with a missing config key",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='convaiinnovations/laya' subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Checkpoint config identity differs. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 with a float where the config has an integer",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024.0, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 2 with a config that is a list",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": [1, 2]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='convaiinnovations/laya' subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Checkpoint config identity differs. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 with no config",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\"}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='convaiinnovations/laya' subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Checkpoint config identity differs. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 with a null config",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": null}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='convaiinnovations/laya' subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Checkpoint config identity differs. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 with nothing recorded",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path=None subfolder=None but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Checkpoint config identity differs. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 with a model that is not a string",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": 7, \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path=7 subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 with a model that is a list",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": [\"a\", \"b\"], \"subfolder\": {\"k\": 1.5}, \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path=['a', 'b'] subfolder={'k': 1.5} but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 with a quote in the model",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"it's\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path=\"it's\" subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 for the English checkpoint",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": null, \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": null,+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 2 for the English checkpoint recorded with a subfolder",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": null,+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='convaiinnovations/laya' subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder=None. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "version 2 for a local directory",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"onnx/multilingual\", \"subfolder\": null, \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "onnx/multilingual",+ "subfolder": null,+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": []+ },+ {+ "label": "version 2 for a local directory recorded for the hub",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "onnx/multilingual",+ "subfolder": null,+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='convaiinnovations/laya' subfolder='multilingual' but this agent is model_id_or_path='onnx/multilingual' subfolder=None. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "an identity warning and a clamp warning",+ "json": "{\"version\": 2, \"temperature\": [9.0, 1.0, 1.0], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"x\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 5.0,+ 1.0,+ 1.0+ ],+ "temperature_by_options": {+ "choice:3-5": 2.5+ },+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='x' subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Loading the temperatures anyway."+ },+ {+ "category": "RuntimeWarning",+ "message": "laya: calibration temperatures are invalid or outside [0.5, 5]; using temperature[0]=9.0 -> 5. Treat confidence from the affected entries as uncalibrated."+ }+ ]+ },+ {+ "label": "payload is a list",+ "json": "[1, 2, 3]",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must be an object, got list",+ "warnings": []+ },+ {+ "label": "payload is a number",+ "json": "3",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must be an object, got int",+ "warnings": []+ },+ {+ "label": "payload is a float",+ "json": "1.5",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must be an object, got float",+ "warnings": []+ },+ {+ "label": "payload is a string",+ "json": "\"abc\"",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must be an object, got str",+ "warnings": []+ },+ {+ "label": "payload is null",+ "json": "null",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must be an object, got NoneType",+ "warnings": []+ },+ {+ "label": "payload is true",+ "json": "true",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must be an object, got bool",+ "warnings": []+ },+ {+ "label": "payload is empty",+ "json": "{}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must contain temperature: [3 floats]",+ "warnings": []+ },+ {+ "label": "version is a bool",+ "json": "{\"version\": true, \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got True",+ "warnings": []+ },+ {+ "label": "version is false",+ "json": "{\"version\": false, \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got False",+ "warnings": []+ },+ {+ "label": "version is a list",+ "json": "{\"version\": [2], \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got [2]",+ "warnings": []+ },+ {+ "label": "version is an object",+ "json": "{\"version\": {\"v\": 2}, \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got {'v': 2}",+ "warnings": []+ },+ {+ "label": "version is not whole",+ "json": "{\"version\": 1.9, \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got 1.9",+ "warnings": []+ },+ {+ "label": "version is not whole and large",+ "json": "{\"version\": 2.5, \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got 2.5",+ "warnings": []+ },+ {+ "label": "version is a word",+ "json": "{\"version\": \"abc\", \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got 'abc'",+ "warnings": []+ },+ {+ "label": "version is an empty string",+ "json": "{\"version\": \"\", \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got ''",+ "warnings": []+ },+ {+ "label": "version is a decimal string",+ "json": "{\"version\": \"1.5\", \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got '1.5'",+ "warnings": []+ },+ {+ "label": "version is a double underscore string",+ "json": "{\"version\": \"1__2\", \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got '1__2'",+ "warnings": []+ },+ {+ "label": "version has a quote",+ "json": "{\"version\": \"it's\", \"temperature\": [1.5, 2.0, 1.25]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got \"it's\"",+ "warnings": []+ },+ {+ "label": "version is NaN",+ "json": "{\"version\": NaN, \"temperature\": [1.0, 1.0, 1.0]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got nan",+ "warnings": []+ },+ {+ "label": "version is Infinity",+ "json": "{\"version\": Infinity, \"temperature\": [1.0, 1.0, 1.0]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got inf",+ "warnings": []+ },+ {+ "label": "version is an overflowing float",+ "json": "{\"version\": 1e999, \"temperature\": [1.0, 1.0, 1.0]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON version must be an integer, got inf",+ "warnings": []+ },+ {+ "label": "version is a huge whole float",+ "json": "{\"version\": 1e300, \"temperature\": [1.0, 1.0, 1.0]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.0,+ 1.0,+ 1.0+ ],+ "temperature_by_options": {},+ "binning_map": null+ },+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path=None subfolder=None but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Checkpoint config identity differs. Loading the temperatures anyway."+ }+ ]+ },+ {+ "label": "temperature is missing",+ "json": "{\"version\": 1}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must contain temperature: [3 floats]",+ "warnings": []+ },+ {+ "label": "temperature is null",+ "json": "{\"temperature\": null}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must contain temperature: [3 floats]",+ "warnings": []+ },+ {+ "label": "temperature has two values",+ "json": "{\"temperature\": [1.0, 2.0]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must contain temperature: [3 floats]",+ "warnings": []+ },+ {+ "label": "temperature has four values",+ "json": "{\"temperature\": [1.0, 2.0, 3.0, 4.0]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must contain temperature: [3 floats]",+ "warnings": []+ },+ {+ "label": "temperature is empty",+ "json": "{\"temperature\": []}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must contain temperature: [3 floats]",+ "warnings": []+ },+ {+ "label": "temperature is a number",+ "json": "{\"temperature\": 2.0}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must contain temperature: [3 floats]",+ "warnings": []+ },+ {+ "label": "temperature is a string of three",+ "json": "{\"temperature\": \"abc\"}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must contain temperature: [3 floats]",+ "warnings": []+ },+ {+ "label": "temperature is an object of three",+ "json": "{\"temperature\": {\"a\": 1, \"b\": 2, \"c\": 3}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON must contain temperature: [3 floats]",+ "warnings": []+ },+ {+ "label": "buckets are a list",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": [1.0]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON temperature_by_options must be an object of bucket -> float, got list",+ "warnings": []+ },+ {+ "label": "buckets are a number",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": 3}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON temperature_by_options must be an object of bucket -> float, got int",+ "warnings": []+ },+ {+ "label": "buckets are a float",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": 1.5}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON temperature_by_options must be an object of bucket -> float, got float",+ "warnings": []+ },+ {+ "label": "buckets are a string",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": \"x\"}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON temperature_by_options must be an object of bucket -> float, got str",+ "warnings": []+ },+ {+ "label": "buckets are false",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": false}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON temperature_by_options must be an object of bucket -> float, got bool",+ "warnings": []+ },+ {+ "label": "buckets are an empty list",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": []}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON temperature_by_options must be an object of bucket -> float, got list",+ "warnings": []+ },+ {+ "label": "binning map is a list",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": []}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map must be an object of bucket -> {bins, values}, got list",+ "warnings": []+ },+ {+ "label": "binning map is a number",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": 3}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map must be an object of bucket -> {bins, values}, got int",+ "warnings": []+ },+ {+ "label": "binning map is a string",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": \"x\"}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map must be an object of bucket -> {bins, values}, got str",+ "warnings": []+ },+ {+ "label": "binning map is false",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": false}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map must be an object of bucket -> {bins, values}, got bool",+ "warnings": []+ },+ {+ "label": "entry is a list",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": [0.1, 0.9]}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must be an object with \"bins\" and \"values\", got list",+ "warnings": []+ },+ {+ "label": "entry is null",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": null}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must be an object with \"bins\" and \"values\", got NoneType",+ "warnings": []+ },+ {+ "label": "entry is a number",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": 0.5}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must be an object with \"bins\" and \"values\", got float",+ "warnings": []+ },+ {+ "label": "entry is a string",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": \"x\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must be an object with \"bins\" and \"values\", got str",+ "warnings": []+ },+ {+ "label": "entry is empty",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have an integer \"bins\" >= 1, got None",+ "warnings": []+ },+ {+ "label": "entry is true",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": true}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must be an object with \"bins\" and \"values\", got bool",+ "warnings": []+ },+ {+ "label": "a quote in a refused bucket name",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"it's\": 3}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map[\"it's\"] must be an object with \"bins\" and \"values\", got int",+ "warnings": []+ },+ {+ "label": "bins is missing",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have an integer \"bins\" >= 1, got None",+ "warnings": []+ },+ {+ "label": "bins is null",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": null, \"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have an integer \"bins\" >= 1, got None",+ "warnings": []+ },+ {+ "label": "bins is zero",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 0, \"values\": []}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have an integer \"bins\" >= 1, got 0",+ "warnings": []+ },+ {+ "label": "bins is negative",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": -2, \"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have an integer \"bins\" >= 1, got -2",+ "warnings": []+ },+ {+ "label": "bins is a whole float",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2.0, \"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have an integer \"bins\" >= 1, got 2.0",+ "warnings": []+ },+ {+ "label": "bins is a float",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2.5, \"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have an integer \"bins\" >= 1, got 2.5",+ "warnings": []+ },+ {+ "label": "bins is a string",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": \"2\", \"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have an integer \"bins\" >= 1, got '2'",+ "warnings": []+ },+ {+ "label": "bins is true",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": true, \"values\": [0.1]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have an integer \"bins\" >= 1, got True",+ "warnings": []+ },+ {+ "label": "bins is a list",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": [2], \"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have an integer \"bins\" >= 1, got [2]",+ "warnings": []+ },+ {+ "label": "bins is a huge integer",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 1000000000000000000000000000000, \"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have \"values\" of length \"bins\" (1000000000000000000000000000000)",+ "warnings": []+ },+ {+ "label": "bins is a negative huge integer",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": -1000000000000000000000000000000, \"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have an integer \"bins\" >= 1, got -1000000000000000000000000000000",+ "warnings": []+ },+ {+ "label": "bins is NaN",+ "json": "{\"temperature\": [1.0, 1.0, 1.0], \"binning_map\": {\"a\": {\"bins\": NaN, \"values\": [0.5]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['a'] must have an integer \"bins\" >= 1, got nan",+ "warnings": []+ },+ {+ "label": "values is missing",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have \"values\" of length \"bins\" (2)",+ "warnings": []+ },+ {+ "label": "values is null",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": null}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have \"values\" of length \"bins\" (2)",+ "warnings": []+ },+ {+ "label": "values is too short",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 3, \"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have \"values\" of length \"bins\" (3)",+ "warnings": []+ },+ {+ "label": "values is too long",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 1, \"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have \"values\" of length \"bins\" (1)",+ "warnings": []+ },+ {+ "label": "values is a string",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 3, \"values\": \"abc\"}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have \"values\" of length \"bins\" (3)",+ "warnings": []+ },+ {+ "label": "values is an object of the right size",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": {\"a\": 0.1, \"b\": 0.9}}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have \"values\" of length \"bins\" (2)",+ "warnings": []+ },+ {+ "label": "values is a number",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 1, \"values\": 0.5}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] must have \"values\" of length \"bins\" (1)",+ "warnings": []+ },+ {+ "label": "a value is null",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [0.1, null]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be numbers, got [0.1, None]",+ "warnings": []+ },+ {+ "label": "a value is a string",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [0.1, \"0.9\"]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be numbers, got [0.1, '0.9']",+ "warnings": []+ },+ {+ "label": "a value is a word",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [0.1, \"abc\"]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be numbers, got [0.1, 'abc']",+ "warnings": []+ },+ {+ "label": "a value is true",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [0.1, true]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be numbers, got [0.1, True]",+ "warnings": []+ },+ {+ "label": "a value is false",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [false, 0.5]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be numbers, got [False, 0.5]",+ "warnings": []+ },+ {+ "label": "a value is a list",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [0.1, [0.9]]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be numbers, got [0.1, [0.9]]",+ "warnings": []+ },+ {+ "label": "a value is an object",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [{\"v\": 0.1}, 0.5]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be numbers, got [{'v': 0.1}, 0.5]",+ "warnings": []+ },+ {+ "label": "a value is NaN",+ "json": "{\"temperature\": [1.0, 1.0, 1.0], \"binning_map\": {\"a\": {\"bins\": 2, \"values\": [0.1, NaN]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['a'] values must be finite, got [0.1, nan]",+ "warnings": []+ },+ {+ "label": "a value is Infinity",+ "json": "{\"temperature\": [1.0, 1.0, 1.0], \"binning_map\": {\"a\": {\"bins\": 2, \"values\": [0.1, Infinity]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['a'] values must be finite, got [0.1, inf]",+ "warnings": []+ },+ {+ "label": "a value is negative Infinity",+ "json": "{\"temperature\": [1.0, 1.0, 1.0], \"binning_map\": {\"a\": {\"bins\": 1, \"values\": [-Infinity]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['a'] values must be finite, got [-inf]",+ "warnings": []+ },+ {+ "label": "a value overflows to Infinity",+ "json": "{\"temperature\": [1.0, 1.0, 1.0], \"binning_map\": {\"a\": {\"bins\": 1, \"values\": [1e999]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['a'] values must be finite, got [inf]",+ "warnings": []+ },+ {+ "label": "a value is below zero",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [-0.1, 0.5]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be in [0, 1], got [-0.1, 0.5]",+ "warnings": []+ },+ {+ "label": "a value is above one",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [0.5, 1.0000001]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be in [0, 1], got [0.5, 1.0000001]",+ "warnings": []+ },+ {+ "label": "a value is an integer above one",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [0.5, 2]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be in [0, 1], got [0.5, 2]",+ "warnings": []+ },+ {+ "label": "a value is -0.0",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 1, \"values\": [-0.0]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 1.5,+ 2.0,+ 1.25+ ],+ "temperature_by_options": {},+ "binning_map": {+ "choice:3-5": {+ "bins": 1,+ "values": [+ -0.0+ ]+ }+ }+ },+ "warnings": []+ },+ {+ "label": "a value is a huge integer",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [0.5, 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "OverflowError",+ "error_message": "int too large to convert to float",+ "warnings": []+ },+ {+ "label": "a huge integer value after a bad one",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 3, \"values\": [null, 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, 0.5]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be numbers, got [None, 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, 0.5]",+ "warnings": []+ },+ {+ "label": "a bad value after a huge integer",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 3, \"values\": [10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, null, 0.5]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "OverflowError",+ "error_message": "int too large to convert to float",+ "warnings": []+ },+ {+ "label": "a string value before a list value",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [\"0.5\", [1]]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be numbers, got ['0.5', [1]]",+ "warnings": []+ },+ {+ "label": "an out-of-range value before a finite one",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [2.0, 0.5]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be in [0, 1], got [2.0, 0.5]",+ "warnings": []+ },+ {+ "label": "an out-of-range value before a non-finite one",+ "json": "{\"temperature\": [1.0, 1.0, 1.0], \"binning_map\": {\"a\": {\"bins\": 2, \"values\": [2.0, NaN]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['a'] values must be in [0, 1], got [2.0, nan]",+ "warnings": []+ },+ {+ "label": "the second entry is the bad one",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 4, \"values\": [0.1, 0.4, 0.6, 0.9]}, \"noul:2\": {\"bins\": 2, \"values\": [0.1, 5.0]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['noul:2'] values must be in [0, 1], got [0.1, 5.0]",+ "warnings": []+ },+ {+ "label": "the first entry is the bad one",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"binning_map\": {\"choice:3-5\": {\"bins\": 2, \"values\": [0.1, 5.0]}, \"noul:2\": {\"bins\": 2, \"values\": [0.2, \"x\"]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['choice:3-5'] values must be in [0, 1], got [0.1, 5.0]",+ "warnings": []+ },+ {+ "label": "a refusal beats a temperature warning",+ "json": "{\"temperature\": [9.0, 1.0, 1.0], \"binning_map\": {\"a\": {\"bins\": 1, \"values\": [3.0]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['a'] values must be in [0, 1], got [3.0]",+ "warnings": []+ },+ {+ "label": "a refusal beats an identity warning",+ "json": "{\"version\": 2, \"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"convaiinnovations/laya\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}, \"binning_map\": {\"a\": {\"bins\": 1, \"values\": [3.0]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "ValueError",+ "error_message": "calibration JSON binning_map['a'] values must be in [0, 1], got [3.0]",+ "warnings": []+ },+ {+ "label": "a duplicate key keeps its last value",+ "json": "{\"temperature\": [1.0, 2.0, 3.0], \"temperature\": [4.0, 5.0, 1.5], \"binning_map\": {\"a\": {\"bins\": 1, \"values\": [0.5]}, \"a\": {\"bins\": 2, \"values\": [0.1, 0.9]}}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "result": {+ "temperature": [+ 4.0,+ 5.0,+ 1.5+ ],+ "temperature_by_options": {},+ "binning_map": {+ "a": {+ "bins": 2,+ "values": [+ 0.1,+ 0.9+ ]+ }+ }+ },+ "warnings": []+ },+ {+ "label": "a temperature that is a huge integer",+ "json": "{\"temperature\": [10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, 1.0, 1.0]}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "OverflowError",+ "error_message": "int too large to convert to float",+ "warnings": []+ },+ {+ "label": "a bucket temperature that is a huge integer",+ "json": "{\"temperature\": [1.5, 2.0, 1.25], \"temperature_by_options\": {\"noul:2\": 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "OverflowError",+ "error_message": "int too large to convert to float",+ "warnings": []+ },+ {+ "label": "a huge integer after an identity warning",+ "json": "{\"version\": 2, \"temperature\": [10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, 1.0, 1.0], \"temperature_by_options\": {\"choice:3-5\": 2.5}, \"model_id_or_path\": \"x\", \"subfolder\": \"multilingual\", \"config\": {\"encoder\": \"jhu-clsp/mmBERT-base\", \"max_len\": 1024, \"head_max_len\": 192, \"option_layout\": \"sequential\"}}",+ "identity": {+ "model_id_or_path": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "cfg": {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [+ 1.2,+ 1.3,+ 1.4+ ],+ "temperature_by_options": {+ "choice:2": 1.1+ }+ }+ },+ "error_type": "OverflowError",+ "error_message": "int too large to convert to float",+ "warnings": [+ {+ "category": "UserWarning",+ "message": "calibration was fitted for model_id_or_path='x' subfolder='multilingual' but this agent is model_id_or_path='convaiinnovations/laya' subfolder='multilingual'. Loading the temperatures anyway."+ }+ ]+ }+ ],+ "lookups": [+ {+ "label": "choice:3-5 at 0.0",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.0,+ "result": 0.1+ },+ {+ "label": "choice:3-5 at 1e-300",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 1e-300,+ "result": 0.1+ },+ {+ "label": "choice:3-5 at 0.1",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.1,+ "result": 0.1+ },+ {+ "label": "choice:3-5 at 0.2499999999999999",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.2499999999999999,+ "result": 0.1+ },+ {+ "label": "choice:3-5 at 0.25",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.25,+ "result": 0.4+ },+ {+ "label": "choice:3-5 at 0.2500000000000001",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.2500000000000001,+ "result": 0.4+ },+ {+ "label": "choice:3-5 at 0.3",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.3,+ "result": 0.4+ },+ {+ "label": "choice:3-5 at 0.4999999999999999",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.4999999999999999,+ "result": 0.4+ },+ {+ "label": "choice:3-5 at 0.5",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.5,+ "result": 0.6+ },+ {+ "label": "choice:3-5 at 0.5000000000000001",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.5000000000000001,+ "result": 0.6+ },+ {+ "label": "choice:3-5 at 0.6",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.6,+ "result": 0.6+ },+ {+ "label": "choice:3-5 at 0.7499999999999999",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.7499999999999999,+ "result": 0.6+ },+ {+ "label": "choice:3-5 at 0.75",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.75,+ "result": 0.9+ },+ {+ "label": "choice:3-5 at 0.7500000000000001",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.7500000000000001,+ "result": 0.9+ },+ {+ "label": "choice:3-5 at 0.9",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.9,+ "result": 0.9+ },+ {+ "label": "choice:3-5 at 0.9999999999999999",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 0.9999999999999999,+ "result": 0.9+ },+ {+ "label": "choice:3-5 at 1.0",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 1.0,+ "result": 0.9+ },+ {+ "label": "choice:3-5 at -0.0",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": -0.0,+ "result": 0.1+ },+ {+ "label": "choice:3-5 at -0.1",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": -0.1,+ "result": 0.1+ },+ {+ "label": "choice:3-5 at -1e+300",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": -1e+300,+ "result": 0.1+ },+ {+ "label": "choice:3-5 at 1.0000000000000002",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 1.0000000000000002,+ "result": 0.9+ },+ {+ "label": "choice:3-5 at 1.5",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 1.5,+ "result": 0.9+ },+ {+ "label": "choice:3-5 at 7.0",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 7.0,+ "result": 0.9+ },+ {+ "label": "choice:3-5 at 1e+300",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": 1e+300,+ "result": 0.9+ },+ {+ "label": "one bin at 0.0",+ "map": "main",+ "bucket": "score:2",+ "confidence": 0.0,+ "result": 0.5+ },+ {+ "label": "one bin at 0.5",+ "map": "main",+ "bucket": "score:2",+ "confidence": 0.5,+ "result": 0.5+ },+ {+ "label": "one bin at 1.0",+ "map": "main",+ "bucket": "score:2",+ "confidence": 1.0,+ "result": 0.5+ },+ {+ "label": "miss at 0.0",+ "map": "main",+ "bucket": "score:3-5",+ "confidence": 0.0,+ "result": 0.0+ },+ {+ "label": "miss at 0.123456789",+ "map": "main",+ "bucket": "score:3-5",+ "confidence": 0.123456789,+ "result": 0.123456789+ },+ {+ "label": "miss at 0.5",+ "map": "main",+ "bucket": "score:3-5",+ "confidence": 0.5,+ "result": 0.5+ },+ {+ "label": "miss at 1.0",+ "map": "main",+ "bucket": "score:3-5",+ "confidence": 1.0,+ "result": 1.0+ },+ {+ "label": "miss at 1.5",+ "map": "main",+ "bucket": "score:3-5",+ "confidence": 1.5,+ "result": 1.5+ },+ {+ "label": "miss at -0.25",+ "map": "main",+ "bucket": "score:3-5",+ "confidence": -0.25,+ "result": -0.25+ },+ {+ "label": "miss on an empty bucket name",+ "map": "main",+ "bucket": "",+ "confidence": 0.4,+ "result": 0.4+ },+ {+ "label": "miss on a nearly right name",+ "map": "main",+ "bucket": "choice:3-5 ",+ "confidence": 0.4,+ "result": 0.4+ },+ {+ "label": "miss on another case",+ "map": "main",+ "bucket": "CHOICE:3-5",+ "confidence": 0.4,+ "result": 0.4+ },+ {+ "label": "empty map",+ "map": "empty",+ "bucket": "choice:3-5",+ "confidence": 0.4,+ "result": 0.4+ },+ {+ "label": "ten bins at 0.1",+ "map": "main",+ "bucket": "choice:6-10",+ "confidence": 0.1,+ "result": 0.15+ },+ {+ "label": "ten bins at 0.2",+ "map": "main",+ "bucket": "choice:6-10",+ "confidence": 0.2,+ "result": 0.25+ },+ {+ "label": "ten bins at 0.3",+ "map": "main",+ "bucket": "choice:6-10",+ "confidence": 0.3,+ "result": 0.35+ },+ {+ "label": "ten bins at 0.7",+ "map": "main",+ "bucket": "choice:6-10",+ "confidence": 0.7,+ "result": 0.75+ },+ {+ "label": "ten bins at 0.57",+ "map": "main",+ "bucket": "choice:6-10",+ "confidence": 0.57,+ "result": 0.55+ },+ {+ "label": "ten bins at 0.9",+ "map": "main",+ "bucket": "choice:6-10",+ "confidence": 0.9,+ "result": 0.95+ },+ {+ "label": "ten bins at 0.95",+ "map": "main",+ "bucket": "choice:6-10",+ "confidence": 0.95,+ "result": 0.95+ },+ {+ "label": "ten bins at 0.99",+ "map": "main",+ "bucket": "choice:6-10",+ "confidence": 0.99,+ "result": 0.95+ },+ {+ "label": "ten bins at 0.999999",+ "map": "main",+ "bucket": "choice:6-10",+ "confidence": 0.999999,+ "result": 0.95+ },+ {+ "label": "hundred bins at 0.01",+ "map": "main",+ "bucket": "choice:11+",+ "confidence": 0.01,+ "result": 0.37+ },+ {+ "label": "hundred bins at 0.07",+ "map": "main",+ "bucket": "choice:11+",+ "confidence": 0.07,+ "result": 0.59+ },+ {+ "label": "hundred bins at 0.29",+ "map": "main",+ "bucket": "choice:11+",+ "confidence": 0.29,+ "result": 0.36+ },+ {+ "label": "hundred bins at 0.57",+ "map": "main",+ "bucket": "choice:11+",+ "confidence": 0.57,+ "result": 0.72+ },+ {+ "label": "hundred bins at 0.58",+ "map": "main",+ "bucket": "choice:11+",+ "confidence": 0.58,+ "result": 0.09+ },+ {+ "label": "hundred bins at 0.99",+ "map": "main",+ "bucket": "choice:11+",+ "confidence": 0.99,+ "result": 0.63+ },+ {+ "label": "hundred bins at 1.0",+ "map": "main",+ "bucket": "choice:11+",+ "confidence": 1.0,+ "result": 0.63+ },+ {+ "label": "two bins at 0.5",+ "map": "main",+ "bucket": "noul:2",+ "confidence": 0.5,+ "result": 0.75+ },+ {+ "label": "two bins at 0.7499999999999999",+ "map": "main",+ "bucket": "noul:2",+ "confidence": 0.7499999999999999,+ "result": 0.75+ },+ {+ "label": "two bins at 0.75",+ "map": "main",+ "bucket": "noul:2",+ "confidence": 0.75,+ "result": 0.75+ },+ {+ "label": "two bins at 1.0",+ "map": "main",+ "bucket": "noul:2",+ "confidence": 1.0,+ "result": 0.75+ },+ {+ "label": "nan",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": "nan",+ "error_type": "ValueError",+ "error_message": "cannot convert float NaN to integer"+ },+ {+ "label": "inf",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": "inf",+ "error_type": "OverflowError",+ "error_message": "cannot convert float infinity to integer"+ },+ {+ "label": "-inf",+ "map": "main",+ "bucket": "choice:3-5",+ "confidence": "-inf",+ "error_type": "OverflowError",+ "error_message": "cannot convert float infinity to integer"+ },+ {+ "label": "nan on a miss",+ "map": "main",+ "bucket": "score:3-5",+ "confidence": "nan",+ "result": "nan"+ },+ {+ "label": "inf on a miss",+ "map": "main",+ "bucket": "score:3-5",+ "confidence": "inf",+ "result": "inf"+ }+ ]+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json b/laya-dotnet/tests/Laya.Tests/golden/routing/meta.jsonindex 8e6d3260..3c88f37a 100644--- a/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json@@ -19,6 +19,8 @@"confidence_probe_reuse": 2,"confidence_probe_jev": 96,"lang_temperatures_probe": 84,+ "calibration_probe_payloads": 152,+ "calibration_probe_lookups": 62,"router_forwarding_probe_predict": 14,"router_forwarding_probe_long": 12,"router_forwarding_probe_batch": 7,diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_batch.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_batch.jsonnew file mode 100644index 00000000..2196e7b4--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_batch.json@@ -0,0 +1,584 @@+{+ "name": "calibration_batch",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "Thanks, that is all sorted now.",+ {+ "notes": "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. ",+ "message": "Please cancel my account today."+ },+ "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den Betrag."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "batch_size": 3,+ "sort_by_length": true+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_widths": [+ 91,+ 110,+ 144,+ 34,+ 163,+ 42,+ 98,+ 56+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6134,+ "technical": 0.2284,+ "other": 0.1582+ },+ "confidence": 0.1546,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6103,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0798,+ "1": 0.2302,+ "2": 0.6901+ },+ "confidence": 0.2756,+ "answer_confidence": 0.6901,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9646,+ "confidence": 0.9646,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5992,+ "technical": 0.2495,+ "other": 0.1513+ },+ "confidence": 0.1453,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.5732,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0803,+ "1": 0.2662,+ "2": 0.6535+ },+ "confidence": 0.2419,+ "answer_confidence": 0.6535,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9459,+ "confidence": 0.9459,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 322,+ "output_tokens": 0,+ "state_tokens": 76,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5877,+ "technical": 0.2629,+ "other": 0.1494+ },+ "confidence": 0.1374,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.5584,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0774,+ "1": 0.2868,+ "2": 0.6358+ },+ "confidence": 0.2316,+ "answer_confidence": 0.6358,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9156,+ "confidence": 0.9156,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 424,+ "output_tokens": 0,+ "state_tokens": 110,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.3648,+ "technical": 0.2527,+ "other": 0.3825+ },+ "confidence": 0.0142,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 0.9436,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.3534,+ "1": 0.3496,+ "2": 0.297+ },+ "confidence": 0.0028,+ "answer_confidence": 0.3534,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.1027,+ "confidence": 0.8973,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 94,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6373,+ "technical": 0.2327,+ "other": 0.1301+ },+ "confidence": 0.1884,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6338,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0665,+ "1": 0.2333,+ "2": 0.7002+ },+ "confidence": 0.2998,+ "answer_confidence": 0.7002,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9792,+ "confidence": 0.9792,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 481,+ "output_tokens": 0,+ "state_tokens": 129,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.1832,+ "technical": 0.3175,+ "other": 0.4993+ },+ "confidence": 0.0698,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.1522,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.3001,+ "1": 0.2476,+ "2": 0.4523+ },+ "confidence": 0.03,+ "answer_confidence": 0.4523,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.0004,+ "confidence": 0.9996,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 118,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.2714,+ "technical": 0.3355,+ "other": 0.3931+ },+ "confidence": 0.0102,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.522,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1143,+ "1": 0.2494,+ "2": 0.6363+ },+ "confidence": 0.1972,+ "answer_confidence": 0.6363,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9561,+ "confidence": 0.9561,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 286,+ "output_tokens": 0,+ "state_tokens": 64,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "technical",+ "probabilities": {+ "refund": 0.218,+ "technical": 0.3953,+ "other": 0.3867+ },+ "confidence": 0.0294,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 0.9888,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.3312,+ "1": 0.3488,+ "2": 0.32+ },+ "confidence": 0.0006,+ "answer_confidence": 0.3488,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.1032,+ "confidence": 0.8968,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 160,+ "output_tokens": 0,+ "state_tokens": 22,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 9,+ 56+ ],+ [+ 9,+ 110+ ],+ [+ 6,+ 163+ ]+ ]+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_batch_lang_override.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_batch_lang_override.jsonnew file mode 100644index 00000000..7218aabf--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_batch_lang_override.json@@ -0,0 +1,457 @@+{+ "name": "calibration_batch_lang_override",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "lang": "pt"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_widths": [+ 91,+ 110,+ 144,+ 34,+ 163+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8298,+ "technical": 0.1151,+ "other": 0.0552+ },+ "confidence": 0.4871,+ "answer_confidence": 0.8298,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6514,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0669,+ "1": 0.2147,+ "2": 0.7184+ },+ "confidence": 0.3183,+ "answer_confidence": 0.7184,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9646,+ "confidence": 0.9646,+ "answer_confidence": 0.9646,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8084,+ "technical": 0.1401,+ "other": 0.0515+ },+ "confidence": 0.4537,+ "answer_confidence": 0.8084,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6116,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0677,+ "1": 0.253,+ "2": 0.6793+ },+ "confidence": 0.2785,+ "answer_confidence": 0.6793,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9459,+ "confidence": 0.9459,+ "answer_confidence": 0.9459,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 322,+ "output_tokens": 0,+ "state_tokens": 76,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7907,+ "technical": 0.1583,+ "other": 0.0511+ },+ "confidence": 0.4271,+ "answer_confidence": 0.7907,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.595,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0651,+ "1": 0.2749,+ "2": 0.66+ },+ "confidence": 0.2654,+ "answer_confidence": 0.66,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9156,+ "confidence": 0.9156,+ "answer_confidence": 0.9156,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 424,+ "output_tokens": 0,+ "state_tokens": 110,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.3877,+ "technical": 0.1861,+ "other": 0.4262+ },+ "confidence": 0.05,+ "answer_confidence": 0.4262,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 0.9381,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.3554,+ "1": 0.3511,+ "2": 0.2935+ },+ "confidence": 0.0033,+ "answer_confidence": 0.3554,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.1027,+ "confidence": 0.8973,+ "answer_confidence": 0.8973,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 94,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.8511,+ "technical": 0.1134,+ "other": 0.0355+ },+ "confidence": 0.5426,+ "answer_confidence": 0.8511,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6735,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0546,+ "1": 0.2173,+ "2": 0.7281+ },+ "confidence": 0.3432,+ "answer_confidence": 0.7281,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9792,+ "confidence": 0.9792,+ "answer_confidence": 0.9792,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 481,+ "output_tokens": 0,+ "state_tokens": 129,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 15,+ 163+ ]+ ]+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_batch_min_confidence.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_batch_min_confidence.jsonnew file mode 100644index 00000000..9d22e66f--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_batch_min_confidence.json@@ -0,0 +1,433 @@+{+ "name": "calibration_batch_min_confidence",+ "states": [+ "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ {+ "subject": "Double charged on invoice 88213",+ "body": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ },+ [+ {+ "role": "user",+ "content": "Hi, I need help with my invoice."+ },+ {+ "role": "assistant",+ "content": "Of course, what is wrong?"+ },+ {+ "role": "user",+ "content": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "",+ "Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Routine account check completed. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ ],+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "min_confidence": 0.6+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_widths": [+ 91,+ 110,+ 144,+ 34,+ 163+ ],+ "results": [+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6134,+ "technical": 0.2284,+ "other": 0.1582+ },+ "confidence": 0.1546,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.6103,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0798,+ "1": 0.2302,+ "2": 0.6901+ },+ "confidence": 0.2756,+ "answer_confidence": 0.6901,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9646,+ "confidence": 0.9646,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5992,+ "technical": 0.2495,+ "other": 0.1513+ },+ "confidence": 0.1453,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.5732,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0803,+ "1": 0.2662,+ "2": 0.6535+ },+ "confidence": 0.2419,+ "answer_confidence": 0.6535,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9459,+ "confidence": 0.9459,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 322,+ "output_tokens": 0,+ "state_tokens": 76,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.5877,+ "technical": 0.2629,+ "other": 0.1494+ },+ "confidence": 0.1374,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.5584,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0774,+ "1": 0.2868,+ "2": 0.6358+ },+ "confidence": 0.2316,+ "answer_confidence": 0.6358,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9156,+ "confidence": 0.9156,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 424,+ "output_tokens": 0,+ "state_tokens": 110,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.3648,+ "technical": 0.2527,+ "other": 0.3825+ },+ "confidence": 0.0142,+ "answer_confidence": 0.31,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 0.9436,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.3534,+ "1": 0.3496,+ "2": 0.297+ },+ "confidence": 0.0028,+ "answer_confidence": 0.3534,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.1027,+ "confidence": 0.8973,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 94,+ "output_tokens": 0,+ "state_tokens": 0,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6373,+ "technical": 0.2327,+ "other": 0.1301+ },+ "confidence": 0.1884,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.6338,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0665,+ "1": 0.2333,+ "2": 0.7002+ },+ "confidence": 0.2998,+ "answer_confidence": 0.7002,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9792,+ "confidence": 0.9792,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 481,+ "output_tokens": 0,+ "state_tokens": 129,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "runs": [+ [+ 15,+ 163+ ]+ ]+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_buckets.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_buckets.jsonnew file mode 100644index 00000000..781ab917--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_buckets.json@@ -0,0 +1,1727 @@+{+ "name": "calibration_buckets",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "pair": {+ "type": "choice",+ "instructions": "Which of the two?",+ "criteria": {+ "first": "the first option",+ "second": "the second option"+ }+ },+ "wide": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices and payments",+ "technical": "bugs and outages",+ "account": "logins and seats",+ "sales": "quotes and upgrades",+ "legal": "contracts and privacy",+ "shipping": "delivery and returns",+ "other": "anything else"+ }+ },+ "scale": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "none",+ "low",+ "mild",+ "high",+ "severe",+ "critical"+ ]+ },+ "flag": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "per_question": [+ {+ "id": "pair",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which of the two?",+ "head_text": "choice question: Which of the two?",+ "options": [+ "first: the first option",+ "second: the second option"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 10,+ 16+ ]+ },+ {+ "id": "wide",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices and payments",+ "technical: bugs and outages",+ "account: logins and seats",+ "sales: quotes and upgrades",+ "legal: contracts and privacy",+ "shipping: delivery and returns",+ "other: anything else"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ]+ },+ {+ "id": "scale",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: none",+ "level 1: low",+ "level 2: mild",+ "level 3: high",+ "level 4: severe",+ "level 5: critical"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36+ ]+ },+ {+ "id": "flag",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 10,+ 16,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ],+ [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36,+ 0+ ],+ [+ 14,+ 24,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 0,+ 1,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ -1.2013487815856934,+ 1.2118793725967407,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ],+ [+ 2.3425981998443604,+ -1.7018845081329346,+ -1.21029531955719,+ -2.12393856048584,+ -0.3992990255355835,+ -2.7612226009368896,+ -1.8578884601593018+ ],+ [+ -2.9551873207092285,+ -1.9968430995941162,+ -1.738126277923584,+ 1.6236977577209473,+ 2.8419923782348633,+ 2.1461334228515625,+ -10000.0+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4451.27099609375,+ -3639.0888671875+ ],+ [+ 4672.62841796875,+ -3815.82080078125+ ],+ [+ 4143.38427734375,+ -3392.93896484375+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "pair": {+ "type": "choice",+ "choice": "second",+ "probabilities": {+ "first": 0.2074,+ "second": 0.7926+ },+ "confidence": 0.2635,+ "answer_confidence": 0.7926,+ "action": {+ "act_probability": 1.0+ }+ },+ "wide": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9843,+ "technical": 0.0012,+ "account": 0.0026,+ "sales": 0.0006,+ "legal": 0.0102,+ "shipping": 0.0002,+ "other": 0.0009+ },+ "confidence": 0.9496,+ "answer_confidence": 0.9843,+ "action": {+ "act_probability": 1.0+ }+ },+ "scale": {+ "type": "score",+ "score": 3.5084,+ "legend": {+ "0": "none",+ "1": "low",+ "2": "mild",+ "3": "high",+ "4": "severe",+ "5": "critical"+ },+ "probabilities": {+ "0": 0.0479,+ "1": 0.066,+ "2": 0.0719,+ "3": 0.2206,+ "4": 0.3311,+ "5": 0.2625+ },+ "confidence": 0.1267,+ "answer_confidence": 0.44,+ "action": {+ "act_probability": 1.0+ }+ },+ "flag": {+ "type": "noul",+ "noul": 0.9646,+ "confidence": 0.9646,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 386,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_buckets_skip_binning.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_buckets_skip_binning.jsonnew file mode 100644index 00000000..12a0b9ec--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_buckets_skip_binning.json@@ -0,0 +1,1786 @@+{+ "name": "calibration_lang_buckets_skip_binning",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "pair": {+ "type": "choice",+ "instructions": "Which of the two?",+ "criteria": {+ "first": "the first option",+ "second": "the second option"+ }+ },+ "wide": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices and payments",+ "technical": "bugs and outages",+ "account": "logins and seats",+ "sales": "quotes and upgrades",+ "legal": "contracts and privacy",+ "shipping": "delivery and returns",+ "other": "anything else"+ }+ },+ "scale": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "none",+ "low",+ "mild",+ "high",+ "severe",+ "critical"+ ]+ },+ "flag": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "per_question": [+ {+ "id": "pair",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which of the two?",+ "head_text": "choice question: Which of the two?",+ "options": [+ "first: the first option",+ "second: the second option"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 10,+ 16+ ]+ },+ {+ "id": "wide",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices and payments",+ "technical: bugs and outages",+ "account: logins and seats",+ "sales: quotes and upgrades",+ "legal: contracts and privacy",+ "shipping: delivery and returns",+ "other: anything else"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ]+ },+ {+ "id": "scale",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: none",+ "level 1: low",+ "level 2: mild",+ "level 3: high",+ "level 4: severe",+ "level 5: critical"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36+ ]+ },+ {+ "id": "flag",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 10,+ 16,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ],+ [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36,+ 0+ ],+ [+ 14,+ 24,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 0,+ 1,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ -1.2013487815856934,+ 1.2118793725967407,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ],+ [+ 2.3425981998443604,+ -1.7018845081329346,+ -1.21029531955719,+ -2.12393856048584,+ -0.3992990255355835,+ -2.7612226009368896,+ -1.8578884601593018+ ],+ [+ -2.9551873207092285,+ -1.9968430995941162,+ -1.738126277923584,+ 1.6236977577209473,+ 2.8419923782348633,+ 2.1461334228515625,+ -10000.0+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4451.27099609375,+ -3639.0888671875+ ],+ [+ 4672.62841796875,+ -3815.82080078125+ ],+ [+ 4143.38427734375,+ -3392.93896484375+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "pair": {+ "type": "choice",+ "choice": "second",+ "probabilities": {+ "first": 0.3091,+ "second": 0.6909+ },+ "confidence": 0.1079,+ "answer_confidence": 0.6909,+ "action": {+ "act_probability": 1.0+ }+ },+ "wide": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9843,+ "technical": 0.0012,+ "account": 0.0026,+ "sales": 0.0006,+ "legal": 0.0102,+ "shipping": 0.0002,+ "other": 0.0009+ },+ "confidence": 0.9496,+ "answer_confidence": 0.9843,+ "action": {+ "act_probability": 1.0+ }+ },+ "scale": {+ "type": "score",+ "score": 3.7886,+ "legend": {+ "0": "none",+ "1": "low",+ "2": "mild",+ "3": "high",+ "4": "severe",+ "5": "critical"+ },+ "probabilities": {+ "0": 0.0221,+ "1": 0.0357,+ "2": 0.0406,+ "3": 0.2179,+ "4": 0.4008,+ "5": 0.283+ },+ "confidence": 0.2249,+ "answer_confidence": 0.4008,+ "action": {+ "act_probability": 1.0+ }+ },+ "flag": {+ "type": "noul",+ "noul": 0.9242,+ "confidence": 0.9242,+ "answer_confidence": 0.9242,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 386,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "it"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_miss_bins.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_miss_bins.jsonnew file mode 100644index 00000000..4a0fdd74--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_miss_bins.json@@ -0,0 +1,1625 @@+{+ "name": "calibration_lang_miss_bins",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.9911494255065918,+ -1.6601243019104004,+ -1.7693676948547363,+ -10000.0+ ],+ [+ -2.449965715408325,+ 0.2764223515987396,+ 2.5646214485168457,+ 1.5527721643447876+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0+ ],+ [+ -1.2292495965957642,+ 0.15064242482185364,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4736.37255859375,+ -3868.318359375+ ],+ [+ 4346.94384765625,+ -3559.03759765625+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ],+ [+ 4845.4169921875,+ -3968.390625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.7007,+ "technical": 0.153,+ "account": 0.1462+ },+ "confidence": 0.2558,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 2.0351,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.049,+ "1": 0.1693,+ "2": 0.4791,+ "3": 0.3025+ },+ "confidence": 0.1612,+ "answer_confidence": 0.4791,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9646,+ "confidence": 0.9646,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.8488,+ "confidence": 0.8488,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "ja"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_null_entry_inherits_checkpoint.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_null_entry_inherits_checkpoint.jsonnew file mode 100644index 00000000..e73b51af--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_null_entry_inherits_checkpoint.json@@ -0,0 +1,1625 @@+{+ "name": "calibration_lang_null_entry_inherits_checkpoint",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.9911494255065918,+ -1.6601243019104004,+ -1.7693676948547363,+ -10000.0+ ],+ [+ -2.449965715408325,+ 0.2764223515987396,+ 2.5646214485168457,+ 1.5527721643447876+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0+ ],+ [+ -1.2292495965957642,+ 0.15064242482185364,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4736.37255859375,+ -3868.318359375+ ],+ [+ 4346.94384765625,+ -3559.03759765625+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ],+ [+ 4845.4169921875,+ -3968.390625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9506,+ "technical": 0.026,+ "account": 0.0234+ },+ "confidence": 0.7898,+ "answer_confidence": 0.9506,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 2.1678,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0053,+ "1": 0.0737,+ "2": 0.6688,+ "3": 0.2522+ },+ "confidence": 0.3966,+ "answer_confidence": 0.6688,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9242,+ "confidence": 0.9242,+ "answer_confidence": 0.9242,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.7866,+ "confidence": 0.7866,+ "answer_confidence": 0.7866,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "fr"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_override_skips_binning.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_override_skips_binning.jsonnew file mode 100644index 00000000..8820daa9--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_lang_override_skips_binning.json@@ -0,0 +1,1625 @@+{+ "name": "calibration_lang_override_skips_binning",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.9911494255065918,+ -1.6601243019104004,+ -1.7693676948547363,+ -10000.0+ ],+ [+ -2.449965715408325,+ 0.2764223515987396,+ 2.5646214485168457,+ 1.5527721643447876+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0+ ],+ [+ -1.2292495965957642,+ 0.15064242482185364,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4736.37255859375,+ -3868.318359375+ ],+ [+ 4346.94384765625,+ -3559.03759765625+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ],+ [+ 4845.4169921875,+ -3968.390625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.7612,+ "technical": 0.1226,+ "account": 0.1161+ },+ "confidence": 0.3491,+ "answer_confidence": 0.7612,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.9457,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0794,+ "1": 0.197,+ "2": 0.4223,+ "3": 0.3014+ },+ "confidence": 0.1008,+ "answer_confidence": 0.4223,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.8536,+ "confidence": 0.8536,+ "answer_confidence": 0.8536,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.715,+ "confidence": 0.715,+ "answer_confidence": 0.715,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "de"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_long_lang_override.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_long_lang_override.jsonnew file mode 100644index 00000000..27792e8b--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_long_lang_override.json@@ -0,0 +1,437 @@+{+ "name": "calibration_long_lang_override",+ "state": "The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "window": 96,+ "batch_size": 3,+ "lang": "de"+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_tokens": 2037,+ "effective_window": 96,+ "effective_stride": 48,+ "effective_room": 990,+ "windows": [+ {+ "token_start": 0,+ "token_end": 96,+ "text": "The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 48,+ "token_end": 144,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 96,+ "token_end": 192,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 144,+ "token_end": 240,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 192,+ "token_end": 288,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 240,+ "token_end": 336,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 288,+ "token_end": 384,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 336,+ "token_end": 432,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 384,+ "token_end": 480,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 432,+ "token_end": 528,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 480,+ "token_end": 576,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 528,+ "token_end": 624,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 576,+ "token_end": 672,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 624,+ "token_end": 720,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 672,+ "token_end": 768,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 720,+ "token_end": 816,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 768,+ "token_end": 864,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 816,+ "token_end": 912,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 864,+ "token_end": 960,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 912,+ "token_end": 1008,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 960,+ "token_end": 1056,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1008,+ "token_end": 1104,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1056,+ "token_end": 1152,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1104,+ "token_end": 1200,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1152,+ "token_end": 1248,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1200,+ "token_end": 1296,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1248,+ "token_end": 1344,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1296,+ "token_end": 1392,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1344,+ "token_end": 1440,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1392,+ "token_end": 1488,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 1440,+ "token_end": 1536,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 1488,+ "token_end": 1584,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1536,+ "token_end": 1632,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1584,+ "token_end": 1680,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1632,+ "token_end": 1728,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1680,+ "token_end": 1776,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1728,+ "token_end": 1824,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1776,+ "token_end": 1872,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1824,+ "token_end": 1920,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1872,+ "token_end": 1968,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1920,+ "token_end": 2016,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the"+ },+ {+ "token_start": 1968,+ "token_end": 2037,+ "text": ". The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "result": {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.7309,+ "technical": 0.1201,+ "other": 0.149+ },+ "confidence": 0.3016,+ "answer_confidence": 0.7309,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 40,+ "token_start": 1920,+ "token_end": 2016,+ "count": 42+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.405,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.1396,+ "1": 0.3159,+ "2": 0.5446+ },+ "confidence": 0.1172,+ "answer_confidence": 0.5446,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 41,+ "token_start": 1968,+ "token_end": 2037,+ "count": 42+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.8477,+ "confidence": 0.8477,+ "answer_confidence": 0.8477,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 41,+ "token_start": 1968,+ "token_end": 2037,+ "count": 42+ }+ }+ },+ "usage": {+ "input_tokens": 15963,+ "output_tokens": 0,+ "state_tokens": 4005,+ "state_tokens_dropped": 0,+ "truncated": 0,+ "truncated_questions": [],+ "windows": 42+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_long_scan.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_long_scan.jsonnew file mode 100644index 00000000..e119ec21--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_long_scan.json@@ -0,0 +1,380 @@+{+ "name": "calibration_long_scan",+ "state": "The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {+ "window": 96,+ "batch_size": 3+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_tokens": 2037,+ "effective_window": 96,+ "effective_stride": 48,+ "effective_room": 990,+ "windows": [+ {+ "token_start": 0,+ "token_end": 96,+ "text": "The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 48,+ "token_end": 144,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 96,+ "token_end": 192,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 144,+ "token_end": 240,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 192,+ "token_end": 288,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 240,+ "token_end": 336,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 288,+ "token_end": 384,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 336,+ "token_end": 432,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 384,+ "token_end": 480,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 432,+ "token_end": 528,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 480,+ "token_end": 576,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 528,+ "token_end": 624,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 576,+ "token_end": 672,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 624,+ "token_end": 720,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 672,+ "token_end": 768,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 720,+ "token_end": 816,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 768,+ "token_end": 864,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 816,+ "token_end": 912,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 864,+ "token_end": 960,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 912,+ "token_end": 1008,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 960,+ "token_end": 1056,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1008,+ "token_end": 1104,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1056,+ "token_end": 1152,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1104,+ "token_end": 1200,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1152,+ "token_end": 1248,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1200,+ "token_end": 1296,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1248,+ "token_end": 1344,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1296,+ "token_end": 1392,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1344,+ "token_end": 1440,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1392,+ "token_end": 1488,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked"+ },+ {+ "token_start": 1440,+ "token_end": 1536,+ "text": ". The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded"+ },+ {+ "token_start": 1488,+ "token_end": 1584,+ "text": " the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates."+ },+ {+ "token_start": 1536,+ "token_end": 1632,+ "text": " routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the"+ },+ {+ "token_start": 1584,+ "token_end": 1680,+ "text": " The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine"+ },+ {+ "token_start": 1632,+ "token_end": 1728,+ "text": " account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The"+ },+ {+ "token_start": 1680,+ "token_end": 1776,+ "text": " status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account"+ },+ {+ "token_start": 1728,+ "token_end": 1824,+ "text": " agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status"+ },+ {+ "token_start": 1776,+ "token_end": 1872,+ "text": " and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent"+ },+ {+ "token_start": 1824,+ "token_end": 1920,+ "text": " updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and"+ },+ {+ "token_start": 1872,+ "token_end": 1968,+ "text": " checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates"+ },+ {+ "token_start": 1920,+ "token_end": 2016,+ "text": " recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the"+ },+ {+ "token_start": 1968,+ "token_end": 2037,+ "text": ". The agent checked the account and recorded routine status updates. Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account."+ }+ ],+ "result": {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "other",+ "probabilities": {+ "refund": 0.2473,+ "technical": 0.2316,+ "other": 0.5211+ },+ "confidence": 0.0679,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 1,+ "token_start": 48,+ "token_end": 144,+ "count": 42+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.517,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0957,+ "1": 0.2916,+ "2": 0.6127+ },+ "confidence": 0.1953,+ "answer_confidence": 0.6127,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 41,+ "token_start": 1968,+ "token_end": 2037,+ "count": 42+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9615,+ "confidence": 0.9615,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "window": {+ "index": 41,+ "token_start": 1968,+ "token_end": 2037,+ "count": 42+ }+ }+ },+ "usage": {+ "input_tokens": 15963,+ "output_tokens": 0,+ "state_tokens": 4005,+ "state_tokens_dropped": 0,+ "truncated": 0,+ "truncated_questions": [],+ "windows": 42+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_long_single_window.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_long_single_window.jsonnew file mode 100644index 00000000..4e686c41--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_long_single_window.json@@ -0,0 +1,148 @@+{+ "name": "calibration_long_single_window",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "topic": {+ "type": "choice",+ "instructions": "What is the request about?",+ "criteria": {+ "refund": "money back, duplicate charge",+ "technical": "bugs and outages",+ "other": null+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today"+ ]+ },+ "cancel": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "options": {},+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ },+ "state_tokens": 57,+ "effective_window": 760,+ "effective_stride": 380,+ "effective_room": 990,+ "windows": [],+ "result": {+ "model": "laya-rl-agent-onnx",+ "answers": {+ "topic": {+ "type": "choice",+ "choice": "refund",+ "probabilities": {+ "refund": 0.6134,+ "technical": 0.2284,+ "other": 0.1582+ },+ "confidence": 0.1546,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 1.6103,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today"+ },+ "probabilities": {+ "0": 0.0798,+ "1": 0.2302,+ "2": 0.6901+ },+ "confidence": 0.2756,+ "answer_confidence": 0.6901,+ "action": {+ "act_probability": 1.0+ }+ },+ "cancel": {+ "type": "noul",+ "noul": 0.9646,+ "confidence": 0.9646,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 265,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": [],+ "windows": 1+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence.jsonnew file mode 100644index 00000000..f918c7a1--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence.json@@ -0,0 +1,1578 @@+{+ "name": "calibration_min_confidence",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.9911494255065918,+ -1.6601243019104004,+ -1.7693676948547363,+ -10000.0+ ],+ [+ -2.449965715408325,+ 0.2764223515987396,+ 2.5646214485168457,+ 1.5527721643447876+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0+ ],+ [+ -1.2292495965957642,+ 0.15064242482185364,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4736.37255859375,+ -3868.318359375+ ],+ [+ 4346.94384765625,+ -3559.03759765625+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ],+ [+ 4845.4169921875,+ -3968.390625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.7007,+ "technical": 0.153,+ "account": 0.1462+ },+ "confidence": 0.2558,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 2.0351,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.049,+ "1": 0.1693,+ "2": 0.4791,+ "3": 0.3025+ },+ "confidence": 0.1612,+ "answer_confidence": 0.4791,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9646,+ "confidence": 0.9646,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.8488,+ "confidence": 0.8488,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.6+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence_buckets.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence_buckets.jsonnew file mode 100644index 00000000..a51a9c33--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence_buckets.json@@ -0,0 +1,1739 @@+{+ "name": "calibration_min_confidence_buckets",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "pair": {+ "type": "choice",+ "instructions": "Which of the two?",+ "criteria": {+ "first": "the first option",+ "second": "the second option"+ }+ },+ "wide": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices and payments",+ "technical": "bugs and outages",+ "account": "logins and seats",+ "sales": "quotes and upgrades",+ "legal": "contracts and privacy",+ "shipping": "delivery and returns",+ "other": "anything else"+ }+ },+ "scale": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "none",+ "low",+ "mild",+ "high",+ "severe",+ "critical"+ ]+ },+ "flag": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ }+ },+ "per_question": [+ {+ "id": "pair",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which of the two?",+ "head_text": "choice question: Which of the two?",+ "options": [+ "first: the first option",+ "second: the second option"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 10,+ 16+ ]+ },+ {+ "id": "wide",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices and payments",+ "technical: bugs and outages",+ "account: logins and seats",+ "sales: quotes and upgrades",+ "legal: contracts and privacy",+ "shipping: delivery and returns",+ "other: anything else"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ]+ },+ {+ "id": "scale",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: none",+ "level 1: low",+ "level 2: mild",+ "level 3: high",+ "level 4: severe",+ "level 5: critical"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36+ ]+ },+ {+ "id": "flag",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 273,+ 253,+ 767,+ 32,+ 50282,+ 50284,+ 806,+ 27,+ 253,+ 806,+ 4500,+ 50284,+ 1273,+ 27,+ 253,+ 1273,+ 4500,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 285,+ 10762,+ 50284,+ 7681,+ 27,+ 19775,+ 285,+ 562,+ 1131,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 285,+ 13512,+ 50284,+ 6224,+ 27,+ 19101,+ 285,+ 34232,+ 50284,+ 4320,+ 27,+ 12712,+ 285,+ 11068,+ 50284,+ 15076,+ 27,+ 6742,+ 285,+ 6548,+ 50284,+ 643,+ 27,+ 2712,+ 2010,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 5293,+ 50284,+ 1268,+ 337,+ 27,+ 1698,+ 50284,+ 1268,+ 374,+ 27,+ 11134,+ 50284,+ 1268,+ 495,+ 27,+ 1029,+ 50284,+ 1268,+ 577,+ 27,+ 5460,+ 50284,+ 1268,+ 608,+ 27,+ 4619,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 10,+ 16,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 11,+ 18,+ 25,+ 32,+ 38,+ 44,+ 50+ ],+ [+ 11,+ 16,+ 21,+ 26,+ 31,+ 36,+ 0+ ],+ [+ 14,+ 24,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ true,+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ false,+ false,+ false,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 0,+ 1,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ -1.2013487815856934,+ 1.2118793725967407,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ],+ [+ 2.3425981998443604,+ -1.7018845081329346,+ -1.21029531955719,+ -2.12393856048584,+ -0.3992990255355835,+ -2.7612226009368896,+ -1.8578884601593018+ ],+ [+ -2.9551873207092285,+ -1.9968430995941162,+ -1.738126277923584,+ 1.6236977577209473,+ 2.8419923782348633,+ 2.1461334228515625,+ -10000.0+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4451.27099609375,+ -3639.0888671875+ ],+ [+ 4672.62841796875,+ -3815.82080078125+ ],+ [+ 4143.38427734375,+ -3392.93896484375+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "pair": {+ "type": "choice",+ "choice": "second",+ "probabilities": {+ "first": 0.2074,+ "second": 0.7926+ },+ "confidence": 0.2635,+ "answer_confidence": 0.7926,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "wide": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.9843,+ "technical": 0.0012,+ "account": 0.0026,+ "sales": 0.0006,+ "legal": 0.0102,+ "shipping": 0.0002,+ "other": 0.0009+ },+ "confidence": 0.9496,+ "answer_confidence": 0.9843,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ },+ "scale": {+ "type": "score",+ "score": 3.5084,+ "legend": {+ "0": "none",+ "1": "low",+ "2": "mild",+ "3": "high",+ "4": "severe",+ "5": "critical"+ },+ "probabilities": {+ "0": 0.0479,+ "1": 0.066,+ "2": 0.0719,+ "3": 0.2206,+ "4": 0.3311,+ "5": 0.2625+ },+ "confidence": 0.1267,+ "answer_confidence": 0.44,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "flag": {+ "type": "noul",+ "noul": 0.9646,+ "confidence": 0.9646,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.5+ }+ },+ "usage": {+ "input_tokens": 386,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": 0.5+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence_lang_override.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence_lang_override.jsonnew file mode 100644index 00000000..715dcde8--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence_lang_override.json@@ -0,0 +1,1635 @@+{+ "name": "calibration_min_confidence_lang_override",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.9911494255065918,+ -1.6601243019104004,+ -1.7693676948547363,+ -10000.0+ ],+ [+ -2.449965715408325,+ 0.2764223515987396,+ 2.5646214485168457,+ 1.5527721643447876+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0+ ],+ [+ -1.2292495965957642,+ 0.15064242482185364,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4736.37255859375,+ -3868.318359375+ ],+ [+ 4346.94384765625,+ -3559.03759765625+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ],+ [+ 4845.4169921875,+ -3968.390625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.7612,+ "technical": 0.1226,+ "account": 0.1161+ },+ "confidence": 0.3491,+ "answer_confidence": 0.7612,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "urgency": {+ "type": "score",+ "score": 1.9457,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.0794,+ "1": 0.197,+ "2": 0.4223,+ "3": 0.3014+ },+ "confidence": 0.1008,+ "answer_confidence": 0.4223,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.6+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.8536,+ "confidence": 0.8536,+ "answer_confidence": 0.8536,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.715,+ "confidence": 0.715,+ "answer_confidence": 0.715,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "lang": "de",+ "min_confidence": 0.6+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence_map.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence_map.jsonnew file mode 100644index 00000000..c058d562--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_min_confidence_map.json@@ -0,0 +1,1582 @@+{+ "name": "calibration_min_confidence_map",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.9911494255065918,+ -1.6601243019104004,+ -1.7693676948547363,+ -10000.0+ ],+ [+ -2.449965715408325,+ 0.2764223515987396,+ 2.5646214485168457,+ 1.5527721643447876+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0+ ],+ [+ -1.2292495965957642,+ 0.15064242482185364,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4736.37255859375,+ -3868.318359375+ ],+ [+ 4346.94384765625,+ -3559.03759765625+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ],+ [+ 4845.4169921875,+ -3968.390625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.7007,+ "technical": 0.153,+ "account": 0.1462+ },+ "confidence": 0.2558,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.7+ },+ "urgency": {+ "type": "score",+ "score": 2.0351,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.049,+ "1": 0.1693,+ "2": 0.4791,+ "3": 0.3025+ },+ "confidence": 0.1612,+ "answer_confidence": 0.4791,+ "action": {+ "act_probability": 1.0+ },+ "low_confidence": true,+ "abstention": "abstained",+ "abstention_threshold": 0.5+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9646,+ "confidence": 0.9646,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.8488,+ "confidence": 0.8488,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ },+ "abstention": "passed",+ "abstention_threshold": 0.6+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "options": {+ "min_confidence": {+ "choice:3-5": 0.7,+ "score:3-5": 0.5,+ "noul:2": 0.6+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_plain.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_plain.jsonnew file mode 100644index 00000000..2c8a7c7e--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_plain.json@@ -0,0 +1,1566 @@+{+ "name": "calibration_plain",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.9911494255065918,+ -1.6601243019104004,+ -1.7693676948547363,+ -10000.0+ ],+ [+ -2.449965715408325,+ 0.2764223515987396,+ 2.5646214485168457,+ 1.5527721643447876+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0+ ],+ [+ -1.2292495965957642,+ 0.15064242482185364,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4736.37255859375,+ -3868.318359375+ ],+ [+ 4346.94384765625,+ -3559.03759765625+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ],+ [+ 4845.4169921875,+ -3968.390625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.7007,+ "technical": 0.153,+ "account": 0.1462+ },+ "confidence": 0.2558,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 2.0351,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.049,+ "1": 0.1693,+ "2": 0.4791,+ "3": 0.3025+ },+ "confidence": 0.1612,+ "answer_confidence": 0.4791,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9646,+ "confidence": 0.9646,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.8488,+ "confidence": 0.8488,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_table_without_lang.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_table_without_lang.jsonnew file mode 100644index 00000000..c67676e4--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_calibration_table_without_lang.json@@ -0,0 +1,1622 @@+{+ "name": "calibration_table_without_lang",+ "state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "serialized_state": "Subject: Double charged on invoice 88213\n\nHi, I was billed twice this month for the Pro plan and the second charge has not been refunded. This is the third time I have written in. If it is not fixed today I am cancelling the account.",+ "questions": {+ "department": {+ "type": "choice",+ "instructions": "Which department should handle this?",+ "criteria": {+ "billing": "invoices, payments, double charges, refunds",+ "technical": "bugs, outages, integration errors",+ "account": "logins, seats, plan changes"+ }+ },+ "urgency": {+ "type": "score",+ "instructions": "How urgent is this request?",+ "criteria": [+ "not urgent",+ "this week",+ "today",+ "drop everything"+ ]+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "The customer threatens to cancel the account."+ },+ "needs_human": {+ "type": "noul",+ "instructions": "This requires a human agent rather than an automated reply.",+ "criteria": {+ "false": "a template reply resolves it",+ "true": "a person must intervene"+ }+ }+ },+ "per_question": [+ {+ "id": "department",+ "type": "choice",+ "qtype": 0,+ "instructions": "Which department should handle this?",+ "head_text": "choice question: Which department should handle this?",+ "options": [+ "billing: invoices, payments, double charges, refunds",+ "technical: bugs, outages, integration errors",+ "account: logins, seats, plan changes"+ ],+ "input_ids": [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 24,+ 34+ ]+ },+ {+ "id": "urgency",+ "type": "score",+ "qtype": 1,+ "instructions": "How urgent is this request?",+ "head_text": "score question: How urgent is this request?",+ "options": [+ "level 0: not urgent",+ "level 1: this week",+ "level 2: today",+ "level 3: drop everything"+ ],+ "input_ids": [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 11,+ 17,+ 23,+ 28+ ]+ },+ {+ "id": "churn_risk",+ "type": "noul",+ "qtype": 2,+ "instructions": "The customer threatens to cancel the account.",+ "head_text": "noul question: The customer threatens to cancel the account.",+ "options": [+ "false: no, the statement does not hold",+ "true: yes, the statement holds"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 14,+ 24+ ]+ },+ {+ "id": "needs_human",+ "type": "noul",+ "qtype": 2,+ "instructions": "This requires a human agent rather than an automated reply.",+ "head_text": "noul question: This requires a human agent rather than an automated reply.",+ "options": [+ "false: a template reply resolves it",+ "true: a person must intervene"+ ],+ "input_ids": [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ "markers": [+ 17,+ 26+ ]+ }+ ],+ "batch": {+ "input_ids": [+ [+ 50281,+ 22122,+ 1953,+ 27,+ 6758,+ 7811,+ 943,+ 6016,+ 436,+ 32,+ 50282,+ 50284,+ 33484,+ 27,+ 29838,+ 1271,+ 13,+ 10762,+ 13,+ 4021,+ 7260,+ 13,+ 1275,+ 41748,+ 50284,+ 7681,+ 27,+ 19775,+ 13,+ 562,+ 1131,+ 13,+ 9554,+ 6332,+ 50284,+ 2395,+ 27,+ 2412,+ 968,+ 13,+ 13512,+ 13,+ 2098,+ 2544,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282+ ],+ [+ 50281,+ 18891,+ 1953,+ 27,+ 1359,+ 21007,+ 310,+ 436,+ 2748,+ 32,+ 50282,+ 50284,+ 1268,+ 470,+ 27,+ 417,+ 21007,+ 50284,+ 1268,+ 337,+ 27,+ 436,+ 2129,+ 50284,+ 1268,+ 374,+ 27,+ 3063,+ 50284,+ 1268,+ 495,+ 27,+ 5926,+ 3253,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 380,+ 7731,+ 37654,+ 281,+ 14002,+ 253,+ 2395,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 642,+ 13,+ 253,+ 3908,+ 1057,+ 417,+ 2186,+ 50284,+ 2032,+ 27,+ 4754,+ 13,+ 253,+ 3908,+ 6556,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ],+ [+ 50281,+ 79,+ 3941,+ 1953,+ 27,+ 831,+ 4419,+ 247,+ 1966,+ 5570,+ 2581,+ 685,+ 271,+ 16644,+ 12252,+ 15,+ 50282,+ 50284,+ 3221,+ 27,+ 247,+ 7646,+ 12252,+ 501,+ 14503,+ 352,+ 50284,+ 2032,+ 27,+ 247,+ 1436,+ 1364,+ 32014,+ 50282,+ 18961,+ 27,+ 18603,+ 6636,+ 327,+ 45156,+ 854,+ 3507,+ 1012,+ 187,+ 187,+ 12764,+ 13,+ 309,+ 369,+ 47045,+ 7019,+ 436,+ 1770,+ 323,+ 253,+ 1294,+ 2098,+ 285,+ 253,+ 1273,+ 4179,+ 556,+ 417,+ 644,+ 23005,+ 264,+ 15,+ 831,+ 310,+ 253,+ 2626,+ 673,+ 309,+ 452,+ 3542,+ 275,+ 15,+ 1310,+ 352,+ 310,+ 417,+ 4229,+ 3063,+ 309,+ 717,+ 476,+ 68,+ 3485,+ 253,+ 2395,+ 15,+ 50282,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283,+ 50283+ ]+ ],+ "attention_mask": [+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ],+ [+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 1,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0,+ 0+ ]+ ],+ "marker_pos": [+ [+ 11,+ 24,+ 34,+ 0+ ],+ [+ 11,+ 17,+ 23,+ 28+ ],+ [+ 14,+ 24,+ 0,+ 0+ ],+ [+ 17,+ 26,+ 0,+ 0+ ]+ ],+ "marker_mask": [+ [+ true,+ true,+ true,+ false+ ],+ [+ true,+ true,+ true,+ true+ ],+ [+ true,+ true,+ false,+ false+ ],+ [+ true,+ true,+ false,+ false+ ]+ ],+ "qtype": [+ 0,+ 1,+ 2,+ 2+ ]+ },+ "marker_axis_padded": false,+ "logits": [+ [+ 1.9911494255065918,+ -1.6601243019104004,+ -1.7693676948547363,+ -10000.0+ ],+ [+ -2.449965715408325,+ 0.2764223515987396,+ 2.5646214485168457,+ 1.5527721643447876+ ],+ [+ -2.3436622619628906,+ 0.30052450299263,+ -10000.0,+ -10000.0+ ],+ [+ -1.2292495965957642,+ 0.15064242482185364,+ -10000.0,+ -10000.0+ ]+ ],+ "act_logits": [+ [+ 4736.37255859375,+ -3868.318359375+ ],+ [+ 4346.94384765625,+ -3559.03759765625+ ],+ [+ 4573.56103515625,+ -3744.32958984375+ ],+ [+ 4845.4169921875,+ -3968.390625+ ]+ ],+ "result": {+ "model": "laya-rl-agent",+ "answers": {+ "department": {+ "type": "choice",+ "choice": "billing",+ "probabilities": {+ "billing": 0.7007,+ "technical": 0.153,+ "account": 0.1462+ },+ "confidence": 0.2558,+ "answer_confidence": 0.92,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgency": {+ "type": "score",+ "score": 2.0351,+ "legend": {+ "0": "not urgent",+ "1": "this week",+ "2": "today",+ "3": "drop everything"+ },+ "probabilities": {+ "0": 0.049,+ "1": 0.1693,+ "2": 0.4791,+ "3": 0.3025+ },+ "confidence": 0.1612,+ "answer_confidence": 0.4791,+ "action": {+ "act_probability": 1.0+ }+ },+ "churn_risk": {+ "type": "noul",+ "noul": 0.9646,+ "confidence": 0.9646,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ },+ "needs_human": {+ "type": "noul",+ "noul": 0.8488,+ "confidence": 0.8488,+ "answer_confidence": 0.81,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 379,+ "output_tokens": 0,+ "state_tokens": 57,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ "lang_temperatures": {+ "de": {+ "temperature": [+ 2.0,+ 3.0,+ 1.5+ ]+ },+ "fr": null,+ "es": {+ "temperature_by_options": {+ "choice:3-5": 2.5,+ "noul:2": 0.7+ }+ },+ "pt": {+ "temperature": [+ 1.2,+ 4.0,+ 0.8+ ],+ "temperature_by_options": {+ "score:3-5": 2.0,+ "choice:6-10": 3.0+ }+ },+ "it": {+ "temperature": null,+ "temperature_by_options": {+ "choice:2": 3.0,+ "choice:6-10": 0.6,+ "score:6-10": 2.0+ }+ },+ "nl": {+ "temperature": [+ 9.0,+ 0.1,+ 1.0+ ]+ },+ "sv-SE": {+ "temperature": [+ 1.5,+ 1.5,+ 1.5+ ]+ },+ "SV": {+ "temperature": [+ 3.5,+ 3.5,+ 3.5+ ]+ }+ },+ "calibration": {+ "file": "calibration_fixture.json",+ "payload": {+ "version": 1,+ "temperature": [+ 1.8,+ 2.2,+ 1.3+ ],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {+ "bins": 4,+ "values": [+ 0.47,+ 0.31,+ 0.92,+ 0.68+ ]+ },+ "noul:2": {+ "bins": 4,+ "values": [+ 0.12,+ 0.34,+ 0.57,+ 0.81+ ]+ },+ "score:6-10": {+ "bins": 3,+ "values": [+ 0.44,+ 0.18,+ 0.79+ ]+ },+ "choice:11+": {+ "bins": 2,+ "values": [+ 0.5,+ 0.9+ ]+ },+ "score:2": {+ "bins": 1,+ "values": [+ 0.5+ ]+ }+ }+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/index.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/index.jsonindex 263d884f..db8b9283 100644--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/index.json+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/index.json@@ -653,5 +653,96 @@"file": "case_long_lang_miss.json","expects_error": false,"kind": "long"+ },+ {+ "name": "calibration_plain",+ "file": "case_calibration_plain.json",+ "expects_error": false+ },+ {+ "name": "calibration_buckets",+ "file": "case_calibration_buckets.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_override_skips_binning",+ "file": "case_calibration_lang_override_skips_binning.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_miss_bins",+ "file": "case_calibration_lang_miss_bins.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_null_entry_inherits_checkpoint",+ "file": "case_calibration_lang_null_entry_inherits_checkpoint.json",+ "expects_error": false+ },+ {+ "name": "calibration_lang_buckets_skip_binning",+ "file": "case_calibration_lang_buckets_skip_binning.json",+ "expects_error": false+ },+ {+ "name": "calibration_table_without_lang",+ "file": "case_calibration_table_without_lang.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence",+ "file": "case_calibration_min_confidence.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence_map",+ "file": "case_calibration_min_confidence_map.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence_buckets",+ "file": "case_calibration_min_confidence_buckets.json",+ "expects_error": false+ },+ {+ "name": "calibration_min_confidence_lang_override",+ "file": "case_calibration_min_confidence_lang_override.json",+ "expects_error": false+ },+ {+ "name": "calibration_batch",+ "file": "case_calibration_batch.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "calibration_batch_lang_override",+ "file": "case_calibration_batch_lang_override.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "calibration_batch_min_confidence",+ "file": "case_calibration_batch_min_confidence.json",+ "expects_error": false,+ "kind": "batch"+ },+ {+ "name": "calibration_long_scan",+ "file": "case_calibration_long_scan.json",+ "expects_error": false,+ "kind": "long"+ },+ {+ "name": "calibration_long_single_window",+ "file": "case_calibration_long_single_window.json",+ "expects_error": false,+ "kind": "long"+ },+ {+ "name": "calibration_long_lang_override",+ "file": "case_calibration_long_lang_override.json",+ "expects_error": false,+ "kind": "long"}]diff --git a/laya-dotnet/tools/calibration_cases.py b/laya-dotnet/tools/calibration_cases.pynew file mode 100644index 00000000..3ff0a249--- /dev/null+++ b/laya-dotnet/tools/calibration_cases.py@@ -0,0 +1,271 @@+"""Synthetic inputs for the model-free calibration probe, `routing/calibration_probe.json`.++`dump_routing_golden.build_calibration_probe` feeds each payload to Python's own+`laya.calibrate.apply_calibration_payload` and each lookup to `laya.calibrate.apply_binning_map`, and records+what was installed, or the exception it raised, plus every warning it emitted (text and category), so a port is+checked against the shipping code on the same inputs without loading a model. This list is SDK-neutral: it is+the shared corpus for the .NET and Rust ports, and each port keeps a verbatim copy.++A payload case is `(label, text, identity)`. `text` is the JSON document, written as text because a payload may+hold what Python's `json.load` accepts and a typed value cannot state (`NaN`, `Infinity`, an integer past+float range, a duplicate key); a port reads it with its Python-json reader. `identity` names an entry of+`IDENTITIES`: the `model_id_or_path`, `subfolder` and parsed `rl_agent_config.json` of the agent the payload is+applied to, which a version 2 file is compared against. An SDK that has no equivalent of+Python's `model_id_or_path` (the .NET and Rust SDKs have none) does not emit that comparison's warning: its tests skip the+recorded warnings whose text starts "calibration was fitted for model_id_or_path=" and assert every other one.++A lookup case is `(label, map name, bucket, confidence)` for `apply_binning_map`; the name picks an entry of `MAPS`,+which are already validated maps, so only a valid one is written. A non-finite `confidence` is written as "nan"/"inf"/"-inf".+"""+import json++# The agent a payload is applied to. `cfg` is a checkpoint config: its two temperature fields are not part of the+# identity a calibration file records.+CFG = {+ "encoder": "jhu-clsp/mmBERT-base",+ "max_len": 1024,+ "head_max_len": 192,+ "option_layout": "sequential",+ "temperature": [1.2, 1.3, 1.4],+ "temperature_by_options": {"choice:2": 1.1},+}+IDENTITIES = {+ "multilingual": {"model_id_or_path": "convaiinnovations/laya", "subfolder": "multilingual", "cfg": CFG},+ "english": {"model_id_or_path": "convaiinnovations/laya", "subfolder": None, "cfg": CFG},+ "local": {"model_id_or_path": "onnx/multilingual", "subfolder": None, "cfg": CFG},+}++# The config a version 2 file records: the checkpoint config without the two temperature fields.+RECORDED_CFG = {k: v for k, v in CFG.items() if k not in ("temperature", "temperature_by_options")}++T3 = [1.5, 2.0, 1.25]+MAP4 = {"bins": 4, "values": [0.1, 0.4, 0.6, 0.9]}+++def _p(label, payload, identity="multilingual"):+ return (label, json.dumps(payload), identity)+++def _raw(label, text, identity="multilingual"):+ return (label, text, identity)+++def _v2(**extra):+ """A version 2 payload recorded for the multilingual checkpoint, with `extra` replacing or adding fields."""+ body = {"version": 2, "temperature": T3, "temperature_by_options": {"choice:3-5": 2.5},+ "model_id_or_path": "convaiinnovations/laya", "subfolder": "multilingual", "config": RECORDED_CFG}+ body.update(extra)+ return body+++def _bin(entry, other=None):+ body = {"temperature": T3, "binning_map": {"choice:3-5": entry}}+ if other is not None:+ body["binning_map"]["noul:2"] = other+ return body+++PAYLOAD_CASES = [+ # --- accepted ------------------------------------------------------------------------------+ _p("temperatures only", {"temperature": T3}),+ _p("temperatures and buckets", {"temperature": T3, "temperature_by_options": {"choice:3-5": 2.5, "noul:2": 0.8}}),+ _p("null buckets", {"temperature": T3, "temperature_by_options": None}),+ _p("empty buckets", {"temperature": T3, "temperature_by_options": {}}),+ _p("bucket names are not validated", {"temperature": T3, "temperature_by_options": {"bogus": 2.0, "": 3.0}}),+ _p("integers are numbers", {"temperature": [2, 3, 1], "temperature_by_options": {"noul:2": 2}}),+ _p("version 1", {"version": 1, "temperature": T3}),+ _p("null version is 1", {"version": None, "temperature": T3}),+ _p("a binning map", {"temperature": T3, "binning_map": {+ "choice:3-5": MAP4, "noul:2": {"bins": 2, "values": [0.2, 0.8]}, "score:2": {"bins": 1, "values": [0.5]}}}),+ _p("null binning map clears", {"temperature": T3, "binning_map": None}),+ _p("empty binning map is kept", {"temperature": T3, "binning_map": {}}),+ _p("values at the bounds", _bin({"bins": 3, "values": [0.0, 1.0, 0.5]})),+ _p("integer values", _bin({"bins": 2, "values": [0, 1]})),+ _p("a bucket name is not validated", {"temperature": T3, "binning_map": {"nonsense": MAP4}}),+ _p("a quote in a bucket name", {"temperature": T3, "binning_map": {"it's": MAP4}}),+ # --- clamping and its warning ---------------------------------------------------------------+ _p("clamp high and low", {"temperature": [9.0, 0.1, 1.0]}),+ _p("clamp boundaries are accepted", {"temperature": [0.5, 5.0, 1.0]}),+ _p("clamp a bucket", {"temperature": T3, "temperature_by_options": {"choice:3-5": 99.0, "noul:2": 0.01, "score:3-5": 2.0}}),+ _p("clamp temperatures and buckets in one warning",+ {"temperature": [9.0, 1.0, 0.0], "temperature_by_options": {"choice:2": -1.0, "noul:2": 7.5}}),+ _p("negative zero", {"temperature": [-0.0, 1.0, 1.0]}),+ _raw("non-finite temperatures", '{"temperature": [NaN, 1.0, Infinity], "temperature_by_options": {"noul:2": -Infinity}}'),+ _p("a bool is not a temperature", {"temperature": [True, 1.5, False]}),+ _p("a numeric string is read as float() reads it", {"temperature": ["2.5", " 1e1 ", "abc"]}),+ _p("null and nested values are 1.0", {"temperature": [None, [1.0], {"a": 1.0}]}),+ _p("bucket values that are not numbers",+ {"temperature": T3, "temperature_by_options": {"choice:2": True, "noul:2": None, "score:2": "2.5", "choice:6-10": 7}}),+ _p("a quote in a clamped bucket name", {"temperature": T3, "temperature_by_options": {"it's": 99.0}}),+ # --- version and the checkpoint it records ---------------------------------------------------+ _p("version 2 for this checkpoint", _v2()),+ _p("version 2 written as a float", _v2(version=2.0)),+ _p("version 2 as a string", _v2(version="2")),+ _p("version 2 as a padded string", _v2(version=" 2 ")),+ _p("version 2 as a signed string", _v2(version="+2")),+ _p("version 2 with an underscore", _v2(version="0_2")),+ _p("version 2 in fullwidth digits", _v2(version="2")),+ _p("version 3 is checked too", _v2(version=3)),+ _p("a huge version is checked", _v2(version=10 ** 40)),+ _p("version 1 is never checked", _v2(version=1, model_id_or_path="elsewhere")),+ _p("version 0 is never checked", _v2(version=0, model_id_or_path="elsewhere")),+ _p("negative version is never checked", _v2(version=-3, model_id_or_path="elsewhere")),+ _p("version 2 for another model", _v2(model_id_or_path="convaiinnovations/laya-multilingual")),+ _p("version 2 for another subfolder", _v2(subfolder="typed-decisions")),+ _p("version 2 recorded without a subfolder", _v2(subfolder=None)),+ _p("version 2 with another config", _v2(config=dict(RECORDED_CFG, max_len=512))),+ _p("version 2 with an extra config key", _v2(config=dict(RECORDED_CFG, extra=1))),+ _p("version 2 with a missing config key", _v2(config={k: v for k, v in RECORDED_CFG.items() if k != "encoder"})),+ _p("version 2 with a float where the config has an integer", _v2(config=dict(RECORDED_CFG, max_len=1024.0))),+ _p("version 2 with a config that is a list", _v2(config=[1, 2])),+ _p("version 2 with no config", {k: v for k, v in _v2().items() if k != "config"}),+ _p("version 2 with a null config", _v2(config=None)),+ _p("version 2 with nothing recorded", {"version": 2, "temperature": T3}),+ _p("version 2 with a model that is not a string", _v2(model_id_or_path=7)),+ _p("version 2 with a model that is a list", _v2(model_id_or_path=["a", "b"], subfolder={"k": 1.5})),+ _p("version 2 with a quote in the model", _v2(model_id_or_path="it's")),+ _p("version 2 for the English checkpoint", _v2(subfolder=None), "english"),+ _p("version 2 for the English checkpoint recorded with a subfolder", _v2(), "english"),+ _p("version 2 for a local directory", _v2(model_id_or_path="onnx/multilingual", subfolder=None), "local"),+ _p("version 2 for a local directory recorded for the hub", _v2(), "local"),+ _p("an identity warning and a clamp warning", _v2(temperature=[9.0, 1.0, 1.0], model_id_or_path="x")),+ # --- refusals: the payload ------------------------------------------------------------------+ _p("payload is a list", [1, 2, 3]),+ _p("payload is a number", 3),+ _p("payload is a float", 1.5),+ _p("payload is a string", "abc"),+ _p("payload is null", None),+ _p("payload is true", True),+ _p("payload is empty", {}),+ # --- refusals: version ----------------------------------------------------------------------+ _p("version is a bool", {"version": True, "temperature": T3}),+ _p("version is false", {"version": False, "temperature": T3}),+ _p("version is a list", {"version": [2], "temperature": T3}),+ _p("version is an object", {"version": {"v": 2}, "temperature": T3}),+ _p("version is not whole", {"version": 1.9, "temperature": T3}),+ _p("version is not whole and large", {"version": 2.5, "temperature": T3}),+ _p("version is a word", {"version": "abc", "temperature": T3}),+ _p("version is an empty string", {"version": "", "temperature": T3}),+ _p("version is a decimal string", {"version": "1.5", "temperature": T3}),+ _p("version is a double underscore string", {"version": "1__2", "temperature": T3}),+ _p("version has a quote", {"version": "it's", "temperature": T3}),+ _raw("version is NaN", '{"version": NaN, "temperature": [1.0, 1.0, 1.0]}'),+ _raw("version is Infinity", '{"version": Infinity, "temperature": [1.0, 1.0, 1.0]}'),+ _raw("version is an overflowing float", '{"version": 1e999, "temperature": [1.0, 1.0, 1.0]}'),+ _raw("version is a huge whole float", '{"version": 1e300, "temperature": [1.0, 1.0, 1.0]}'),+ # --- refusals: temperatures -----------------------------------------------------------------+ _p("temperature is missing", {"version": 1}),+ _p("temperature is null", {"temperature": None}),+ _p("temperature has two values", {"temperature": [1.0, 2.0]}),+ _p("temperature has four values", {"temperature": [1.0, 2.0, 3.0, 4.0]}),+ _p("temperature is empty", {"temperature": []}),+ _p("temperature is a number", {"temperature": 2.0}),+ _p("temperature is a string of three", {"temperature": "abc"}),+ _p("temperature is an object of three", {"temperature": {"a": 1, "b": 2, "c": 3}}),+ _p("buckets are a list", {"temperature": T3, "temperature_by_options": [1.0]}),+ _p("buckets are a number", {"temperature": T3, "temperature_by_options": 3}),+ _p("buckets are a float", {"temperature": T3, "temperature_by_options": 1.5}),+ _p("buckets are a string", {"temperature": T3, "temperature_by_options": "x"}),+ _p("buckets are false", {"temperature": T3, "temperature_by_options": False}),+ _p("buckets are an empty list", {"temperature": T3, "temperature_by_options": []}),+ # --- refusals: the binning map --------------------------------------------------------------+ _p("binning map is a list", {"temperature": T3, "binning_map": []}),+ _p("binning map is a number", {"temperature": T3, "binning_map": 3}),+ _p("binning map is a string", {"temperature": T3, "binning_map": "x"}),+ _p("binning map is false", {"temperature": T3, "binning_map": False}),+ _p("entry is a list", {"temperature": T3, "binning_map": {"choice:3-5": [0.1, 0.9]}}),+ _p("entry is null", {"temperature": T3, "binning_map": {"choice:3-5": None}}),+ _p("entry is a number", {"temperature": T3, "binning_map": {"choice:3-5": 0.5}}),+ _p("entry is a string", {"temperature": T3, "binning_map": {"choice:3-5": "x"}}),+ _p("entry is empty", {"temperature": T3, "binning_map": {"choice:3-5": {}}}),+ _p("entry is true", {"temperature": T3, "binning_map": {"choice:3-5": True}}),+ _p("a quote in a refused bucket name", {"temperature": T3, "binning_map": {"it's": 3}}),+ _p("bins is missing", _bin({"values": [0.1, 0.9]})),+ _p("bins is null", _bin({"bins": None, "values": [0.1, 0.9]})),+ _p("bins is zero", _bin({"bins": 0, "values": []})),+ _p("bins is negative", _bin({"bins": -2, "values": [0.1, 0.9]})),+ _p("bins is a whole float", _bin({"bins": 2.0, "values": [0.1, 0.9]})),+ _p("bins is a float", _bin({"bins": 2.5, "values": [0.1, 0.9]})),+ _p("bins is a string", _bin({"bins": "2", "values": [0.1, 0.9]})),+ _p("bins is true", _bin({"bins": True, "values": [0.1]})),+ _p("bins is a list", _bin({"bins": [2], "values": [0.1, 0.9]})),+ _p("bins is a huge integer", _bin({"bins": 10 ** 30, "values": [0.1, 0.9]})),+ _p("bins is a negative huge integer", _bin({"bins": -10 ** 30, "values": [0.1, 0.9]})),+ _raw("bins is NaN", '{"temperature": [1.0, 1.0, 1.0], "binning_map": {"a": {"bins": NaN, "values": [0.5]}}}'),+ _p("values is missing", _bin({"bins": 2})),+ _p("values is null", _bin({"bins": 2, "values": None})),+ _p("values is too short", _bin({"bins": 3, "values": [0.1, 0.9]})),+ _p("values is too long", _bin({"bins": 1, "values": [0.1, 0.9]})),+ _p("values is a string", _bin({"bins": 3, "values": "abc"})),+ _p("values is an object of the right size", _bin({"bins": 2, "values": {"a": 0.1, "b": 0.9}})),+ _p("values is a number", _bin({"bins": 1, "values": 0.5})),+ _p("a value is null", _bin({"bins": 2, "values": [0.1, None]})),+ _p("a value is a string", _bin({"bins": 2, "values": [0.1, "0.9"]})),+ _p("a value is a word", _bin({"bins": 2, "values": [0.1, "abc"]})),+ _p("a value is true", _bin({"bins": 2, "values": [0.1, True]})),+ _p("a value is false", _bin({"bins": 2, "values": [False, 0.5]})),+ _p("a value is a list", _bin({"bins": 2, "values": [0.1, [0.9]]})),+ _p("a value is an object", _bin({"bins": 2, "values": [{"v": 0.1}, 0.5]})),+ _raw("a value is NaN", '{"temperature": [1.0, 1.0, 1.0], "binning_map": {"a": {"bins": 2, "values": [0.1, NaN]}}}'),+ _raw("a value is Infinity",+ '{"temperature": [1.0, 1.0, 1.0], "binning_map": {"a": {"bins": 2, "values": [0.1, Infinity]}}}'),+ _raw("a value is negative Infinity",+ '{"temperature": [1.0, 1.0, 1.0], "binning_map": {"a": {"bins": 1, "values": [-Infinity]}}}'),+ _raw("a value overflows to Infinity",+ '{"temperature": [1.0, 1.0, 1.0], "binning_map": {"a": {"bins": 1, "values": [1e999]}}}'),+ _p("a value is below zero", _bin({"bins": 2, "values": [-0.1, 0.5]})),+ _p("a value is above one", _bin({"bins": 2, "values": [0.5, 1.0000001]})),+ _p("a value is an integer above one", _bin({"bins": 2, "values": [0.5, 2]})),+ _p("a value is -0.0", _bin({"bins": 1, "values": [-0.0]})),+ _p("a value is a huge integer", _bin({"bins": 2, "values": [0.5, 10 ** 400]})),+ _p("a huge integer value after a bad one", _bin({"bins": 3, "values": [None, 10 ** 400, 0.5]})),+ _p("a bad value after a huge integer", _bin({"bins": 3, "values": [10 ** 400, None, 0.5]})),+ _p("a string value before a list value", _bin({"bins": 2, "values": ["0.5", [1]]})),+ _p("an out-of-range value before a finite one", _bin({"bins": 2, "values": [2.0, 0.5]})),+ _raw("an out-of-range value before a non-finite one",+ '{"temperature": [1.0, 1.0, 1.0], "binning_map": {"a": {"bins": 2, "values": [2.0, NaN]}}}'),+ _p("the second entry is the bad one", _bin(MAP4, {"bins": 2, "values": [0.1, 5.0]})),+ _p("the first entry is the bad one", _bin({"bins": 2, "values": [0.1, 5.0]}, {"bins": 2, "values": [0.2, "x"]})),+ _p("a refusal beats a temperature warning",+ {"temperature": [9.0, 1.0, 1.0], "binning_map": {"a": {"bins": 1, "values": [3.0]}}}),+ _p("a refusal beats an identity warning", _v2(binning_map={"a": {"bins": 1, "values": [3.0]}})),+ _raw("a duplicate key keeps its last value",+ '{"temperature": [1.0, 2.0, 3.0], "temperature": [4.0, 5.0, 1.5], "binning_map": {"a": {"bins": 1, "values": [0.5]}, '+ '"a": {"bins": 2, "values": [0.1, 0.9]}}}'),+ # --- an integer too large for a float -------------------------------------------------------+ _p("a temperature that is a huge integer", {"temperature": [10 ** 400, 1.0, 1.0]}),+ _p("a bucket temperature that is a huge integer", {"temperature": T3, "temperature_by_options": {"noul:2": 10 ** 400}}),+ _p("a huge integer after an identity warning", _v2(temperature=[10 ** 400, 1.0, 1.0], model_id_or_path="x")),+]++BIN4 = {"bins": 4, "values": [0.1, 0.4, 0.6, 0.9]}+BIN1 = {"bins": 1, "values": [0.5]}+BIN10 = {"bins": 10, "values": [round(i / 10 + 0.05, 2) for i in range(10)]}+BIN100 = {"bins": 100, "values": [round((i * 37 % 100) / 100, 2) for i in range(100)]}+MAP = {"choice:3-5": BIN4, "noul:2": {"bins": 2, "values": [0.25, 0.75]}, "score:2": BIN1, "choice:6-10": BIN10,+ "choice:11+": BIN100}++MAPS = {"main": MAP, "empty": {}}++LOOKUP_CASES = (+ # bin edges and the ends+ [("choice:3-5 at %r" % c, "main", "choice:3-5", c)+ for c in (0.0, 1e-300, 0.1, 0.2499999999999999, 0.25, 0.2500000000000001, 0.3, 0.4999999999999999, 0.5,+ 0.5000000000000001, 0.6, 0.7499999999999999, 0.75, 0.7500000000000001, 0.9, 0.9999999999999999, 1.0)]+ # outside [0, 1]: clipped by the bin index, not rejected+ + [("choice:3-5 at %r" % c, "main", "choice:3-5", c) for c in (-0.0, -0.1, -1e300, 1.0000000000000002, 1.5, 7.0, 1e300)]+ # one bin takes everything+ + [("one bin at %r" % c, "main", "score:2", c) for c in (0.0, 0.5, 1.0)]+ # a bucket the map was not fit for passes through, with whatever the caller holds+ + [("miss at %r" % c, "main", "score:3-5", c) for c in (0.0, 0.123456789, 0.5, 1.0, 1.5, -0.25)]+ + [("miss on an empty bucket name", "main", "", 0.4), ("miss on a nearly right name", "main", "choice:3-5 ", 0.4),+ ("miss on another case", "main", "CHOICE:3-5", 0.4), ("empty map", "empty", "choice:3-5", 0.4)]+ # ten bins: float products that fall just under a whole number are truncated, not rounded+ + [("ten bins at %r" % c, "main", "choice:6-10", c) for c in (0.1, 0.2, 0.3, 0.7, 0.57, 0.9, 0.95, 0.99, 0.999999)]+ + [("hundred bins at %r" % c, "main", "choice:11+", c) for c in (0.01, 0.07, 0.29, 0.57, 0.58, 0.99, 1.0)]+ + [("two bins at %r" % c, "main", "noul:2", c) for c in (0.5, 0.7499999999999999, 0.75, 1.0)]+ # a value float() takes and int() refuses+ + [("nan", "main", "choice:3-5", float("nan")), ("inf", "main", "choice:3-5", float("inf")),+ ("-inf", "main", "choice:3-5", float("-inf")), ("nan on a miss", "main", "score:3-5", float("nan")),+ ("inf on a miss", "main", "score:3-5", float("inf"))]+)diff --git a/laya-dotnet/tools/calibration_fixture.json b/laya-dotnet/tools/calibration_fixture.jsonnew file mode 100644index 00000000..31d82880--- /dev/null+++ b/laya-dotnet/tools/calibration_fixture.json@@ -0,0 +1,17 @@+{+ "version": 1,+ "temperature": [1.8, 2.2, 1.3],+ "temperature_by_options": {+ "choice:3-5": 2.4,+ "noul:2": 0.8,+ "choice:6-10": 0.6,+ "score:6-10": 3.0+ },+ "binning_map": {+ "choice:3-5": {"bins": 4, "values": [0.47, 0.31, 0.92, 0.68]},+ "noul:2": {"bins": 4, "values": [0.12, 0.34, 0.57, 0.81]},+ "score:6-10": {"bins": 3, "values": [0.44, 0.18, 0.79]},+ "choice:11+": {"bins": 2, "values": [0.5, 0.9]},+ "score:2": {"bins": 1, "values": [0.5]}+ }+}diff --git a/laya-dotnet/tools/dump_golden.py b/laya-dotnet/tools/dump_golden.pyindex cc99abba..c08a3fa3 100644--- a/laya-dotnet/tools/dump_golden.py+++ b/laya-dotnet/tools/dump_golden.py@@ -45,6 +45,9 @@ from golden_cases import ( # noqa: E402BATCH_ERROR_CASES,BUDGET_CASES,BUDGET_ERROR_CASES,+ CALIBRATION_BATCH_CASES,+ CALIBRATION_CASES,+ CALIBRATION_LONG_CASES,CASES,ERROR_CASES,LANG_BATCH_CASES,@@ -267,6 +270,31 @@ def with_lang_temperatures(target, case: dict) -> None:case.get("lang_temperatures"), target.cfg.get("temperature", [1.0, 1.0, 1.0]))+def with_calibration(target, case: dict) -> None:+ """Install the case's calibration file exactly as the `calibration=` constructor option does.++ `load_calibration` is the method both `Agent` and `ONNXAgent` expose: it reads the file and hands the payload to+ `apply_calibration_payload`, which installs the temperatures and the binning map. The checkpoint's own+ temperatures are put back first, and the binning map cleared, so a case without a file is answered as an agent+ built without the option -- the agent and its shims are shared by every case. The file is looked for next to+ this script.+ """+ cfg = target.cfg+ target.temperature = [clamp_temperature(t) for t in cfg.get("temperature", [1.0, 1.0, 1.0])]+ target.temperature_by_options = {+ k: clamp_temperature(v) for k, v in cfg.get("temperature_by_options", {}).items()+ }+ target.binning_map = None+ if case.get("calibration"):+ target.load_calibration(os.path.join(TOOLS, case["calibration"]))+++def calibration_record(case: dict) -> dict:+ """The file a case was recorded with: its name under tools/ and its parsed content."""+ with open(os.path.join(TOOLS, case["calibration"]), encoding="utf-8") as f:+ return {"file": case["calibration"], "payload": json.load(f)}++def self_check(agent: Agent, onnx_dir: str) -> dict:"""Refuse to emit goldens unless the session reproduces the bundled PyTorch reference."""path = os.path.join(onnx_dir, "fixtures.npz")@@ -333,6 +361,7 @@ def record_case(agent: Agent, case: dict) -> dict:options = case.get("options")per_question = describe_questions(agent, state, questions, options)with_lang_temperatures(agent, case)+ with_calibration(agent, case)result = agent.system_one(state, questions, **(options or {}))last = agent.model.lastrec = {@@ -358,6 +387,8 @@ def record_case(agent: Agent, case: dict) -> dict:# Only for a case that carries a table, so every earlier golden stays byte-identical.if case.get("lang_temperatures") is not None:rec["lang_temperatures"] = case["lang_temperatures"]+ if case.get("calibration"):+ rec["calibration"] = calibration_record(case)return rec@@ -414,6 +445,7 @@ def record_shortlist_case(agent: Agent, case: dict) -> dict:def record_error_case(agent: Agent, case: dict) -> dict:options = case.get("options")with_lang_temperatures(agent, case)+ with_calibration(agent, case)try:agent.system_one(case["state"], case["questions"], **(options or {}))except ValueError as e:@@ -428,6 +460,8 @@ def record_error_case(agent: Agent, case: dict) -> dict:rec["options"] = optionsif case.get("lang_temperatures") is not None:rec["lang_temperatures"] = case["lang_temperatures"]+ if case.get("calibration"):+ rec["calibration"] = calibration_record(case)return recraise SystemExit("case %r was expected to raise ValueError but did not" % case["name"])@@ -442,11 +476,14 @@ def record_long_case(agent: Agent, case: dict, expects_error: bool = False) -> donnx.temperature = agent.temperatureonnx.temperature_by_options = agent.temperature_by_optionswith_lang_temperatures(onnx, case)+ with_calibration(onnx, case)options = case.get("options", {})state, questions = case.get("state"), case["questions"]rec = {"name": case["name"], "state": state, "questions": questions, "options": options}if case.get("lang_temperatures") is not None:rec["lang_temperatures"] = case["lang_temperatures"]+ if case.get("calibration"):+ rec["calibration"] = calibration_record(case)if expects_error:try:onnx.predict_long(state, questions, **options)@@ -512,6 +549,7 @@ def record_batch_case(agent: Agent, case: dict, expects_error: bool = False) ->onnx.temperature = agent.temperatureonnx.temperature_by_options = agent.temperature_by_optionswith_lang_temperatures(onnx, case)+ with_calibration(onnx, case)onnx.hooks = []onnx.model_id = "golden"states, questions = case["states"], case["questions"]@@ -519,6 +557,8 @@ def record_batch_case(agent: Agent, case: dict, expects_error: bool = False) ->rec = {"name": case["name"], "states": states, "questions": questions, "options": options}if case.get("lang_temperatures") is not None:rec["lang_temperatures"] = case["lang_temperatures"]+ if case.get("calibration"):+ rec["calibration"] = calibration_record(case)if expects_error:expected = {"TypeError": TypeError, "ValueError": ValueError}[case["error_type"]]try:@@ -689,6 +729,26 @@ def run_checkpoint(onnx_dir: str, out_dir: str) -> int:index.append({"name": case["name"], "file": "case_%s.json" % case["name"],"expects_error": False, "kind": "long"})+ # Calibration files and histogram binning, appended last for the same reason. The shared agent and shims have the+ # file reset by the next case; this puts the checkpoint's own temperatures back for whatever runs after.+ for cases, expects_error in ((CALIBRATION_CASES, False),):+ for case in cases:+ rec = (record_error_case if expects_error else record_case)(agent, case)+ dump(rec, os.path.join(out_dir, "case_%s.json" % case["name"]))+ index.append({"name": case["name"], "file": "case_%s.json" % case["name"],+ "expects_error": expects_error})+ for case in CALIBRATION_BATCH_CASES:+ rec = record_batch_case(agent, case)+ dump(rec, os.path.join(out_dir, "case_%s.json" % case["name"]))+ index.append({"name": case["name"], "file": "case_%s.json" % case["name"],+ "expects_error": False, "kind": "batch"})+ for case in CALIBRATION_LONG_CASES:+ rec = record_long_case(agent, case)+ dump(rec, os.path.join(out_dir, "case_%s.json" % case["name"]))+ index.append({"name": case["name"], "file": "case_%s.json" % case["name"],+ "expects_error": False, "kind": "long"})+ with_calibration(agent, {})+dump(index, os.path.join(out_dir, "index.json"))print("done: %d cases -> %s" % (len(index), out_dir))return 0diff --git a/laya-dotnet/tools/dump_routing_golden.py b/laya-dotnet/tools/dump_routing_golden.pyindex 7d18e525..ebc4f13f 100644--- a/laya-dotnet/tools/dump_routing_golden.py+++ b/laya-dotnet/tools/dump_routing_golden.py@@ -27,6 +27,11 @@ corpus from `laya-dotnet/tools/routing_cases.py` and writes one JSON file per pr_map, with the error text), _option_bucket, resolve_min_confidence,apply_confidence_gate (incl. reused payloads, #910) and jev_confidence.Non-finite floats are written as "nan"/"inf"/"-inf" strings.+ calibration_probe.json laya.calibrate.apply_calibration_payload and apply_binning_map over the synthetic+ inputs of tools/calibration_cases.py: what a payload installs (clamped temperatures,+ binning map) or the exception (type and text), every warning raised (text and+ category), and the recalibrated confidence per lookup (bin edges, 0.0, 1.0, a+ bucket the map lacks). No model.lang_temperatures_probe.json laya.common.resolve_lang_temperatures over the synthetic inputs oftools/lang_temperature_cases.py: the resolved table (key normalisation, lastduplicate wins, inheriting the checkpoint's temperature, clamping) or the@@ -59,6 +64,7 @@ import jsonimport mathimport osimport sys+import typesimport warningsTOOLS = os.path.dirname(os.path.abspath(__file__))@@ -87,11 +93,13 @@ from routing_cases import ( # noqa: E402SHORTLIST_CASES,)+import calibration_cases as calc # noqa: E402import confidence_cases as cc # noqa: E402import lang_temperature_cases as ltc # noqa: E402import hashing_embedder # noqa: E402from laya import confidence # noqa: E402+from laya.calibrate import CALIBRATION_VERSION, apply_binning_map, apply_calibration_payload # noqa: E402from laya.common import TEMP_MAX, TEMP_MIN, resolve_lang_temperatures # noqa: E402from laya.email import clean_email_body, email_state # noqa: E402from laya.lang import analyse, detect_script, latin_profile, script_profile, state_text # noqa: E402@@ -342,6 +350,50 @@ def build_lang_temperatures_probe():return {"temp_min": TEMP_MIN, "temp_max": TEMP_MAX, "cases": cases}+# ============================================================================ calibration_probe+def _recorded_warnings(caught):+ return [{"category": w.category.__name__, "message": str(w.message)} for w in caught]+++def _apply_recorded(text, identity):+ """`apply_calibration_payload` on `json.loads(text)` for an agent with `identity`: what it installed, or the error."""+ agent = types.SimpleNamespace(**identity)+ with warnings.catch_warnings(record=True) as caught:+ warnings.simplefilter("always")+ try:+ apply_calibration_payload(agent, json.loads(text))+ outcome = {"result": {+ "temperature": agent.temperature,+ "temperature_by_options": agent.temperature_by_options,+ "binning_map": agent.binning_map,+ }}+ except Exception as e: # ValueError by contract; an OverflowError is a finding, so everything is recorded+ outcome = {"error_type": type(e).__name__, "error_message": str(e)}+ outcome["warnings"] = _recorded_warnings(caught)+ return outcome+++def _lookup_recorded(binning_map, bucket, confidence_value):+ try:+ return {"result": apply_binning_map(confidence_value, bucket, binning_map)}+ except Exception as e:+ return {"error_type": type(e).__name__, "error_message": str(e)}+++def build_calibration_probe():+ payloads = [+ dict(label=label, json=text, identity=calc.IDENTITIES[identity], **_apply_recorded(text, calc.IDENTITIES[identity]))+ for label, text, identity in calc.PAYLOAD_CASES+ ]+ lookups = [+ dict(label=label, map=name, bucket=bucket, confidence=value,+ **_lookup_recorded(calc.MAPS[name], bucket, value))+ for label, name, bucket, value in calc.LOOKUP_CASES+ ]+ return {"temp_min": TEMP_MIN, "temp_max": TEMP_MAX, "version": CALIBRATION_VERSION,+ "maps": calc.MAPS, "payloads": payloads, "lookups": lookups}++# ============================================================================ router_forwarding_probeclass _StubAgent:"""An agent that answers nothing and records the keyword arguments each entry point received.@@ -598,6 +650,7 @@ def main() -> int:route_batch_probe = build_route_batch_probe()confidence_probe = build_confidence_probe()lang_temperatures_probe = build_lang_temperatures_probe()+ calibration_probe = build_calibration_probe()router_forwarding_probe = build_router_forwarding_probe()email_probe = build_email_probe()shortlist_probe = build_shortlist_probe()@@ -629,6 +682,8 @@ def main() -> int:"confidence_probe_reuse": len(confidence_probe["reuse"]),"confidence_probe_jev": len(confidence_probe["jev"]),"lang_temperatures_probe": len(lang_temperatures_probe["cases"]),+ "calibration_probe_payloads": len(calibration_probe["payloads"]),+ "calibration_probe_lookups": len(calibration_probe["lookups"]),"router_forwarding_probe_predict": len(router_forwarding_probe["predict"]),"router_forwarding_probe_long": len(router_forwarding_probe["long"]),"router_forwarding_probe_batch": len(router_forwarding_probe["batch"]),@@ -651,6 +706,9 @@ def main() -> int:results["lang_temperatures_probe.json"] = dump(_encode_floats(lang_temperatures_probe), os.path.join(out_dir, "lang_temperatures_probe.json"), args.force)+ results["calibration_probe.json"] = dump(+ _encode_floats(calibration_probe), os.path.join(out_dir, "calibration_probe.json"), args.force+ )results["router_forwarding_probe.json"] = dump(router_forwarding_probe, os.path.join(out_dir, "router_forwarding_probe.json"), args.force)diff --git a/laya-dotnet/tools/golden_cases.py b/laya-dotnet/tools/golden_cases.pyindex 9ad54c79..2adc81e5 100644--- a/laya-dotnet/tools/golden_cases.py+++ b/laya-dotnet/tools/golden_cases.py@@ -717,3 +717,79 @@ LANG_LONG_CASES = [+ QUICKSTART_STATE),"options": {"window": 96, "batch_size": 3, "lang": "ja"}},]+++# ---------------------------------------------------------------------------------------------+# Calibration files (`calibration=` / `load_calibration`) and histogram binning.+#+# The cases below are SDK-neutral: the Rust SDK keeps a verbatim copy. `calibration` names a file under+# tools/ (here `calibration_fixture.json`, hand-made: temperatures, per-bucket temperatures and a binning map+# with several buckets) that `dump_golden.with_calibration` reads through Python's own `load_calibration` on the+# torch Agent and on every ONNX shim, and the file name and its parsed content are recorded in the case JSON+# (`"calibration": {"file", "payload"}`), so a port loads the very same file. The file is installed for these+# cases only, so a port builds one extra engine per checkpoint with it. A binned bucket is a hit, any other+# (here `choice:2`, `choice:6-10`, `score:3-5`) is a miss that keeps the temperature-scaled value. The map is+# deliberately not monotone, so binning changes which window `predict_long` picks and whether an answer passes+# the `min_confidence` gate. Edge values of the lookup (0.0, 1.0, bin edges) are pinned model-free by+# `routing/calibration_probe.json`.+# ---------------------------------------------------------------------------------------------++CALIBRATION_FILE = "calibration_fixture.json"++# `system_one` with the file: hit and miss buckets, a language the table holds (temperatures replaced, binning+# skipped), a language it does not (calibration kept, binned), a null entry (inherits the *checkpoint's*+# temperature, not the file's), and the abstention gate on binned values.+CALIBRATION_CASES = [+ {"name": "calibration_plain", "state": QUICKSTART_STATE, "questions": LANG_QUESTIONS,+ "calibration": CALIBRATION_FILE},+ {"name": "calibration_buckets", "state": QUICKSTART_STATE, "questions": LANG_BUCKET_QUESTIONS,+ "calibration": CALIBRATION_FILE},+ {"name": "calibration_lang_override_skips_binning", "state": QUICKSTART_STATE, "questions": LANG_QUESTIONS,+ "calibration": CALIBRATION_FILE, "lang_temperatures": LANG_TEMPERATURES, "options": {"lang": "de"}},+ {"name": "calibration_lang_miss_bins", "state": QUICKSTART_STATE, "questions": LANG_QUESTIONS,+ "calibration": CALIBRATION_FILE, "lang_temperatures": LANG_TEMPERATURES, "options": {"lang": "ja"}},+ {"name": "calibration_lang_null_entry_inherits_checkpoint", "state": QUICKSTART_STATE,+ "questions": LANG_QUESTIONS, "calibration": CALIBRATION_FILE, "lang_temperatures": LANG_TEMPERATURES,+ "options": {"lang": "fr"}},+ {"name": "calibration_lang_buckets_skip_binning", "state": QUICKSTART_STATE,+ "questions": LANG_BUCKET_QUESTIONS, "calibration": CALIBRATION_FILE, "lang_temperatures": LANG_TEMPERATURES,+ "options": {"lang": "it"}},+ {"name": "calibration_table_without_lang", "state": QUICKSTART_STATE, "questions": LANG_QUESTIONS,+ "calibration": CALIBRATION_FILE, "lang_temperatures": LANG_TEMPERATURES},+ {"name": "calibration_min_confidence", "state": QUICKSTART_STATE, "questions": LANG_QUESTIONS,+ "calibration": CALIBRATION_FILE, "options": {"min_confidence": 0.6}},+ {"name": "calibration_min_confidence_map", "state": QUICKSTART_STATE, "questions": LANG_QUESTIONS,+ "calibration": CALIBRATION_FILE,+ "options": {"min_confidence": {"choice:3-5": 0.7, "score:3-5": 0.5, "noul:2": 0.6}}},+ {"name": "calibration_min_confidence_buckets", "state": QUICKSTART_STATE, "questions": LANG_BUCKET_QUESTIONS,+ "calibration": CALIBRATION_FILE, "options": {"min_confidence": 0.5}},+ {"name": "calibration_min_confidence_lang_override", "state": QUICKSTART_STATE, "questions": LANG_QUESTIONS,+ "calibration": CALIBRATION_FILE, "lang_temperatures": LANG_TEMPERATURES,+ "options": {"lang": "de", "min_confidence": 0.6}},+]++# `predict_batch` with the file: plain, a language it overrides, and the gate.+CALIBRATION_BATCH_CASES = [+ {"name": "calibration_batch", "states": BATCH_MIXED_STATES, "questions": BATCH_QUESTIONS,+ "calibration": CALIBRATION_FILE, "options": {"batch_size": 3, "sort_by_length": True}},+ {"name": "calibration_batch_lang_override", "states": BATCH_MIXED_STATES[:5], "questions": BATCH_QUESTIONS,+ "calibration": CALIBRATION_FILE, "lang_temperatures": LANG_TEMPERATURES, "options": {"lang": "pt"}},+ {"name": "calibration_batch_min_confidence", "states": BATCH_MIXED_STATES[:5], "questions": BATCH_QUESTIONS,+ "calibration": CALIBRATION_FILE, "options": {"min_confidence": 0.6}},+]++# `predict_long` with the file: a multi-window scan (the best window is chosen on the binned value), a state that+# fits one window, and a language that skips the binning.+CALIBRATION_LONG_CASES = [+ {"name": "calibration_long_scan", "calibration": CALIBRATION_FILE, "questions": LONG_QUESTIONS,+ "state": ("The agent checked the account and recorded routine status updates. " * 180+ + QUICKSTART_STATE),+ "options": {"window": 96, "batch_size": 3}},+ {"name": "calibration_long_single_window", "calibration": CALIBRATION_FILE, "questions": LONG_QUESTIONS,+ "state": QUICKSTART_STATE, "options": {}},+ {"name": "calibration_long_lang_override", "calibration": CALIBRATION_FILE,+ "lang_temperatures": LANG_TEMPERATURES, "questions": LONG_QUESTIONS,+ "state": ("The agent checked the account and recorded routine status updates. " * 180+ + QUICKSTART_STATE),+ "options": {"window": 96, "batch_size": 3, "lang": "de"}},+]