Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(dotnet): follow Python 0.4.12 language detection, routing, email and config
kevin-gatimu69af085cef0c
diff --git a/laya-dotnet/CHANGELOG.md b/laya-dotnet/CHANGELOG.mdindex 890ffd51..ed7666a4 100644--- a/laya-dotnet/CHANGELOG.md+++ b/laya-dotnet/CHANGELOG.md@@ -11,6 +11,15 @@ goldens: 0.4.12).### Added+- `LangGuess` (a language code or a callback of the state), Python's `lang_guess`: per call through new+ `LayaRouter.Route`, `Predict` and `PredictLong` overloads, and for every request through+ `LayaRouterOptions.LangGuess`. Precedence is explicit `lang`, then `lang_guess`, then detection;+ a hint that names no language falls through. Existing signatures are unchanged.+- `LayaEmail.State` overload taking Python's `max_chars`, consumed (never a state field) and applied only+ when cleaning.+- `LayaConfig.TemperatureRawValues` / `TemperatureByOptionsRawValues` and+ `Calibration.ClampTemperature(object?)` / `TemperatureRejected`: Python's `clamp_temperature` and+ `temperature_rejected` on the JSON value itself.- `LayaEngine.PredictLong` and `LayaRouter.PredictLong`, ported from Python`ONNXAgent.predict_long`: overlapping token windows, batched inference, deciding-windowattribution, per-type aggregation and summed scan diagnostics.@@ -26,6 +35,38 @@ goldens: 0.4.12).### Fixed+- Language detection: a line or state written in capitals is no longer named a language on acronym-shaped+ evidence (`MON DES EST LA`, `STORES LOS ANGELES LAS VEGAS`), as Python's `_shouted` rules; combining+ marks count with their script only when they have a combining class, so Devanagari matras read as the+ script; the per-field read falls through to the diacritic rule when a named language is disbelieved; the+ `\w`/`\d`/`\s`/`isupper`/`strip` classes follow Python, including for characters outside the BMP.+- Checkpoint config JSON is read by a reader that follows CPython's `json.loads` (`PythonJsonReader`): the bare+ `NaN`, `Infinity` and `-Infinity` are accepted in value position only (`{NaN: 1}`, `nan`, `+Infinity`, `01`,+ trailing commas, comments and single quotes stay errors), integer and float literals keep their Python type+ (`-0` is the int `0`, `7.0` a float, a 30-digit int prints in full), floats are bit-identical to `float()`,+ and no marker text stands in for a token. `tokenizer_config.json` is read the same way. A temperature given as+ an integer too large for a float fails the load with Python's `OverflowError: int too large to convert to+ float` (`InvalidDataException`), while a float literal such as `1e400` reads as infinity and falls back to 1.0.+- A warning is traced once per process for the same text and call site, as Python's default filter does+ (`stacklevel`): the site is the caller of `LayaEngine.Create` / `FromDirectory` (temperature warning) or of+ `PredictLong` (window and stride warnings), through the router the router's own call. The `predict_long`+ window and stride warnings now carry Python's texts.+- `rl_agent_config.json` and `tokenizer_config.json` are read as strict UTF-8 and a BOM is rejected, as+ Python's `json.load(open(path))` does on a UTF-8 locale (it opens them without an `encoding`, so on a default+ Windows install it would use the ANSI code page; the two agree on ASCII files).+- Email patterns count and bound code points, not UTF-16 units, so characters outside the BMP next to a+ keyword or inside a `.{0,300}` window behave as in Python.+- `byte[]` states are decoded as strict UTF-8 for detection, and invalid UTF-8 is no text, as in Python;+ serializing one for a prediction throws, as Python's `TypeError` does.+- Route: a blank, whitespace, `C`, `C.UTF-8`, `POSIX`, `und`, `zxx`, `mul` or `.` `lang` falls through to+ detection, and locale codes (`en_US`, `en_US.UTF-8`, ` EN `) normalise as Python does; reasons quote+ caller text with Python's `repr` (non-printable characters escaped).+- Email cleaning: a negative `maxChars` slices as Python does (including the `max_chars * 4` pre-cut);+ U+001C..U+001F count as whitespace, `\b` follows Python's `\w`, and case-insensitive matching treats+ `i`, `I`, `ı` and `İ` as one letter, as `re.I` does.+- Checkpoint temperatures are read as Python does: a numeric string (`"2.5"`, `" 2.5 "`, `"1e1"`) is a+ number, and `true`, `null`, lists, dictionaries and non-numeric strings fall back to 1.0. The load+ warning is one message with Python's text, listing each entry as `name=repr -> applied`.- `LayaRouterOptions.MaxLoaded` defaults to 2, as Python's `Router(max_loaded=2)` does, so arouter that alternates two languages no longer reloads a checkpoint on every request.- Undetermined Latin text uses the router's configured default, including its route reason.@@ -45,7 +86,5 @@ goldens: 0.4.12).- Python-only opt-in features are not yet ported: abstention (`min_confidence`),`lang_temperatures`, hooks, batching APIs, question option ordering and digest controls.-- `LayaEmail.State` does not expose Python's `max_chars` keyword; use `CleanBody` with a custom- budget and pass the cleaned body with `clean: false`.- New Python usage diagnostics (`state_tokens`, `truncated`, `state_tokens_dropped` and`truncated_questions`) are populated for `PredictLong`, but not ordinary `Predict`.diff --git a/laya-dotnet/README.md b/laya-dotnet/README.mdindex b588e20d..ac0644e3 100644--- a/laya-dotnet/README.md+++ b/laya-dotnet/README.md@@ -543,6 +543,15 @@ Console.WriteLine(result.Routing!.Reason); // "English Latin text"or testing the routing logic in isolation. `Preload()` builds every checkpoint up front for aserver or demo, so no request pays a cold load.+**Language hints.** Precedence is explicit `model` > `task` > detected typed-decisions workflow (opt-in) >+explicit `lang` > `LangGuess` > detection > default. An explicit `lang` decides only when its code names a+language (`en`, `EN`, `en-US`, `en_US`, `en_US.UTF-8`, `pt-BR`); blank, `C`, `C.UTF-8`, `POSIX`, `und`, `zxx`+and `mul` fall through to detection. A `LangGuess` is Python's `lang_guess`, a code or a callback of the state+(a callback returning `null` abstains, so a language-identification model can decline); pass one per call+to the `Route`/`Predict`/`PredictLong` overloads that end in `LangGuess? langGuess`, or install one for every+request with `new LayaRouterOptions { LangGuess = LangGuess.FromCallback(state => Identify(state)) }`.+Route reasons quote caller text with Python's `repr`, escaping non-printable characters.+**Email cleaning** strips quoted history, signatures and disclaimers before you ask any questions:```csharp@@ -551,6 +560,11 @@ var state = LayaEmail.State(subject, rawBody, sender: "[email protected]");var answers = router.Predict(state, LayaPresets.Email());```+`LayaEmail.State(subject, body, sender, clean, extra, maxChars)` takes Python's `max_chars` budget (default 3000).+It is a slice budget, applied as Python slices: `0` gives an empty body and a negative value cuts from the end+(`5 * |maxChars|` characters in all), so pass a number larger than the body to mean "no limit". It is ignored+when `clean` is `false` and is never added to the state as a field.+**Embedding shortlist** narrows a high-cardinality choice question before the model sees it, usinga caller-supplied embedding function (bring your own bi-encoder — this SDK does not include one):@@ -656,8 +670,9 @@ The contents of `rl_agent_config.json`. Missing keys fall back to the same defau| `int HeadMaxLen` | Budget for instructions plus all option fragments (256 on multilingual). || `IReadOnlyList<double> Temperature` | Per-question-type temperature `[choice, score, noul]`, clamped. Always 3 entries. || `IReadOnlyDictionary<string, double> TemperatureByOptions` | Per-bucket temperature, clamped. Keys look like `choice:3-5` (buckets `2`, `3-5`, `6-10`, `11+`). |-| `IReadOnlyList<double> TemperatureRaw` / `TemperatureByOptionsRaw` | What the checkpoint shipped, before clamping. |-| `IReadOnlyList<string> ClampedTemperatures` | Every temperature that was clamped into `[0.5, 5.0]`. A non-empty list means confidence from those buckets is uncalibrated. Each entry is also logged at load time via `Trace.TraceWarning`. |+| `IReadOnlyList<double> TemperatureRaw` / `TemperatureByOptionsRaw` | What the checkpoint shipped, before clamping, as numbers (a value that is not a number reads `NaN`). |+| `IReadOnlyList<object?> TemperatureRawValues` / `IReadOnlyDictionary<string, object?> TemperatureByOptionsRawValues` | The same entries as the JSON held them (`long`, `double`, `string`, `bool`, `null`, list). |+| `IReadOnlyList<string> ClampedTemperatures` | Every temperature that was clamped into `[0.5, 5.0]`. A non-empty list means confidence from those buckets is uncalibrated. Entries read `name=<Python repr of the value> -> <applied>`, as Python's warning lists them. One `Trace.TraceWarning` listing every entry is raised at load time. |```csharpforeach (var warning in engine.Config.ClampedTemperatures)@@ -932,8 +947,6 @@ disclaimer, signature and reply-header rules, including Portuguese, Spanish andPython, so default predictions are unaffected. They are planned as follow-ups.- **Other Python-only APIs are not ported:** hooks, batching APIs, question option ordering anddigest controls.-- **Custom email-state body budgets:** `LayaEmail.State` does not expose Python's `max_chars`- keyword. Call `LayaEmail.CleanBody(body, maxChars)` and pass its result with `clean: false`.- **Usage diagnostics:** Python's `state_tokens`, `truncated`, `state_tokens_dropped` and`truncated_questions` are exposed for `PredictLong` only; ordinary `Predict` still reportsjust input/output tokens.diff --git a/laya-dotnet/src/Laya/Calibration.cs b/laya-dotnet/src/Laya/Calibration.csindex 9e22096b..7a8a6437 100644--- a/laya-dotnet/src/Laya/Calibration.cs+++ b/laya-dotnet/src/Laya/Calibration.cs@@ -23,6 +23,30 @@ public static class Calibrationpublic static double ClampTemperature(double t) =>double.IsNaN(t) || double.IsInfinity(t) ? 1.0 : Math.Min(TempMax, Math.Max(TempMin, t));+ /// <summary>+ /// <see cref="ClampTemperature(double)"/> for a value read from JSON, exactly as Python's+ /// <c>clamp_temperature</c> treats it: a <see cref="bool"/> is 1.0, and so is anything+ /// <c>float()</c> refuses (<see langword="null"/>, a list, a dictionary, a non-numeric string).+ /// A numeric string such as <c>"2.5"</c>, <c>" 2.5 "</c> or <c>"1e1"</c> is read the way+ /// <c>float()</c> reads it.+ /// </summary>+ public static double ClampTemperature(object? t)+ {+ if (t is bool) return 1.0;+ return PythonFloat.TryConvert(t, out var value) ? ClampTemperature(value) : 1.0;+ }++ /// <summary>+ /// Python's <c>temperature_rejected</c>: whether <paramref name="raw"/> is not the number+ /// <see cref="ClampTemperature(object?)"/> turned into <paramref name="applied"/>. A bool, or a+ /// value that will not convert, is rejected; so is a number that had to move.+ /// </summary>+ public static bool TemperatureRejected(object? raw, double applied)+ {+ if (raw is bool) return true;+ return !PythonFloat.TryConvert(raw, out var value) || value != applied;+ }+/// <summary>/// The calibration bucket key for a question type and option count, e.g. <c>"choice:3-5"</c>./// </summary>diff --git a/laya-dotnet/src/Laya/LangGuess.cs b/laya-dotnet/src/Laya/LangGuess.csnew file mode 100644index 00000000..ada14408--- /dev/null+++ b/laya-dotnet/src/Laya/LangGuess.cs@@ -0,0 +1,69 @@+namespace Laya;++/// <summary>+/// A caller-supplied answer to "can the English checkpoint read this state?" -- Python's+/// <c>lang_guess</c>: a language code, or a callback taking the state.+/// </summary>+/// <remarks>+/// <para>+/// It is an opt-in hint, checked after an explicit <c>lang</c> and before the built-in detection.+/// Any code that is not English routes to the multilingual checkpoint. A code that names no+/// language (blank, <c>C</c>, <c>POSIX</c>, <c>C.UTF-8</c>, <c>und</c>, <c>zxx</c>, <c>mul</c>), or+/// a callback returning <see langword="null"/>, abstains, and routing falls through to detection;+/// that is how a language-identification model that is not confident says so.+/// </para>+/// <para>+/// Pass one per call (the <c>langGuess</c> overloads of <see cref="LayaRouter.Route(object?, QuestionSet?, string?, string?, string?, LangGuess?)"/>)+/// or install one for every request with <see cref="LayaRouterOptions.LangGuess"/>.+/// </para>+/// </remarks>+public sealed class LangGuess+{+ private readonly string? _code;+ private readonly Func<object?, string?>? _callback;++ private LangGuess(string? code, Func<object?, string?>? callback)+ {+ _code = code;+ _callback = callback;+ }++ /// <summary>A hint that is a language code such as <c>"en"</c>, <c>"en-US"</c>, <c>"en_US.UTF-8"</c> or <c>"pt-BR"</c>.</summary>+ public static LangGuess FromCode(string? code) => new(code, null);++ /// <summary>+ /// A hint that reads the state and answers with a language code, or <see langword="null"/> to abstain.+ /// </summary>+ public static LangGuess FromCallback(Func<object?, string?> callback)+ {+ ArgumentNullException.ThrowIfNull(callback);+ return new(null, callback);+ }++ /// <summary>A code string converts to a code hint.</summary>+ public static implicit operator LangGuess(string code) => FromCode(code);++ /// <summary>Python's <c>_resolve_hint</c>: English / not English, or <see langword="null"/> when the hint names nothing.</summary>+ internal bool? Resolve(object? state) =>+ EnglishFromCode(_callback is null ? _code : _callback(state));++ // Codes that are valid $LANG values but name no language, so they answer nothing about the+ // state: router.py:242 _LANGUAGE_AGNOSTIC_CODES.+ private static readonly string[] LanguageAgnosticCodes = ["c", "posix", "und", "zxx", "mul"];++ /// <summary>+ /// Python's <c>_english_from_code</c> (router.py:245): <see langword="true"/> for an English code,+ /// <see langword="false"/> for any other language, <see langword="null"/> when the code identifies nothing.+ /// Accepts <c>"en"</c>, <c>"EN"</c>, <c>"en-US"</c>, <c>"en_US"</c> and <c>"en_US.UTF-8"</c>.+ /// </summary>+ internal static bool? EnglishFromCode(string? value)+ {+ if (value is null) return null;+ var code = LanguageDetection.PythonLower(PythonStr.Strip(value));+ if (code.Length == 0) return null;+ code = code.Split('.', 2)[0]; // en_US.UTF-8 -> en_US+ var primary = code.Replace('_', '-').Split('-', 2)[0]; // en_US -> en+ if (primary.Length == 0 || Array.IndexOf(LanguageAgnosticCodes, primary) >= 0) return null;+ return primary is "en" or "eng" or "english";+ }+}diff --git a/laya-dotnet/src/Laya/LanguageDetection.cs b/laya-dotnet/src/Laya/LanguageDetection.csindex 71ea19e1..44508cf9 100644--- a/laya-dotnet/src/Laya/LanguageDetection.cs+++ b/laya-dotnet/src/Laya/LanguageDetection.cs@@ -178,6 +178,8 @@ public static class LanguageDetection},};+ private static readonly UTF8Encoding StrictUtf8 = new(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true);+private static readonly HashSet<string> NordicOverlapWords = new(StringComparer.Ordinal){ "får", "hej", "ja", "jo", "kommer", "mig", "min", "nej", "om", "skulle", "tack", "vi" };private static readonly HashSet<string> ShortSwedishWords = new(StringComparer.Ordinal)@@ -188,10 +190,6 @@ public static class LanguageDetection};private static readonly HashSet<string> EnglishCollisionWords = new(StringComparer.Ordinal){ "care", "come", "do", "im", "per", "plus", "son", "todo" };- private static readonly Regex Identifier = new(@"(?<![\w-])[\w-]*(?:[.@][\w-]+)+");- private static readonly Regex CodeLine = new(@"[=;{}\[\]]|\w\(");- private static readonly Regex Joined = new(@"[^\W_][._/\\][^\W_]");- private static readonly Regex LetterRun = new(@"[^\W\d_]{2,}");// Letters that ordinary English does not use. This is the signal that catches a Latin-script// language for which no stopword list is held at all (Romanian, Polish, Czech, Turkish,@@ -252,6 +250,11 @@ public static class LanguageDetectioncase string s:leaves.Add(s);return;+ // bytes: strict UTF-8, and invalid UTF-8 contributes no text (lang.py:253-255).+ case byte[] bytes:+ try { leaves.Add(StrictUtf8.GetString(bytes)); }+ catch (DecoderFallbackException) { }+ return;// Dict-like before generically enumerable, mirroring PythonJson's dispatch order.case IEnumerable<KeyValuePair<string, object?>> kvpSeq:foreach (var kv in kvpSeq) CollectText(kv.Value, leaves, depth + 1);@@ -390,7 +393,7 @@ public static class LanguageDetection{ArgumentNullException.ThrowIfNull(text);- var words = ExtractWords(PythonLower(Identifier.Replace(text, " ").Replace("İ", "i"))).ToList();+ var words = ExtractWords(PythonLower(ReplaceIdentifiers(text).Replace("İ", "i"))).ToList();var lowered = PythonLower(text);var diac = 0;@@ -489,14 +492,14 @@ public static class LanguageDetection{if (seen >= 4000) break;var sample = SliceCodePoints(segment, 4000 - seen);- seen += sample.EnumerateRunes().Count();+ seen += CodePointCount(sample);var language = NamedProseLanguage(sample);if (language is not null){result = result with{Language = language, IsEnglish = false, LanguageUndecided = false,- MixedSegment = sample.Trim(),+ MixedSegment = PythonStr.Strip(sample),};break;}@@ -507,30 +510,10 @@ public static class LanguageDetectionvar bestCount = -1;foreach (var leaf in leaves){- LanguageAnalysis? candidate = null;- var lineCount = -1;- foreach (var line in leaf.Split('\n'))- {- if (line.EnumerateRunes().Count() < 7) continue;- var sample = SliceCodePoints(line, 4000);- if (string.IsNullOrWhiteSpace(sample) || CodeLine.IsMatch(sample)) continue;- var detection = AnalyseText(sample);- if (detection.IsEnglish) continue;- var alphaCount = sample.EnumerateRunes().Count(Rune.IsLetter);- if (detection.Language is not null and not "en")- {- if (NamedProseLanguage(sample) is null) continue;- }- else if (detection.Script is not "latin" and not "unknown")- {- if (!HasNonLatinWords(sample) || alphaCount < 10) continue;- }- else if (!(detection.LanguageUndecided && detection.DiacriticRate >= NonEnDiacriticRate- && ExtractWords(sample).Count >= 4)) continue;- if (alphaCount > lineCount) { lineCount = alphaCount; candidate = detection; }- }+ var candidate = LeafNonEnglish(leaf);+ if (candidate is null) continue;var count = SliceCodePoints(leaf, 4000).EnumerateRunes().Count(Rune.IsLetter);- if (candidate is not null && count > bestCount) { bestCount = count; best = candidate; }+ if (count > bestCount) { bestCount = count; best = candidate; }}return best is null ? result : result with{@@ -538,30 +521,85 @@ public static class LanguageDetection};}+ // _leaf_non_english: a string value that is itself not safe for the English checkpoint.+ private static LanguageAnalysis? LeafNonEnglish(string leaf)+ {+ var bestN = -1;+ LanguageAnalysis? best = null;+ foreach (var line in leaf.Split('\n'))+ {+ // A line this short cannot be selected (see lang.py's note), so skip it first.+ if (CodePointCount(line) < 7) continue;+ var sample = SliceCodePoints(line, 4000);+ if (PythonStr.Strip(sample).Length == 0 || HasCodeSyntax(sample)) continue;+ var det = AnalyseText(sample);+ if (det.IsEnglish) continue;+ // A named language still has to survive NamedProseLanguage, but a veto is not a verdict+ // of English: the line falls through to the diacritic branch below.+ var named = det.Language is not (null or "en") && NamedProseLanguage(sample) is not null;+ var alphaCount = sample.EnumerateRunes().Count(Rune.IsLetter);+ if (!named)+ {+ if (det.Script is not ("latin" or "unknown"))+ {+ if (!(HasNonLatinWords(sample) && alphaCount >= NonLatinMinLetters)) continue;+ }+ else if (!(det.DiacriticRate >= NonEnDiacriticRate && ExtractWords(sample).Count >= 4)) continue;+ }+ if (alphaCount > bestN) { bestN = alphaCount; best = det; }+ }+ return best;+ }++ // _shouted: a *cased* segment written entirely in capitals.+ private static bool Shouted(string text)+ {+ var upper = false;+ foreach (var rune in text.EnumerateRunes())+ {+ if (PythonStr.IsLower(rune)) return false;+ if (PythonStr.IsUpper(rune)) upper = true;+ }+ return upper;+ }++ // _shouted_evidence: whether an all-caps line's evidence for `lang` is more than acronym-shaped tokens.+ private static bool ShoutedEvidence(List<string> tokens, string lang, double diacriticRate)+ {+ if (!tokens.Any(t => CodePointCount(t) >= ShoutedMinWord)) return false;+ if (diacriticRate >= NonEnDiacriticRate) return true;+ var stop = Stop.GetValueOrDefault(lang);+ return stop is not null+ && tokens.Select(PythonLower).Any(w => stop.Contains(w) && CodePointCount(w) >= ShoutedMinStopword);+ }+private static string? NamedProseLanguage(string segment){- if (string.IsNullOrWhiteSpace(segment) || CodeLine.IsMatch(segment)) return null;- var prose = string.Join(' ', segment.Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries)- .Where(token => !Joined.IsMatch(token)));- if (prose.EnumerateRunes().Any(Rune.IsLower))- prose = LetterRun.Replace(prose, match => match.Value.EnumerateRunes().Any(Rune.IsUpper)- && !match.Value.EnumerateRunes().Any(Rune.IsLower) ? " " : match.Value);+ if (PythonStr.Strip(segment).Length == 0 || HasCodeSyntax(segment)) return null;+ var prose = string.Join(' ', PythonStr.Split(segment).Where(token => !IsJoinedToken(token)));+ var shouted = Shouted(prose);+ if (!shouted) prose = BlankUpperRuns(prose);var tokens = ExtractWords(prose);if (tokens.Count < 4) return null;- var language = LatinProfile(prose).Language;+ var profile = LatinProfile(prose);+ var language = profile.Language;if (language is null or "en") return null;+ if (shouted && !ShoutedEvidence(tokens, language, profile.DiacriticRate)) return null;return tokens.Select(PythonLower).Distinct().Count(w => Stop[language].Contains(w)) >= 2? language : null;}+ private static bool NonLatinCarries(double nonLatin, int nonLatinLetters) =>+ nonLatin >= NonLatinFraction+ || (nonLatin >= NonLatinMinFraction && nonLatinLetters >= NonLatinMinLetters);+private static LanguageAnalysis AnalyseText(string text){var prof = ScriptProfile(text);var script = DetectScript(text);var nonLatin = prof.Count > 0 ? Calibration.Round4(1.0 - prof.GetValueOrDefault("latin", 0.0)) : 0.0;- var nonLatinLetters = Math.Round(nonLatin * text.EnumerateRunes().Count(Rune.IsLetter));- if (script == "latin" && HasNonLatinWords(text)- && (nonLatin >= 0.2 || (nonLatin >= 0.1 && nonLatinLetters >= 10)))+ var nonLatinLetters = (int)Math.Round(nonLatin * text.EnumerateRunes().Count(Rune.IsLetter));+ if (script == "latin" && HasNonLatinWords(text) && NonLatinCarries(nonLatin, nonLatinLetters))script = prof.Where(p => p.Key != "latin").MaxBy(p => p.Value).Key;if (script == "unknown")@@ -572,6 +610,26 @@ public static class LanguageDetectionvar profLat = LatinProfile(text);var lang = profLat.Language;+ // The acronym bar belongs here too, not only in the segment scan (lang.py:689-736). Neither+ // bar applies when the state is carried by a caseless script.+ if (lang is not (null or "en") && !NonLatinCarries(nonLatin, nonLatinLetters))+ {+ if (Shouted(text))+ {+ if (!ShoutedEvidence(ExtractWords(text), lang, profLat.DiacriticRate)) lang = null;+ }+ else+ {+ // Mixed-case text: re-take the verdict without the acronym-shaped all-caps runs.+ var caps = LetterRuns(text).Select(r => text.Substring(r.Start, r.Length))+ .Where(PythonStr.StrIsUpper).ToList();+ if (caps.Count > 0 && !caps.Any(c => CodePointCount(c) >= ShoutedMinWord))+ {+ var blanked = BlankUpperRuns(text);+ if (blanked != text) lang = LatinProfile(blanked).Language;+ }+ }+ }// Undecided is not English. Treating it as English sends every Latin-script language no// stopwords are held for to the checkpoint that cannot read it, silently.var undecided = lang is null;@@ -590,8 +648,7 @@ public static class LanguageDetectionvar upper = false;foreach (var rune in text.EnumerateRunes()){- if (Rune.GetUnicodeCategory(rune) is UnicodeCategory.NonSpacingMark or UnicodeCategory.SpacingCombiningMark)- continue;+ if (PythonStr.IsCombining(rune.Value)) continue;var cp = rune.Value;var next = cp < 0x0250 || (cp >= 0x1E00 && cp <= 0x1EFF)|| (cp >= 0xFF21 && cp <= 0xFF3A) || (cp >= 0xFF41 && cp <= 0xFF5A)@@ -600,11 +657,133 @@ public static class LanguageDetectionif (count >= 2 && !upper) return true;script = next;count = next is null ? 0 : 1;- upper = Rune.IsUpper(rune);+ upper = PythonStr.IsUpper(rune);}return count >= 2 && !upper;}+ // ── Python-compatible scanners ───────────────────────────────────────────+ // `re`'s \w, \s and \d differ from .NET's (marks, connector punctuation, U+001C..U+001F) and+ // .NET regexes work on UTF-16 units, not code points, so the patterns lang.py runs are written+ // over code-point predicates instead.++ private const int ShoutedMinStopword = 4;+ private const int ShoutedMinWord = 5;+ private const double NonLatinFraction = 0.2;+ private const double NonLatinMinFraction = 0.1;+ private const int NonLatinMinLetters = 10;++ private static int CodePointCount(string s)+ {+ var count = 0;+ foreach (var _ in s.EnumerateRunes()) count++;+ return count;+ }++ /// <summary><c>\w</c>: <c>str.isalnum()</c> or the underscore.</summary>+ private static bool IsWordChar(Rune r) => r.Value == '_' || Rune.IsLetter(r) || Rune.IsNumber(r);++ /// <summary><c>[^\W_]</c>: <c>str.isalnum()</c>.</summary>+ private static bool IsAlnum(Rune r) => Rune.IsLetter(r) || Rune.IsNumber(r);++ /// <summary><c>_IDENTIFIER.sub(" ", text)</c>: <c>(?<![\w-])[\w-]*(?:[.@][\w-]+)+</c>.</summary>+ private static string ReplaceIdentifiers(string text)+ {+ var runes = text.EnumerateRunes().ToArray();+ static bool Ident(Rune r) => IsWordChar(r) || r.Value == '-';+ var sb = new StringBuilder(text.Length);+ var i = 0;+ while (i < runes.Length)+ {+ // The lookbehind only lets a match start at the head of a run, which is also what keeps this linear.+ if (i == 0 || !Ident(runes[i - 1]))+ {+ var j = i;+ while (j < runes.Length && Ident(runes[j])) j++;+ var groups = 0;+ while (j < runes.Length && runes[j].Value is '.' or '@')+ {+ var k = j + 1;+ while (k < runes.Length && Ident(runes[k])) k++;+ if (k == j + 1) break;+ j = k;+ groups++;+ }+ if (groups > 0)+ {+ sb.Append(' ');+ i = j;+ continue;+ }+ }+ sb.Append(runes[i]);+ i++;+ }+ return sb.ToString();+ }++ /// <summary><c>_CODE_LINE.search</c>: <c>[=;{}\[\]]|\w\(</c>.</summary>+ private static bool HasCodeSyntax(string line)+ {+ var prevWord = false;+ foreach (var rune in line.EnumerateRunes())+ {+ if (rune.Value is '=' or ';' or '{' or '}' or '[' or ']' || (rune.Value == '(' && prevWord)) return true;+ prevWord = IsWordChar(rune);+ }+ return false;+ }++ /// <summary><c>_JOINED.search(token)</c>: <c>[^\W_][._/\\][^\W_]</c>.</summary>+ private static bool IsJoinedToken(string token)+ {+ var runes = token.EnumerateRunes().ToArray();+ for (var i = 0; i + 2 < runes.Length; i++)+ if (IsAlnum(runes[i]) && runes[i + 1].Value is '.' or '_' or '/' or '\\' && IsAlnum(runes[i + 2]))+ return true;+ return false;+ }++ /// <summary><c>_LETTER_RUN.finditer</c>: maximal runs of at least two <c>[^\W\d_]</c> characters (UTF-16 offsets).</summary>+ private static List<(int Start, int Length)> LetterRuns(string text)+ {+ var runs = new List<(int, int)>();+ var start = -1;+ var count = 0;+ var offset = 0;+ foreach (var rune in text.EnumerateRunes())+ {+ if (IsWordRune(rune))+ {+ if (start < 0) { start = offset; count = 0; }+ count++;+ }+ else+ {+ if (start >= 0 && count >= 2) runs.Add((start, offset - start));+ start = -1;+ }+ offset += rune.Utf16SequenceLength;+ }+ if (start >= 0 && count >= 2) runs.Add((start, offset - start));+ return runs;+ }++ /// <summary><c>_LETTER_RUN.sub(lambda m: " " if m.group().isupper() else m.group(), text)</c>.</summary>+ private static string BlankUpperRuns(string text)+ {+ var sb = new StringBuilder(text.Length);+ var last = 0;+ foreach (var (start, length) in LetterRuns(text))+ {+ if (!PythonStr.StrIsUpper(text.Substring(start, length))) continue;+ sb.Append(text, last, start - last).Append(' ');+ last = start + length;+ }+ sb.Append(text, last, text.Length - last);+ return sb.ToString();+ }+// ── word extraction ──────────────────────────────────────────────────────/// <summary>diff --git a/laya-dotnet/src/Laya/LayaConfig.cs b/laya-dotnet/src/Laya/LayaConfig.csindex d00fd497..f3bdd654 100644--- a/laya-dotnet/src/Laya/LayaConfig.cs+++ b/laya-dotnet/src/Laya/LayaConfig.cs@@ -1,5 +1,3 @@-using System.Text.Json;-namespace Laya;/// <summary>@@ -12,38 +10,49 @@ namespace Laya;public sealed class LayaConfig{private LayaConfig(string? encoder, int maxLen, int headMaxLen,- IReadOnlyList<double> temperatureRaw,- IReadOnlyDictionary<string, double> temperatureByOptionsRaw)+ IReadOnlyList<object?> temperatureValues,+ IReadOnlyDictionary<string, object?> temperatureByOptionsValues){+ TemperatureRawValues = temperatureValues;+ TemperatureByOptionsRawValues = temperatureByOptionsValues;+ var temperatureRaw = temperatureValues.Select(AsDouble).ToList();+ var temperatureByOptionsRaw = temperatureByOptionsValues+ .ToDictionary(kv => kv.Key, kv => AsDouble(kv.Value), StringComparer.Ordinal);Encoder = encoder;MaxLen = maxLen;HeadMaxLen = headMaxLen;TemperatureRaw = temperatureRaw;TemperatureByOptionsRaw = temperatureByOptionsRaw;- Temperature = temperatureRaw.Select(Calibration.ClampTemperature).ToList();- TemperatureByOptions = temperatureByOptionsRaw+ // clamp_temperature on every entry, with the JSON value as Python holds it (onnx_agent.py:249-252).+ Temperature = temperatureValues.Select(Calibration.ClampTemperature).ToList();+ TemperatureByOptions = temperatureByOptionsValues.ToDictionary(kv => kv.Key, kv => Calibration.ClampTemperature(kv.Value), StringComparer.Ordinal);+ // onnx_agent.py:254-262: "<name>=<repr of the raw value> -> <applied, %g>", by-options entries first.var rejected = new List<string>();- foreach (var kv in temperatureByOptionsRaw)- if (Calibration.ClampTemperature(kv.Value) != kv.Value)- rejected.Add($"{kv.Key}={kv.Value:G4}");- for (var i = 0; i < temperatureRaw.Count; i++)- if (Calibration.ClampTemperature(temperatureRaw[i]) != temperatureRaw[i])- rejected.Add($"temperature[{i}]={temperatureRaw[i]:G4}");+ foreach (var kv in temperatureByOptionsValues)+ if (Calibration.TemperatureRejected(kv.Value, TemperatureByOptions[kv.Key]))+ rejected.Add($"{kv.Key}={PythonFloat.Repr(kv.Value)} -> {PythonFloat.FormatG(TemperatureByOptions[kv.Key])}");+ for (var i = 0; i < temperatureValues.Count; i++)+ if (Calibration.TemperatureRejected(temperatureValues[i], Temperature[i]))+ rejected.Add($"temperature[{i}]={PythonFloat.Repr(temperatureValues[i])} -> {PythonFloat.FormatG(Temperature[i])}");ClampedTemperatures = rejected;- // Mirrors Python's RuntimeWarning. One warning per clamped value so a log scan finds- // them without needing the property; use System.Diagnostics.Trace so the caller can- // suppress them by removing trace listeners rather than needing a new dependency.- foreach (var desc in rejected)- System.Diagnostics.Trace.TraceWarning(- "Laya: temperature {0} is outside [{1}, {2}] and was clamped — " +- "confidence from that bucket is uncalibrated",- desc, Calibration.TempMin, Calibration.TempMax);+ // Mirrors Python's RuntimeWarning, one warning listing every rejected entry, shown once per process+ // for the same text (WarnOnce); it goes through System.Diagnostics.Trace so a caller can suppress+ // it by removing trace listeners.+ if (rejected.Count > 0)+ WarnOnce.Warn(+ $"laya ONNX: this checkpoint ships temperatures outside [{PythonFloat.FormatG(Calibration.TempMin)}, "+ + $"{PythonFloat.FormatG(Calibration.TempMax)}] which would distort "+ + $"confidence; clamping {string.Join(", ", rejected)}. Treat confidence from the affected buckets as uncalibrated.");}+ // The numeric view of a raw JSON value kept for TemperatureRaw: a number as itself, a numeric string+ // as float() reads it, anything else NaN.+ private static double AsDouble(object? value) => PythonFloat.TryConvert(value, out var d) && value is not bool ? d : double.NaN;+/// <summary>The encoder the checkpoint was trained on, for information only.</summary>public string? Encoder { get; }@@ -59,9 +68,23 @@ public sealed class LayaConfig/// <summary>Per <c>(type, option-count)</c> bucket temperature, clamped. Keys as <see cref="Calibration.TempBucket"/> builds them.</summary>public IReadOnlyDictionary<string, double> TemperatureByOptions { get; }- /// <summary>What the checkpoint shipped, before clamping.</summary>+ /// <summary>+ /// What the checkpoint shipped, before clamping, as numbers: an entry that is not a number+ /// (and not a numeric string <c>float()</c> accepts) reads <see cref="double.NaN"/>. See+ /// <see cref="TemperatureRawValues"/> for the values themselves.+ /// </summary>public IReadOnlyList<double> TemperatureRaw { get; }+ /// <summary>+ /// The <c>temperature</c> entries exactly as the JSON held them: a <see cref="long"/> or+ /// <see cref="System.Numerics.BigInteger"/> for an integer, a <see cref="double"/>, a+ /// <see cref="string"/>, a <see cref="bool"/>, <see langword="null"/>, or a list / key-value list.+ /// </summary>+ public IReadOnlyList<object?> TemperatureRawValues { get; }++ /// <summary>The <c>temperature_by_options</c> entries exactly as the JSON held them; see <see cref="TemperatureRawValues"/>.</summary>+ public IReadOnlyDictionary<string, object?> TemperatureByOptionsRawValues { get; }+/// <summary>What the checkpoint shipped, before clamping.</summary>public IReadOnlyDictionary<string, double> TemperatureByOptionsRaw { get; }@@ -74,36 +97,49 @@ public sealed class LayaConfigpublic IReadOnlyList<string> ClampedTemperatures { get; }/// <summary>Read a config from a <c>rl_agent_config.json</c> file.</summary>- public static LayaConfig Load(string path) => Parse(File.ReadAllText(path));+ public static LayaConfig Load(string path) => Parse(PythonJsonReader.ReadAllText(path));- /// <summary>Parse a config from JSON text.</summary>+ /// <summary>Parse a config from JSON text, read as Python's <c>json.loads</c> reads it.</summary>public static LayaConfig Parse(string json){- using var doc = JsonDocument.Parse(json);- var root = doc.RootElement;+ if (PythonJsonReader.Parse(json) is not IReadOnlyDictionary<string, object?> root)+ throw new InvalidDataException("the checkpoint config must be a JSON object");- var temperature = new List<double>();- if (root.TryGetProperty("temperature", out var temp) && temp.ValueKind == JsonValueKind.Array)- foreach (var t in temp.EnumerateArray())- temperature.Add(t.ValueKind == JsonValueKind.Number ? t.GetDouble() : double.NaN);+ // `cfg.get("temperature", [1.0, 1.0, 1.0])`: the list as written, entries as json.loads typed them.+ var temperature = new List<object?>();+ if (root.TryGetValue("temperature", out var temp) && temp is List<object?> list)+ temperature.AddRange(list);// Python indexes self.temperature[qtype] for qtype 0..2, so a short list would be an// IndexError there; pad to 3 with the neutral value rather than failing later per-question.while (temperature.Count < 3) temperature.Add(1.0);- var byOptions = new Dictionary<string, double>(StringComparer.Ordinal);- if (root.TryGetProperty("temperature_by_options", out var tbo) && tbo.ValueKind == JsonValueKind.Object)- foreach (var p in tbo.EnumerateObject())- byOptions[p.Name] = p.Value.ValueKind == JsonValueKind.Number ? p.Value.GetDouble() : double.NaN;+ var byOptions = new Dictionary<string, object?>(StringComparer.Ordinal);+ if (root.TryGetValue("temperature_by_options", out var tbo) && tbo is IEnumerable<KeyValuePair<string, object?>> map)+ foreach (var (key, value) in map)+ byOptions[key] = value;++ // clamp_temperature(<int too large for a float>) raises OverflowError, which nothing catches, so+ // Python fails to load the checkpoint (onnx_agent.py:249-252); a float literal reads as inf instead.+ if (temperature.Concat(byOptions.Values).Any(v => v is System.Numerics.BigInteger big && double.IsInfinity((double)big)))+ throw new InvalidDataException("OverflowError: int too large to convert to float");return new LayaConfig(- root.TryGetProperty("encoder", out var enc) && enc.ValueKind == JsonValueKind.String- ? enc.GetString()- : null,- root.TryGetProperty("max_len", out var ml) && ml.ValueKind == JsonValueKind.Number ? ml.GetInt32() : 512,- root.TryGetProperty("head_max_len", out var hml) && hml.ValueKind == JsonValueKind.Number- ? hml.GetInt32()- : 192,+ root.TryGetValue("encoder", out var enc) && enc is string encoder ? encoder : null,+ IntOr(root, "max_len", 512),+ IntOr(root, "head_max_len", 192),temperature,byOptions);}++ // `cfg.get(key, fallback)` for an integer setting: a JSON number is taken, anything else is the fallback.+ private static int IntOr(IReadOnlyDictionary<string, object?> root, string key, int fallback)+ {+ if (!root.TryGetValue(key, out var v)) return fallback;+ return v switch+ {+ long l when l is >= int.MinValue and <= int.MaxValue => (int)l,+ long or System.Numerics.BigInteger or double => throw new FormatException($"{key} is not a 32-bit integer"),+ _ => fallback,+ };+ }}diff --git a/laya-dotnet/src/Laya/LayaEmail.cs b/laya-dotnet/src/Laya/LayaEmail.csindex 568dedf0..5e755285 100644--- a/laya-dotnet/src/Laya/LayaEmail.cs+++ b/laya-dotnet/src/Laya/LayaEmail.cs@@ -6,90 +6,110 @@ namespace Laya;/// <summary>Email utilities for cleaning and structuring email inputs. Ports <c>laya/email.py</c>.</summary>public static partial class LayaEmail{+ // Python's \b is a boundary between \w and \W with \w = str.isalnum() or "_"; .NET's \w also takes+ // marks and connector punctuation and leaves out Nl/No. [\p{L}\p{N}_] is Python's \w.+ private const string Wb = @"(?:(?<=[\p{L}\p{N}_])(?![\p{L}\p{N}_])|(?<![\p{L}\p{N}_])(?=[\p{L}\p{N}_]))";+// re.match anchors at the start only; every pattern below carries its own leading `^`, so// .NET's Regex.IsMatch (which is not implicitly start-anchored) reproduces that exactly.// `re.I` -> RegexOptions.IgnoreCase.- [GeneratedRegex(@"^\s*On .{0,300}wrote:\s*$", RegexOptions.IgnoreCase)]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*On .{0,300}wrote:[\s\x1c-\x1f]*$", RegexOptions.IgnoreCase)]private static partial Regex QuoteHeaderOnWrote();- [GeneratedRegex(@"^\s*(Em (?=.*\d).{0,300}escreveu:|El (?=.*\d).{0,300}escribi[óo]:|Le (?=.*\d).{0,300}a [eé]crit\s*:)\s*$", RegexOptions.IgnoreCase)]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*(Em (?=.*\d).{0,300}escreveu:|El (?=.*\d).{0,300}escr[iIıİ]b[iIıİ][óo]:|Le (?=.*\d).{0,300}a [eé]cr[iIıİ]t[\s\x1c-\x1f]*:)[\s\x1c-\x1f]*$", RegexOptions.IgnoreCase)]private static partial Regex QuoteHeaderTranslated();- [GeneratedRegex(@"^\s*-{2,}\s*(Original|Forwarded) Message\s*-{2,}", RegexOptions.IgnoreCase)]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*-{2,}[\s\x1c-\x1f]*(Or[iIıİ]g[iIıİ]nal|Forwarded) Message[\s\x1c-\x1f]*-{2,}", RegexOptions.IgnoreCase)]private static partial Regex QuoteHeaderOriginalOrForwarded();- [GeneratedRegex(@"^\s*-{2,}\s*(Mensagem (original|encaminhada)|Mensaje (original|reenviado)|Message d'origine)\s*-{2,}", RegexOptions.IgnoreCase)]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*-{2,}[\s\x1c-\x1f]*(Mensagem (or[iIıİ]g[iIıİ]nal|encam[iIıİ]nhada)|Mensaje (or[iIıİ]g[iIıİ]nal|reenv[iIıİ]ado)|Message d'or[iIıİ]g[iIıİ]ne)[\s\x1c-\x1f]*-{2,}", RegexOptions.IgnoreCase)]private static partial Regex QuoteHeaderTranslatedMessage();- [GeneratedRegex(@"^\s*_{8,}\s*$")]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*_{8,}[\s\x1c-\x1f]*$")]private static partial Regex QuoteHeaderUnderscores();- [GeneratedRegex(@"^\s*(From:|De\s*:)\s.*[@<]", RegexOptions.IgnoreCase)]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*(From:|De[\s\x1c-\x1f]*:)[\s\x1c-\x1f].*[@<]", RegexOptions.IgnoreCase)]private static partial Regex QuoteHeaderFromLine();- [GeneratedRegex(@"^\s*(De|From)\s*:\s+\S", RegexOptions.IgnoreCase)]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*(De|From)[\s\x1c-\x1f]*:[\s\x1c-\x1f]+[^\s\x1c-\x1f]", RegexOptions.IgnoreCase)]private static partial Regex HeaderFromName();- [GeneratedRegex(@"^\s*(Enviad[oa]( em| el)?:\s|Envoy[ée]( le)?\s*:\s|Sent:\s|(Data|Fecha|Date):\s.*\d{4})", RegexOptions.IgnoreCase)]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*(Env[iIıİ]ad[oa]( em| el)?:[\s\x1c-\x1f]|Envoy[ée]( le)?[\s\x1c-\x1f]*:[\s\x1c-\x1f]|Sent:[\s\x1c-\x1f]|(Data|Fecha|Date):[\s\x1c-\x1f].*\d{4})", RegexOptions.IgnoreCase)]private static partial Regex HeaderNext();- [GeneratedRegex(@"^.{0,120}\S@\S+\s+(wrote|escreveu|escribi[óo]|a [eé]crit)\s*:\s*$", RegexOptions.IgnoreCase)]+ [GeneratedRegex(@"^.{0,120}[^\s\x1c-\x1f]@[^\s\x1c-\x1f]+[\s\x1c-\x1f]+(wrote|escreveu|escr[iIıİ]b[iIıİ][óo]|a [eé]cr[iIıİ]t)[\s\x1c-\x1f]*:[\s\x1c-\x1f]*$", RegexOptions.IgnoreCase)]private static partial Regex AttributionTail();- [GeneratedRegex(@"^\s*(On|Em|El|Le) (?=.*\d)", RegexOptions.IgnoreCase)]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*(On|Em|El|Le) (?=.*\d)", RegexOptions.IgnoreCase)]private static partial Regex AttributionHead();- [GeneratedRegex(@"^\s*--\s*$")]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*--[\s\x1c-\x1f]*$")]private static partial Regex SignatureDashDash();- [GeneratedRegex(@"^\s*(?i:best|kind|warmest|warm|many thanks|thanks|thank you|regards|cheers|sincerely)"- + @"(?i:\s+(?:and|&)\s+regards|\s+(?:regards|wishes|again|in advance|a lot|so much|very much))?")]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*(?i:best|k[iIıİ]nd|warmest|warm|many thanks|thanks|thank you|regards|cheers|s[iIıİ]ncerely)"+ + @"(?i:[\s\x1c-\x1f]+(?:and|&)[\s\x1c-\x1f]+regards|[\s\x1c-\x1f]+(?:regards|w[iIıİ]shes|aga[iIıİ]n|[iIıİ]n advance|a lot|so much|very much))?")]private static partial Regex SignatureClosingHead();- [GeneratedRegex(@"^\s*(atenciosamente|att|abraços?|abs|um abraço|cordialmente|grat[oa]|(muito )?obrigad[oa]s?"- + @"( desde já| pela atenção)?|(com os melhores )?cumprimentos|saudações|"- + @"(un )?saludos?( cordiales)?|atentamente|(muchas )?gracias( de antemano)?|"- + @"(bien )?cordialement|salutations( distinguées)?|bien à vous|merci( d'avance)?|bonne journée)[\s,!.]*$",+ [GeneratedRegex(@"^[\s\x1c-\x1f]*(atenc[iIıİ]osamente|att|abraços?|abs|um abraço|cord[iIıİ]almente|grat[oa]|(mu[iIıİ]to )?obr[iIıİ]gad[oa]s?"+ + @"( desde já| pela atenção)?|(com os melhores )?cumpr[iIıİ]mentos|saudações|"+ + @"(un )?saludos?( cord[iIıİ]ales)?|atentamente|(muchas )?grac[iIıİ]as( de antemano)?|"+ + @"(b[iIıİ]en )?cord[iIıİ]alement|salutat[iIıİ]ons( d[iIıİ]st[iIıİ]nguées)?|b[iIıİ]en à vous|merc[iIıİ]( d'avance)?|bonne journée)[\s\x1c-\x1f,!.]*$",RegexOptions.IgnoreCase)]private static partial Regex SignatureTranslated();- private const string Device = @"iphone|ipad|android|ios|mobile|celular|telemóvel|móvil|galaxy|smartphone|samsung|tablet|outlook|yahoo|mail|e-?mail|gmail|windows";+ private const string Device = @"[iIıİ]phone|[iIıİ]pad|andro[iIıİ]d|[iIıİ]os|mob[iIıİ]le|celular|telemóvel|móv[iIıİ]l|galaxy|smartphone|samsung|tablet|outlook|yahoo|ma[iIıİ]l|e-?ma[iIıİ]l|gma[iIıİ]l|w[iIıİ]ndows";- [GeneratedRegex(@"^\s*((enviad[oa] (do|pelo|pela|via|desde|a partir do)( meu| minha| mi)?|sent from( my)?|"- + @"envoy[ée] (depuis|de)( mon| ma| mes)?)"- + @" (" + Device + @")( (" + Device + @"|para|for|no|na|\d+|phone|device|pro|max|mini|plus|using [a-z][a-z0-9_.+-]*))*"- + @"|(obter o|get) outlook (para|for) (ios|android))[\s.!]*$", RegexOptions.IgnoreCase)]+ [GeneratedRegex(@"^[\s\x1c-\x1f]*((env[iIıİ]ad[oa] (do|pelo|pela|v[iIıİ]a|desde|a part[iIıİ]r do)( meu| m[iIıİ]nha| m[iIıİ])?|sent from( my)?|"+ + @"envoy[ée] (depu[iIıİ]s|de)( mon| ma| mes)?)"+ + @" (" + Device + @")( (" + Device + @"|para|for|no|na|\d+|phone|dev[iIıİ]ce|pro|max|m[iIıİ]n[iIıİ]|plus|us[iIıİ]ng [a-zıİ][a-z0-9_.+ıİ-]*))*"+ + @"|(obter o|get) outlook (para|for) ([iIıİ]os|andro[iIıİ]d))[\s\x1c-\x1f.!]*$", RegexOptions.IgnoreCase)]private static partial Regex SignatureSentFromMobile();// .search() semantics: no anchors, matches anywhere in the text.[GeneratedRegex(- @"(\b(e-?mail|message|information|communication|transmission|contents?)\b[^.]{0,60}"- + @"\bconfidential\b[^.]{0,60}\b(intended|solely|addressee|recipient|privileged|disclos|unauthori[sz]ed)|"- + @"\bconfidential\b[^.]{0,60}\b(and (may|is) (also )?privileged)|"- + @"if you (have )?received this (e-?mail|message) in error|"- + @"\b(esta|este) (mensagem|e-?mail|mensaje|correo)\b[^.]{0,80}(confidencia|sigilos|privilegiad)|"- + @"\b(uso exclusivo|exclusivamente|únicamente|unicamente)\b[^.]{0,30}(destinatári|destinatari|pessoa|persona|entidade|entidad)|"- + @"\b(recebeu|recebido|receber) (esta|este) (mensagem|e-?mail)\b[^.]{0,20} por (engano|erro)|"- + @"\b(ha recibido|recibió|recibe) (este|esta) (mensaje|correo)\b[^.]{0,20} por error|"- + @"\bantes de imprimir\b[^.]{0,100}(meio ambiente|medio ambiente|natureza|planeta|realmente necess)|"- + @"\b(meio|medio) ambiente\b[^.]{0,30}antes de imprimir|"- + @"\b(ce|cet|cette) (message|e-?mail|mail|courriel)\b[^.]{0,80}(confidentiel|privil[eé]gi)|"- + @"\bavez re[çc]u (ce|cet|cette) (message|e-?mail|mail)\b[^.]{0,20} par erreur|"- + @"\b(usage exclusif|exclusivement|uniquement)\b[^.]{0,30}destinataire)",+ @"(" + Wb + @"(e-?ma[iIıİ]l|message|[iIıİ]nformat[iIıİ]on|commun[iIıİ]cat[iIıİ]on|transm[iIıİ]ss[iIıİ]on|contents?)" + Wb + @"[^.]{0,60}"+ + @"" + Wb + @"conf[iIıİ]dent[iIıİ]al" + Wb + @"[^.]{0,60}" + Wb + @"([iIıİ]ntended|solely|addressee|rec[iIıİ]p[iIıİ]ent|pr[iIıİ]v[iIıİ]leged|d[iIıİ]sclos|unauthor[iIıİ][sz]ed)|"+ + @"" + Wb + @"conf[iIıİ]dent[iIıİ]al" + Wb + @"[^.]{0,60}" + Wb + @"(and (may|[iIıİ]s) (also )?pr[iIıİ]v[iIıİ]leged)|"+ + @"[iIıİ]f you (have )?rece[iIıİ]ved th[iIıİ]s (e-?ma[iIıİ]l|message) [iIıİ]n error|"+ + @"" + Wb + @"(esta|este) (mensagem|e-?ma[iIıİ]l|mensaje|correo)" + Wb + @"[^.]{0,80}(conf[iIıİ]denc[iIıİ]a|s[iIıİ]g[iIıİ]los|pr[iIıİ]v[iIıİ]leg[iIıİ]ad)|"+ + @"" + Wb + @"(uso exclus[iIıİ]vo|exclus[iIıİ]vamente|ún[iIıİ]camente|un[iIıİ]camente)" + Wb + @"[^.]{0,30}(dest[iIıİ]natár[iIıİ]|dest[iIıİ]natar[iIıİ]|pessoa|persona|ent[iIıİ]dade|ent[iIıİ]dad)|"+ + @"" + Wb + @"(recebeu|receb[iIıİ]do|receber) (esta|este) (mensagem|e-?ma[iIıİ]l)" + Wb + @"[^.]{0,20} por (engano|erro)|"+ + @"" + Wb + @"(ha rec[iIıİ]b[iIıİ]do|rec[iIıİ]b[iIıİ]ó|rec[iIıİ]be) (este|esta) (mensaje|correo)" + Wb + @"[^.]{0,20} por error|"+ + @"" + Wb + @"antes de [iIıİ]mpr[iIıİ]m[iIıİ]r" + Wb + @"[^.]{0,100}(me[iIıİ]o amb[iIıİ]ente|med[iIıİ]o amb[iIıİ]ente|natureza|planeta|realmente necess)|"+ + @"" + Wb + @"(me[iIıİ]o|med[iIıİ]o) amb[iIıİ]ente" + Wb + @"[^.]{0,30}antes de [iIıİ]mpr[iIıİ]m[iIıİ]r|"+ + @"" + Wb + @"(ce|cet|cette) (message|e-?ma[iIıİ]l|ma[iIıİ]l|courr[iIıİ]el)" + Wb + @"[^.]{0,80}(conf[iIıİ]dent[iIıİ]el|pr[iIıİ]v[iIıİ]l[eé]g[iIıİ])|"+ + @"" + Wb + @"avez re[çc]u (ce|cet|cette) (message|e-?ma[iIıİ]l|ma[iIıİ]l)" + Wb + @"[^.]{0,20} par erreur|"+ + @"" + Wb + @"(usage exclus[iIıİ]f|exclus[iIıİ]vement|un[iIıİ]quement)" + Wb + @"[^.]{0,30}dest[iIıİ]nata[iIıİ]re)",RegexOptions.IgnoreCase)]private static partial Regex Disclaimer();// Sentence boundary: split just after a `.`/`!`/`?` and the whitespace that follows it. .NET// Regex supports the lookbehind natively, so this needs no hand-written splitter.- [GeneratedRegex(@"(?<=[.!?])\s+")]+ [GeneratedRegex(@"(?<=[.!?])[\s\x1c-\x1f]+")]private static partial Regex SentenceBoundary();- [GeneratedRegex(@"\n\s*\n")]+ [GeneratedRegex(@"\n[\s\x1c-\x1f]*\n")]private static partial Regex ParagraphBreak();[GeneratedRegex(@"[ \t]+")]private static partial Regex SpaceTabRun();+ // Python's `re` works on code points; .NET's on UTF-16 units, where a character outside the BMP is+ // two units that `.`, `{0,n}` counts and `\b` would treat as two non-word characters. Every+ // pattern here is only asked "does it match", so each such character is first replaced by one BMP+ // stand-in of the same word class (a letter or number by U+4E00, anything else by U+2603), which+ // no pattern mentions.+ private static string CodePoints(string s)+ {+ if (!s.Any(char.IsSurrogate)) return s;+ var sb = new StringBuilder(s.Length);+ foreach (var rune in s.EnumerateRunes())+ sb.Append(rune.IsBmp ? rune.ToString() : Rune.IsLetter(rune) || Rune.IsNumber(rune) ? "一" : "☃");+ return sb.ToString();+ }++ private static bool Hit(this Regex regex, string text) => regex.IsMatch(CodePoints(text));+private static readonly Regex[] QuoteHeaders =[QuoteHeaderOnWrote(), QuoteHeaderTranslated(), QuoteHeaderOriginalOrForwarded(),@@ -102,37 +122,43 @@ public static partial class LayaEmail];/// <summary>Remove quoted email history, signatures and disclaimers to keep input focused.</summary>+ /// <remarks>+ /// <paramref name="maxChars"/> is a slice budget, not a floor, applied twice exactly as in Python:+ /// as the <c>max_chars * 4</c> pre-cut that bounds the regex work and as the final+ /// <c>text[:max_chars]</c>. <c>0</c> answers with the empty string and a negative value trims from+ /// the end (<c>5 * |max_chars|</c> characters off the body in all), as Python's slicing does.+ /// </remarks>public static string CleanBody(string? body, int maxChars = 3000){var text = (body ?? string.Empty).Replace("\r\n", "\n").Replace("\r", "\n").Replace("\\n", "\n");- if (CodePointLength(text) > maxChars * 4)- text = LanguageDetection.SliceCodePoints(text, maxChars * 4);+ if (CodePointLength(text) > (long)maxChars * 4)+ text = SliceTo(text, (long)maxChars * 4);var lines = new List<string>();var source = text.Split('\n');for (var i = 0; i < source.Length; i++){var line = source[i];- if (lines.Count > 0 && Array.Exists(QuoteHeaders, p => p.IsMatch(line))) break;- if (lines.Count > 0 && HeaderFromName().IsMatch(line) && i + 1 < source.Length- && HeaderNext().IsMatch(source[i + 1])) break;- if (lines.Count > 0 && AttributionTail().IsMatch(line))+ if (lines.Count > 0 && Array.Exists(QuoteHeaders, p => p.Hit(line))) break;+ if (lines.Count > 0 && HeaderFromName().Hit(line) && i + 1 < source.Length+ && HeaderNext().Hit(source[i + 1])) break;+ if (lines.Count > 0 && AttributionTail().Hit(line)){- if (AttributionHead().IsMatch(lines[^1])) lines.RemoveAt(lines.Count - 1);+ if (AttributionHead().Hit(lines[^1])) lines.RemoveAt(lines.Count - 1);break;}- if (line.TrimStart().StartsWith('>')) continue;- lines.Add(line.TrimEnd());+ if (PythonStr.LStrip(line).StartsWith('>')) continue;+ lines.Add(PythonStr.RStrip(line));}var cut = lines.Count;var start = Math.Max(1, Math.Min((int)(lines.Count * 0.6), lines.Count - 8));for (var i = start; i < lines.Count; i++){- var stripped = lines[i].Trim();+ var stripped = PythonStr.Strip(lines[i]);var length = CodePointLength(stripped);- if ((length <= 40 && (IsEnglishSignoff(lines[i]) || Array.Exists(SignatureMarkers, p => p.IsMatch(lines[i]))))- || (length <= 60 && SignatureSentFromMobile().IsMatch(lines[i])))+ if ((length <= 40 && (IsEnglishSignoff(lines[i]) || Array.Exists(SignatureMarkers, p => p.Hit(lines[i]))))+ || (length <= 60 && SignatureSentFromMobile().Hit(lines[i]))){cut = i;break;@@ -141,9 +167,9 @@ public static partial class LayaEmaillines = lines.GetRange(0, cut);var paragraphs = ParagraphBreak().Split(string.Join('\n', lines)).Select(StripDisclaimer);- var collapsed = string.Join("\n\n", paragraphs.Select(p => p.Trim()).Where(p => p.Length > 0));+ var collapsed = string.Join("\n\n", paragraphs.Select(PythonStr.Strip).Where(p => p.Length > 0));text = SpaceTabRun().Replace(collapsed, " ");- return LanguageDetection.SliceCodePoints(text, maxChars);+ return SliceTo(text, maxChars);}private static bool IsEnglishSignoff(string line)@@ -156,11 +182,11 @@ public static partial class LayaEmailif (Rune.GetUnicodeCategory(rune) is System.Globalization.UnicodeCategory.NonSpacingMarkor System.Globalization.UnicodeCategory.SpacingCombiningMarkor System.Globalization.UnicodeCategory.EnclosingMark- && tail.Count > 0 && !Rune.IsWhiteSpace(tail[^1])) continue;+ && tail.Count > 0 && !PythonStr.IsSpace(tail[^1].Value)) continue;tail.Add(rune);}var i = 0;- while (i < tail.Count && (Rune.IsWhiteSpace(tail[i]) || tail[i].Value is ',' or ';' or ':' or '!' or '.')) i++;+ while (i < tail.Count && (PythonStr.IsSpace(tail[i].Value) || tail[i].Value is ',' or ';' or ':' or '!' or '.')) i++;var tokens = 0;while (i < tail.Count){@@ -173,7 +199,7 @@ public static partial class LayaEmail|| Rune.GetUnicodeCategory(tail[i]) is System.Globalization.UnicodeCategory.LetterNumberor System.Globalization.UnicodeCategory.OtherNumber|| tail[i].Value is '_' or '\'' or '-')) i++;- while (i < tail.Count && (Rune.IsWhiteSpace(tail[i]) || tail[i].Value is ',' or '.')) i++;+ while (i < tail.Count && (PythonStr.IsSpace(tail[i].Value) || tail[i].Value is ',' or '.')) i++;}return true;}@@ -189,16 +215,16 @@ public static partial class LayaEmail/// </remarks>private static string StripDisclaimer(string paragraph){- if (!Disclaimer().IsMatch(paragraph)) return paragraph; // nothing to do: keep the original line structure+ if (!Disclaimer().Hit(paragraph)) return paragraph; // nothing to do: keep the original line structurevar pieces = new List<string>();- foreach (var part in SentenceBoundary().Split(paragraph).Select(p => p.Trim()).Where(p => p.Length > 0))+ foreach (var part in SentenceBoundary().Split(paragraph).Select(PythonStr.Strip).Where(p => p.Length > 0)){- if (!Disclaimer().IsMatch(part)) { pieces.Add(part); continue; }+ if (!Disclaimer().Hit(part)) { pieces.Add(part); continue; }var buffer = "";- foreach (var line in part.Split('\n').Select(l => l.Trim()).Where(l => l.Length > 0))+ foreach (var line in part.Split('\n').Select(PythonStr.Strip).Where(l => l.Length > 0)){var firstLetter = line.EnumerateRunes().FirstOrDefault(Rune.IsLetter);- if (buffer.Length > 0 && Rune.IsUpper(firstLetter))+ if (buffer.Length > 0 && PythonStr.IsUpper(firstLetter)){pieces.Add(buffer);buffer = line;@@ -207,7 +233,7 @@ public static partial class LayaEmail}if (buffer.Length > 0) pieces.Add(buffer);}- return string.Join(' ', pieces.Where(p => !Disclaimer().IsMatch(p)));+ return string.Join(' ', pieces.Where(p => !Disclaimer().Hit(p)));}/// <summary>Construct a clean state dictionary for email classification.</summary>@@ -226,11 +252,31 @@ public static partial class LayaEmailpublic static IReadOnlyDictionary<string, object?> State(string? subject, string? body, string? sender = null, bool clean = true,IEnumerable<KeyValuePair<string, object?>>? extra = null)+ => State(subject, body, sender, clean, extra, 3000);++ /// <summary>+ /// <see cref="State(string?, string?, string?, bool, IEnumerable{KeyValuePair{string, object?}}?)"/>+ /// with Python's <c>max_chars</c> budget for <see cref="CleanBody"/> (<c>email_state(..., max_chars=...)</c>).+ /// </summary>+ /// <param name="subject">The email subject.</param>+ /// <param name="body">The email body.</param>+ /// <param name="sender">Optional sender, stored under the key <c>"from"</c> when non-empty.</param>+ /// <param name="clean">Whether <paramref name="body"/> is cleaned; when <see langword="false"/> it is kept whole.</param>+ /// <param name="extra">Additional fields; see the shorter overload.</param>+ /// <param name="maxChars">+ /// The slice budget <see cref="CleanBody"/> cuts the body to. It is consumed here, never added to+ /// the state as a field, and ignored when <paramref name="clean"/> is <see langword="false"/>.+ /// Zero gives an empty body and a negative value cuts from the end; pass a budget larger than the+ /// body, never <c>0</c> or a negative, to mean "no limit".+ /// </param>+ public static IReadOnlyDictionary<string, object?> State(+ string? subject, string? body, string? sender, bool clean,+ IEnumerable<KeyValuePair<string, object?>>? extra, int maxChars){var state = new OrderedDictionary<string, object?>(StringComparer.Ordinal){- ["subject"] = (subject ?? string.Empty).Trim(),- ["body"] = clean ? CleanBody(body) : body ?? string.Empty,+ ["subject"] = PythonStr.Strip(subject ?? string.Empty),+ ["body"] = clean ? CleanBody(body, maxChars) : body ?? string.Empty,};if (!string.IsNullOrEmpty(sender)) state["from"] = sender;if (extra is not null)@@ -239,6 +285,14 @@ public static partial class LayaEmailreturn state;}+ /// <summary>Python's <c>s[:n]</c> for any integer <c>n</c>, measured in code points: a negative <c>n</c> drops <c>|n|</c> from the end.</summary>+ private static string SliceTo(string s, long n)+ {+ if (n >= 0) return LanguageDetection.SliceCodePoints(s, (int)Math.Min(n, int.MaxValue));+ var keep = CodePointLength(s) + n;+ return keep <= 0 ? string.Empty : LanguageDetection.SliceCodePoints(s, (int)keep);+ }+private static int CodePointLength(string s){var count = 0;diff --git a/laya-dotnet/src/Laya/LayaEngine.cs b/laya-dotnet/src/Laya/LayaEngine.csindex dde42275..1d973011 100644--- a/laya-dotnet/src/Laya/LayaEngine.cs+++ b/laya-dotnet/src/Laya/LayaEngine.cs@@ -1,3 +1,4 @@+using System.Runtime.CompilerServices;using Laya.Tokenization;using Microsoft.ML.OnnxRuntime;using Microsoft.ML.OnnxRuntime.Tensors;@@ -76,6 +77,7 @@ public sealed class LayaEngine : IDisposable, ILayaLongPredictor/// When the loaded config's encoder contradicts the expected encoder for the requested/// <see cref="LayaOptions.Checkpoint"/> — indicates the wrong artifact was loaded./// </exception>+ [WarnSite, MethodImpl(MethodImplOptions.NoInlining)]public static LayaEngine Create(LayaOptions? options = null){options ??= new LayaOptions();@@ -126,6 +128,7 @@ public sealed class LayaEngine : IDisposable, ILayaLongPredictor}/// <summary>Load an engine from an artifact directory containing either ONNX layout.</summary>+ [WarnSite, MethodImpl(MethodImplOptions.NoInlining)]public static LayaEngine FromDirectory(string directory) =>Create(new LayaOptions { ModelDirectory = directory });@@ -214,6 +217,7 @@ public sealed class LayaEngine : IDisposable, ILayaLongPredictor/// <param name="batchSize">Maximum windows per forward pass; nonpositive or null uses Python's automatic cap.</param>/// <returns>The deciding window's answers and summed usage, including the number of windows.</returns>/// <exception cref="ArgumentException">Unsupported aggregation, a gap-causing stride, or no state room.</exception>+ [WarnSite, MethodImpl(MethodImplOptions.NoInlining)]public LayaResult PredictLong(object state, QuestionSet questions, int? window = null, int? stride = null,string aggregate = "auto", int? batchSize = null){diff --git a/laya-dotnet/src/Laya/LayaRouter.cs b/laya-dotnet/src/Laya/LayaRouter.csindex d8e93a60..452724ff 100644--- a/laya-dotnet/src/Laya/LayaRouter.cs+++ b/laya-dotnet/src/Laya/LayaRouter.cs@@ -51,7 +51,7 @@ public sealed class LayaRouterOptionspublic LayaCheckpoint Default { get; set; } = LayaCheckpoint.Multilingual;/// <summary>- /// Whether <see cref="LayaRouter.Route"/> may pick <see cref="LayaCheckpoint.TypedDecisions"/>+ /// Whether <see cref="LayaRouter.Route(object?, QuestionSet?, string?, string?, string?)"/> may pick <see cref="LayaCheckpoint.TypedDecisions"/>/// on its own when the question ids match one of the four typed-decisions workflow/// signatures. Default <see langword="false"/>: that checkpoint is fine-tuned on specific/// synthetic workflows and must never be a silent default.@@ -60,6 +60,14 @@ public sealed class LayaRouterOptions/// <summary>Whether every checkpoint should be built immediately when the router is constructed.</summary>public bool Preload { get; set; }++ /// <summary>+ /// A language hint applied to every request, Python's <c>Router(lang_guess=...)</c>: a language+ /// code or a callback taking the state. Checked after an explicit <c>lang</c> and a per-call hint,+ /// and before the built-in detection; a hint that names no language falls through to detection.+ /// See <see cref="Laya.LangGuess"/>.+ /// </summary>+ public LangGuess? LangGuess { get; init; }}/// <summary>@@ -140,11 +148,17 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposablepublic LayaCheckpoint Default { get; }/// <summary>- /// Whether <see cref="Route"/> may pick <see cref="LayaCheckpoint.TypedDecisions"/> on its+ /// Whether <see cref="Route(object?, QuestionSet?, string?, string?, string?)"/> may pick <see cref="LayaCheckpoint.TypedDecisions"/> on its/// own when the question ids match a typed-decisions workflow signature./// </summary>public bool AutoTaskDetection { get; }+ /// <summary>+ /// The language hint installed with <see cref="LayaRouterOptions.LangGuess"/>, applied to every+ /// request after a per-call hint, or <see langword="null"/>.+ /// </summary>+ public LangGuess? LangGuess { get; }+/// <summary>Create a router that builds real <see cref="LayaEngine"/> instances on demand.</summary>public LayaRouter(LayaRouterOptions? options = null): this(options, BuildDefaultFactory((options ?? new LayaRouterOptions()).EngineOptions))@@ -164,6 +178,7 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposable_maxLoaded = Math.Max(1, options.MaxLoaded);Default = options.Default;AutoTaskDetection = options.AutoTaskDetection;+ LangGuess = options.LangGuess;if (options.Preload) Preload();}@@ -225,32 +240,56 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposable/// <remarks>/// Precedence: explicit <paramref name="model"/> > explicit <paramref name="task"/> >/// detected workflow (opt-in via <see cref="AutoTaskDetection"/>) > explicit- /// <paramref name="lang"/> > detected script/language > <see cref="Default"/>.+ /// <paramref name="lang"/> > the hint installed with <see cref="LayaRouterOptions.LangGuess"/>+ /// > detected script/language > <see cref="Default"/>. An explicit <paramref name="lang"/>+ /// decides only when its code names a language (blank, <c>C</c>, <c>POSIX</c>, <c>und</c> ... fall through)./// </remarks>public RouteDecision Route(object? state, QuestionSet? questions = null,string? model = null, string? task = null, string? lang = null)+ => Route(state, questions, model, task, lang, null);++ /// <summary>+ /// <see cref="Route(object?, QuestionSet?, string?, string?, string?)"/> with Python's per-call+ /// <c>lang_guess</c> hint: checked after an explicit <paramref name="lang"/> and before the+ /// built-in detection, ahead of the hint installed with <see cref="LayaRouterOptions.LangGuess"/>.+ /// </summary>+ public RouteDecision Route(+ object? state, QuestionSet? questions,+ string? model, string? task, string? lang, LangGuess? langGuess){if (model is not null)- return new RouteDecision(NormaliseName(model), $"explicit model={PyRepr(model)}", null, null);+ return new RouteDecision(NormaliseName(model), $"explicit model={PythonStr.Repr(model)}", null, null);if (task is not null){var normalisedTaskSpelling = task.ToLowerInvariant().Replace("-", "_");var key = NormaliseName(normalisedTaskSpelling == "typed_decisions" ? "typed-decisions" : task);- return new RouteDecision(key, $"explicit task={PyRepr(task)}", null, null);+ return new RouteDecision(key, $"explicit task={PythonStr.Repr(task)}", null, null);}var workflow = MatchTypedDecisionsWorkflow(questions);if (workflow is not null && AutoTaskDetection)return new RouteDecision(LayaCheckpoint.TypedDecisions,- $"question ids match the {PyRepr(workflow)} typed-decisions workflow", null, workflow);+ $"question ids match the {PythonStr.Repr(workflow)} typed-decisions workflow", null, workflow);if (lang is not null){- var primary = lang.ToLowerInvariant().Split('-')[0];- var key = primary is "en" or "eng" or "english" ? LayaCheckpoint.English : LayaCheckpoint.Multilingual;- return new RouteDecision(key, $"explicit lang={PyRepr(lang)}", null, workflow);+ // An explicit `lang` is decisive only when the code names a language. Blank or whitespace+ // resolves to no usable hint, so it falls through to lang_guess/detection (router.py:1242).+ var resolved = LangGuess.EnglishFromCode(lang);+ if (resolved is { } isEnglish)+ return new RouteDecision(isEnglish ? LayaCheckpoint.English : LayaCheckpoint.Multilingual,+ $"explicit lang={PythonStr.Repr(lang)}", null, workflow);+ }++ // Caller-supplied hint, per-call first then the one installed on the router (router.py:1254).+ foreach (var (source, hint) in new[] { ("lang_guess", langGuess), ("Router(lang_guess=...)", LangGuess) })+ {+ if (hint?.Resolve(state) is { } guessEnglish)+ return new RouteDecision(guessEnglish ? LayaCheckpoint.English : LayaCheckpoint.Multilingual,+ $"{source}: the caller identified this as {(guessEnglish ? "English" : "non-English")} text",+ null, workflow);}var det = LanguageDetection.Analyse(state);@@ -271,10 +310,10 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposable{model2 = LayaCheckpoint.Multilingual;reason = det.MixedSegment is { Length: > 0 } mixed- ? $"Latin script, mostly English, but a line or field reads as {PyRepr(det.Language!)} "- + $"({PyRepr(LanguageDetection.SliceCodePoints(mixed, 60))}); the English checkpoint cannot read it"+ ? $"Latin script, mostly English, but a line or field reads as {PythonStr.Repr(det.Language!)} "+ + $"({PythonStr.Repr(LanguageDetection.SliceCodePoints(mixed, 60))}); the English checkpoint cannot read it": det.Language is { } identified- ? $"Latin script but language looks like {PyRepr(identified)}, not English"+ ? $"Latin script but language looks like {PythonStr.Repr(identified)}, not English"// Unidentified Latin-script language: routed on the non-English letters alone,// because no stopword list here covers it.: $"Latin script, language not identified but {FormatPercent0(det.DiacriticRate)}% "@@ -299,35 +338,6 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposableprivate static string FormatPercent0(double fraction) =>Math.Round(100.0 * fraction, MidpointRounding.ToEven).ToString("0", System.Globalization.CultureInfo.InvariantCulture);- /// <summary>Python's <c>%r</c> for a <see cref="string"/>: a quoted, escaped repr.</summary>- private static string PyRepr(string s)- {- var hasSingle = s.Contains('\'');- var hasDouble = s.Contains('"');- var quote = hasSingle && !hasDouble ? '"' : '\'';- var sb = new System.Text.StringBuilder(s.Length + 2);- sb.Append(quote);- foreach (var rune in s.EnumerateRunes())- {- if (rune.Value == quote) { sb.Append('\\').Append(quote); continue; }- switch (rune.Value)- {- case '\\': sb.Append("\\\\"); break;- case '\n': sb.Append("\\n"); break;- case '\r': sb.Append("\\r"); break;- case '\t': sb.Append("\\t"); break;- default:- if (rune.Value < 0x20 || rune.Value == 0x7f)- sb.Append("\\x").Append(rune.Value.ToString("x2"));- else- sb.Append(rune.ToString());- break;- }- }- sb.Append(quote);- return sb.ToString();- }-// ── loading ───────────────────────────────────────────────────────────────/// <summary>@@ -476,7 +486,16 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposablepublic LayaResult Predict(object? state, QuestionSet questions,string? model = null, string? task = null, string? lang = null)- => PredictRouted(state, questions, model, task, lang,+ => Predict(state, questions, model, task, lang, null);++ /// <summary>+ /// <see cref="Predict(object?, QuestionSet, string?, string?, string?)"/> with Python's per-call+ /// <c>lang_guess</c> hint; see <see cref="Route(object?, QuestionSet?, string?, string?, string?, LangGuess?)"/>.+ /// </summary>+ public LayaResult Predict(+ object? state, QuestionSet questions,+ string? model, string? task, string? lang, LangGuess? langGuess)+ => PredictRouted(state, questions, model, task, lang, langGuess,predictor => predictor.Predict(state, questions));LayaResult ILayaLongPredictor.PredictLong(@@ -491,9 +510,20 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposableobject state, QuestionSet questions, int? window = null, int? stride = null,string aggregate = "auto", int? batchSize = null,string? model = null, string? task = null, string? lang = null)+ => PredictLong(state, questions, window, stride, aggregate, batchSize, model, task, lang, null);++ /// <summary>+ /// <see cref="PredictLong(object, QuestionSet, int?, int?, string, int?, string?, string?, string?)"/>+ /// with Python's per-call <c>lang_guess</c> hint; see+ /// <see cref="Route(object?, QuestionSet?, string?, string?, string?, LangGuess?)"/>.+ /// </summary>+ public LayaResult PredictLong(+ object state, QuestionSet questions, int? window, int? stride,+ string aggregate, int? batchSize,+ string? model, string? task, string? lang, LangGuess? langGuess){ArgumentNullException.ThrowIfNull(state);- return PredictRouted(state, questions, model, task, lang, predictor =>+ return PredictRouted(state, questions, model, task, lang, langGuess, predictor =>predictor is ILayaLongPredictor scanner? scanner.PredictLong(state, questions, window, stride, aggregate, batchSize): throw new NotSupportedException(@@ -501,11 +531,11 @@ public sealed class LayaRouter : ILayaLongPredictor, IDisposable}private LayaResult PredictRouted(- object? state, QuestionSet questions, string? model, string? task, string? lang,+ object? state, QuestionSet questions, string? model, string? task, string? lang, LangGuess? langGuess,Func<ILayaPredictor, LayaResult> predict){ArgumentNullException.ThrowIfNull(questions);- var decision = Route(state, questions, model, task, lang);+ var decision = Route(state, questions, model, task, lang, langGuess);Slot slot;lock (_lock)diff --git a/laya-dotnet/src/Laya/LongPrediction.cs b/laya-dotnet/src/Laya/LongPrediction.csindex 216cb5b0..15d86607 100644--- a/laya-dotnet/src/Laya/LongPrediction.cs+++ b/laya-dotnet/src/Laya/LongPrediction.cs@@ -12,23 +12,44 @@ internal static class LongPredictionvar requested = window is > 0 ? window.Value : Math.Max(64, maxLen - headMaxLen - 8);var room = questions.Count == 0 ? requested : questions.Ids.Min(id => SequenceBuilder.StateRoom(tokenizer, questions[id], maxLen, headMaxLen));+ return BudgetForRoom(requested, room, maxLen, headMaxLen, window, stride);+ }++ // window_budget once the smallest room is known (common.py:404 onward).+ internal static (int Window, int Stride, int Room) BudgetForRoom(+ int requested, int room, int maxLen, int headMaxLen, int? window, int? stride)+ {if (room <= 0)throw new ArgumentException($"predict_long: the questions' options fill the whole sequence (max_len={maxLen},"+ $" head_max_len={headMaxLen}), leaving no room for the state; no window can carry any of it."- + " A label set this large is what laya.shortlist.predict_shortlist is for", nameof(questions));+ + " A label set this large is what laya.shortlist.predict_shortlist is for", "questions");var size = Math.Min(requested, room);- if (size < requested && (window is > 0 || requested >= room * 2L))- Trace.TraceWarning(- $"laya: predict_long: scan window capped {requested} -> {size} to fit these questions"- + $" inside max_len={maxLen}; fewer or shorter options or shortlisting reduce scan cost");+ // common.py:410-440: the two texts Python warns with when the window is cut to the room.+ if (size < requested && window is > 0)+ WarnOnce.Warn(+ $"laya: predict_long: window={requested} is wider than the {room} state tokens these questions"+ + $" leave inside max_len={maxLen}, so every window would be truncated to {room} on the way to"+ + $" the model; scanning with window={room} instead");+ else if (size < requested && requested >= room * 2L)+ WarnOnce.Warn(+ $"laya: predict_long: these questions leave only {room} of max_len={maxLen} for the state"+ + $" (their heads take the rest), so the scan window is capped {requested} -> {room} and roughly"+ + $" {PythonFormat1((double)requested / room)}x as many windows -- each a full forward pass -- are needed to read the"+ + " document. Fewer or shorter options, a larger max_len, or"+ + " laya.shortlist.predict_shortlist for a large label set will all cost less than"+ + " scanning at this width");var step = stride is > 0 ? stride.Value : Math.Max(1, size / 2);if (step > size){if (size < requested && stride <= requested){- step = Math.Max(1, size / 2);- Trace.TraceWarning($"laya: predict_long: stride={stride} reduced to {step} after window clamp");+ var reduced = Math.Max(1, size / 2);+ WarnOnce.Warn(+ $"laya: predict_long: stride={step} was a 50% step for the window={requested} you asked for, but"+ + $" the window was reduced to {size} to fit the room these questions leave; scanning with"+ + $" stride={reduced} instead");+ step = reduced;}elsethrow new ArgumentException(@@ -40,6 +61,18 @@ internal static class LongPredictionreturn (size, step, room);}+ // Python's "%.1f": the exact binary value rounded to one decimal, a tie going to the even digit+ // (.NET's "F1" breaks ties away from zero, which differs for x.25 and x.75).+ internal static string PythonFormat1(double value)+ {+ var scaled = value * 10;+ var floor = Math.Floor(scaled);+ // an exact tie needs a multiple of 0.25, whose product with 10 is exact+ if (scaled - floor == 0.5 && value * 4 == Math.Floor(value * 4))+ return ((floor % 2 == 0 ? floor : floor + 1) / 10).ToString("F1", System.Globalization.CultureInfo.InvariantCulture);+ return value.ToString("F1", System.Globalization.CultureInfo.InvariantCulture);+ }+internal static IReadOnlyList<WindowInfo> Spans(int tokens, int window, int stride){var starts = new List<int>();diff --git a/laya-dotnet/src/Laya/PythonFloat.cs b/laya-dotnet/src/Laya/PythonFloat.csnew file mode 100644index 00000000..bf9c4593--- /dev/null+++ b/laya-dotnet/src/Laya/PythonFloat.cs@@ -0,0 +1,90 @@+using System.Globalization;+using System.Numerics;+using System.Text;+using System.Text.RegularExpressions;++namespace Laya;++/// <summary>Python's <c>float(value)</c> and <c>repr(value)</c> for the values a JSON document can hold.</summary>+internal static partial class PythonFloat+{+ // float(str): [sign] (digits [. [digits]] | . digits) [e [sign] digits], underscores only between+ // digits, or inf / infinity / nan in any case. The input is already reduced to ASCII.+ [GeneratedRegex(@"^[+-]?(?:(?:[0-9]+(?:_[0-9]+)*)(?:\.(?:[0-9]+(?:_[0-9]+)*)?)?|\.[0-9]+(?:_[0-9]+)*)(?:[eE][+-]?[0-9]+(?:_[0-9]+)*)?$")]+ private static partial Regex NumberShape();++ /// <summary>+ /// <c>float(text)</c> for a <see cref="string"/>: <see langword="false"/> where Python raises+ /// <c>ValueError</c>. Unicode whitespace is stripped and Unicode decimal digits are read as+ /// their ASCII value, as <c>_PyUnicode_TransformDecimalAndSpaceToASCII</c> does; only the+ /// ASCII whitespace characters are stripped, so a control character such as U+001F is rejected.+ /// </summary>+ internal static bool TryParse(string text, out double value)+ {+ value = 0;+ var ascii = new StringBuilder(text.Length);+ foreach (var rune in text.EnumerateRunes())+ {+ if (rune.Value < 128) ascii.Append((char)rune.Value); // ASCII is left as it is, U+001C..U+001F included+ else if (PythonStr.IsSpace(rune.Value)) ascii.Append(' ');+ else+ {+ var digit = CharUnicodeInfo.GetDecimalDigitValue(rune.ToString(), 0);+ if (digit < 0 || Rune.GetUnicodeCategory(rune) != UnicodeCategory.DecimalDigitNumber) return false;+ ascii.Append((char)('0' + digit));+ }+ }+ var s = ascii.ToString().Trim(' ', '\t', '\n', '\v', '\f', '\r');+ if (s.Length == 0) return false;++ var body = s[0] is '+' or '-' ? s[1..] : s;+ var negative = s[0] == '-';+ if (body.Equals("inf", StringComparison.OrdinalIgnoreCase) || body.Equals("infinity", StringComparison.OrdinalIgnoreCase))+ {+ value = negative ? double.NegativeInfinity : double.PositiveInfinity;+ return true;+ }+ if (body.Equals("nan", StringComparison.OrdinalIgnoreCase))+ {+ value = double.NaN;+ return true;+ }+ if (!NumberShape().IsMatch(s)) return false;+ return double.TryParse(s.Replace("_", ""), NumberStyles.Float, CultureInfo.InvariantCulture, out value);+ }++ /// <summary>+ /// <c>float(raw)</c> for a JSON-derived value; <see langword="false"/> where Python raises+ /// <c>TypeError</c> or <c>ValueError</c> (<see langword="null"/>, lists, dicts, non-numeric strings).+ /// A <see cref="bool"/> is converted too, as <c>float(True)</c> is 1.0; callers that refuse it say so first.+ /// </summary>+ internal static bool TryConvert(object? raw, out double value)+ {+ switch (raw)+ {+ case bool b: value = b ? 1.0 : 0.0; return true;+ case double d: value = d; return true;+ case long l: value = l; return true;+ case BigInteger big: value = (double)big; return true;+ case string s: return TryParse(s, out value);+ default: value = 0; return false;+ }+ }++ /// <summary>Python's <c>repr()</c> of a JSON-derived value (<c>None</c>, <c>True</c>, int, float, str, list, dict).</summary>+ internal static string Repr(object? value) => value switch+ {+ null => "None",+ bool b => b ? "True" : "False",+ long l => l.ToString(CultureInfo.InvariantCulture),+ BigInteger big => big.ToString(CultureInfo.InvariantCulture),+ double d => double.IsNaN(d) ? "nan" : double.IsPositiveInfinity(d) ? "inf" : double.IsNegativeInfinity(d) ? "-inf" : PythonJson.Repr(d),+ string s => PythonStr.Repr(s),+ IReadOnlyList<object?> list => "[" + string.Join(", ", list.Select(Repr)) + "]",+ IEnumerable<KeyValuePair<string, object?>> map => "{" + string.Join(", ", map.Select(kv => PythonStr.Repr(kv.Key) + ": " + Repr(kv.Value))) + "}",+ _ => value.ToString() ?? "None",+ };++ /// <summary>Python's <c>"%g"</c> for a temperature already clamped into [0.5, 5].</summary>+ internal static string FormatG(double value) => value.ToString("G6", CultureInfo.InvariantCulture);+}diff --git a/laya-dotnet/src/Laya/PythonJson.cs b/laya-dotnet/src/Laya/PythonJson.csindex a89633a2..354ce3c9 100644--- a/laya-dotnet/src/Laya/PythonJson.cs+++ b/laya-dotnet/src/Laya/PythonJson.cs@@ -183,6 +183,12 @@ public static class PythonJsonsb.Append(Repr((double)dec));break;+ // Python's bytes/bytearray are not JSON serializable (TypeError); a state that is bytes can be+ // routed (lang.py decodes it) but never predicted on. Under default=str Python writes str(b"..."),+ // which is not ported, so that path keeps the array rendering.+ case byte[] when !fallbackToStr:+ throw new JsonException("Object of type bytes is not JSON serializable");+// Dictionaries: more-specific IEnumerable<KVP> before general IDictionary before general IEnumerable.case IEnumerable<KeyValuePair<string, object?>> kvpEnum:WriteKvpSequence(sb, kvpEnum, escapeNonAscii, fallbackToStr, seen);diff --git a/laya-dotnet/src/Laya/PythonJsonReader.cs b/laya-dotnet/src/Laya/PythonJsonReader.csnew file mode 100644index 00000000..b03f4cce--- /dev/null+++ b/laya-dotnet/src/Laya/PythonJsonReader.cs@@ -0,0 +1,261 @@+using System.Globalization;+using System.Numerics;+using System.Text;+using System.Text.Json;++namespace Laya;++/// <summary>+/// <c>json.loads</c> as CPython's C scanner implements it, producing the values Python would hold:+/// <see cref="long"/> / <see cref="BigInteger"/> for an integer literal, <see cref="double"/> for a float+/// literal (a literal that overflows is infinity), <see cref="string"/>, <see cref="bool"/>,+/// <see langword="null"/>, <c>List<object?></c> and <c>OrderedDictionary<string, object?></c>.+/// </summary>+/// <remarks>+/// <see cref="JsonDocument"/> is stricter than Python in exactly one place that matters here: Python also+/// accepts the bare tokens <c>NaN</c>, <c>Infinity</c> and <c>-Infinity</c> in value position. They are+/// parsed to doubles directly, so no marker text exists that a real string could collide with. Everything+/// else follows <c>json.decoder</c>: whitespace is space, tab, LF and CR only; <c>-?(0|[1-9][0-9]*)</c> with+/// an optional fraction and exponent, so <c>01</c>, <c>+1</c>, <c>.5</c> and <c>1.</c> are errors; strings+/// reject control characters but keep lone surrogates; no trailing commas, comments or single quotes; a+/// duplicate key keeps its first position and its last value. Errors are <see cref="JsonException"/>.+/// </remarks>+internal static class PythonJsonReader+{+ // Python 3.11+ refuses to turn more digits than this into an int (sys.int_info.default_max_str_digits).+ private const int MaxIntDigits = 4300;+ private const int MaxDepth = 1000;++ /// <summary>+ /// Read a JSON file the way Python's <c>json.load(open(path))</c> does. The reference opens these files+ /// without an <c>encoding</c>, so it decodes with the locale's: UTF-8 on Linux and macOS (and with+ /// Python's UTF-8 mode), the ANSI code page on a default Windows install. This reads UTF-8, strictly: invalid bytes are an+ /// error (<c>UnicodeDecodeError</c>), and a leading BOM is kept as U+FEFF so <see cref="Parse"/> rejects it as+ /// <c>json.loads</c> does. The two decodings agree on every ASCII file.+ /// </summary>+ internal static string ReadAllText(string path)+ {+ try { return new UTF8Encoding(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true).GetString(File.ReadAllBytes(path)); }+ catch (DecoderFallbackException e) { throw new JsonException($"{path} is not valid UTF-8: {e.Message}", e); }+ }++ internal static object? Parse(string text)+ {+ if (text.StartsWith('\uFEFF')) throw new JsonException("Unexpected UTF-8 BOM (decode using utf-8-sig): char 0");+ var p = new State(text);+ p.SkipWhitespace();+ var value = p.Value(0);+ p.SkipWhitespace();+ if (p.Pos != text.Length) throw p.Error("Extra data");+ return value;+ }++ /// <summary>+ /// Parse, then re-emit as a <see cref="JsonDocument"/> for code that reads through that API. NaN and+ /// the infinities, which JSON text cannot hold, become <c>null</c>.+ /// </summary>+ internal static JsonDocument ParseDocument(string text)+ {+ var value = Parse(text);+ using var stream = new MemoryStream();+ using (var writer = new Utf8JsonWriter(stream))+ Write(writer, value);+ return JsonDocument.Parse(stream.ToArray());+ }++ private static void Write(Utf8JsonWriter w, object? value)+ {+ switch (value)+ {+ case null: w.WriteNullValue(); break;+ case bool b: w.WriteBooleanValue(b); break;+ case long l: w.WriteNumberValue(l); break;+ case BigInteger big: w.WriteRawValue(big.ToString(CultureInfo.InvariantCulture)); break;+ case double d when double.IsFinite(d): w.WriteNumberValue(d); break;+ case double: w.WriteNullValue(); break;+ case string s: w.WriteStringValue(ReplaceLoneSurrogates(s)); break;+ case List<object?> list:+ w.WriteStartArray();+ foreach (var item in list) Write(w, item);+ w.WriteEndArray();+ break;+ case IEnumerable<KeyValuePair<string, object?>> map:+ w.WriteStartObject();+ foreach (var kv in map) { w.WritePropertyName(ReplaceLoneSurrogates(kv.Key)); Write(w, kv.Value); }+ w.WriteEndObject();+ break;+ }+ }++ private static string ReplaceLoneSurrogates(string s)+ {+ var sb = new StringBuilder(s.Length);+ foreach (var rune in s.EnumerateRunes()) sb.Append(rune.ToString());+ return sb.ToString();+ }++ private sealed class State(string text)+ {+ internal int Pos;++ internal JsonException Error(string message) => new($"{message}: char {Pos}");++ internal void SkipWhitespace()+ {+ while (Pos < text.Length && text[Pos] is ' ' or '\t' or '\n' or '\r') Pos++;+ }++ internal object? Value(int depth)+ {+ if (depth > MaxDepth) throw Error("maximum recursion depth exceeded");+ if (Pos >= text.Length) throw Error("Expecting value");+ switch (text[Pos])+ {+ case '"': Pos++; return String();+ case '{': Pos++; return Object(depth + 1);+ case '[': Pos++; return Array(depth + 1);+ case 'n' when Literal("null"): return null;+ case 't' when Literal("true"): return true;+ case 'f' when Literal("false"): return false;+ case 'N' when Literal("NaN"): return double.NaN;+ case 'I' when Literal("Infinity"): return double.PositiveInfinity;+ case '-' when string.CompareOrdinal(text, Pos, "-Infinity", 0, 9) == 0: Pos += 9; return double.NegativeInfinity;+ default:+ var number = Number();+ return number ?? throw Error("Expecting value");+ }+ }++ private bool Literal(string word)+ {+ if (string.CompareOrdinal(text, Pos, word, 0, word.Length) != 0) return false;+ Pos += word.Length;+ return true;+ }++ private static bool Digit(char c) => c is >= '0' and <= '9';++ // _match_number_unicode: -?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][-+]?[0-9]+)?+ private object? Number()+ {+ var start = Pos;+ var i = Pos;+ if (i < text.Length && text[i] == '-') i++;+ if (i >= text.Length || !Digit(text[i])) return null;+ if (text[i] == '0') i++;+ else while (i < text.Length && Digit(text[i])) i++;+ var integerEnd = i;+ var isFloat = false;+ if (i + 1 < text.Length && text[i] == '.' && Digit(text[i + 1]))+ {+ i += 2;+ while (i < text.Length && Digit(text[i])) i++;+ isFloat = true;+ }+ if (i < text.Length && text[i] is 'e' or 'E')+ {+ var j = i + 1;+ if (j < text.Length && text[j] is '+' or '-') j++;+ if (j < text.Length && Digit(text[j]))+ {+ while (j < text.Length && Digit(text[j])) j++;+ i = j;+ isFloat = true;+ }+ }+ var literal = text[start..i];+ Pos = i;+ if (isFloat)+ return double.Parse(literal, NumberStyles.Float, CultureInfo.InvariantCulture); // correctly rounded; overflow is infinity+ var digits = integerEnd - start - (text[start] == '-' ? 1 : 0);+ if (digits > MaxIntDigits)+ throw new JsonException($"Exceeds the limit ({MaxIntDigits} digits) for integer string conversion: value has {digits} digits");+ return long.TryParse(literal, NumberStyles.AllowLeadingSign, CultureInfo.InvariantCulture, out var l)+ ? (object)l // not a bare `l`: the conditional would widen it to BigInteger+ : BigInteger.Parse(literal, CultureInfo.InvariantCulture);+ }++ // py_scanstring with strict=True: control characters below U+0020 are errors, lone surrogates stay.+ private string String()+ {+ var sb = new StringBuilder();+ while (true)+ {+ if (Pos >= text.Length) throw Error("Unterminated string starting at");+ var c = text[Pos++];+ if (c == '"') return sb.ToString();+ if (c < ' ') throw Error("Invalid control character at");+ if (c != '\\') { sb.Append(c); continue; }+ if (Pos >= text.Length) throw Error("Unterminated string starting at");+ var e = text[Pos++];+ switch (e)+ {+ case '"': sb.Append('"'); break;+ case '\\': sb.Append('\\'); break;+ case '/': sb.Append('/'); break;+ case 'b': sb.Append('\b'); break;+ case 'f': sb.Append('\f'); break;+ case 'n': sb.Append('\n'); break;+ case 'r': sb.Append('\r'); break;+ case 't': sb.Append('\t'); break;+ case 'u': sb.Append(Hex4()); break;+ default: throw Error("Invalid \\escape");+ }+ }+ }++ private char Hex4()+ {+ if (Pos + 4 > text.Length) throw Error("Invalid \\uXXXX escape");+ var value = 0;+ for (var k = 0; k < 4; k++)+ {+ var h = text[Pos + k];+ var digit = h is >= '0' and <= '9' ? h - '0' : h is >= 'a' and <= 'f' ? h - 'a' + 10 : h is >= 'A' and <= 'F' ? h - 'A' + 10 : -1;+ if (digit < 0) throw Error("Invalid \\uXXXX escape");+ value = value * 16 + digit;+ }+ Pos += 4;+ return (char)value; // a high surrogate followed by \uDC00.. joins in the UTF-16 string, as Python joins the pair+ }++ private object Object(int depth)+ {+ var map = new OrderedDictionary<string, object?>(StringComparer.Ordinal);+ SkipWhitespace();+ if (Pos < text.Length && text[Pos] == '}') { Pos++; return map; }+ while (true)+ {+ if (Pos >= text.Length || text[Pos] != '"') throw Error("Expecting property name enclosed in double quotes");+ Pos++;+ var key = String();+ SkipWhitespace();+ if (Pos >= text.Length || text[Pos] != ':') throw Error("Expecting ':' delimiter");+ Pos++;+ SkipWhitespace();+ map[key] = Value(depth);+ SkipWhitespace();+ if (Pos < text.Length && text[Pos] == '}') { Pos++; return map; }+ if (Pos >= text.Length || text[Pos] != ',') throw Error("Expecting ',' delimiter");+ Pos++;+ SkipWhitespace();+ }+ }++ private object Array(int depth)+ {+ var list = new List<object?>();+ SkipWhitespace();+ if (Pos < text.Length && text[Pos] == ']') { Pos++; return list; }+ while (true)+ {+ list.Add(Value(depth));+ SkipWhitespace();+ if (Pos < text.Length && text[Pos] == ']') { Pos++; return list; }+ if (Pos >= text.Length || text[Pos] != ',') throw Error("Expecting ',' delimiter");+ Pos++;+ SkipWhitespace();+ }+ }+ }+}diff --git a/laya-dotnet/src/Laya/PythonStr.cs b/laya-dotnet/src/Laya/PythonStr.csnew file mode 100644index 00000000..a0ce1fae--- /dev/null+++ b/laya-dotnet/src/Laya/PythonStr.cs@@ -0,0 +1,191 @@+using System.Globalization;+using System.Text;++namespace Laya;++/// <summary>+/// Python <c>str</c> predicates that .NET spells differently, so the ports of <c>lang.py</c>,+/// <c>router.py</c> and <c>email.py</c> read the same characters as the reference does.+/// </summary>+/// <remarks>+/// The range tables below were generated from CPython 3.12 (Unicode 15.0.0) as+/// <c>unicodedata.combining(ch) != 0</c>, <c>ch.isupper()</c> / <c>ch.islower()</c> for the+/// characters outside <c>Lu</c> / <c>Ll</c>, and <c>ch.isspace()</c>.+/// </remarks>+internal static class PythonStr+{+ private static readonly (int Lo, int Hi)[] CombiningRanges =+ [+ (0x300, 0x34E), (0x350, 0x36F), (0x483, 0x487), (0x591, 0x5BD), (0x5BF, 0x5BF), (0x5C1, 0x5C2),+ (0x5C4, 0x5C5), (0x5C7, 0x5C7), (0x610, 0x61A), (0x64B, 0x65F), (0x670, 0x670), (0x6D6, 0x6DC),+ (0x6DF, 0x6E4), (0x6E7, 0x6E8), (0x6EA, 0x6ED), (0x711, 0x711), (0x730, 0x74A), (0x7EB, 0x7F3),+ (0x7FD, 0x7FD), (0x816, 0x819), (0x81B, 0x823), (0x825, 0x827), (0x829, 0x82D), (0x859, 0x85B),+ (0x898, 0x89F), (0x8CA, 0x8E1), (0x8E3, 0x8FF), (0x93C, 0x93C), (0x94D, 0x94D), (0x951, 0x954),+ (0x9BC, 0x9BC), (0x9CD, 0x9CD), (0x9FE, 0x9FE), (0xA3C, 0xA3C), (0xA4D, 0xA4D), (0xABC, 0xABC),+ (0xACD, 0xACD), (0xB3C, 0xB3C), (0xB4D, 0xB4D), (0xBCD, 0xBCD), (0xC3C, 0xC3C), (0xC4D, 0xC4D),+ (0xC55, 0xC56), (0xCBC, 0xCBC), (0xCCD, 0xCCD), (0xD3B, 0xD3C), (0xD4D, 0xD4D), (0xDCA, 0xDCA),+ (0xE38, 0xE3A), (0xE48, 0xE4B), (0xEB8, 0xEBA), (0xEC8, 0xECB), (0xF18, 0xF19), (0xF35, 0xF35),+ (0xF37, 0xF37), (0xF39, 0xF39), (0xF71, 0xF72), (0xF74, 0xF74), (0xF7A, 0xF7D), (0xF80, 0xF80),+ (0xF82, 0xF84), (0xF86, 0xF87), (0xFC6, 0xFC6), (0x1037, 0x1037), (0x1039, 0x103A), (0x108D, 0x108D),+ (0x135D, 0x135F), (0x1714, 0x1715), (0x1734, 0x1734), (0x17D2, 0x17D2), (0x17DD, 0x17DD), (0x18A9, 0x18A9),+ (0x1939, 0x193B), (0x1A17, 0x1A18), (0x1A60, 0x1A60), (0x1A75, 0x1A7C), (0x1A7F, 0x1A7F), (0x1AB0, 0x1ABD),+ (0x1ABF, 0x1ACE), (0x1B34, 0x1B34), (0x1B44, 0x1B44), (0x1B6B, 0x1B73), (0x1BAA, 0x1BAB), (0x1BE6, 0x1BE6),+ (0x1BF2, 0x1BF3), (0x1C37, 0x1C37), (0x1CD0, 0x1CD2), (0x1CD4, 0x1CE0), (0x1CE2, 0x1CE8), (0x1CED, 0x1CED),+ (0x1CF4, 0x1CF4), (0x1CF8, 0x1CF9), (0x1DC0, 0x1DFF), (0x20D0, 0x20DC), (0x20E1, 0x20E1), (0x20E5, 0x20F0),+ (0x2CEF, 0x2CF1), (0x2D7F, 0x2D7F), (0x2DE0, 0x2DFF), (0x302A, 0x302F), (0x3099, 0x309A), (0xA66F, 0xA66F),+ (0xA674, 0xA67D), (0xA69E, 0xA69F), (0xA6F0, 0xA6F1), (0xA806, 0xA806), (0xA82C, 0xA82C), (0xA8C4, 0xA8C4),+ (0xA8E0, 0xA8F1), (0xA92B, 0xA92D), (0xA953, 0xA953), (0xA9B3, 0xA9B3), (0xA9C0, 0xA9C0), (0xAAB0, 0xAAB0),+ (0xAAB2, 0xAAB4), (0xAAB7, 0xAAB8), (0xAABE, 0xAABF), (0xAAC1, 0xAAC1), (0xAAF6, 0xAAF6), (0xABED, 0xABED),+ (0xFB1E, 0xFB1E), (0xFE20, 0xFE2F), (0x101FD, 0x101FD), (0x102E0, 0x102E0), (0x10376, 0x1037A), (0x10A0D, 0x10A0D),+ (0x10A0F, 0x10A0F), (0x10A38, 0x10A3A), (0x10A3F, 0x10A3F), (0x10AE5, 0x10AE6), (0x10D24, 0x10D27), (0x10EAB, 0x10EAC),+ (0x10EFD, 0x10EFF), (0x10F46, 0x10F50), (0x10F82, 0x10F85), (0x11046, 0x11046), (0x11070, 0x11070), (0x1107F, 0x1107F),+ (0x110B9, 0x110BA), (0x11100, 0x11102), (0x11133, 0x11134), (0x11173, 0x11173), (0x111C0, 0x111C0), (0x111CA, 0x111CA),+ (0x11235, 0x11236), (0x112E9, 0x112EA), (0x1133B, 0x1133C), (0x1134D, 0x1134D), (0x11366, 0x1136C), (0x11370, 0x11374),+ (0x11442, 0x11442), (0x11446, 0x11446), (0x1145E, 0x1145E), (0x114C2, 0x114C3), (0x115BF, 0x115C0), (0x1163F, 0x1163F),+ (0x116B6, 0x116B7), (0x1172B, 0x1172B), (0x11839, 0x1183A), (0x1193D, 0x1193E), (0x11943, 0x11943), (0x119E0, 0x119E0),+ (0x11A34, 0x11A34), (0x11A47, 0x11A47), (0x11A99, 0x11A99), (0x11C3F, 0x11C3F), (0x11D42, 0x11D42), (0x11D44, 0x11D45),+ (0x11D97, 0x11D97), (0x11F41, 0x11F42), (0x16AF0, 0x16AF4), (0x16B30, 0x16B36), (0x16FF0, 0x16FF1), (0x1BC9E, 0x1BC9E),+ (0x1D165, 0x1D169), (0x1D16D, 0x1D172), (0x1D17B, 0x1D182), (0x1D185, 0x1D18B), (0x1D1AA, 0x1D1AD), (0x1D242, 0x1D244),+ (0x1E000, 0x1E006), (0x1E008, 0x1E018), (0x1E01B, 0x1E021), (0x1E023, 0x1E024), (0x1E026, 0x1E02A), (0x1E08F, 0x1E08F),+ (0x1E130, 0x1E136), (0x1E2AE, 0x1E2AE), (0x1E2EC, 0x1E2EF), (0x1E4EC, 0x1E4EF), (0x1E8D0, 0x1E8D6), (0x1E944, 0x1E94A),+ ];++ private static readonly (int Lo, int Hi)[] OtherUpperRanges =+ [+ (0x2160, 0x216F), (0x24B6, 0x24CF), (0x1F130, 0x1F149), (0x1F150, 0x1F169), (0x1F170, 0x1F189),+ ];++ private static readonly (int Lo, int Hi)[] OtherLowerRanges =+ [+ (0xAA, 0xAA), (0xBA, 0xBA), (0x2B0, 0x2B8), (0x2C0, 0x2C1), (0x2E0, 0x2E4), (0x345, 0x345),+ (0x37A, 0x37A), (0x10FC, 0x10FC), (0x1D2C, 0x1D6A), (0x1D78, 0x1D78), (0x1D9B, 0x1DBF), (0x2071, 0x2071),+ (0x207F, 0x207F), (0x2090, 0x209C), (0x2170, 0x217F), (0x24D0, 0x24E9), (0x2C7C, 0x2C7D), (0xA69C, 0xA69D),+ (0xA770, 0xA770), (0xA7F2, 0xA7F4), (0xA7F8, 0xA7F9), (0xAB5C, 0xAB5F), (0xAB69, 0xAB69), (0x10780, 0x10780),+ (0x10783, 0x10785), (0x10787, 0x107B0), (0x107B2, 0x107BA), (0x1E030, 0x1E06D),+ ];++ private static readonly (int Lo, int Hi)[] SpaceRanges =+ [+ (0x9, 0xD), (0x1C, 0x20), (0x85, 0x85), (0xA0, 0xA0), (0x1680, 0x1680), (0x2000, 0x200A),+ (0x2028, 0x2029), (0x202F, 0x202F), (0x205F, 0x205F), (0x3000, 0x3000),+ ];+++ private static bool InRanges(int cp, (int Lo, int Hi)[] ranges)+ {+ foreach (var (lo, hi) in ranges)+ if (cp >= lo && cp <= hi) return true;+ return false;+ }++ /// <summary><c>ch.isspace()</c>: Unicode White_Space plus U+001C..U+001F, which <c>\s</c> accepts too.</summary>+ internal static bool IsSpace(int cp) => InRanges(cp, SpaceRanges);++ /// <summary><c>unicodedata.combining(ch) != 0</c>: a nonzero canonical combining class.</summary>+ internal static bool IsCombining(int cp) => cp >= 0x300 && InRanges(cp, CombiningRanges);++ /// <summary><c>ch.isupper()</c> for one character.</summary>+ internal static bool IsUpper(Rune r) =>+ Rune.GetUnicodeCategory(r) == UnicodeCategory.UppercaseLetter || InRanges(r.Value, OtherUpperRanges);++ /// <summary><c>ch.islower()</c> for one character.</summary>+ internal static bool IsLower(Rune r) =>+ Rune.GetUnicodeCategory(r) == UnicodeCategory.LowercaseLetter || InRanges(r.Value, OtherLowerRanges);++ private static bool IsTitle(Rune r) => Rune.GetUnicodeCategory(r) == UnicodeCategory.TitlecaseLetter;++ /// <summary><c>s.isupper()</c>: at least one uppercase character and no lowercase or titlecase one.</summary>+ internal static bool StrIsUpper(string s)+ {+ var cased = false;+ foreach (var r in s.EnumerateRunes())+ {+ if (IsLower(r) || IsTitle(r)) return false;+ if (IsUpper(r)) cased = true;+ }+ return cased;+ }++ /// <summary><c>s.strip()</c>.</summary>+ internal static string Strip(string s)+ {+ var start = 0;+ var end = s.Length;+ while (start < end && IsSpace(s[start])) start++;+ while (end > start && IsSpace(s[end - 1])) end--;+ return s[start..end];+ }++ /// <summary><c>s.rstrip()</c>.</summary>+ internal static string RStrip(string s)+ {+ var end = s.Length;+ while (end > 0 && IsSpace(s[end - 1])) end--;+ return s[..end];+ }++ /// <summary><c>s.lstrip()</c>.</summary>+ internal static string LStrip(string s)+ {+ var start = 0;+ while (start < s.Length && IsSpace(s[start])) start++;+ return s[start..];+ }++ /// <summary><c>s.split()</c> with no argument: runs of whitespace separate, empty pieces dropped.</summary>+ internal static List<string> Split(string s)+ {+ var parts = new List<string>();+ var start = -1;+ for (var i = 0; i < s.Length; i++)+ {+ if (IsSpace(s[i]))+ {+ if (start >= 0) { parts.Add(s[start..i]); start = -1; }+ }+ else if (start < 0) start = i;+ }+ if (start >= 0) parts.Add(s[start..]);+ return parts;+ }++ /// <summary><c>ch.isprintable()</c>: not Other (Cc, Cf, Cs, Co, Cn) or Separator (Zl, Zp, Zs) except the ASCII space.</summary>+ internal static bool IsPrintable(Rune r)+ {+ if (r.Value == ' ') return true;+ return Rune.GetUnicodeCategory(r) is not (UnicodeCategory.Control or UnicodeCategory.Format+ or UnicodeCategory.Surrogate or UnicodeCategory.PrivateUse or UnicodeCategory.OtherNotAssigned+ or UnicodeCategory.LineSeparator or UnicodeCategory.ParagraphSeparator or UnicodeCategory.SpaceSeparator);+ }++ /// <summary>Python's <c>repr()</c> of a <see cref="string"/> (and so <c>%r</c>).</summary>+ internal static string Repr(string s)+ {+ var quote = s.Contains('\'') && !s.Contains('"') ? '"' : '\'';+ var sb = new StringBuilder(s.Length + 2);+ sb.Append(quote);+ for (var i = 0; i < s.Length; i++)+ {+ int cp;+ if (char.IsHighSurrogate(s[i]) && i + 1 < s.Length && char.IsLowSurrogate(s[i + 1]))+ {+ cp = char.ConvertToUtf32(s[i], s[i + 1]);+ i++;+ }+ else cp = s[i]; // a lone surrogate is not printable, as in Python+ if (cp == quote || cp == '\\') { sb.Append('\\').Append((char)cp); continue; }+ switch (cp)+ {+ case '\n': sb.Append("\\n"); continue;+ case '\r': sb.Append("\\r"); continue;+ case '\t': sb.Append("\\t"); continue;+ }+ if (cp is >= 0xD800 and <= 0xDFFF) sb.Append("\\u").Append(cp.ToString("x4"));+ else if (!IsPrintable(new Rune(cp)))+ sb.Append(cp < 0x100 ? "\\x" + cp.ToString("x2") : cp < 0x10000 ? "\\u" + cp.ToString("x4") : "\\U" + cp.ToString("x8"));+ else sb.Append(char.ConvertFromUtf32(cp));+ }+ sb.Append(quote);+ return sb.ToString();+ }+}diff --git a/laya-dotnet/src/Laya/Tokenization/HfTokenizer.cs b/laya-dotnet/src/Laya/Tokenization/HfTokenizer.csindex aef0e31c..54ed5244 100644--- a/laya-dotnet/src/Laya/Tokenization/HfTokenizer.cs+++ b/laya-dotnet/src/Laya/Tokenization/HfTokenizer.cs@@ -327,8 +327,8 @@ public sealed class HfTokenizer : ILayaTokenizer, IDisposable// 2a. Fused layout: tokenizer_config.json says "cls_token": "[CLS]" (for ModernBERT)// or "cls_token": "<bos>" (for mmBERT). HuggingFace sometimes stores these as// plain strings and sometimes as {"content": "...", ...} dicts.- using var f = File.OpenRead(configPath);- using var cfg = JsonDocument.Parse(f);+ // Python reads this file with json.load, which accepts NaN / Infinity / -Infinity tokens.+ using var cfg = PythonJsonReader.ParseDocument(PythonJsonReader.ReadAllText(configPath));var r = cfg.RootElement;clsStr = RequireTokenString(r, "cls_token", configPath);sepStr = RequireTokenString(r, "sep_token", configPath);diff --git a/laya-dotnet/src/Laya/WarnOnce.cs b/laya-dotnet/src/Laya/WarnOnce.csnew file mode 100644index 00000000..9a16c2f0--- /dev/null+++ b/laya-dotnet/src/Laya/WarnOnce.cs@@ -0,0 +1,62 @@+using System.Collections.Concurrent;+using System.Diagnostics;+using System.Runtime.CompilerServices;++namespace Laya;++/// <summary>+/// Marks a public entry whose <em>caller</em> is the call site of any warning raised below it: Python's+/// <c>warnings.warn(..., stacklevel=2 or 3)</c> attributes a warning to the code that called the public+/// method, not to the library line that raised it. Such a method must also be+/// <c>[MethodImpl(MethodImplOptions.NoInlining)]</c> so its frame exists in a Release build.+/// </summary>+[AttributeUsage(AttributeTargets.Method)]+internal sealed class WarnSiteAttribute : Attribute;++/// <summary>+/// Python's default warnings filter shows an identical warning once per (text, category, module, line) of+/// the frame <c>stacklevel</c> selects. The category is always <c>RuntimeWarning</c> here; the frame is found by+/// walking the stack to the nearest <see cref="WarnSiteAttribute"/> entry and taking its caller, keyed by+/// that caller's method and IL offset, which tells call sites apart exactly without a PDB. Reached through+/// <see cref="LayaRouter"/> the caller is the router's own call, as Python attributes it to <c>router.py</c>.+/// </summary>+internal static class WarnOnce+{+ private static readonly ConcurrentDictionary<(string Site, string Text), byte> Seen = new();++ /// <summary>Trace <paramref name="text"/> as a warning unless this call site already did. Thread-safe.</summary>+ [MethodImpl(MethodImplOptions.NoInlining)]+ internal static void Warn(string text)+ {+ if (Seen.TryAdd((Site(), text), 0))+ Trace.TraceWarning("{0}", text);+ }++ /// <summary>Forget every warning shown so far (tests only).</summary>+ internal static void Reset() => Seen.Clear();++ private static bool IsEntry(StackFrame frame) =>+ frame.GetMethod() is { } method && method.IsDefined(typeof(WarnSiteAttribute), false);++ // Library plumbing between a public entry and the warning; with no entry on the stack (a config parsed+ // or a window budgeted directly) the first frame outside it is the site.+ private static bool IsPlumbing(StackFrame frame) =>+ frame.GetMethod()?.DeclaringType is { } type+ && (type == typeof(WarnOnce) || type == typeof(LayaConfig) || type == typeof(LongPrediction));++ private static string Site()+ {+ var frames = new StackTrace(false).GetFrames();+ var i = 0;+ while (i < frames.Length && !IsEntry(frames[i])) i++;+ if (i < frames.Length)+ while (i < frames.Length && IsEntry(frames[i])) i++; // overloads that call each other are one entry+ else+ {+ i = 0;+ while (i < frames.Length && IsPlumbing(frames[i])) i++;+ }+ if (i >= frames.Length || frames[i].GetMethod() is not { } method) return "";+ return $"{method.Module.ModuleVersionId}:{method.MetadataToken}:{frames[i].GetILOffset()}";+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/LangGuessTests.cs b/laya-dotnet/tests/Laya.Tests/LangGuessTests.csnew file mode 100644index 00000000..ab933ed3--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/LangGuessTests.cs@@ -0,0 +1,135 @@+using System.Text;+using System.Text.Json;++namespace Laya.Tests;++/// <summary>+/// The parts of the Python parity port the golden probes cannot carry: a <c>lang_guess</c> callback,+/// byte-string states, and <c>max_chars</c> beside <c>clean=False</c>. Expected values were produced by+/// running the Python reference (<c>Router._route</c>, <c>lang.state_text</c>, <c>lang.analyse</c>,+/// <c>email_state</c>) on the same inputs.+/// </summary>+public sealed class LangGuessTests+{+ private sealed class NoopPredictor : ILayaPredictor+ {+ public LayaResult Predict(object? state, QuestionSet questions) =>+ new("noop", [], new Dictionary<string, Answer>(StringComparer.Ordinal), new Usage(0, 0));+ }++ private static LayaRouter NewRouter(LangGuess? installed = null) =>+ new(new LayaRouterOptions { LangGuess = installed }, _ => new NoopPredictor());++ private const string NonEnglish = "lang_guess: the caller identified this as non-English text";+ private const string Undecided = "Latin script, language not identified and no non-English letters; using default (multilingual)";++ [Fact]+ public void CallbackAnswersWithACode()+ {+ using var router = NewRouter();+ var decision = router.Route("hello", null, null, null, null, LangGuess.FromCallback(_ => "pt"));+ Assert.Equal(LayaCheckpoint.Multilingual, decision.Model);+ Assert.Equal(NonEnglish, decision.Reason);+ Assert.Null(decision.Detection);+ }++ [Theory]+ [InlineData(null)]+ [InlineData("C")]+ public void CallbackThatAbstainsFallsThroughToDetection(string? code)+ {+ using var router = NewRouter();+ var decision = router.Route("hello", null, null, null, null, LangGuess.FromCallback(_ => code));+ Assert.Equal(Undecided, decision.Reason);+ Assert.NotNull(decision.Detection);+ }++ [Fact]+ public void CallbackReceivesTheState()+ {+ object? seen = null;+ using var router = NewRouter();+ var state = new Dictionary<string, object?> { ["text"] = "hola" };+ router.Route(state, null, null, null, null, LangGuess.FromCallback(s => { seen = s; return null; }));+ Assert.Same(state, seen);+ }++ [Fact]+ public void RouterInstalledHintAppliesToEveryRequestAndPerCallOverridesIt()+ {+ using var router = NewRouter(LangGuess.FromCallback(s => s is string t && t.Contains("hola") ? "es" : "en"));+ Assert.Equal("Router(lang_guess=...): the caller identified this as non-English text", router.Route("hola amigo").Reason);+ Assert.Equal("Router(lang_guess=...): the caller identified this as English text",+ router.Route("Care este ora in Tokyo?").Reason);+ Assert.Equal(NonEnglish, router.Route("hola amigo", null, null, null, null, "pt").Reason);+ }++ [Fact]+ public void ExplicitLangBeatsAHintButABlankLangDoesNot()+ {+ using var router = NewRouter();+ Assert.Equal("explicit lang='en'", router.Route("hola", null, null, null, "en", "pt").Reason);+ Assert.Equal(NonEnglish, router.Route("hola", null, null, null, " ", "pt").Reason);+ }++ [Fact]+ public void ReasonQuotesCallerTextLikePythonRepr() =>+ Assert.Equal("explicit lang='x\\u200by'", NewRouter().Route("x", lang: "xy").Reason);++ [Fact]+ public void PredictAcceptsAPerCallHint()+ {+ using var router = NewRouter();+ var result = router.Predict("hello", new QuestionSet(), null, null, null, LangGuess.FromCode("fr"));+ Assert.Equal(NonEnglish, result.Routing!.Reason);+ }++ // ── bytes states (lang.py:253-255) ────────────────────────────────────────++ [Fact]+ public void ValidUtf8BytesAreReadAsText()+ {+ Assert.Equal("hola mundo", LanguageDetection.StateText(Encoding.UTF8.GetBytes("hola mundo")));+ var analysis = LanguageDetection.Analyse(Encoding.UTF8.GetBytes("Preciso de ajuda com a minha fatura, nao consigo pagar hoje"));+ Assert.Equal("latin", analysis.Script);+ Assert.Equal("pt", analysis.Language);+ Assert.False(analysis.IsEnglish);+ }++ [Fact]+ public void InvalidUtf8BytesAreNoText()+ {+ Assert.Equal("", LanguageDetection.StateText(new byte[] { 0x63, 0x61, 0x66, 0xC3, 0xA9, 0x20, 0xFF, 0x20, 0x6F, 0x6B }));+ Assert.Equal("unknown", LanguageDetection.Analyse(new byte[] { 0xFF, 0xFE }).Script);+ }++ [Fact]+ public void BytesNestedInAStateAreReadToo() =>+ Assert.Equal("ab c", LanguageDetection.StateText(new Dictionary<string, object?> { ["a"] = "ab"u8.ToArray(), ["b"] = "c" }));++ [Fact]+ public void BytesAreNotJsonSerializableSoPredictionRejectsThem() =>+ Assert.Throws<JsonException>(() => PythonJson.State(new byte[] { 1, 2 }));++ // ── email_state(max_chars=...) ────────────────────────────────────────────++ [Fact]+ public void MaxCharsIsIgnoredWhenNotCleaningAndIsNeverAField()+ {+ var state = LayaEmail.State(" Locked out ", "My account is locked and I need help right now.", null, false, null, 5);+ Assert.Equal("My account is locked and I need help right now.", state["body"]);+ Assert.Equal(["subject", "body"], state.Keys);+ Assert.Equal("Locked out", state["subject"]);+ }++ [Theory]+ [InlineData(10, "My account")]+ [InlineData(0, "")]+ [InlineData(-5, "My account is locked ")]+ public void MaxCharsBudgetCutsTheCleanedBody(int maxChars, string body)+ {+ // clean_email_body("My account is locked and I need help right now.", n): 0 -> "", -5 -> 5*5 characters off the end.+ var state = LayaEmail.State("s", "My account is locked and I need help right now.", null, true, null, maxChars);+ Assert.Equal(body, state["body"]);+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/LayaEmailTests.cs b/laya-dotnet/tests/Laya.Tests/LayaEmailTests.csindex 263f5040..3711dab8 100644--- a/laya-dotnet/tests/Laya.Tests/LayaEmailTests.cs+++ b/laya-dotnet/tests/Laya.Tests/LayaEmailTests.cs@@ -168,10 +168,16 @@ public sealed class LayaEmailTestsvar body = e.GetProperty("body").GetString();var sender = e.GetProperty("sender").ValueKind == JsonValueKind.Null ? null : e.GetProperty("sender").GetString();var clean = e.GetProperty("clean").GetBoolean();- var extra = e.GetProperty("extra").EnumerateObject()- .Select(p => new KeyValuePair<string, object?>(p.Name, RoutingGoldenData.ToClr(p.Value)));-- var got = LayaEmail.State(subject, body, sender, clean, extra);+ // max_chars is a named parameter of Python's email_state, so it is consumed, not an extra field.+ var maxChars = 3000;+ var extra = new List<KeyValuePair<string, object?>>();+ foreach (var p in e.GetProperty("extra").EnumerateObject())+ {+ if (p.Name == "max_chars") maxChars = p.Value.GetInt32();+ else extra.Add(new KeyValuePair<string, object?>(p.Name, RoutingGoldenData.ToClr(p.Value)));+ }++ var got = LayaEmail.State(subject, body, sender, clean, extra, maxChars);var want = e.GetProperty("result");Assert.Equal(want.EnumerateObject().Select(p => p.Name), got.Keys);diff --git a/laya-dotnet/tests/Laya.Tests/LayaRouterTests.cs b/laya-dotnet/tests/Laya.Tests/LayaRouterTests.csindex 086b9dc4..89a17f2f 100644--- a/laya-dotnet/tests/Laya.Tests/LayaRouterTests.cs+++ b/laya-dotnet/tests/Laya.Tests/LayaRouterTests.cs@@ -454,8 +454,11 @@ public sealed class LayaRouterTestsvar model = OptString(routeKwargs, "model");var task = OptString(routeKwargs, "task");var lang = OptString(routeKwargs, "lang");+ var guess = OptString(routeKwargs, "lang_guess");+ var langGuess = guess is null ? null : LangGuess.FromCode(guess);- var options = new LayaRouterOptions();+ var installed = OptString(routerKwargs, "lang_guess");+ var options = new LayaRouterOptions { LangGuess = installed is null ? null : LangGuess.FromCode(installed) };if (routerKwargs.TryGetProperty("auto_task_detection", out var atd)) options.AutoTaskDetection = atd.GetBoolean();if (routerKwargs.TryGetProperty("default", out var def)) options.Default = LayaRouter.NormaliseName(def.GetString()!);// "models" and "standalone_repos" affect only the repo string, which RouteDecision does@@ -463,7 +466,7 @@ public sealed class LayaRouterTests// to the model selection under test here and are intentionally not applied.using var router = NewFakeRouter(options);- var decision = router.Route(state, questions, model, task, lang);+ var decision = router.Route(state, questions, model, task, lang, langGuess);var want = e.GetProperty("decision");Assert.Equal(want.GetProperty("model").GetString(), CheckpointName(decision.Model));diff --git a/laya-dotnet/tests/Laya.Tests/PythonJsonReaderTests.cs b/laya-dotnet/tests/Laya.Tests/PythonJsonReaderTests.csnew file mode 100644index 00000000..70be4800--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/PythonJsonReaderTests.cs@@ -0,0 +1,635 @@+using System.Diagnostics;+using System.Numerics;+using System.Text.Json;++namespace Laya.Tests;++/// <summary>+/// <see cref="PythonJsonReader"/> against CPython 3.12 <c>json.loads</c>, the warning repr of each JSON literal,+/// float parsing bit for bit, and the once-per-process warning filter. Every expectation below was produced by+/// running the Python reference (<c>json.loads</c>, <c>float</c>, <c>laya.common.clamp_temperature</c>,+/// <c>window_budget</c> under <c>warnings.simplefilter("always")</c>) on the same input; regenerate with the+/// scratch generator rather than editing by hand.+/// </summary>+public sealed class PythonJsonReaderTests+{+ public static TheoryData<string, string?, bool> Accepts() => new()+ {+ { "nan", null, false },+ { "NaN", "nan", true },+ { "+Infinity", null, false },+ { "Infinity", "inf", true },+ { "-Infinity", "-inf", true },+ { "-NaN", null, false },+ { "Infinityx", null, false },+ { "-Infinityx", null, false },+ { "01", null, false },+ { "-", null, false },+ { "-0", "0", true },+ { "-0.0", "-0.0", true },+ { "1.", null, false },+ { ".5", null, false },+ { "1e5", "100000.0", true },+ { "1E+2", "100.0", true },+ { "1e", null, false },+ { "1e+", null, false },+ { "0e0", "0.0", true },+ { "-01", null, false },+ { "[1,]", null, false },+ { "[,1]", null, false },+ { "[1 2]", null, false },+ { "[]", "[]", true },+ { "[ ]", "[]", true },+ { "{}", "{}", true },+ { "{\"a\":1,}", null, false },+ { "{\"a\" 1}", null, false },+ { "{'a': 1}", null, false },+ { "['a']", null, false },+ { "{NaN: 1}", null, false },+ { "{1: 1}", null, false },+ { "{\"a\": NaN}", "{'a': nan}", true },+ { "[NaN, Infinity, -Infinity]", "[nan, inf, -inf]", true },+ { "{\"a\": -Infinity, \"b\": [Infinity]}", "{'a': -inf, 'b': [inf]}", true },+ { "{\"a\": nan}", null, false },+ { "[nul]", null, false },+ { "[tru]", null, false },+ { "[True]", null, false },+ { "[null, true, false]", "[None, True, False]", true },+ { "\"abc", null, false },+ { "\"ab\\qc\"", null, false },+ { "\"\\ud83d\\ude00\"", "'\ud83d\ude00'", true },+ { "\"\\ud800\"", "'\\ud800'", true },+ { "\"\\u12\"", null, false },+ { "\"a\u0009b\"", null, false },+ { "\"a\\tb\"", "'a\\tb'", true },+ { "\"\u0001\"", null, false },+ { "/*c*/1", null, false },+ { "// c\u000a1", null, false },+ { "1 // c", null, false },+ { " \u0009\u000a\u000d[1]\u000d\u000a", "[1]", true },+ { "\u00a0[1]", null, false },+ { "[1]\u00a0", null, false },+ { "\ufeff[1]", null, false },+ { "", null, false },+ { " ", null, false },+ { "[1] x", null, false },+ { "[1][2]", null, false },+ { "{\"a\":1,\"a\":2}", "{'a': 2}", true },+ { "{\"a\":1,\"b\":2,\"a\":3}", "{'a': 3, 'b': 2}", true },+ { "123456789012345678901234567890", "123456789012345678901234567890", true },+ { "-123456789012345678901234567890", "-123456789012345678901234567890", true },+ { "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111", null, true },+ { "11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111", null, false },+ { "9223372036854775807", "9223372036854775807", true },+ { "9223372036854775808", "9223372036854775808", true },+ { "-9223372036854775808", "-9223372036854775808", true },+ { "-9223372036854775809", "-9223372036854775809", true },+ { "1.0", "1.0", true },+ { "7.0", "7.0", true },+ { "0.1", "0.1", true },+ { "1e400", "inf", true },+ { "-1e400", "-inf", true },+ { "1e-400", "0.0", true },+ { "[[[[1]]]]", "[[[[1]]]]", true },+ { "{\"a\": {\"b\": [1, {\"c\": null}]}}", "{'a': {'b': [1, {'c': None}]}}", true },+ { "\"Infinity\"", "'Infinity'", true },+ { "\"NaN\"", "'NaN'", true },+ { "[\"\\ud83d\"]", "['\\ud83d']", true },+ { "{\"\\u00e9\": \"\\u00e9\"}", "{'\u00e9': '\u00e9'}", true },+ { "2.5e", null, false },+ { "2.5E-3", "0.0025", true },+ { "\u0663", null, false },+ { "[\u0661]", null, false },+ { "\uff11", null, false },+ };++ [Theory]+ [MemberData(nameof(Accepts))]+ public void AcceptsAndRejectsExactlyWhenPythonDoes(string text, string? repr, bool accepted)+ {+ if (!accepted)+ {+ Assert.ThrowsAny<JsonException>(() => PythonJsonReader.Parse(text));+ return;+ }+ var value = PythonJsonReader.Parse(text);+ if (repr is not null) Assert.Equal(repr, PythonFloat.Repr(value));+ }++ public static TheoryData<string, long, string> Floats() => new()+ {+ { "2.2250738585072011e-308", 4503599627370495L, "2.225073858507201e-308" },+ { "2.2250738585072014e-308", 4503599627370496L, "2.2250738585072014e-308" },+ { "0.1", 4591870180066957722L, "0.1" },+ { "0.3", 4599075939470750515L, "0.3" },+ { "0.30000000000000004", 4599075939470750516L, "0.30000000000000004" },+ { "1.7976931348623157e308", 9218868437227405311L, "1.7976931348623157e+308" },+ { "1.7976931348623158e308", 9218868437227405311L, "1.7976931348623157e+308" },+ { "1.7976931348623159e308", 9218868437227405312L, "inf" },+ { "9007199254740993.0", 4845873199050653696L, "9007199254740992.0" },+ { "9007199254740993e0", 4845873199050653696L, "9007199254740992.0" },+ { "9007199254740992.5", 4845873199050653696L, "9007199254740992.0" },+ { "5e-324", 1L, "5e-324" },+ { "4.9406564584124654e-324", 1L, "5e-324" },+ { "2.4703282292062327e-324", 0L, "0.0" },+ { "2.4703282292062328e-324", 1L, "5e-324" },+ { "1e-323", 2L, "1e-323" },+ { "1.00000000000000011102230246251565404236316680908203125", 4607182418800017408L, "1.0" },+ { "1.00000000000000011102230246251565404236316680908203126", 4607182418800017409L, "1.0000000000000002" },+ { "8.41e21", 4934958393344299186L, "8.41e+21" },+ { "7.0e-10", 4469837215314003459L, "7e-10" },+ { "123456789012345678901234567890e-10", 4892433759222981601L, "1.2345678901234567e+19" },+ { "4.35", 4616583683022153318L, "4.35" },+ { "0.000001", 4517329193108106637L, "1e-06" },+ { "1e22", 4936209963552724370L, "1e+22" },+ { "1e23", 4950912855330343670L, "1e+23" },+ { "1e21", 4921056587992461136L, "1e+21" },+ { "1e-5", 4532020583610935537L, "1e-05" },+ { "1e16", 4846369599423283200L, "1e+16" },+ { "1e15", 4831355200913801216L, "1000000000000000.0" },+ { "123456789.123456789", 4728057454355442549L, "123456789.12345679" },+ { "-0.0", -9223372036854775808L, "-0.0" },+ { "0.0", 0L, "0.0" },+ { "100.0", 4636737291354636288L, "100.0" },+ { "2.5", 4612811918334230528L, "2.5" },+ { "0.5", 4602678819172646912L, "0.5" },+ { "5.0", 4617315517961601024L, "5.0" },+ { "1.5e300", 9097811302482466869L, "1.5e+300" },+ { "3.14159265358979323846264338327950288", 4614256656552045848L, "3.141592653589793" },+ { "10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.0", 9218868437227405312L, "inf" },+ { "0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001", 0L, "0.0" },+ { "1e308", 9214871658872686752L, "1e+308" },+ { "1e309", 9218868437227405312L, "inf" },+ { "589.773e-66", 3661160551921729496L, "5.89773e-64" },+ { "9890240870969119.84e-47", 4143325121061772819L, "9.89024087096912e-32" },+ { "70.0033592", 4634626465411991810L, "70.0033592" },+ { "98725432407308880.76", 4861050737830851493L, "9.872543240730888e+16" },+ { "5761058061281496373e+107", 6488345599367672440L, "5.761058061281496e+125" },+ { "-21409026285024355765949748639e-68", -5209918890524461499L, "-2.1409026285024356e-40" },+ { "-49558865774533561827692011.0e-247", -7927274691076673688L, "-4.955886577453356e-222" },+ { "1070241417.79", 4742261044085595832L, "1070241417.79" },+ { "142973103200e-146", 2589746254242631162L, "1.429731032e-135" },+ { "-245586195999701292.0e+169", -1827881155231506903L, "-2.455861959997013e+186" },+ { "5515451689549098e+38", 5410802935147893406L, "5.515451689549098e+53" },+ { "2174151.70638386378173234614e-39", 4118141512346634435L, "2.174151706383864e-33" },+ { "-90052157802223625946388384098.0e+210", -1041553556867938829L, "-9.005215780222362e+238" },+ { "18.577544364e+269", 8650326167602130540L, "1.8577544364e+270" },+ { "-343.379e+302", -60506935699092264L, "-3.43379e+304" },+ { "1094e+102", 6178524366699236661L, "1.094e+105" },+ { "93468655916.186969246451", 4770933967592107997L, "93468655916.18697" },+ { "4.6", 4616865157998863974L, "4.6" },+ { "1036300192629047180132.0", 4921333536483588510L, "1.0363001926290472e+21" },+ { "-6482203456254545.5", -4380023833602608558L, "-6482203456254546.0" },+ { "584501114346614378.0e+246", 8553056883673417103L, "5.845011143466144e+263" },+ { "-812935950329914212297.5", -4303742634299861103L, "-8.129359503299142e+20" },+ { "756128707606555e+218", 8090815721794448434L, "7.56128707606555e+232" },+ { "8.3454822753665398514537271457", 4620887706912954378L, "8.345482275366539" },+ { "827002066984754.515e+2", 4860049161224049404L, "8.270020669847546e+16" },+ { "286732114859191630413170.0e+171", 7516345396997012067L, "2.867321148591916e+194" },+ { "2293573137504623974985115941.39e+145", 7185419934915850854L, "2.293573137504624e+172" },+ { "5956547941627243923118.11435316e+71", 5994931679605799054L, "5.956547941627244e+92" },+ { "-109853.23e-208", -7652867289783984088L, "-1.0985323e-203" },+ { "78208072976786455144e-230", 1463798063640768485L, "7.820807297678646e-211" },+ { "984388456717607885.0", 4876081732005612488L, "9.84388456717608e+17" },+ { "9.065242645513447959e+11", 4785745451415300873L, "906524264551.3448" },+ { "616.4e-299", 175675555446034971L, "6.164e-297" },+ { "193709049088815348057109453365.0e+146", 7229285078893848047L, "1.9370904908881534e+175" },+ { "448672292245580362309655e+184", 7713526195429197804L, "4.486722922455804e+207" },+ { "8683378656433886565108326.3e+283", 9214059114725894834L, "8.683378656433887e+307" },+ { "-28.4181e-85", -5866478163108931173L, "-2.84181e-84" },+ { "356576.0e+228", 8101049532934230075L, "3.56576e+233" },+ { "26496678585940525652.0", 4897378110994754292L, "2.6496678585940525e+19" },+ { "4075.847", 4661181297412794221L, "4075.847" },+ { "1.2178213189", 4608163398810648798L, "1.2178213189" },+ { "79214968058166.07531656", 4814914159987412357L, "79214968058166.08" },+ { "954050.1500", 4696442440478674125L, "954050.15" },+ { "-8976010571e-110", -6113318437113755177L, "-8.976010571e-101" },+ { "39.55178653618156437606496315", 4630763236406567664L, "39.55178653618157" },+ { "97095057824.8922882", 4771171627467596909L, "97095057824.89229" },+ { "72999.150585231338818086490629e-270", 640307887851073669L, "7.299915058523134e-266" },+ { "9061496597394310321297800.941e-126", 3095182002697426466L, "9.061496597394311e-102" },+ { "24619.3350e-192", 1800151515651920268L, "2.4619335e-188" },+ { "6314486095813.13510103625016", 4798296037284320394L, "6314486095813.135" },+ { "86101857.47976950557063e+51", 5488602628152256209L, "8.61018574797695e+58" },+ { "-9228406647553151.7491e+278", -218672533017604270L, "-9.228406647553152e+293" },+ { "-109116263664436163761961952.5", -4227032247252742235L, "-1.0911626366443617e+26" },+ { "-108.0e+25", -4212014816622836652L, "-1.08e+27" },+ { "175905.0e-252", 915266163476354630L, "1.75905e-247" },+ { "-892308217983694516192177909.0e+249", -488141606772896842L, "-8.923082179836945e+275" },+ { "566622208781851001e-178", 2209435534181310078L, "5.66622208781851e-161" },+ { "452800311895686290894792629868.0e-197", 2103487415330792755L, "4.528003118956863e-168" },+ { "1.063e-109", 2976866746370904503L, "1.063e-109" },+ { "-2673455e+36", -3981550870848177602L, "-2.673455e+42" },+ { "-66.605e-251", -8344304590302386922L, "-6.6605e-250" },+ { "925937158701.0e-35", 4262203249624775218L, "9.25937158701e-24" },+ { "29422736699.34649e+228", 8174617011458726069L, "2.942273669934649e+238" },+ { "101037977942856208.1301755983e-137", 2811617633051290470L, "1.0103797794285621e-120" },+ { "6.0", 4618441417868443648L, "6.0" },+ { "9.0e-220", 1329781493142383157L, "9e-220" },+ { "67634678919493557584989669483e+211", 8194853033308816974L, "6.763467891949355e+239" },+ { "702982603311441710284686632.0e-134", 3004044460287497697L, "7.029826033114417e-108" },+ { "815e-255", 835389832106102001L, "8.15e-253" },+ { "9766676350389724.630400508e+169", 7374358766207213511L, "9.766676350389724e+184" },+ { "-33205335789386189589964448.642e-262", -8154436352499320006L, "-3.320533578938619e-237" },+ { "824.7232", 4650465557803754429L, "824.7232" },+ { "2e-301", 108408437193569761L, "2e-301" },+ { "-1.178893106510238019359e+252", -845387005382895447L, "-1.178893106510238e+252" },+ { "-43655914131839848.0e+121", -2557265014845614752L, "-4.365591413183985e+137" },+ { "0.9938056755411020e-35", 4083194199714898251L, "9.93805675541102e-36" },+ { "710500.48325268947893055e-23", 4350585052196504161L, "7.105004832526895e-18" },+ { "686240811847e+209", 7910657241990298994L, "6.86240811847e+220" },+ { "679622257854033598.0e-205", 1806766875166682218L, "6.796222578540336e-188" },+ { "-7e-289", -8927461087006752900L, "-7e-289" },+ { "30702319630037594023513e-245", 1278012107044104135L, "3.0702319630037592e-223" },+ { "1461537561921258853812.672209", 4923506116429650887L, "1.4615375619212588e+21" },+ { "62815381205888002175.0", 4903080688165338527L, "6.2815381205888e+19" },+ { "-9544.176484812597718553915516e-224", -7907969144704734809L, "-9.544176484812598e-221" },+ { "5.7076014885587062543823e-53", 3825207549279336301L, "5.707601488558706e-53" },+ { "2209125363642.0e-265", 826949312482712499L, "2.209125363642e-253" },+ { "1254014211346568.149967618053", 4832874857386557985L, "1254014211346568.2" },+ { "860887042490279377421.608405238e+174", 7523146718194340208L, "8.608870424902794e+194" },+ { "-40.7360686023e+228", -1181387509002387742L, "-4.07360686023e+229" },+ { "-29940542806192093569.792496", -4325153138696874160L, "-2.9940542806192095e+19" },+ { "7398798278076082.4334347135e+111", 6504847345485541630L, "7.398798278076083e+126" },+ { "-4.5997e+89", -3274988697286904235L, "-4.5997e+89" },+ { "81385107.001069753242e+198", 7687707508636978020L, "8.138510700106975e+205" },+ { "1.448885406", 4609204018947211062L, "1.448885406" },+ { "3.1e-326", 0L, "0.0" },+ { "5602.818565224318691e-328", 0L, "0.0" },+ { "1052.7859812208216130e+239", 8227957510357109448L, "1.0527859812208216e+242" },+ { "6406683254176726224698.0", 4933047881616180983L, "6.406683254176727e+21" },+ { "6164444212.5268252e+4", 4813231290996353568L, "61644442125268.25" },+ { "3098522319958316587745298e-108", 3357455367953734192L, "3.098522319958317e-84" },+ { "-8.335", -4602490230938250772L, "-8.335" },+ { "9795650524228224.5", 4846267424685397312L, "9795650524228224.0" },+ { "8.803e+208", 7733000584992601379L, "8.803e+208" },+ { "-40.4754e+2", -4562252987193276498L, "-4047.54" },+ { "3244.25740767e-74", 3552400129709224697L, "3.24425740767e-71" },+ { "4.888896784054e-27", 4213176179556737267L, "4.888896784054e-27" },+ { "287121071301879032571.0e-269", 888491831123499208L, "2.8712107130187903e-249" },+ { "2.667e-247", 918125189786468461L, "2.667e-247" },+ { "4.1e+258", 8476074615440912419L, "4.1e+258" },+ { "201468628786749.0e+33", 5314710545822263247L, "2.01468628786749e+47" },+ { "-46624500e+109", -2871138759936641890L, "-4.66245e+116" },+ { "-67005471.564448369e-128", -6414340721194690622L, "-6.7005471564448365e-121" },+ { "34028800569731647400907.5", 4944058289878429476L, "3.402880056973165e+22" },+ { "16951922792118695.0e-284", 600777099768678075L, "1.6951922792118696e-268" },+ { "84401390207367115115034126102.5", 5039822046976206397L, "8.440139020736712e+28" },+ { "29.6086480596433068665402", 4629027311205151006L, "29.608648059643308" },+ { "106921231602280537550264.0e+285", 9215218441707426373L, "1.0692123160228053e+308" },+ { "129230277299152998284216e+217", 8199030327693238427L, "1.29230277299153e+240" },+ { "1553257291099189753e+172", 7452300297099932243L, "1.5532572910991898e+190" },+ { "4494291574322159225375204.0e-71", 3913746587349763910L, "4.494291574322159e-47" },+ { "40031996724480682520e-129", 2970143182746377635L, "4.0031996724480683e-110" },+ { "8188e-216", 1433848121983751043L, "8.188e-213" },+ { "534.539e-204", 1595667330951917634L, "5.34539e-202" },+ { "35971902786626390752494945.0", 4989356549151033225L, "3.597190278662639e+25" },+ { "0e-281", 0L, "0.0" },+ { "-682368482150667510117784768e+114", -2509447815885517798L, "-6.823684821506675e+140" },+ { "640419741.860009635e-141", 2629283826755081122L, "6.404197418600096e-133" },+ { "3867.83640e+220", 7951879328509240763L, "3.8678364e+223" },+ { "7484084994110360e+114", 6549808393743915945L, "7.48408499411036e+129" },+ { "226947377103.54566963", 4776748617038284249L, "226947377103.5457" },+ { "482577188729681.0e+89", 6157981773288755775L, "4.82577188729681e+103" },+ { "103e+251", 8392029547058071552L, "1.03e+253" },+ { "4365.5", 4661521932712148992L, "4365.5" },+ { "-4512.6364295269155707e-44", -5219827205513723182L, "-4.5126364295269156e-41" },+ { "448266066116924326410170055e+207", 8102551607070509961L, "4.4826606611692435e+233" },+ { "5.19722e+244", 8268171589802821021L, "5.19722e+244" },+ { "-39131082994071626.0", -4368103852802492663L, "-3.9131082994071624e+16" },+ { "407035748452268.2813538408e+300", 9218868437227405312L, "inf" },+ { "527993861.0", 4737637875791691776L, "527993861.0" },+ { "-60907153604348718e-96", -5801512463652081565L, "-6.090715360434872e-80" },+ { "-7900758140089280526542175413.0e+4", -4139043653209105069L, "-7.90075814008928e+31" },+ { "7625540528025970792853.1e-329", 12220732555163875L, "7.625540528025971e-308" },+ { "4476804528917239647583.0e-89", 3599308095700970046L, "4.47680452891724e-68" },+ { "116.2675e+17", 4892082719720425660L, "1.162675e+19" },+ { "787.90e-92", 3274130948369826539L, "7.879e-90" },+ { "41394131928838786182935e-104", 3389276706488298453L, "4.1394131928838786e-82" },+ { "-653.36283e-221", -7880609436539950119L, "-6.5336283e-219" },+ { "955794988208664771687346.0e+191", 7823243632544452582L, "9.557949882086648e+214" },+ { "17754507637222071162381875e+247", 8679845126018147466L, "1.775450763722207e+272" },+ { "7.68e+195", 7537421850789065372L, "7.68e+195" },+ { "8282734863971946419860335.0e-263", 1045189542936631201L, "8.282734863971947e-239" },+ { "3.74625606e+246", 8295952952511376097L, "3.74625606e+246" },+ { "50285.95e-158", 2313434100645552143L, "5.028595e-154" },+ { "5885669970437227285774.92e-60", 4035231547988945579L, "5.885669970437227e-39" },+ { "375369467390947253.0e-185", 2102148746146269705L, "3.7536946739094724e-168" },+ { "6122.4562", 4663453726483542127L, "6122.4562" },+ { "720.1809e+69", 5681878342431414333L, "7.201809e+71" },+ { "560840.2578625272264e+14", 4902258989505715080L, "5.608402578625272e+19" },+ { "2.1e+131", 6571577934564546031L, "2.1e+131" },+ { "50984094523449e+204", 7863984994275004437L, "5.0984094523449e+217" },+ { "7104325.92287798999108e+113", 6400067620129659834L, "7.10432592287799e+119" },+ { "68754142819712478990052954.0e+157", 7342193561072622828L, "6.875414281971248e+182" },+ { "6371135377620827668511.0e-285", 669257222946182605L, "6.371135377620828e-264" },+ { "86.6719915e-139", 2556300397935060845L, "8.66719915e-138" },+ { "-924847124631e-136", -6471869011214791394L, "-9.24847124631e-125" },+ { "783763203245432683477359324e-241", 1403720925441841835L, "7.837632032454327e-215" },+ { "33013808597453.66732629574952e+41", 5422681294914886070L, "3.301380859745367e+54" },+ { "5232506686.5", 4752280688179806208L, "5232506686.5" },+ { "7530696e-30", 4261026773984002807L, "7.530696e-24" },+ { "2601136e+99", 6184046376648300779L, "2.601136e+105" },+ { "718784768300756617568.14448e-204", 1866864073262966569L, "7.187847683007566e-184" },+ { "148522773723924508475718e+212", 8125249407171543644L, "1.485227737239245e+235" },+ { "102584274921242367.2282441693e-160", 2467976006030512767L, "1.0258427492124238e-143" },+ { "42861464006636e+76", 5947828761292558790L, "4.2861464006636e+89" },+ { "2014800757348487.0e+267", 8830362297817994366L, "2.014800757348487e+282" },+ { "830083.5", 4695377575063584768L, "830083.5" },+ { "-10710250632410346.7352087243259", -4376647312115287435L, "-1.0710250632410346e+16" },+ { "-86786996481861727835861e-303", -8806449666860697700L, "-8.678699648186173e-281" },+ { "72298747.0", 4724624296264073216L, "72298747.0" },+ { "3.0e+308", 9218868437227405312L, "inf" },+ { "106347261478462469789542676.0e-300", 508190658297005694L, "1.0634726147846248e-274" },+ { "548207.364", 4692956277492299071L, "548207.364" },+ { "336344063227909494072513889.0e-304", 455730275051153250L, "3.363440632279095e-278" },+ { "76120650225431013839366081273.8e+304", 9218868437227405312L, "inf" },+ { "3.0e+132", 6589021296160631994L, "3e+132" },+ { "2682540.0e-322", 54295214L, "2.68254e-316" },+ { "2563458070685408450447.0e+186", 7710145140987739075L, "2.5634580706854084e+207" },+ { "-31587694393529104224.738610482e+4", -4264705513488230457L, "-3.1587694393529107e+23" },+ { "697532084472580e+0", 4828935457589581856L, "697532084472580.0" },+ { "6382609957350453420168813860e+41", 5636163289936670526L, "6.382609957350454e+68" },+ { "698361.34572929904460289472328", 4694246090374082113L, "698361.345729299" },+ { "73496914.7102e-149", 2495536008417178287L, "7.34969147102e-142" },+ { "-47606.0e-288", -8855113716859869028L, "-4.7606e-284" },+ { "6075.62917237797270389723e-143", 2524029061219851261L, "6.075629172377973e-140" },+ { "-25930253551497.0e-112", -6091488245390213054L, "-2.5930253551497e-99" },+ { "196749581454.0e-135", 2756385367201602600L, "1.96749581454e-124" },+ { "50505023755247012e-36", 4318336429831471741L, "5.050502375524701e-20" },+ { "83477775675082701478947576630e+259", 8914292473251887725L, "8.34777756750827e+287" },+ { "7.3e-13", 4425260804279883197L, "7.3e-13" },+ { "-620.5", -4574702845315514368L, "-620.5" },+ };++ [Theory]+ [MemberData(nameof(Floats))]+ public void FloatLiteralsAreBitIdenticalToPythonAndReprTheSame(string literal, long bits, string repr)+ {+ var value = Assert.IsType<double>(PythonJsonReader.Parse(literal));+ Assert.Equal(bits, BitConverter.DoubleToInt64Bits(value));+ Assert.Equal(repr, PythonFloat.Repr(value));+ }++ [Theory]+ [InlineData("-0", 0.5, "temperature[0]=0 -> 0.5")]+ [InlineData("0", 0.5, "temperature[0]=0 -> 0.5")]+ [InlineData("7", 5.0, "temperature[0]=7 -> 5")]+ [InlineData("7.0", 5.0, "temperature[0]=7.0 -> 5")]+ [InlineData("-0.0", 0.5, "temperature[0]=-0.0 -> 0.5")]+ [InlineData("1", 1.0, "")]+ [InlineData("1.0", 1.0, "")]+ [InlineData("2.5", 2.5, "")]+ [InlineData("100000000000000000000000000000", 5.0, "temperature[0]=100000000000000000000000000000 -> 5")]+ [InlineData("-100000000000000000000000000000", 0.5, "temperature[0]=-100000000000000000000000000000 -> 0.5")]+ [InlineData("100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", 5.0, "temperature[0]=100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 -> 5")]+ [InlineData("1e400", 1.0, "temperature[0]=inf -> 1")]+ [InlineData("-1e400", 1.0, "temperature[0]=-inf -> 1")]+ [InlineData("0.1006", 0.5, "temperature[0]=0.1006 -> 0.5")]+ [InlineData("1E2", 5.0, "temperature[0]=100.0 -> 5")]+ [InlineData("5e0", 5.0, "")]+ [InlineData("0.5", 0.5, "")]+ [InlineData("10", 5.0, "temperature[0]=10 -> 5")]+ [InlineData("100000000000000000000.0", 5.0, "temperature[0]=1e+20 -> 5")]+ [InlineData("1e22", 5.0, "temperature[0]=1e+22 -> 5")]+ [InlineData("1e-7", 0.5, "temperature[0]=1e-07 -> 0.5")]+ [InlineData("true", 1.0, "temperature[0]=True -> 1")]+ [InlineData("null", 1.0, "temperature[0]=None -> 1")]+ [InlineData("\"2.5\"", 2.5, "")]+ [InlineData("\"x\"", 1.0, "temperature[0]='x' -> 1")]+ [InlineData("[1]", 1.0, "temperature[0]=[1] -> 1")]+ [InlineData("NaN", 1.0, "temperature[0]=nan -> 1")]+ [InlineData("Infinity", 1.0, "temperature[0]=inf -> 1")]+ [InlineData("-Infinity", 1.0, "temperature[0]=-inf -> 1")]+ public void WarningEntryUsesTheLiteralsPythonType(string literal, double applied, string entry)+ {+ // Distinct text per row is not needed: the entry list is on the config, the trace is deduplicated separately.+ var config = LayaConfig.Parse("{\"temperature\": [" + literal + ", 1, 1]}");+ Assert.Equal(applied, config.Temperature[0]);+ Assert.Equal<IEnumerable<string>>(entry.Length == 0 ? [] : new[] { entry }, config.ClampedTemperatures);+ }++ [Fact]+ public void ARealStringIsNeverReadAsNaNOrInfinity()+ {+ var config = LayaConfig.Parse("{\"temperature\": [\"NaN\", \"Infinity\", \"\\uE000laya-nan\"], \"encoder\": \"NaN\"}");+ Assert.Equal("NaN", config.Encoder);+ Assert.Equal(["temperature[0]='NaN' -> 1", "temperature[1]='Infinity' -> 1", "temperature[2]='\\ue000laya-nan' -> 1"], config.ClampedTemperatures);+ }++ [Fact]+ public void ConfigRejectsMalformedJsonAsPythonDoes()+ {+ foreach (var bad in new[] { "{NaN: 1}", "{\"temperature\": [nan]}", "{\"temperature\": [1,]}", "{\"temperature\": [01]}", "{'a': 1}" })+ Assert.ThrowsAny<JsonException>(() => LayaConfig.Parse(bad));+ Assert.Throws<InvalidDataException>(() => LayaConfig.Parse("[1]"));+ }++ [Fact]+ public void IntegerLiteralsKeepPythonsIntTypeAndFloatLiteralsStayFloats()+ {+ Assert.IsType<long>(PythonJsonReader.Parse("7"));+ Assert.IsType<long>(PythonJsonReader.Parse("-0"));+ Assert.IsType<BigInteger>(PythonJsonReader.Parse("9223372036854775808"));+ Assert.IsType<double>(PythonJsonReader.Parse("7.0"));+ Assert.IsType<double>(PythonJsonReader.Parse("7e0"));+ }+}++/// <summary>Python's default warnings filter: one trace per distinct (site, text) per process.</summary>+public sealed class WarnOnceTests+{+ private sealed class Capture : TraceListener+ {+ private readonly object _gate = new();+ private readonly List<string> _messages = [];+ private readonly int? _thread;+ public Capture(int? onlyThread) => _thread = onlyThread;+ public List<string> Messages { get { lock (_gate) return [.. _messages]; } }+ public override void Write(string? message) { lock (_gate) _messages.Add(message ?? ""); }+ public override void WriteLine(string? message) { lock (_gate) _messages.Add(message ?? ""); }+ public override void TraceEvent(TraceEventCache? eventCache, string source, TraceEventType eventType, int id, string? format, params object?[]? args)+ {+ // Other test classes warn from their own threads and call sites; only the ones raised here count.+ if (_thread is { } thread && Environment.CurrentManagedThreadId != thread) return;+ lock (_gate) _messages.Add(args is null ? format ?? "" : string.Format(format ?? "", args));+ }+ }++ private static List<string> Capturing(Action action, string needle = "laya", bool anyThread = false)+ {+ var capture = new Capture(anyThread ? null : Environment.CurrentManagedThreadId);+ Trace.Listeners.Add(capture);+ try { action(); }+ finally { Trace.Listeners.Remove(capture); }+ return capture.Messages.Where(m => m.StartsWith("laya", StringComparison.Ordinal) && m.Contains(needle)).ToList();+ }++ private const string Config = "{\"temperature\": [0.2, 1, 1]}";++ // ── call sites (Python keys the default filter on the frame stacklevel selects) ──++ private static string TempConfig(string value) => "{\"temperature\": [" + value + ", 1, 1]}";++ [Fact]+ public void TheSameCallSiteInALoopWarnsOnce()+ {+ WarnOnce.Reset();+ var messages = Capturing(() => { for (var i = 0; i < 5; i++) LayaConfig.Parse(Config); }, "temperature[0]=0.2 ");+ Assert.Single(messages);+ Assert.StartsWith("laya ONNX: this checkpoint ships temperatures outside [0.5, 5]", messages[0]);+ }++ [Fact]+ public void TwoCallSitesWarnTwiceForTheSameText()+ {+ WarnOnce.Reset();+ var messages = Capturing(() =>+ {+ LayaConfig.Parse(Config);+ LayaConfig.Parse(Config);+ }, "temperature[0]=0.2 ");+ Assert.Equal(2, messages.Count);+ }++ [Fact]+ public void ANewTextFromTheSameSiteWarnsAgain()+ {+ WarnOnce.Reset();+ var messages = Capturing(() =>+ {+ foreach (var value in new[] { "0.3", "0.3", "0.4" }) LayaConfig.Parse(TempConfig(value));+ }, "temperature[0]=0.");+ Assert.Equal(2, messages.Count(m => m.Contains("temperature[0]=0.3 ")) + messages.Count(m => m.Contains("temperature[0]=0.4 ")));+ Assert.Single(messages, m => m.Contains("temperature[0]=0.3 "));+ }++ // A public entry is attributed to its caller, and overloads that call each other are one entry.+ [WarnSite, System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.NoInlining)]+ private static void EntryOverload(string text) => EntryCore(text);++ [WarnSite, System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.NoInlining)]+ private static void EntryCore(string text) => WarnOnce.Warn(text);++ [Fact]+ public void AnEntryIsKeyedByItsCallerNotByTheLibraryLine()+ {+ WarnOnce.Reset();+ var once = Capturing(() => { for (var i = 0; i < 4; i++) EntryOverload("laya: site test loop"); }, "site test loop");+ Assert.Single(once);+ var twice = Capturing(() =>+ {+ EntryOverload("laya: site test lines");+ EntryOverload("laya: site test lines");+ }, "site test lines");+ Assert.Equal(2, twice.Count);+ }++ // Through the router the site is the router's own call into the checkpoint, as Python attributes it to router.py.+ private sealed class WarningScanner : ILayaPredictor, ILayaLongPredictor+ {+ public LayaResult Predict(object? state, QuestionSet questions) => throw new NotSupportedException();++ [WarnSite, System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.NoInlining)]+ public LayaResult PredictLong(object state, QuestionSet questions, int? window = null, int? stride = null,+ string aggregate = "auto", int? batchSize = null)+ {+ WarnOnce.Warn("laya: predict_long: via the router");+ return new LayaResult("scanner", [], new Dictionary<string, Answer>(StringComparer.Ordinal), new Usage(0, 0));+ }+ }++ [Fact]+ public void CallsThroughTheRouterFromDifferentUserLinesWarnOnce()+ {+ WarnOnce.Reset();+ using var router = new LayaRouter(new LayaRouterOptions(), _ => new WarningScanner());+ var messages = Capturing(() =>+ {+ router.PredictLong("hello", new QuestionSet());+ router.PredictLong("hello again", new QuestionSet());+ }, "via the router");+ Assert.Single(messages);+ }++ [Fact]+ public void ConcurrentLoadsWarnOnce()+ {+ WarnOnce.Reset();+ var messages = Capturing(() => Parallel.For(0, 32, _ => LayaConfig.Parse(Config)), "temperature[0]=0.2 ", anyThread: true);+ Assert.Single(messages);+ }++ public static TheoryData<int, int, int, int?, int?, string, string[]> Scenarios() => new()+ {+ { 100, 512, 192, 1000, null, "100,50,100", new string[] { "laya: predict_long: window=1000 is wider than the 100 state tokens these questions leave inside max_len=512, so every window would be truncated to 100 on the way to the model; scanning with window=100 instead" } },+ { 100, 512, 192, null, null, "100,50,100", new string[] { "laya: predict_long: these questions leave only 100 of max_len=512 for the state (their heads take the rest), so the scan window is capped 312 -> 100 and roughly 3.1x as many windows -- each a full forward pass -- are needed to read the document. Fewer or shorter options, a larger max_len, or laya.shortlist.predict_shortlist for a large label set will all cost less than scanning at this width" } },+ { 24, 174, 100, null, null, "24,12,24", new string[] { "laya: predict_long: these questions leave only 24 of max_len=174 for the state (their heads take the rest), so the scan window is capped 66 -> 24 and roughly 2.8x as many windows -- each a full forward pass -- are needed to read the document. Fewer or shorter options, a larger max_len, or laya.shortlist.predict_shortlist for a large label set will all cost less than scanning at this width" } },+ { 32, 180, 100, null, null, "32,16,32", new string[] { "laya: predict_long: these questions leave only 32 of max_len=180 for the state (their heads take the rest), so the scan window is capped 72 -> 32 and roughly 2.2x as many windows -- each a full forward pass -- are needed to read the document. Fewer or shorter options, a larger max_len, or laya.shortlist.predict_shortlist for a large label set will all cost less than scanning at this width" } },+ { 100, 512, 192, 1000, 500, "100,50,100", new string[] { "laya: predict_long: window=1000 is wider than the 100 state tokens these questions leave inside max_len=512, so every window would be truncated to 100 on the way to the model; scanning with window=100 instead", "laya: predict_long: stride=500 was a 50% step for the window=1000 you asked for, but the window was reduced to 100 to fit the room these questions leave; scanning with stride=50 instead" } },+ { 200, 512, 192, null, null, "200,100,200", new string[] { } },+ { 100, 512, 192, 500, 400, "100,50,100", new string[] { "laya: predict_long: window=500 is wider than the 100 state tokens these questions leave inside max_len=512, so every window would be truncated to 100 on the way to the model; scanning with window=100 instead", "laya: predict_long: stride=400 was a 50% step for the window=500 you asked for, but the window was reduced to 100 to fit the room these questions leave; scanning with stride=50 instead" } },+ { 150, 512, 192, null, null, "150,75,150", new string[] { "laya: predict_long: these questions leave only 150 of max_len=512 for the state (their heads take the rest), so the scan window is capped 312 -> 150 and roughly 2.1x as many windows -- each a full forward pass -- are needed to read the document. Fewer or shorter options, a larger max_len, or laya.shortlist.predict_shortlist for a large label set will all cost less than scanning at this width" } },+ { 104, 512, 192, null, null, "104,52,104", new string[] { "laya: predict_long: these questions leave only 104 of max_len=512 for the state (their heads take the rest), so the scan window is capped 312 -> 104 and roughly 3.0x as many windows -- each a full forward pass -- are needed to read the document. Fewer or shorter options, a larger max_len, or laya.shortlist.predict_shortlist for a large label set will all cost less than scanning at this width" } },+ { 399, 512, 192, 400, 399, "399,399,399", new string[] { "laya: predict_long: window=400 is wider than the 399 state tokens these questions leave inside max_len=512, so every window would be truncated to 399 on the way to the model; scanning with window=399 instead" } },+ { 100, 512, 192, 1000, 2000, "ERR", new string[] { "laya: predict_long: window=1000 is wider than the 100 state tokens these questions leave inside max_len=512, so every window would be truncated to 100 on the way to the model; scanning with window=100 instead" } },+ };++ [Theory]+ [MemberData(nameof(Scenarios))]+ public void PredictLongWarningsHavePythonsTextAndAreShownOnce(int room, int maxLen, int head, int? window, int? stride, string result, string[] expected)+ {+ var requested = window is > 0 ? window.Value : Math.Max(64, maxLen - head - 8);+ WarnOnce.Reset();+ (int, int, int) got = default;+ var threw = false;+ void Run() { try { var b = LongPrediction.BudgetForRoom(requested, room, maxLen, head, window, stride); got = (b.Window, b.Stride, b.Room); } catch (ArgumentException) { threw = true; } }+ var first = Capturing(Run);+ Assert.Equal(result == "ERR", threw);+ if (!threw) Assert.Equal(result, $"{got.Item1},{got.Item2},{got.Item3}");+ Assert.Equal(expected, first.Where(m => m.StartsWith("laya: predict_long")).ToArray());+ var second = Capturing(Run);+ Assert.DoesNotContain(second, m => m.StartsWith("laya: predict_long"));+ }+}++/// <summary>+/// Config files are read like Python's <c>json.load(open(path))</c>: strict UTF-8, a BOM is an error+/// (<c>json.loads</c>: "Unexpected UTF-8 BOM (decode using utf-8-sig)"), invalid bytes are an error.+/// Checked against CPython 3.12 reading the same bytes with <c>encoding="utf-8"</c>.+/// </summary>+public sealed class ConfigFileEncodingTests+{+ private static string Write(byte[] bytes)+ {+ var path = Path.Combine(Path.GetTempPath(), "laya-cfg-" + Guid.NewGuid().ToString("N") + ".json");+ File.WriteAllBytes(path, bytes);+ return path;+ }++ private static readonly byte[] Bom = [0xEF, 0xBB, 0xBF];++ [Fact]+ public void AValidUtf8FileLoads()+ {+ var path = Write("{\"encoder\": \"café\", \"max_len\": 64}"u8.ToArray());+ try { Assert.Equal("café", LayaConfig.Load(path).Encoder); }+ finally { File.Delete(path); }+ }++ [Fact]+ public void ABomIsRejectedLikePython()+ {+ var path = Write([.. Bom, .. "{\"max_len\": 64}"u8.ToArray()]);+ try { Assert.Contains("Unexpected UTF-8 BOM", Assert.Throws<JsonException>(() => LayaConfig.Load(path)).Message); }+ finally { File.Delete(path); }+ }++ [Fact]+ public void InvalidUtf8IsRejectedLikePython()+ {+ var path = Write([.. "{\"encoder\": \"caf"u8.ToArray(), 0xE9, .. "\"}"u8.ToArray()]);+ try { Assert.Throws<JsonException>(() => LayaConfig.Load(path)); }+ finally { File.Delete(path); }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/TemperatureConfigTests.cs b/laya-dotnet/tests/Laya.Tests/TemperatureConfigTests.csnew file mode 100644index 00000000..1e999a7a--- /dev/null+++ b/laya-dotnet/tests/Laya.Tests/TemperatureConfigTests.cs@@ -0,0 +1,131 @@+using System.Diagnostics;++namespace Laya.Tests;++/// <summary>+/// <c>clamp_temperature</c> / <c>temperature_rejected</c> on the JSON values a checkpoint's+/// <c>rl_agent_config.json</c> can hold. Every expectation below was produced by running the Python+/// functions in <c>laya/common.py</c> (CPython 3.12) on <c>json.loads(value)</c>, with the entry text+/// built as <c>onnx_agent.py</c> builds it: <c>"%s=%r -> %g"</c>.+/// </summary>+public sealed class TemperatureConfigTests+{+ [Theory]+ [InlineData("2.5", 2.5, false, "temperature[0]=2.5 -> 2.5")]+ [InlineData("\"2.5\"", 2.5, false, "temperature[0]='2.5' -> 2.5")]+ [InlineData("\" 2.5 \"", 2.5, false, "temperature[0]=' 2.5 ' -> 2.5")]+ [InlineData("\"2_5\"", 5.0, true, "temperature[0]='2_5' -> 5")]+ [InlineData("\"1e1\"", 5.0, true, "temperature[0]='1e1' -> 5")]+ [InlineData("\"0x1\"", 1.0, true, "temperature[0]='0x1' -> 1")]+ [InlineData("\"\\uff12.\\uff15\"", 2.5, false, "temperature[0]='2.5' -> 2.5")]+ [InlineData("true", 1.0, true, "temperature[0]=True -> 1")]+ [InlineData("false", 1.0, true, "temperature[0]=False -> 1")]+ [InlineData("null", 1.0, true, "temperature[0]=None -> 1")]+ [InlineData("[1]", 1.0, true, "temperature[0]=[1] -> 1")]+ [InlineData("{\"a\": 1}", 1.0, true, "temperature[0]={'a': 1} -> 1")]+ [InlineData("\"nan\"", 1.0, true, "temperature[0]='nan' -> 1")]+ [InlineData("\"-Infinity\"", 1.0, true, "temperature[0]='-Infinity' -> 1")]+ [InlineData("\"5.\"", 5.0, false, "temperature[0]='5.' -> 5")]+ [InlineData("\".5\"", 0.5, false, "temperature[0]='.5' -> 0.5")]+ [InlineData("\"1__0\"", 1.0, true, "temperature[0]='1__0' -> 1")]+ [InlineData("\"_1\"", 1.0, true, "temperature[0]='_1' -> 1")]+ [InlineData("\"\"", 1.0, true, "temperature[0]='' -> 1")]+ [InlineData("\"abc\"", 1.0, true, "temperature[0]='abc' -> 1")]+ [InlineData("7", 5.0, true, "temperature[0]=7 -> 5")]+ [InlineData("0.1006", 0.5, true, "temperature[0]=0.1006 -> 0.5")]+ [InlineData("1e999", 1.0, true, "temperature[0]=inf -> 1")]+ [InlineData("\"\\u001f2.5\\u001f\"", 1.0, true, "temperature[0]='\\x1f2.5\\x1f' -> 1")]+ [InlineData("\"+3\"", 3.0, false, "temperature[0]='+3' -> 3")]+ [InlineData("\"1_e5\"", 1.0, true, "temperature[0]='1_e5' -> 1")]+ [InlineData("\"1e1_0\"", 5.0, true, "temperature[0]='1e1_0' -> 5")]+ [InlineData("\"\\u0663\"", 3.0, false, "temperature[0]='٣' -> 3")]+ [InlineData("\"1_000.0_1\"", 5.0, true, "temperature[0]='1_000.0_1' -> 5")]+ [InlineData("\"1.5e+0\"", 1.5, false, "temperature[0]='1.5e+0' -> 1.5")]+ [InlineData("\" \\n3\\t\"", 3.0, false, "temperature[0]=' \\n3\\t' -> 3")]+ [InlineData("\"- 3\"", 1.0, true, "temperature[0]='- 3' -> 1")]+ [InlineData("\"inf\"", 1.0, true, "temperature[0]='inf' -> 1")]+ [InlineData("\"NaN\"", 1.0, true, "temperature[0]='NaN' -> 1")]+ [InlineData("\"1,5\"", 1.0, true, "temperature[0]='1,5' -> 1")]+ [InlineData("0", 0.5, true, "temperature[0]=0 -> 0.5")]+ [InlineData("-2", 0.5, true, "temperature[0]=-2 -> 0.5")]+ [InlineData("\"0.2\"", 0.5, true, "temperature[0]='0.2' -> 0.5")]+ [InlineData("10.0", 5.0, true, "temperature[0]=10.0 -> 5")]+ [InlineData("[\"a\", 1]", 1.0, true, "temperature[0]=['a', 1] -> 1")]+ [InlineData("{\"k\": [true, null]}", 1.0, true, "temperature[0]={'k': [True, None]} -> 1")]+ [InlineData("\"it's\"", 1.0, true, "temperature[0]=\"it's\" -> 1")]+ public void ConfigEntryMatchesPython(string json, double applied, bool rejected, string entry)+ {+ var config = LayaConfig.Parse("{\"temperature\": [" + json + ", 1, 1]}");+ Assert.Equal(applied, config.Temperature[0]);+ Assert.Equal(applied, Calibration.ClampTemperature(config.TemperatureRawValues[0]));+ Assert.Equal(rejected, Calibration.TemperatureRejected(config.TemperatureRawValues[0], applied));+ Assert.Equal<IEnumerable<string>>(rejected ? new[] { entry } : [], config.ClampedTemperatures);+ }++ private sealed class Capture : TraceListener+ {+ public List<string> Messages { get; } = [];+ public override void Write(string? message) => Messages.Add(message ?? "");+ public override void WriteLine(string? message) => Messages.Add(message ?? "");+ public override void TraceEvent(TraceEventCache? eventCache, string source, TraceEventType eventType, int id, string? format, params object?[]? args) =>+ Messages.Add(args is null ? format ?? "" : string.Format(format ?? "", args));+ }++ [Fact]+ public void WarnsOnceWithPythonsTextListingEveryRejectedEntry()+ {+ // Expected text from onnx_agent.py:254-262 run on the same document.+ const string json = "{\"temperature\": [true, \"2.5\", 0.1, 9, null, \"x\"], "+ + "\"temperature_by_options\": {\"choice:11+\": 0.1006, \"noul:2\": \"3\"}}";+ const string warning = "laya ONNX: this checkpoint ships temperatures outside [0.5, 5] which would distort confidence; "+ + "clamping choice:11+=0.1006 -> 0.5, temperature[0]=True -> 1, temperature[2]=0.1 -> 0.5, temperature[3]=9 -> 5, "+ + "temperature[4]=None -> 1, temperature[5]='x' -> 1. Treat confidence from the affected buckets as uncalibrated.";++ var capture = new Capture();+ Trace.Listeners.Add(capture);+ LayaConfig config;+ try { config = LayaConfig.Parse(json); }+ finally { Trace.Listeners.Remove(capture); }++ Assert.Equal<IEnumerable<double>>(new[] { 1.0, 2.5, 0.5, 5.0, 1.0, 1.0 }, config.Temperature);+ Assert.Equal(0.5, config.TemperatureByOptions["choice:11+"]);+ Assert.Equal(3.0, config.TemperatureByOptions["noul:2"]);+ // Other tests may warn through the same global listener meanwhile; this document warns exactly once.+ Assert.Single(capture.Messages, m => m == warning);+ }++ [Fact]+ public void AcceptedTemperaturesDoNotWarn()+ {+ var config = LayaConfig.Parse("{\"temperature\": [1.0, 2.5, \"3\"], \"temperature_by_options\": {\"choice:2\": 1.5}}");+ Assert.Empty(config.ClampedTemperatures);+ Assert.Equal<IEnumerable<double>>(new[] { 1.0, 2.5, 3.0 }, config.Temperature);+ }++ // Python: json.loads accepts the bare tokens NaN / Infinity / -Infinity (-> float), so the entry clamps to 1.0.+ [Fact]+ public void BareNaNAndInfinityTokensLoadAsPythonDoes()+ {+ var config = LayaConfig.Parse("{\"temperature\": [NaN, 1.2, 1], \"temperature_by_options\": {\"a\": Infinity, \"b\": -Infinity, \"c\": \"NaN\"}}");+ Assert.Equal<IEnumerable<double>>(new[] { 1.0, 1.2, 1.0 }, config.Temperature);+ Assert.Equal<IEnumerable<string>>(+ ["a=inf -> 1", "b=-inf -> 1", "c='NaN' -> 1", "temperature[0]=nan -> 1"], config.ClampedTemperatures);+ }++ // Python: float literals that overflow read as inf; 10**308 is an int that still converts, 10**309 does not.+ [Fact]+ public void OverflowingFloatLiteralIsInfinityButAHugeIntegerFailsTheLoad()+ {+ var ok = LayaConfig.Parse("{\"temperature\": [1, 1e400, 1" + new string('0', 308) + "]}");+ Assert.Equal<IEnumerable<double>>(new[] { 1.0, 1.0, 5.0 }, ok.Temperature);+ Assert.Equal("temperature[1]=inf -> 1", ok.ClampedTemperatures[0]);+ Assert.Equal("temperature[2]=1" + new string('0', 308) + " -> 5", ok.ClampedTemperatures[1]);++ foreach (var huge in new[] { "1" + new string('0', 309), "-1" + new string('0', 309) })+ {+ var inList = Assert.Throws<InvalidDataException>(() => LayaConfig.Parse("{\"temperature\": [1, 1, " + huge + "]}"));+ Assert.Equal("OverflowError: int too large to convert to float", inList.Message);+ Assert.Throws<InvalidDataException>(() => LayaConfig.Parse("{\"temperature_by_options\": {\"x\": " + huge + "}}"));+ }+ }+}diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/email_probe.json b/laya-dotnet/tests/Laya.Tests/golden/routing/email_probe.jsonindex adbee5ab..f34f54ca 100644--- a/laya-dotnet/tests/Laya.Tests/golden/routing/email_probe.json+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/email_probe.json@@ -419,6 +419,1032 @@"body": "Hi team,\n\nMy account was charged twice for the Pro plan this month and the refund has not arrived yet. Could you please look into this today?\n\nBest regards,\nPriya Sharma\nSent from my iPhone\n\nThis email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender.\n\nOn Tue, Sep 22, 2026 at 9:14 AM, Support <[email protected]> wrote:\n> Hi Priya, thanks for reaching out, we are looking into it.\n> - Support Team\n","max_chars": 3000,"result": "Hi team,\n\nMy account was charged twice for the Pro plan this month and the refund has not arrived yet. Could you please look into this today?"+ },+ {+ "label": "p0411/budget_default_cut",+ "body": "word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word ",+ "max_chars": 3000,+ "result": "word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word "+ },+ {+ "label": "p0411/budget_small",+ "body": "Please refund the duplicate charge on invoice 4411 today.",+ "max_chars": 10,+ "result": "Please ref"+ },+ {+ "label": "p0411/budget_one",+ "body": "Hello",+ "max_chars": 1,+ "result": "H"+ },+ {+ "label": "p0411/budget_zero",+ "body": "Please refund it.",+ "max_chars": 0,+ "result": ""+ },+ {+ "label": "p0411/budget_zero_empty_body",+ "body": "",+ "max_chars": 0,+ "result": ""+ },+ {+ "label": "p0411/budget_negative_one",+ "body": "Please refund it today and thanks for the help.",+ "max_chars": -1,+ "result": "Please refund it today and thanks for the "+ },+ {+ "label": "p0411/budget_negative_ten",+ "body": "Please refund it today and thanks for the help with the invoice.",+ "max_chars": -10,+ "result": "Please refund "+ },+ {+ "label": "p0411/budget_negative_beyond_body",+ "body": "Please refund it.",+ "max_chars": -100,+ "result": ""+ },+ {+ "label": "p0411/budget_larger_than_body",+ "body": "Please refund it.",+ "max_chars": 100000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/budget_pre_cut_hides_footer",+ "body": "Request line.\nfiller line that is not a footer.\nfiller line that is not a footer.\nfiller line that is not a footer.\n\n\nThis email is confidential and intended solely for the named addressee.",+ "max_chars": 25,+ "result": "Request line.\nfiller line"+ },+ {+ "label": "p0411/budget_pre_cut_hides_quote",+ "body": "Short request.\nMore detail here.\nOn Mon, Jan 1, 2024 at 9:00 AM Jane <[email protected]> wrote:\nOld text.",+ "max_chars": 12,+ "result": "Short reques"+ },+ {+ "label": "p0411/budget_pre_cut_multibyte",+ "body": "退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 ",+ "max_chars": 30,+ "result": "退款请求 退款请求 退款请求 退款请求 退款请求 退款请求 "+ },+ {+ "label": "p0411/budget_boundary_exact",+ "body": "abcde",+ "max_chars": 5,+ "result": "abcde"+ },+ {+ "label": "p0411/budget_negative_multiline",+ "body": "line one\nline two\nline three\nline four",+ "max_chars": -2,+ "result": "line one\nline two\nline three"+ },+ {+ "label": "p0411/quote_first_line_kept",+ "body": "On Mon, Jan 1, 2024 Jane wrote:\nPlease refund it.",+ "max_chars": 3000,+ "result": "On Mon, Jan 1, 2024 Jane wrote:\nPlease refund it."+ },+ {+ "label": "p0411/quote_en_wrote",+ "body": "Please refund it.\nOn Mon, Jan 1, 2024 at 9:00 AM Jane <[email protected]> wrote:\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/quote_en_wrote_trailing_space",+ "body": "Please refund it.\nON TUE, SEP 22 JANE WROTE: \nOld request.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/quote_pt_no_date_is_body",+ "body": "Preciso de ajuda.\nEm resposta ao que você escreveu:\nMais detalhes aqui.",+ "max_chars": 3000,+ "result": "Preciso de ajuda.\nEm resposta ao que você escreveu:\nMais detalhes aqui."+ },+ {+ "label": "p0411/quote_pt_date",+ "body": "Preciso de ajuda.\nEm 10/09/2024 às 10:00, João escreveu:\nOld request.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/quote_pt_unicode_digits",+ "body": "Preciso de ajuda.\nEm ١٠/٠٩ João escreveu:\nOld request.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/quote_es_no_date_is_body",+ "body": "Necesito ayuda.\nComo usted escribió:\nEl resto del texto.",+ "max_chars": 3000,+ "result": "Necesito ayuda.\nComo usted escribió:\nEl resto del texto."+ },+ {+ "label": "p0411/quote_es_no_date_el",+ "body": "Necesito ayuda.\nEl cliente que escribió:\nEl resto del texto.",+ "max_chars": 3000,+ "result": "Necesito ayuda.\nEl cliente que escribió:\nEl resto del texto."+ },+ {+ "label": "p0411/quote_es_date_ascii_o",+ "body": "Necesito ayuda.\nEl 10/09 Juan escribio:\nOld request.",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/quote_fr_no_date_is_body",+ "body": "Je demande de l'aide.\nLe rapport que vous avez écrit :\nLe reste du texte.",+ "max_chars": 3000,+ "result": "Je demande de l'aide.\nLe rapport que vous avez écrit :\nLe reste du texte."+ },+ {+ "label": "p0411/quote_fr_no_space_before_colon",+ "body": "Je demande de l'aide.\nLe 10/09 Jean a écrit:\nOld request.",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/quote_fr_ecrit_ascii",+ "body": "Je demande de l'aide.\nLe 10/09 Jean a ecrit :\nOld request.",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/quote_original_message",+ "body": "Please refund it.\n-----Original Message-----\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/quote_forwarded_message",+ "body": "Please refund it.\n---- Forwarded Message ----\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/quote_mensagem_original",+ "body": "Preciso de ajuda.\n-----Mensagem original-----\nOld request.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/quote_mensagem_encaminhada",+ "body": "Preciso de ajuda.\n----- Mensagem encaminhada -----\nOld request.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/quote_mensaje_original",+ "body": "Necesito ayuda.\n-----Mensaje original-----\nOld request.",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/quote_mensaje_reenviado",+ "body": "Necesito ayuda.\n-----Mensaje reenviado-----\nOld request.",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/quote_message_d_origine",+ "body": "Je demande de l'aide.\n-----Message d'origine-----\nOld request.",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/quote_underscores",+ "body": "Please refund it.\n________\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/quote_underscores_seven",+ "body": "Please refund it.\n_______\nStill here.",+ "max_chars": 3000,+ "result": "Please refund it.\n_______\nStill here."+ },+ {+ "label": "p0411/quote_from_address",+ "body": "Please refund it.\nFrom: Jane <[email protected]>\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/quote_from_prose_with_at",+ "body": "Please refund it.\nFrom: the desk at 5pm we meet\nStill here.",+ "max_chars": 3000,+ "result": "Please refund it.\nFrom: the desk at 5pm we meet\nStill here."+ },+ {+ "label": "p0411/quote_de_address",+ "body": "Preciso de ajuda.\nDe: Maria <[email protected]>\nOld request.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/quote_de_french_spaced",+ "body": "Je demande de l'aide.\nDe : Marie <[email protected]>\nOld request.",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/quote_de_dates_is_body",+ "body": "Quero folga.\nDe: 10/09 a 15/09\nPara: 20/09\nObrigada.",+ "max_chars": 3000,+ "result": "Quero folga.\nDe: 10/09 a 15/09\nPara: 20/09"+ },+ {+ "label": "p0411/quote_de_name_enviado",+ "body": "Preciso de ajuda.\nDe: Maria Souza\nEnviado: segunda-feira\nOld request.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/quote_de_name_enviada_em",+ "body": "Preciso de ajuda.\nDe: Maria Souza\nEnviada em: segunda\nOld request.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/quote_de_name_fecha",+ "body": "Necesito ayuda.\nDe: Maria Souza\nFecha: 10 de septiembre de 2024\nOld request.",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/quote_de_name_data_no_year",+ "body": "Preciso de ajuda.\nDe: Maria Souza\nData: 10/09\nStill here.",+ "max_chars": 3000,+ "result": "Preciso de ajuda.\nDe: Maria Souza\nData: 10/09\nStill here."+ },+ {+ "label": "p0411/quote_de_name_para_only",+ "body": "Quero folga.\nDe: 10/09\nPara: 15/09\nObrigada.",+ "max_chars": 3000,+ "result": "Quero folga.\nDe: 10/09\nPara: 15/09"+ },+ {+ "label": "p0411/quote_de_french_envoye",+ "body": "Je demande de l'aide.\nDe : Marie Dupont\nEnvoyé : lundi 10 septembre\nOld request.",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/quote_from_name_date",+ "body": "Please refund it.\nFrom: Jane Smith\nDate: Mon, 10 Sep 2018 10:00\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/quote_from_name_last_line",+ "body": "Please refund it.\nFrom: Jane Smith",+ "max_chars": 3000,+ "result": "Please refund it.\nFrom: Jane Smith"+ },+ {+ "label": "p0411/quote_from_name_first_line",+ "body": "From: Jane Smith\nSent: Monday\nOld request.",+ "max_chars": 3000,+ "result": "From: Jane Smith\nSent: Monday\nOld request."+ },+ {+ "label": "p0411/quote_gt_lines_dropped",+ "body": "Please refund it.\n> quoted one\n > quoted two\nStill mine.",+ "max_chars": 3000,+ "result": "Please refund it.\nStill mine."+ },+ {+ "label": "p0411/quote_attribution_tail_en",+ "body": "Please refund it.\nOn Mon, 10/09/2018, Jane Smith\[email protected]> wrote:\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/quote_attribution_tail_pt",+ "body": "Preciso de ajuda.\nEm 10/09/2018, Fulano de Tal\[email protected]> escreveu:\nOld request.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/quote_attribution_tail_es",+ "body": "Necesito ayuda.\nEl 10/09/2018, Fulano de Tal\[email protected]> escribió:\nOld request.",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/quote_attribution_tail_fr",+ "body": "Je demande de l'aide.\nLe 10/09/2018, Fulano\[email protected]> a écrit :\nOld request.",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/quote_attribution_tail_no_head_date",+ "body": "Please refund it.\nOn the other hand\[email protected]> wrote:\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it.\nOn the other hand"+ },+ {+ "label": "p0411/quote_attribution_tail_unrelated_head",+ "body": "Please refund it.\nWith thanks 10/09\[email protected]> wrote:\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it.\nWith thanks 10/09"+ },+ {+ "label": "p0411/quote_attribution_tail_first_line",+ "body": "[email protected]> wrote:\nOld request.",+ "max_chars": 3000,+ "result": "[email protected]> wrote:\nOld request."+ },+ {+ "label": "p0411/quote_attribution_tail_too_long",+ "body": "Please refund it.\nOn 10/09\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@y.com> wrote:\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it.\nOn 10/09\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@y.com> wrote:\nOld request."+ },+ {+ "label": "p0411/quote_crlf",+ "body": "Please refund it.\r\nOn Mon, Jan 1, 2024 Jane wrote:\r\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/quote_lone_cr",+ "body": "Please refund it.\rOn Mon, Jan 1, 2024 Jane wrote:\rOld request.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/quote_literal_backslash_n",+ "body": "Please refund it.\\nOn Mon, Jan 1, 2024 Jane wrote:\\nOld request.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_dashes",+ "body": "Line one.\nLine two.\nLine three.\nLine four.\nLine five.\n--\nJohn Smith\nCEO",+ "max_chars": 3000,+ "result": "Line one.\nLine two.\nLine three.\nLine four.\nLine five."+ },+ {+ "label": "p0411/sig_dashes_trailing_space",+ "body": "Line one.\nLine two.\nLine three.\nLine four.\nLine five.\n-- \nJohn Smith",+ "max_chars": 3000,+ "result": "Line one.\nLine two.\nLine three.\nLine four.\nLine five."+ },+ {+ "label": "p0411/sig_best_regards",+ "body": "Please refund it.\nBest regards,\nJohn Smith",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_kind_regards_name",+ "body": "Please refund it.\nKind regards, John",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_thanks_and_regards",+ "body": "Please refund it.\nThanks and regards, José",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_thanks_ampersand_regards",+ "body": "Please refund it.\nThanks & regards,\nJosé",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_warmest_wishes",+ "body": "Please refund it.\nWarmest wishes, Pat",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_many_thanks_in_advance",+ "body": "Please refund it.\nMany thanks in advance!",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_thank_you_so_much",+ "body": "Please refund it.\nThank you so much, Priya Sharma",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_three_name_tokens",+ "body": "Please refund it.\nThanks, Anna Maria Lopez",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_four_name_tokens_is_prose",+ "body": "Please refund it.\nThanks, Anna Maria Lopez Garcia",+ "max_chars": 3000,+ "result": "Please refund it.\nThanks, Anna Maria Lopez Garcia"+ },+ {+ "label": "p0411/sig_thanks_for_is_prose",+ "body": "Please refund it.\nThanks for the quick reply",+ "max_chars": 3000,+ "result": "Please refund it.\nThanks for the quick reply"+ },+ {+ "label": "p0411/sig_thanks_digits",+ "body": "Please refund it.\nThanks, Anna 2",+ "max_chars": 3000,+ "result": "Please refund it.\nThanks, Anna 2"+ },+ {+ "label": "p0411/sig_apostrophe_hyphen_name",+ "body": "Please refund it.\nRegards, D'Arcy-Smith",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_lowercase_name_is_prose",+ "body": "Please refund it.\nThanks, anna",+ "max_chars": 3000,+ "result": "Please refund it.\nThanks, anna"+ },+ {+ "label": "p0411/sig_caseless_name",+ "body": "Please refund it.\nRegards, 山田",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_cyrillic_name",+ "body": "Please refund it.\nSincerely, Дмитрий",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_name_with_semicolon_is_prose",+ "body": "Please refund it.\nThanks; Anna",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_cheers_bang",+ "body": "Please refund it.\nCheers!!",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/sig_too_long_is_not_cut",+ "body": "Please refund it.\nBest regards, Averyveryverylongnamethatgoes",+ "max_chars": 3000,+ "result": "Please refund it.\nBest regards, Averyveryverylongnamethatgoes"+ },+ {+ "label": "p0411/sig_pt_atenciosamente",+ "body": "Preciso de ajuda.\nAtenciosamente,\nMaria",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/sig_pt_att",+ "body": "Preciso de ajuda.\nAtt.\nMaria",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/sig_pt_abraco",+ "body": "Preciso de ajuda.\nAbraços!\nMaria",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/sig_pt_um_abraco",+ "body": "Preciso de ajuda.\nUm abraço\nMaria",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/sig_pt_obrigado_desde_ja",+ "body": "Preciso de ajuda.\nObrigado desde já!\nMaria",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/sig_pt_muito_obrigada_atencao",+ "body": "Preciso de ajuda.\nMuito obrigada pela atenção.\nMaria",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/sig_pt_obrigado_request_is_prose",+ "body": "Preciso de ajuda.\nObrigado pelo retorno, mas ainda preciso do boleto.",+ "max_chars": 3000,+ "result": "Preciso de ajuda.\nObrigado pelo retorno, mas ainda preciso do boleto."+ },+ {+ "label": "p0411/sig_pt_cumprimentos",+ "body": "Preciso de ajuda.\nCom os melhores cumprimentos,\nMaria",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/sig_es_saludos_cordiales",+ "body": "Necesito ayuda.\nSaludos cordiales,\nJuan",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/sig_es_un_saludo",+ "body": "Necesito ayuda.\nUn saludo\nJuan",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/sig_es_gracias_de_antemano",+ "body": "Necesito ayuda.\nMuchas gracias de antemano.\nJuan",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/sig_es_atentamente",+ "body": "Necesito ayuda.\nAtentamente,\nJuan",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/sig_fr_cordialement",+ "body": "Je demande de l'aide.\nCordialement,\nMarie",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/sig_fr_bien_cordialement",+ "body": "Je demande de l'aide.\nBien cordialement\nMarie",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/sig_fr_salutations_distinguees",+ "body": "Je demande de l'aide.\nSalutations distinguées,\nMarie",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/sig_fr_bien_a_vous",+ "body": "Je demande de l'aide.\nBien à vous\nMarie",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/sig_fr_merci_d_avance",+ "body": "Je demande de l'aide.\nMerci d'avance !\nMarie",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/sig_fr_bonne_journee",+ "body": "Je demande de l'aide.\nBonne journée.\nMarie",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/sig_fr_merci_request_is_prose",+ "body": "Je demande de l'aide.\nMerci pour votre aide, mais il reste un problème.",+ "max_chars": 3000,+ "result": "Je demande de l'aide.\nMerci pour votre aide, mais il reste un problème."+ },+ {+ "label": "p0411/sig_scan_starts_late",+ "body": "Cheers,\nJohn\nBody line 0.\nBody line 1.\nBody line 2.\nBody line 3.\nBody line 4.\nBody line 5.\nBody line 6.\nBody line 7.\nBody line 8.\nBody line 9.\nBody line 10.\nBody line 11.",+ "max_chars": 3000,+ "result": "Cheers,\nJohn\nBody line 0.\nBody line 1.\nBody line 2.\nBody line 3.\nBody line 4.\nBody line 5.\nBody line 6.\nBody line 7.\nBody line 8.\nBody line 9.\nBody line 10.\nBody line 11."+ },+ {+ "label": "p0411/sig_scan_window_ten_lines",+ "body": "Body line 0.\nBody line 1.\nBody line 2.\nBody line 3.\nBody line 4.\nThanks,\nMore 0.\nMore 1.\nMore 2.\nMore 3.",+ "max_chars": 3000,+ "result": "Body line 0.\nBody line 1.\nBody line 2.\nBody line 3.\nBody line 4."+ },+ {+ "label": "p0411/sig_in_short_body_first_line_never_cut",+ "body": "Thanks,\nPlease refund it.",+ "max_chars": 3000,+ "result": "Thanks,\nPlease refund it."+ },+ {+ "label": "p0411/sig_longer_than_forty_is_not_cut",+ "body": "Line one.\nLine two.\nLine three.\nLine four.\nLine five.\nRegards, Someone With A Very Long Name Indeed Yes",+ "max_chars": 3000,+ "result": "Line one.\nLine two.\nLine three.\nLine four.\nLine five.\nRegards, Someone With A Very Long Name Indeed Yes"+ },+ {+ "label": "p0411/device_sent_from_iphone",+ "body": "Please refund it.\nSent from my iPhone",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/device_sent_from_galaxy",+ "body": "Please refund it.\nSent from my Samsung Galaxy S21 Ultra 5G.",+ "max_chars": 3000,+ "result": "Please refund it.\nSent from my Samsung Galaxy S21 Ultra 5G."+ },+ {+ "label": "p0411/device_sent_from_using",+ "body": "Please refund it.\nSent from Outlook using mail-client_1.2",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/device_samsung_pt",+ "body": "Preciso de ajuda.\nEnviado do meu smartphone Samsung Galaxy.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/device_pt_celular_request",+ "body": "Preciso de ajuda.\nEnviado do meu celular o comprovante ontem.",+ "max_chars": 3000,+ "result": "Preciso de ajuda.\nEnviado do meu celular o comprovante ontem."+ },+ {+ "label": "p0411/device_get_outlook_ios",+ "body": "Please refund it.\nGet Outlook for iOS",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/device_obter_outlook_android",+ "body": "Preciso de ajuda.\nObter o Outlook para Android",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/device_fr_iphone",+ "body": "Je demande de l'aide.\nEnvoyé depuis mon iPhone",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/device_fr_de_ipad",+ "body": "Je demande de l'aide.\nEnvoyé de mon iPad.",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/device_fr_mentioned_in_prose",+ "body": "Bonjour,\nEnvoyé depuis mon iPhone par erreur.\nMerci de rembourser la facture.",+ "max_chars": 3000,+ "result": "Bonjour,\nEnvoyé depuis mon iPhone par erreur.\nMerci de rembourser la facture."+ },+ {+ "label": "p0411/device_es_movil",+ "body": "Necesito ayuda.\nEnviado desde mi móvil",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/device_pt_mi",+ "body": "Preciso de ajuda.\nEnviado via mi Android",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/device_longer_than_sixty",+ "body": "Please refund it.\nSent from my iPhone phone phone phone phone phone phone phone phone phone phone phone phone ",+ "max_chars": 3000,+ "result": "Please refund it.\nSent from my iPhone phone phone phone phone phone phone phone phone phone phone phone phone"+ },+ {+ "label": "p0411/disclaimer_en_standard",+ "body": "Please refund it.\n\nThis email is confidential and intended solely for the named addressee.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/disclaimer_en_wrapped",+ "body": "Please refund it.\n\nThis email and its contents are\nconfidential and intended solely for the addressee.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/disclaimer_en_privileged",+ "body": "Please refund it.\n\nThis information is confidential and may be privileged.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/disclaimer_en_received_in_error",+ "body": "Please refund it.\n\nIf you have received this email in error please delete it.",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/disclaimer_en_question_is_body",+ "body": "Is this confidential?",+ "max_chars": 3000,+ "result": "Is this confidential?"+ },+ {+ "label": "p0411/disclaimer_en_fused_lines",+ "body": "My account is locked\nThis email is confidential and intended solely for the addressee.",+ "max_chars": 3000,+ "result": "My account is locked"+ },+ {+ "label": "p0411/disclaimer_en_inline",+ "body": "My account is locked. This email is confidential and intended solely for the addressee. Please help me.",+ "max_chars": 3000,+ "result": "My account is locked. Please help me."+ },+ {+ "label": "p0411/disclaimer_pt",+ "body": "Preciso de ajuda.\n\nEsta mensagem é confidencial e destinada exclusivamente ao destinatário.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/disclaimer_pt_request_confidencial",+ "body": "Preciso do contrato confidencial.",+ "max_chars": 3000,+ "result": "Preciso do contrato confidencial."+ },+ {+ "label": "p0411/disclaimer_pt_print",+ "body": "Preciso de ajuda.\n\nAntes de imprimir, pense no meio ambiente.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/disclaimer_pt_print_request",+ "body": "Antes de imprimir o boleto, confira o valor.",+ "max_chars": 3000,+ "result": "Antes de imprimir o boleto, confira o valor."+ },+ {+ "label": "p0411/disclaimer_pt_ambiente_first",+ "body": "Preciso de ajuda.\n\nPense no meio ambiente antes de imprimir.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/disclaimer_pt_recebeu_engano",+ "body": "Preciso de ajuda.\n\nSe você recebeu esta mensagem por engano, apague-a.",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "p0411/disclaimer_es",+ "body": "Necesito ayuda.\n\nEste mensaje es confidencial y de uso exclusivo del destinatario.",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/disclaimer_es_recibido_error",+ "body": "Necesito ayuda.\n\nSi ha recibido este mensaje por error, bórrelo.",+ "max_chars": 3000,+ "result": "Necesito ayuda."+ },+ {+ "label": "p0411/disclaimer_fr",+ "body": "Je demande de l'aide.\n\nCe message est confidentiel et réservé au destinataire.",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/disclaimer_fr_recu_erreur",+ "body": "Je demande de l'aide.\n\nSi vous avez reçu ce message par erreur, supprimez-le.",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/disclaimer_fr_request_confidentiel",+ "body": "Je veux le contrat confidentiel.",+ "max_chars": 3000,+ "result": "Je veux le contrat confidentiel."+ },+ {+ "label": "p0411/disclaimer_fr_usage_exclusif",+ "body": "Je demande de l'aide.\n\nUsage exclusif du destinataire.",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "p0411/disclaimer_mixed_paragraph",+ "body": "Please help me. The email is confidential and intended solely for you. Thanks!",+ "max_chars": 3000,+ "result": "Please help me. Thanks!"+ },+ {+ "label": "p0411/disclaimer_uppercase_split",+ "body": "Locked out\nThis email is confidential and intended solely for you.\nMore text.",+ "max_chars": 3000,+ "result": "Locked out More text."+ },+ {+ "label": "p0411/disclaimer_caseless_wrapped",+ "body": "退款请求\n\n这封邮件是机密的 This email is\nconfidential and intended solely for you.",+ "max_chars": 3000,+ "result": "退款请求"+ },+ {+ "label": "p0411/disclaimer_sentence_split_ws",+ "body": "Hello there. This email is confidential and intended solely for you. Bye!",+ "max_chars": 3000,+ "result": "Hello there. Bye!"+ },+ {+ "label": "p0411/whitespace_collapse",+ "body": "Please refund\t\tit.\n\n\n\nThanks a lot",+ "max_chars": 3000,+ "result": "Please refund it."+ },+ {+ "label": "p0411/empty_body",+ "body": "",+ "max_chars": 3000,+ "result": ""+ },+ {+ "label": "p0411/only_quoted",+ "body": "> a\n> b",+ "max_chars": 3000,+ "result": ""+ },+ {+ "label": "p0411/only_whitespace",+ "body": " \n\t\n ",+ "max_chars": 3000,+ "result": ""+ },+ {+ "label": "p0411/unit_separator_blank_line",+ "body": "First\n\u001f\nSecond",+ "max_chars": 3000,+ "result": "First\n\nSecond"+ },+ {+ "label": "p0411/unicode_whitespace_line",+ "body": "First\n \nSecond\n \n\nThird",+ "max_chars": 3000,+ "result": "First\n\nSecond\n\nThird"+ },+ {+ "label": "pyre/boundary_after_no_superscript",+ "body": "Please help me.\n\nThis email² is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me.\n\nThis email² is confidential and intended solely for you."+ },+ {+ "label": "pyre/boundary_before_no_superscript",+ "body": "Please help me.\n\nThis ²email is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me.\n\nThis ²email is confidential and intended solely for you."+ },+ {+ "label": "pyre/boundary_after_combining_mark",+ "body": "Please help me.\n\nThis emaiĺ is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/boundary_before_combining_mark",+ "body": "Please help me.\n\nThis ́email is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/boundary_after_connector_punctuation",+ "body": "Please help me.\n\nThis email‿ is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/boundary_after_zero_width_joiner",+ "body": "Please help me.\n\nThis email is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/boundary_after_circled_letter",+ "body": "Please help me.\n\nThis emailⒶ is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/boundary_after_roman_numeral",+ "body": "Preciso de ajuda.\n\nEsta mensagemⅠ é confidencial.",+ "max_chars": 3000,+ "result": "Preciso de ajuda.\n\nEsta mensagemⅠ é confidencial."+ },+ {+ "label": "pyre/ignorecase_dotted_capital_i_disclaimer",+ "body": "Please help me.\n\nİf you received this email in error, delete it.",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/ignorecase_dotless_i_disclaimer",+ "body": "Please help me.\n\nIf you received this email ın error, delete it.",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/ignorecase_dotless_i_device_footer",+ "body": "Please help me.\nSent from my ıPhone",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/ignorecase_dotted_i_device_range",+ "body": "Please help me.\nSent from my iPhone using İOS",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/ignorecase_dotless_i_signoff",+ "body": "Please help me.\nKınd regards,\nJohn",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/ignorecase_dotless_i_pt_signoff",+ "body": "Preciso de ajuda.\nCordıalmente,\nMaria",+ "max_chars": 3000,+ "result": "Preciso de ajuda."+ },+ {+ "label": "pyre/ignorecase_dotted_i_original_message",+ "body": "Please help me.\n-----Orİginal Message-----\nOld request.",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/ignorecase_dotless_i_french_signoff",+ "body": "Je demande de l'aide.\nBonne journée,\nMarıe",+ "max_chars": 3000,+ "result": "Je demande de l'aide."+ },+ {+ "label": "pyre/ignorecase_long_s_and_kelvin_sign",+ "body": "Please help me.\n\nThis email is confidential and intended ſolely for you, thankK you.",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/astral_boundary_after_letter",+ "body": "Please help me.\n\nThis email𝐀 is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me.\n\nThis email𝐀 is confidential and intended solely for you."+ },+ {+ "label": "pyre/astral_boundary_before_letter",+ "body": "Please help me.\n\nThis 𝐀email is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me.\n\nThis 𝐀email is confidential and intended solely for you."+ },+ {+ "label": "pyre/astral_boundary_after_emoji",+ "body": "Please help me.\n\nThis email😀 is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/astral_dot_run_within_bound",+ "body": "Please help me.\n\nThis email 𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀 is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/astral_dot_run_past_bound",+ "body": "Please help me.\n\nThis email 𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀 is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me.\n\nThis email 𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀 is confidential and intended solely for you."+ },+ {+ "label": "pyre/astral_quote_header_within_bound",+ "body": "Please help me.\nOn Monday 1 Jan, 𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀 wrote:\n> old request",+ "max_chars": 3000,+ "result": "Please help me."+ },+ {+ "label": "pyre/astral_quote_header_past_bound",+ "body": "Please help me.\nOn Monday 1 Jan, 𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀 wrote:\n> old request\nStill mine.",+ "max_chars": 3000,+ "result": "Please help me.\nOn Monday 1 Jan, 𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀𝐀 wrote:\nStill mine."+ },+ {+ "label": "pyre/astral_digit_boundary",+ "body": "Please help me.\n\nThis email𝟎 is confidential and intended solely for you.",+ "max_chars": 3000,+ "result": "Please help me.\n\nThis email𝟎 is confidential and intended solely for you."}],"state": [@@ -528,6 +1554,213 @@"subject": "सहायता चाहिए","body": "मुझसे दो बार शुल्क लिया गया, कृपया रिफंड करें।"}+ },+ {+ "label": "p0411/max_chars_extra_small",+ "subject": "Locked out",+ "body": "My account is locked and I need help right now.",+ "sender": null,+ "clean": true,+ "extra": {+ "max_chars": 10+ },+ "result": {+ "subject": "Locked out",+ "body": "My account"+ }+ },+ {+ "label": "p0411/max_chars_extra_zero",+ "subject": "Locked out",+ "body": "My account is locked.",+ "sender": null,+ "clean": true,+ "extra": {+ "max_chars": 0+ },+ "result": {+ "subject": "Locked out",+ "body": ""+ }+ },+ {+ "label": "p0411/max_chars_extra_negative",+ "subject": "Locked out",+ "body": "My account is locked and I need help right now.",+ "sender": null,+ "clean": true,+ "extra": {+ "max_chars": -5+ },+ "result": {+ "subject": "Locked out",+ "body": "My account is locked "+ }+ },+ {+ "label": "p0411/max_chars_extra_large",+ "subject": "Locked out",+ "body": "My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. ",+ "sender": null,+ "clean": true,+ "extra": {+ "max_chars": 100000+ },+ "result": {+ "subject": "Locked out",+ "body": "My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked."+ }+ },+ {+ "label": "p0411/max_chars_default_cut",+ "subject": "Locked out",+ "body": "My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. ",+ "sender": null,+ "clean": true,+ "extra": {},+ "result": {+ "subject": "Locked out",+ "body": "My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My account is locked. My accou"+ }+ },+ {+ "label": "p0411/max_chars_ignored_when_not_clean",+ "subject": "Locked out",+ "body": "My account is locked and I need help right now.",+ "sender": null,+ "clean": false,+ "extra": {+ "max_chars": 5+ },+ "result": {+ "subject": "Locked out",+ "body": "My account is locked and I need help right now."+ }+ },+ {+ "label": "p0411/max_chars_with_other_extra",+ "subject": "Locked out",+ "body": "My account is locked and I need help.",+ "sender": "[email protected]",+ "clean": true,+ "extra": {+ "max_chars": 7,+ "priority": "high"+ },+ "result": {+ "subject": "Locked out",+ "body": "My acco",+ "from": "[email protected]",+ "priority": "high"+ }+ },+ {+ "label": "p0411/sender_empty_dropped",+ "subject": "Locked out",+ "body": "My account is locked.",+ "sender": "",+ "clean": true,+ "extra": {},+ "result": {+ "subject": "Locked out",+ "body": "My account is locked."+ }+ },+ {+ "label": "p0411/extra_overrides_from",+ "subject": "Locked out",+ "body": "My account is locked.",+ "sender": "[email protected]",+ "clean": true,+ "extra": {+ "from": "[email protected]"+ },+ "result": {+ "subject": "Locked out",+ "body": "My account is locked.",+ "from": "[email protected]"+ }+ },+ {+ "label": "p0411/extra_non_string_values",+ "subject": "Locked out",+ "body": "My account is locked.",+ "sender": null,+ "clean": true,+ "extra": {+ "n": 1,+ "f": 1.5,+ "b": false,+ "l": [+ 1,+ "x"+ ],+ "d": {+ "k": "v"+ }+ },+ "result": {+ "subject": "Locked out",+ "body": "My account is locked.",+ "n": 1,+ "f": 1.5,+ "b": false,+ "l": [+ 1,+ "x"+ ],+ "d": {+ "k": "v"+ }+ }+ },+ {+ "label": "p0411/subject_none_like_empty",+ "subject": "",+ "body": "My account is locked.",+ "sender": null,+ "clean": true,+ "extra": {},+ "result": {+ "subject": "",+ "body": "My account is locked."+ }+ },+ {+ "label": "p0411/subject_unicode_whitespace",+ "subject": " Locked out ",+ "body": "My account is locked.",+ "sender": null,+ "clean": true,+ "extra": {},+ "result": {+ "subject": "Locked out",+ "body": "My account is locked."+ }+ },+ {+ "label": "p0411/state_quote_cleaned",+ "subject": "Locked out",+ "body": "Please unlock it.\nOn Mon, Jan 1, 2024 Jane wrote:\nOld text.",+ "sender": null,+ "clean": true,+ "extra": {},+ "result": {+ "subject": "Locked out",+ "body": "Please unlock it."+ }+ },+ {+ "label": "p0411/state_unclean_keeps_quote",+ "subject": "Locked out",+ "body": "Please unlock it.\nOn Mon, Jan 1, 2024 Jane wrote:\nOld text.",+ "sender": null,+ "clean": false,+ "extra": {},+ "result": {+ "subject": "Locked out",+ "body": "Please unlock it.\nOn Mon, Jan 1, 2024 Jane wrote:\nOld text."+ }}]}diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/lang_probe.json b/laya-dotnet/tests/Laya.Tests/golden/routing/lang_probe.jsonindex 39a5351a..088d5e19 100644--- a/laya-dotnet/tests/Laya.Tests/golden/routing/lang_probe.json+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/lang_probe.json@@ -4243,5 +4243,2581 @@"non_latin_fraction": 0.0,"mixed_segment": null}+ },+ {+ "label": "p0411/swedish_sentence",+ "state": "Hej, jag behöver hjälp med min faktura",+ "state_text": "Hej, jag behöver hjälp med min faktura",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "sv",+ "english_hits": 0,+ "diacritic_rate": 0.05263157894736842,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "sv",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0526,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/swedish_ascii",+ "state": "jag behover hjalp med min faktura",+ "state_text": "jag behover hjalp med min faktura",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "sv",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "sv",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/swedish_two_words",+ "state": "lösenord fel",+ "state_text": "lösenord fel",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "sv",+ "english_hits": 0,+ "diacritic_rate": 0.08333333333333333,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "sv",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0833,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/swedish_one_word",+ "state": "lösenord",+ "state_text": "lösenord",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.125,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.125,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/swedish_inte_kan",+ "state": "kan inte logga in på appen idag",+ "state_text": "kan inte logga in på appen idag",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "sv",+ "english_hits": 1,+ "diacritic_rate": 0.03225806451612903,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "sv",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0323,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/swedish_inte_kan_not_first",+ "state": "jag vet att kan inte logga in",+ "state_text": "jag vet att kan inte logga in",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "sv",+ "english_hits": 1,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "sv",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/nordic_overlap_tack",+ "state": "tack så mycket",+ "state_text": "tack så mycket",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.07142857142857142,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0714,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/nordic_overlap_with_english",+ "state": "hej please send me the invoice",+ "state_text": "hej please send me the invoice",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 2,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/danish_sentence",+ "state": "jeg har brug for hjælp til min faktura",+ "state_text": "jeg har brug for hjælp til min faktura",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "sv",+ "english_hits": 1,+ "diacritic_rate": 0.02631578947368421,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "sv",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0263,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/norwegian_sentence",+ "state": "jeg trenger hjelp med fakturaen min",+ "state_text": "jeg trenger hjelp med fakturaen min",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/rescue_loanword",+ "state": "Please send the café menu to the customer today",+ "state_text": "Please send the café menu to the customer today",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 4,+ "diacritic_rate": 0.02127659574468085,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0213,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/rescue_two_accents",+ "state": "Please send the café and the résumé to the customer today",+ "state_text": "Please send the café and the résumé to the customer today",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 6,+ "diacritic_rate": 0.05263157894736842,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0526,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/rescue_vetoed_by_rate",+ "state": "ñandú ñandú ñandú the please",+ "state_text": "ñandú ñandú ñandú the please",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 2,+ "diacritic_rate": 0.21428571428571427,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.2143,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/rescue_name",+ "state": "Please forward this to José and Zoë at the office",+ "state_text": "Please forward this to José and Zoë at the office",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 6,+ "diacritic_rate": 0.04081632653061224,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0408,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/rescue_one_english_word",+ "state": "Der Kunde hat nicht please",+ "state_text": "Der Kunde hat nicht please",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 1,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/modifier_letters",+ "state": "ʰʲʷ hello there",+ "state_text": "ʰʲʷ hello there",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.7692307692307693,+ "other": 0.23076923076923078+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 0.7692307692307693,+ "other": 0.23076923076923078+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.2308,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/modifier_only",+ "state": "ʰʲʷ",+ "state_text": "ʰʲʷ",+ "detect_script": "other",+ "script_profile": {+ "other": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "other",+ "script_profile": {+ "other": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 1.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/ipa_only",+ "state": "ɐɑɒɓ",+ "state_text": "ɐɑɒɓ",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/fullwidth_only",+ "state": "HELLO world",+ "state_text": "HELLO world",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/bopomofo_only",+ "state": "ㄅㄆㄇㄈ",+ "state_text": "ㄅㄆㄇㄈ",+ "detect_script": "other",+ "script_profile": {+ "other": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "other",+ "script_profile": {+ "other": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 1.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/halfwidth_katakana",+ "state": "アイウエオ",+ "state_text": "アイウエオ",+ "detect_script": "other",+ "script_profile": {+ "other": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "other",+ "script_profile": {+ "other": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 1.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/cjk_extension_b",+ "state": "𠀀𠀁𠀂",+ "state_text": "𠀀𠀁𠀂",+ "detect_script": "other",+ "script_profile": {+ "other": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "other",+ "script_profile": {+ "other": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 1.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/unclaimed_with_latin",+ "state": "please help ㄅㄆㄇㄈㄉㄊ today",+ "state_text": "please help ㄅㄆㄇㄈㄉㄊ today",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.7142857142857143,+ "other": 0.2857142857142857+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 1,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 0.7142857142857143,+ "other": 0.2857142857142857+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.2857,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/unclaimed_dominant",+ "state": "ok ㄅㄆㄇㄈㄉㄊㄋㄌㄍㄎ",+ "state_text": "ok ㄅㄆㄇㄈㄉㄊㄋㄌㄍㄎ",+ "detect_script": "other",+ "script_profile": {+ "latin": 0.16666666666666666,+ "other": 0.8333333333333334+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "other",+ "script_profile": {+ "latin": 0.16666666666666666,+ "other": 0.8333333333333334+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.8333,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/tie_named_vs_latin",+ "state": "abc αβγ",+ "state_text": "abc αβγ",+ "detect_script": "greek",+ "script_profile": {+ "latin": 0.5,+ "greek": 0.5+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "greek",+ "script_profile": {+ "latin": 0.5,+ "greek": 0.5+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.5,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/tie_two_scripts",+ "state": "αβγ абв",+ "state_text": "αβγ абв",+ "detect_script": "greek",+ "script_profile": {+ "greek": 0.5,+ "cyrillic": 0.5+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "greek",+ "script_profile": {+ "greek": 0.5,+ "cyrillic": 0.5+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 1.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/devanagari_matras",+ "state": "please call मु मु मु मु मु मु now",+ "state_text": "please call मु मु मु मु मु मु now",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.6842105263157895,+ "devanagari": 0.3157894736842105+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 1,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "devanagari",+ "script_profile": {+ "latin": 0.6842105263157895,+ "devanagari": 0.3157894736842105+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.3158,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/devanagari_single_letters",+ "state": "Please call क ख ग घ ङ च छ ज झ ञ now",+ "state_text": "Please call क ख ग घ ङ च छ ज झ ञ now",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.5652173913043478,+ "devanagari": 0.43478260869565216+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 1,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 0.5652173913043478,+ "devanagari": 0.43478260869565216+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.4348,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/devanagari_danda",+ "state": "Please read this ।।।।।।।।।।।।",+ "state_text": "Please read this ।।।।।।।।।।।।",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/cyrillic_combining_name",+ "state": "Please send it to Влади́мир Петро́вич today",+ "state_text": "Please send it to Влади́мир Петро́вич today",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.5428571428571428,+ "cyrillic": 0.45714285714285713+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 3,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 0.5428571428571428,+ "cyrillic": 0.45714285714285713+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.4571,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/cyrillic_lower_word",+ "state": "Please send it to привет мир today ok",+ "state_text": "Please send it to привет мир today ok",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.7,+ "cyrillic": 0.3+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 3,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "cyrillic",+ "script_profile": {+ "latin": 0.7,+ "cyrillic": 0.3+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.3,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/arabic_harakat",+ "state": "Please read مَرْحَبًا بِكُمْ في الموقع today",+ "state_text": "Please read مَرْحَبًا بِكُمْ في الموقع today",+ "detect_script": "arabic",+ "script_profile": {+ "latin": 0.4838709677419355,+ "arabic": 0.5161290322580645+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 1,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "arabic",+ "script_profile": {+ "latin": 0.4838709677419355,+ "arabic": 0.5161290322580645+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.5161,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/thai_tone_marks",+ "state": "Please read สวัสดีครับ ขอบคุณมาก today",+ "state_text": "Please read สวัสดีครับ ขอบคุณมาก today",+ "detect_script": "thai",+ "script_profile": {+ "latin": 0.5,+ "thai": 0.5+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 1,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "thai",+ "script_profile": {+ "latin": 0.5,+ "thai": 0.5+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.5,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/vietnamese_combining",+ "state": "Xin chào, tôi can̂ giúp đỡ về hoá đơn",+ "state_text": "Xin chào, tôi can̂ giúp đỡ về hoá đơn",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 3,+ "diacritic_rate": 0.10256410256410256,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.1026,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/greek_final_sigma",+ "state": "Please check ΟΔΥΣΣΕΥΣ today",+ "state_text": "Please check ΟΔΥΣΣΕΥΣ today",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.6666666666666666,+ "greek": 0.3333333333333333+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 1,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 0.6666666666666666,+ "greek": 0.3333333333333333+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.3333,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/non_latin_promotion_min_letters",+ "state": "ok xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ありがとうございます",+ "state_text": "ok xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ありがとうございます",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.8913043478260869,+ "kana": 0.10869565217391304+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "kana",+ "script_profile": {+ "latin": 0.8913043478260869,+ "kana": 0.10869565217391304+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.1087,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/non_latin_fraction_edge",+ "state": "Please яя aaaaaaaa",+ "state_text": "Please яя aaaaaaaa",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.875,+ "cyrillic": 0.125+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 0.875,+ "cyrillic": 0.125+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.125,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/identifier_domains",+ "state": "github.com acme.com v1.2.3 la o e un [email protected]",+ "state_text": "github.com acme.com v1.2.3 la o e un [email protected]",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/identifier_trailing_dot",+ "state": "arrivato. la fattura. non e arrivata ancora.",+ "state_text": "arrivato. la fattura. non e arrivata ancora.",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "it",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "it",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/identifier_double_dot",+ "state": "a..b la fattura non arrivata",+ "state_text": "a..b la fattura non arrivata",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "it",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "it",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/identifier_leading_at",+ "state": "@home la fattura non e arrivata ancora",+ "state_text": "@home la fattura non e arrivata ancora",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "it",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "it",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/identifier_hyphen",+ "state": "e-mail la fattura non e arrivata ancora x-y.z",+ "state_text": "e-mail la fattura non e arrivata ancora x-y.z",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "it",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "it",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/dotted_I",+ "state": "İstanbul'da bir sorun var ve yardım istiyorum",+ "state_text": "İstanbul'da bir sorun var ve yardım istiyorum",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "az",+ "english_hits": 0,+ "diacritic_rate": 0.021739130434782608,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "az",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0217,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/dotted_I_english",+ "state": "I am in İstanbul and please send the receipt",+ "state_text": "I am in İstanbul and please send the receipt",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 5,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/code_line_state",+ "state": "x = 5 el la los las para que no",+ "state_text": "x = 5 el la los las para que no",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "es",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "es",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/call_syntax_state",+ "state": "foo(bar) para que no el la los",+ "state_text": "foo(bar) para que no el la los",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "es",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "es",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/titlecase_digraph",+ "state": "Džungla and the please are here",+ "state_text": "Džungla and the please are here",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 4,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/shouted_acronyms",+ "state": "MON DES EST LA",+ "state_text": "MON DES EST LA",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "fr",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/shouted_acronyms_under_english",+ "state": "Please look at this\nMON DES EST LA",+ "state_text": "Please look at this\nMON DES EST LA",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 3,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/shouted_acronyms_two_english_lines",+ "state": "Please look at this\nand send me the details.\nMON DES EST LA",+ "state_text": "Please look at this\nand send me the details.\nMON DES EST LA",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 5,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/shouted_address",+ "state": "STORES LOS ANGELES LAS VEGAS EL PASO CLOSED",+ "state_text": "STORES LOS ANGELES LAS VEGAS EL PASO CLOSED",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "es",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/shouted_german_umlaut",+ "state": "WIE SPÄT IST ES IN KÖLN",+ "state_text": "WIE SPÄT IST ES IN KÖLN",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "de",+ "english_hits": 1,+ "diacritic_rate": 0.08695652173913043,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.087,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/shouted_german_ascii",+ "state": "WIE SPAET IST ES IN KOELN",+ "state_text": "WIE SPAET IST ES IN KOELN",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "de",+ "english_hits": 1,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/shouted_portuguese_prose",+ "state": "EU NAO CONSIGO ENTRAR NA MINHA CONTA",+ "state_text": "EU NAO CONSIGO ENTRAR NA MINHA CONTA",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "pt",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/shouted_german_prose",+ "state": "ICH HABE MEIN PAKET NOCH NICHT BEKOMMEN",+ "state_text": "ICH HABE MEIN PAKET NOCH NICHT BEKOMMEN",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "de",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "de",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/shouted_french_prose",+ "state": "JE VOUDRAIS UN REMBOURSEMENT POUR LA COMMANDE",+ "state_text": "JE VOUDRAIS UN REMBOURSEMENT POUR LA COMMANDE",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "fr",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "fr",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/shouted_cyrillic_acronyms",+ "state": "ПРИВЕТ MON DES EST LA",+ "state_text": "ПРИВЕТ MON DES EST LA",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.6470588235294118,+ "cyrillic": 0.35294117647058826+ },+ "latin_profile": {+ "language": "fr",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 0.6470588235294118,+ "cyrillic": 0.35294117647058826+ },+ "language": "fr",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.3529,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/emphasis_german",+ "state": "sag mir das HEUTIGE DATUM",+ "state_text": "sag mir das HEUTIGE DATUM",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "de",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "de",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/emphasis_spanish",+ "state": "quiero cancelar mi PEDIDO POR FAVOR",+ "state_text": "quiero cancelar mi PEDIDO POR FAVOR",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "es",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "es",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/emphasis_acronym_mixed",+ "state": "quero cancelar o pedido MON DES EST LA por favor",+ "state_text": "quero cancelar o pedido MON DES EST LA por favor",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "fr",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/emphasis_long_run",+ "state": "please cancel the order and send a REFUND for it",+ "state_text": "please cancel the order and send a REFUND for it",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 5,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/caseless_with_caps",+ "state": "ありがとう MON DES EST LA",+ "state_text": "ありがとう MON DES EST LA",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.6875,+ "kana": 0.3125+ },+ "latin_profile": {+ "language": "fr",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "kana",+ "script_profile": {+ "latin": 0.6875,+ "kana": 0.3125+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.3125,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/hindi_caps",+ "state": "मुझे MON DES EST LA चाहिए",+ "state_text": "मुझे MON DES EST LA चाहिए",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.6875,+ "devanagari": 0.3125+ },+ "latin_profile": {+ "language": "fr",+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "devanagari",+ "script_profile": {+ "latin": 0.6875,+ "devanagari": 0.3125+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.3125,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/shouted_english",+ "state": "PLEASE SEND ME THE RECEIPT FOR THE ORDER",+ "state_text": "PLEASE SEND ME THE RECEIPT FOR THE ORDER",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 4,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/segment_after_cap",+ "state": "Please review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nEu nao consigo entrar na minha conta.",+ "state_text": "Please review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and sen",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 327,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/segment_within_cap",+ "state": "Please review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nEu nao consigo entrar na minha conta.",+ "state_text": "Please review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nPlease review the error and send me the details.\nEu nao consigo entrar na minha conta.",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 40,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "Eu nao consigo entrar na minha conta."+ }+ },+ {+ "label": "p0411/segment_shouted_foreign",+ "state": "Please review the error and send me the details.\nEU NAO CONSIGO ENTRAR NA MINHA CONTA",+ "state_text": "Please review the error and send me the details.\nEU NAO CONSIGO ENTRAR NA MINHA CONTA",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 4,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "EU NAO CONSIGO ENTRAR NA MINHA CONTA"+ }+ },+ {+ "label": "p0411/segment_acronym_line",+ "state": "Please review the error and send me the details.\nMON DES EST LA",+ "state_text": "Please review the error and send me the details.\nMON DES EST LA",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 4,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/segment_joined_tokens",+ "state": "Please review the error and send me the details.\nNav/Com OS/2 ESA/UN C:\\DOS\\mode radio listing",+ "state_text": "Please review the error and send me the details.\nNav/Com OS/2 ESA/UN C:\\DOS\\mode radio listing",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 4,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/segment_two_same_word",+ "state": "Please review the error and send me the details.\ncom com com com com com",+ "state_text": "Please review the error and send me the details.\ncom com com com com com",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "pt",+ "english_hits": 4,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/segment_call_syntax",+ "state": "Please review the error and send me the details.\nround(el, 2) para que no tengo el",+ "state_text": "Please review the error and send me the details.\nround(el, 2) para que no tengo el",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 4,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/segment_list_state",+ "state": [+ "Please review the error and send me the details.",+ "Quiero cancelar mi pedido por favor ahora mismo"+ ],+ "state_text": "Please review the error and send me the details. Quiero cancelar mi pedido por favor ahora mismo",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 4,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "es",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "Quiero cancelar mi pedido por favor ahora mismo"+ }+ },+ {+ "label": "p0411/segment_nested_dict",+ "state": {+ "ticket": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "customer": {+ "msg": "ich habe mein paket noch nicht bekommen"+ }+ }+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ich habe mein paket noch nicht bekommen",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 12,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "de",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "ich habe mein paket noch nicht bekommen"+ }+ },+ {+ "label": "p0411/segment_too_deep",+ "state": {+ "a": {+ "b": {+ "c": {+ "d": {+ "e": {+ "f": {+ "g": "ich habe mein paket noch nicht bekommen"+ }+ }+ }+ }+ }+ },+ "n": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. "+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 12,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/segment_unicode_strip",+ "state": "Please review the error and send me the details.\n Eu nao consigo entrar na minha conta. ",+ "state_text": "Please review the error and send me the details.\n Eu nao consigo entrar na minha conta. ",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 4,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "Eu nao consigo entrar na minha conta."+ }+ },+ {+ "label": "p0411/leaf_name_field",+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "name": "José"+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. José",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 20,+ "diacritic_rate": 0.004,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.004,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/leaf_short_value",+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "x": "žluť"+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. žluť",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 20,+ "diacritic_rate": 0.008,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.008,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/leaf_nonlatin_short",+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "x": "你好吗"+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. 你好吗",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.9848484848484849,+ "han": 0.015151515151515152+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 20,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 0.9848484848484849,+ "han": 0.015151515151515152+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0152,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/leaf_nonlatin_ten",+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "x": "我的账户无法登录请帮助我吧"+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. 我的账户无法登录请帮助我吧",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.9375,+ "han": 0.0625+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 20,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 0.9375,+ "han": 0.0625+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0625,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/leaf_cyrillic_name",+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "x": "Дмитрий Петрович Савицкий"+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Дмитрий Петрович Савицкий",+ "detect_script": "latin",+ "script_profile": {+ "latin": 0.8944954128440367,+ "cyrillic": 0.10550458715596331+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 20,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 0.8944954128440367,+ "cyrillic": 0.10550458715596331+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.1055,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/leaf_code_value",+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "x": "x = żółć żółć żółć żółć"+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. x = żółć żółć żółć żółć",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 20,+ "diacritic_rate": 0.05947955390334572,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0595,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/leaf_diacritics_four_words",+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "x": "żółć gęślą jaźń łódź"+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. żółć gęślą jaźń łódź",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 20,+ "diacritic_rate": 0.045112781954887216,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0451,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/leaf_two_candidates",+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "a": "Quiero cancelar mi pedido por favor",+ "b": "Ich habe mein Paket noch nicht bekommen und möchte es zurück"+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Quiero cancelar mi pedido por favor Ich habe mein Paket noch nicht bekommen und möchte es zurück",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 20,+ "diacritic_rate": 0.005847953216374269,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "es",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0058,+ "non_latin_fraction": 0.0,+ "mixed_segment": "Quiero cancelar mi pedido por favor"+ }+ },+ {+ "label": "p0411/leaf_shouted_foreign",+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "x": "EU NAO CONSIGO ENTRAR NA MINHA CONTA"+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. EU NAO CONSIGO ENTRAR NA MINHA CONTA",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 20,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "EU NAO CONSIGO ENTRAR NA MINHA CONTA"+ }+ },+ {+ "label": "p0411/leaf_shouted_acronyms",+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "x": "MON DES EST LA"+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. MON DES EST LA",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 20,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/leaf_number_and_none",+ "state": {+ "n": 5,+ "m": null,+ "t": true,+ "s": "Please review the error and send me the details."+ },+ "state_text": "Please review the error and send me the details.",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 4,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/leaf_long_second_field",+ "state": {+ "a": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "b": "Quiero cancelar mi pedido por favor ahora mismo"+ },+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and sen",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 327,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "es",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/empty_string",+ "state": "",+ "state_text": "",+ "detect_script": "unknown",+ "script_profile": {},+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "unknown",+ "script_profile": {},+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/whitespace_only",+ "state": " \n\t ",+ "state_text": " \n\t ",+ "detect_script": "unknown",+ "script_profile": {},+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "unknown",+ "script_profile": {},+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/digits_only",+ "state": "12345 67890",+ "state_text": "12345 67890",+ "detect_script": "unknown",+ "script_profile": {},+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "unknown",+ "script_profile": {},+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/state_text_cap_multibyte",+ "state": "çççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççç",+ "state_text": "çççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççççç",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": null,+ "english_hits": 0,+ "diacritic_rate": 1.0,+ "looks_non_english": true+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 1.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }+ },+ {+ "label": "p0411/roman_numerals",+ "state": "Ⅳ Ⅴ Ⅵ Ⅶ please",+ "state_text": "Ⅳ Ⅴ Ⅵ Ⅶ please",+ "detect_script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "latin_profile": {+ "language": "en",+ "english_hits": 1,+ "diacritic_rate": 0.0,+ "looks_non_english": false+ },+ "analyse": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ }}]diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json b/laya-dotnet/tests/Laya.Tests/golden/routing/meta.jsonindex 00db9d20..f96af525 100644--- a/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json@@ -7,10 +7,10 @@"shortlist_probe": "Non-finite floats (NaN, +Inf, -Inf) inside a case's 'vectors' matrix are encoded as the JSON strings \"nan\", \"inf\", \"-inf\" because JSON has no native literal for them; a case with vectors=null is a k>=n passthrough case where embed_fn must never be called."},"counts": {- "lang_probe": 154,- "route_probe": 74,- "email_probe_clean": 70,- "email_probe_state": 8,+ "lang_probe": 246,+ "route_probe": 142,+ "email_probe_clean": 241,+ "email_probe_state": 22,"shortlist_probe": 22,"hashing_embedder_probe": 22}diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/route_probe.json b/laya-dotnet/tests/Laya.Tests/golden/routing/route_probe.jsonindex a1a70f12..f914d4df 100644--- a/laya-dotnet/tests/Laya.Tests/golden/routing/route_probe.json+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/route_probe.json@@ -2036,5 +2036,1416 @@},"workflow": null}+ },+ {+ "label": "p0411/lang_blank",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Hello there",+ "lang": ""+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, language not identified and no non-English letters; using default (multilingual)",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_blank_foreign_text",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Hindi मुझसे दो बार शुल्क लिया गया",+ "lang": " "+ },+ "decision": {+ "model": "multilingual",+ "reason": "non-Latin script (devanagari, 72% of letters); the English checkpoint cannot read it",+ "detection": {+ "script": "devanagari",+ "script_profile": {+ "latin": 0.2777777777777778,+ "devanagari": 0.7222222222222222+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.7222,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_C",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Hello there my friend",+ "lang": "C"+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_C_UTF8",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Hello there my friend",+ "lang": "C.UTF-8"+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_POSIX",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "二重に請求されました",+ "lang": "POSIX"+ },+ "decision": {+ "model": "multilingual",+ "reason": "non-Latin script (kana, 100% of letters); the English checkpoint cannot read it",+ "detection": {+ "script": "kana",+ "script_profile": {+ "han": 0.4,+ "kana": 0.6+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 1.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_und",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Hello there my friend",+ "lang": "und"+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_zxx",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Hello there my friend",+ "lang": "zxx"+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_mul",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Hello there my friend",+ "lang": "mul"+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_only_dot",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Hello there my friend",+ "lang": "."+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_posix_underscore",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hola",+ "lang": "en_US"+ },+ "decision": {+ "model": "english",+ "reason": "explicit lang='en_US'",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_posix_encoding",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hola",+ "lang": "en_US.UTF-8"+ },+ "decision": {+ "model": "english",+ "reason": "explicit lang='en_US.UTF-8'",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_en_dot_encoding",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hola",+ "lang": "en.UTF-8"+ },+ "decision": {+ "model": "english",+ "reason": "explicit lang='en.UTF-8'",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_uppercase_spaces",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hola",+ "lang": " EN "+ },+ "decision": {+ "model": "english",+ "reason": "explicit lang=' EN '",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_zh_CN",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hello",+ "lang": "zh_CN"+ },+ "decision": {+ "model": "multilingual",+ "reason": "explicit lang='zh_CN'",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_unknown_code",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hello",+ "lang": "qq"+ },+ "decision": {+ "model": "multilingual",+ "reason": "explicit lang='qq'",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_quote_in_code",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hello",+ "lang": "it's"+ },+ "decision": {+ "model": "multilingual",+ "reason": "explicit lang=\"it's\"",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_nonprintable_code",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hello",+ "lang": "xy"+ },+ "decision": {+ "model": "multilingual",+ "reason": "explicit lang='x\\u200by'",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_beats_lang_guess",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hello",+ "lang": "en",+ "lang_guess": "ro"+ },+ "decision": {+ "model": "english",+ "reason": "explicit lang='en'",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_blank_then_guess",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hello",+ "lang": "",+ "lang_guess": "ro"+ },+ "decision": {+ "model": "multilingual",+ "reason": "lang_guess: the caller identified this as non-English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/lang_C_then_guess",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hello",+ "lang": "C",+ "lang_guess": "ro"+ },+ "decision": {+ "model": "multilingual",+ "reason": "lang_guess: the caller identified this as non-English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/typed_workflow_beats_lang_guess",+ "router_kwargs": {+ "auto_task_detection": true+ },+ "route_kwargs": {+ "state": "x",+ "questions": {+ "action": {+ "type": "noul",+ "instructions": "x"+ },+ "category": {+ "type": "noul",+ "instructions": "x"+ },+ "churn_risk": {+ "type": "noul",+ "instructions": "x"+ },+ "needs_human": {+ "type": "noul",+ "instructions": "x"+ },+ "urgency": {+ "type": "noul",+ "instructions": "x"+ }+ },+ "lang_guess": "ro"+ },+ "decision": {+ "model": "typed-decisions",+ "reason": "question ids match the 'customer_service' typed-decisions workflow",+ "detection": null,+ "workflow": "customer_service"+ }+ },+ {+ "label": "p0411/task_beats_lang_guess",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "x",+ "task": "typed_decisions",+ "lang_guess": "ro"+ },+ "decision": {+ "model": "typed-decisions",+ "reason": "explicit task='typed_decisions'",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/model_beats_lang_guess",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "x",+ "model": "english",+ "lang_guess": "ro"+ },+ "decision": {+ "model": "english",+ "reason": "explicit model='english'",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_ro",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "ro"+ },+ "decision": {+ "model": "multilingual",+ "reason": "lang_guess: the caller identified this as non-English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_en",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "en"+ },+ "decision": {+ "model": "english",+ "reason": "lang_guess: the caller identified this as English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_en_US",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "en_US"+ },+ "decision": {+ "model": "english",+ "reason": "lang_guess: the caller identified this as English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_en_UTF8",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "en_US.UTF-8"+ },+ "decision": {+ "model": "english",+ "reason": "lang_guess: the caller identified this as English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_de_DE",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "de-DE"+ },+ "decision": {+ "model": "multilingual",+ "reason": "lang_guess: the caller identified this as non-English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_RO_upper",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "RO"+ },+ "decision": {+ "model": "multilingual",+ "reason": "lang_guess: the caller identified this as non-English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_spaces_en",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": " en "+ },+ "decision": {+ "model": "english",+ "reason": "lang_guess: the caller identified this as English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_unknown_code",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "qq"+ },+ "decision": {+ "model": "multilingual",+ "reason": "lang_guess: the caller identified this as non-English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_blank_abstains",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": ""+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_spaces_abstain",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": " "+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_C_abstains",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "C"+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_CUTF8_abstains",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "C.UTF-8"+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_POSIX_abstains",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "POSIX"+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_und_abstains",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Hindi मुझसे दो बार",+ "lang_guess": "und"+ },+ "decision": {+ "model": "multilingual",+ "reason": "non-Latin script (devanagari, 55% of letters); the English checkpoint cannot read it",+ "detection": {+ "script": "devanagari",+ "script_profile": {+ "latin": 0.45454545454545453,+ "devanagari": 0.5454545454545454+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.5455,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_dot_abstains",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "."+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_dot_utf8",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": ".UTF-8"+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_overrides_script",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "二重に請求されました",+ "lang_guess": "en"+ },+ "decision": {+ "model": "english",+ "reason": "lang_guess: the caller identified this as English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/guess_overrides_default",+ "router_kwargs": {+ "default": "english"+ },+ "route_kwargs": {+ "state": "",+ "lang_guess": "fr"+ },+ "decision": {+ "model": "multilingual",+ "reason": "lang_guess: the caller identified this as non-English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/installed_guess_ro",+ "router_kwargs": {+ "lang_guess": "ro"+ },+ "route_kwargs": {+ "state": "Care este ora in Tokyo?"+ },+ "decision": {+ "model": "multilingual",+ "reason": "Router(lang_guess=...): the caller identified this as non-English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/installed_guess_en_script",+ "router_kwargs": {+ "lang_guess": "en"+ },+ "route_kwargs": {+ "state": "二重に請求されました"+ },+ "decision": {+ "model": "english",+ "reason": "Router(lang_guess=...): the caller identified this as English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/installed_guess_C_abstains",+ "router_kwargs": {+ "lang_guess": "C"+ },+ "route_kwargs": {+ "state": "Care este ora in Tokyo?"+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/installed_guess_overridden",+ "router_kwargs": {+ "lang_guess": "ro"+ },+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "en"+ },+ "decision": {+ "model": "english",+ "reason": "lang_guess: the caller identified this as English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/installed_guess_after_abstaining_call",+ "router_kwargs": {+ "lang_guess": "ro"+ },+ "route_kwargs": {+ "state": "Care este ora in Tokyo?",+ "lang_guess": "C"+ },+ "decision": {+ "model": "multilingual",+ "reason": "Router(lang_guess=...): the caller identified this as non-English text",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/installed_guess_lang_wins",+ "router_kwargs": {+ "lang_guess": "ro"+ },+ "route_kwargs": {+ "state": "hello",+ "lang": "en"+ },+ "decision": {+ "model": "english",+ "reason": "explicit lang='en'",+ "detection": null,+ "workflow": null+ }+ },+ {+ "label": "p0411/reason_nordic_undecided",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "hej min vän"+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, language not identified but 9% non-English letters; not safe for the English checkpoint",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0909,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/reason_undecided_english_default",+ "router_kwargs": {+ "default": "english"+ },+ "route_kwargs": {+ "state": "Hello"+ },+ "decision": {+ "model": "english",+ "reason": "Latin script, language not identified and no non-English letters; using default (english)",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/reason_undecided_multilingual_default",+ "router_kwargs": {+ "default": "multilingual"+ },+ "route_kwargs": {+ "state": "Hello"+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, language not identified and no non-English letters; using default (multilingual)",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/reason_unknown_default_english",+ "router_kwargs": {+ "default": "english"+ },+ "route_kwargs": {+ "state": "12345"+ },+ "decision": {+ "model": "english",+ "reason": "no letters detected in state; using default (english)",+ "detection": {+ "script": "unknown",+ "script_profile": {},+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/reason_unknown_default_typed",+ "router_kwargs": {+ "default": "typed-decisions"+ },+ "route_kwargs": {+ "state": "12345"+ },+ "decision": {+ "model": "typed-decisions",+ "reason": "no letters detected in state; using default (typed-decisions)",+ "detection": {+ "script": "unknown",+ "script_profile": {},+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/reason_shouted_acronyms",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "MON DES EST LA"+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, language not identified and no non-English letters; using default (multilingual)",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/reason_unidentified_latin",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "žluťoučký kůň úpěl ďábelské ódy"+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, language not identified but 39% non-English letters; not safe for the English checkpoint",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.3871,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/reason_percent_rounding",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "ok ㄅㄆㄇㄈㄉㄊㄋㄌㄍㄎ"+ },+ "decision": {+ "model": "multilingual",+ "reason": "non-Latin script (other, 83% of letters); the English checkpoint cannot read it",+ "detection": {+ "script": "other",+ "script_profile": {+ "latin": 0.16666666666666666,+ "other": 0.8333333333333334+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.8333,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/reason_non_latin_promotion",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Please ありがとうありがとうありがとうありがとう"+ },+ "decision": {+ "model": "multilingual",+ "reason": "non-Latin script (kana, 77% of letters); the English checkpoint cannot read it",+ "detection": {+ "script": "kana",+ "script_profile": {+ "latin": 0.23076923076923078,+ "kana": 0.7692307692307693+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.7692,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/mixed_segment_long_line",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Please review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nEu nao consigo entrar na minha conta e depois eu nao sei mais o que fazer com isso tudo."+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, mostly English, but a line or field reads as 'pt' ('Eu nao consigo entrar na minha conta e depois eu nao sei mai'); the English checkpoint cannot read it",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "Eu nao consigo entrar na minha conta e depois eu nao sei mais o que fazer com isso tudo."+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/mixed_segment_quote_char",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Please review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nEu nao consigo entrar na minha conta, don't ask me e depois"+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, mostly English, but a line or field reads as 'pt' (\"Eu nao consigo entrar na minha conta, don't ask me e depois\"); the English checkpoint cannot read it",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "Eu nao consigo entrar na minha conta, don't ask me e depois"+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/mixed_segment_double_quotes",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Please review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nEu nao consigo entrar na \"minha\" conta, don't ask me e depois"+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, mostly English, but a line or field reads as 'pt' ('Eu nao consigo entrar na \"minha\" conta, don\\'t ask me e depoi'); the English checkpoint cannot read it",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "Eu nao consigo entrar na \"minha\" conta, don't ask me e depois"+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/mixed_segment_nonprintable",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Please review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nEu nao consigo entrar na minha\u0007 conta e depois eu nao sei mais o que fazer com isso"+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, mostly English, but a line or field reads as 'pt' ('Eu nao consigo\\u200b entrar na minha\\x07 conta e depois eu nao sei m'); the English checkpoint cannot read it",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "Eu nao consigo entrar na minha\u0007 conta e depois eu nao sei mais o que fazer com isso"+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/mixed_segment_astral",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Please review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nEu nao consigo 😀 entrar na minha conta e depois eu nao sei mais o que fazer com isso tudo"+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, mostly English, but a line or field reads as 'pt' ('Eu nao consigo 😀 entrar na minha conta e depois eu nao sei m'); the English checkpoint cannot read it",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "Eu nao consigo 😀 entrar na minha conta e depois eu nao sei mais o que fazer com isso tudo"+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/mixed_segment_field",+ "router_kwargs": {},+ "route_kwargs": {+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "body": "Quiero cancelar mi pedido por favor ahora"+ }+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, mostly English, but a line or field reads as 'es' ('Quiero cancelar mi pedido por favor ahora'); the English checkpoint cannot read it",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "es",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "Quiero cancelar mi pedido por favor ahora"+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/mixed_segment_shouted_rejected",+ "router_kwargs": {},+ "route_kwargs": {+ "state": "Please review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nMON DES EST LA"+ },+ "decision": {+ "model": "english",+ "reason": "English Latin text",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "en",+ "is_english": true,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/mixed_segment_customer_default",+ "router_kwargs": {+ "default": "english"+ },+ "route_kwargs": {+ "state": "Please review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nPlease review the error and send me the details of this and what you would like to do.\nEu nao consigo entrar na minha conta e depois eu nao sei mais o que fazer."+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, mostly English, but a line or field reads as 'pt' ('Eu nao consigo entrar na minha conta e depois eu nao sei mai'); the English checkpoint cannot read it",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": "pt",+ "is_english": false,+ "language_undecided": false,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": "Eu nao consigo entrar na minha conta e depois eu nao sei mais o que fazer."+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/leaf_flip_no_mixed_segment",+ "router_kwargs": {},+ "route_kwargs": {+ "state": {+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",+ "x": "žluťoučký kůň úpěl ďábelské ódy"+ }+ },+ "decision": {+ "model": "multilingual",+ "reason": "Latin script, language not identified but 4% non-English letters; not safe for the English checkpoint",+ "detection": {+ "script": "latin",+ "script_profile": {+ "latin": 1.0+ },+ "language": null,+ "is_english": false,+ "language_undecided": true,+ "diacritic_rate": 0.0433,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/state_none",+ "router_kwargs": {},+ "route_kwargs": {+ "state": null+ },+ "decision": {+ "model": "multilingual",+ "reason": "no letters detected in state; using default (multilingual)",+ "detection": {+ "script": "unknown",+ "script_profile": {},+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/state_number",+ "router_kwargs": {},+ "route_kwargs": {+ "state": 5+ },+ "decision": {+ "model": "multilingual",+ "reason": "no letters detected in state; using default (multilingual)",+ "detection": {+ "script": "unknown",+ "script_profile": {},+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }+ },+ {+ "label": "p0411/state_empty_dict",+ "router_kwargs": {},+ "route_kwargs": {+ "state": {}+ },+ "decision": {+ "model": "multilingual",+ "reason": "no letters detected in state; using default (multilingual)",+ "detection": {+ "script": "unknown",+ "script_profile": {},+ "language": null,+ "is_english": true,+ "language_undecided": true,+ "diacritic_rate": 0.0,+ "non_latin_fraction": 0.0,+ "mixed_segment": null+ },+ "workflow": null+ }}]diff --git a/laya-dotnet/tools/routing_cases.py b/laya-dotnet/tools/routing_cases.pyindex 14fab542..53a08e03 100644--- a/laya-dotnet/tools/routing_cases.py+++ b/laya-dotnet/tools/routing_cases.py@@ -15,8 +15,7 @@ Four sections, one per probe file:SHORTLIST_CASES -> shortlist_probe.json (shortlist.shortlist_choice)Plus the single source of truth for the combined routing sample's own fixed inputs-(SAMPLE_* below), so the samples can be diffed against a Python-recorded-golden the same way the existing `Laya.Sample` is.+(SAMPLE_* below), so the samples can be diffed against a Python-recorded golden."""# ============================================================================ lang_probe@@ -236,6 +235,126 @@ LANG_STATES = [# ---- English control long enough to stay decisively English despite length ----("long/english_over_1000","Please refund the duplicate charge on invoice 4411 today. " * 20),++ # ---- Python 0.4.11 behaviours neither SDK's upstream corpus pinned ----+ # Nordic / Swedish rules and the short-Swedish branch+ ("p0411/swedish_sentence", "Hej, jag behöver hjälp med min faktura"),+ ("p0411/swedish_ascii", "jag behover hjalp med min faktura"),+ ("p0411/swedish_two_words", "lösenord fel"),+ ("p0411/swedish_one_word", "lösenord"),+ ("p0411/swedish_inte_kan", "kan inte logga in på appen idag"),+ ("p0411/swedish_inte_kan_not_first", "jag vet att kan inte logga in"),+ ("p0411/nordic_overlap_tack", "tack så mycket"),+ ("p0411/nordic_overlap_with_english", "hej please send me the invoice"),+ ("p0411/danish_sentence", "jeg har brug for hjælp til min faktura"),+ ("p0411/norwegian_sentence", "jeg trenger hjelp med fakturaen min"),+ # English function words vs. one accented loanword (English rescue)+ ("p0411/rescue_loanword", "Please send the café menu to the customer today"),+ ("p0411/rescue_two_accents", "Please send the café and the résumé to the customer today"),+ ("p0411/rescue_vetoed_by_rate", "ñandú ñandú ñandú the please"),+ ("p0411/rescue_name", "Please forward this to José and Zoë at the office"),+ ("p0411/rescue_one_english_word", "Der Kunde hat nicht please"),+ # Latin cutoff (U+02B0), fullwidth Latin, modifier letters and unclaimed letters+ ("p0411/modifier_letters", "\u02b0\u02b2\u02b7 hello there"),+ ("p0411/modifier_only", "\u02b0\u02b2\u02b7"),+ ("p0411/ipa_only", "\u0250\u0251\u0252\u0253"),+ ("p0411/fullwidth_only", "HELLO world"),+ ("p0411/bopomofo_only", "ㄅㄆㄇㄈ"),+ ("p0411/halfwidth_katakana", "アイウエオ"),+ ("p0411/cjk_extension_b", "\U00020000\U00020001\U00020002"),+ ("p0411/unclaimed_with_latin", "please help ㄅㄆㄇㄈㄉㄊ today"),+ ("p0411/unclaimed_dominant", "ok ㄅㄆㄇㄈㄉㄊㄋㄌㄍㄎ"),+ ("p0411/tie_named_vs_latin", "abc αβγ"),+ ("p0411/tie_two_scripts", "αβγ абв"),+ # non-Latin words: combining classes, single letters, names, punctuation in script ranges+ ("p0411/devanagari_matras", "please call म\u0941 म\u0941 म\u0941 म\u0941 म\u0941 म\u0941 now"),+ ("p0411/devanagari_single_letters", "Please call क ख ग घ ङ च छ ज झ ञ now"),+ ("p0411/devanagari_danda", "Please read this \u0964\u0964\u0964\u0964\u0964\u0964\u0964\u0964\u0964\u0964\u0964\u0964"),+ ("p0411/cyrillic_combining_name", "Please send it to Влади\u0301мир Петро\u0301вич today"),+ ("p0411/cyrillic_lower_word", "Please send it to привет мир today ok"),+ ("p0411/arabic_harakat", "Please read مَرْحَبًا بِكُمْ في الموقع today"),+ ("p0411/thai_tone_marks", "Please read สวัสดีครับ ขอบคุณมาก today"),+ ("p0411/vietnamese_combining", "Xin cha\u0300o, to\u0302i can\u0302 gi\u00fap \u0111\u1ee1 v\u1ec1 ho\u00e1 \u0111\u01a1n"),+ ("p0411/greek_final_sigma", "Please check \u039f\u0394\u03a5\u03a3\u03a3\u0395\u03a5\u03a3 today"),+ ("p0411/non_latin_promotion_min_letters", "ok " + "x" * 80 + " " + "ありがとうございます"),+ ("p0411/non_latin_fraction_edge", "Please " + "я" * 2 + " " + "a" * 8),+ # identifiers, code, compounds+ ("p0411/identifier_domains", "github.com acme.com v1.2.3 la o e un [email protected]"),+ ("p0411/identifier_trailing_dot", "arrivato. la fattura. non e arrivata ancora."),+ ("p0411/identifier_double_dot", "a..b la fattura non arrivata"),+ ("p0411/identifier_leading_at", "@home la fattura non e arrivata ancora"),+ ("p0411/identifier_hyphen", "e-mail la fattura non e arrivata ancora x-y.z"),+ ("p0411/dotted_I", "\u0130stanbul'da bir sorun var ve yard\u0131m istiyorum"),+ ("p0411/dotted_I_english", "I am in \u0130stanbul and please send the receipt"),+ ("p0411/code_line_state", "x = 5 el la los las para que no"),+ ("p0411/call_syntax_state", "foo(bar) para que no el la los"),+ ("p0411/titlecase_digraph", "\u01c5ungla and the please are here"),+ # all-caps (shouted) handling and acronym lines+ ("p0411/shouted_acronyms", "MON DES EST LA"),+ ("p0411/shouted_acronyms_under_english", "Please look at this\nMON DES EST LA"),+ ("p0411/shouted_acronyms_two_english_lines", "Please look at this\nand send me the details.\nMON DES EST LA"),+ ("p0411/shouted_address", "STORES LOS ANGELES LAS VEGAS EL PASO CLOSED"),+ ("p0411/shouted_german_umlaut", "WIE SPÄT IST ES IN KÖLN"),+ ("p0411/shouted_german_ascii", "WIE SPAET IST ES IN KOELN"),+ ("p0411/shouted_portuguese_prose", "EU NAO CONSIGO ENTRAR NA MINHA CONTA"),+ ("p0411/shouted_german_prose", "ICH HABE MEIN PAKET NOCH NICHT BEKOMMEN"),+ ("p0411/shouted_french_prose", "JE VOUDRAIS UN REMBOURSEMENT POUR LA COMMANDE"),+ ("p0411/shouted_cyrillic_acronyms", "ПРИВЕТ MON DES EST LA"),+ ("p0411/emphasis_german", "sag mir das HEUTIGE DATUM"),+ ("p0411/emphasis_spanish", "quiero cancelar mi PEDIDO POR FAVOR"),+ ("p0411/emphasis_acronym_mixed", "quero cancelar o pedido MON DES EST LA por favor"),+ ("p0411/emphasis_long_run", "please cancel the order and send a REFUND for it"),+ ("p0411/caseless_with_caps", "ありがとう MON DES EST LA"),+ ("p0411/hindi_caps", "मुझे MON DES EST LA चाहिए"),+ ("p0411/shouted_english", "PLEASE SEND ME THE RECEIPT FOR THE ORDER"),+ # segment scan: field/line boundaries, 4000-character cap, code lines, shouted segments+ ("p0411/segment_after_cap",+ "Please review the error and send me the details.\n" * 100 + "Eu nao consigo entrar na minha conta."),+ ("p0411/segment_within_cap",+ "Please review the error and send me the details.\n" * 10 + "Eu nao consigo entrar na minha conta."),+ ("p0411/segment_shouted_foreign",+ "Please review the error and send me the details.\nEU NAO CONSIGO ENTRAR NA MINHA CONTA"),+ ("p0411/segment_acronym_line",+ "Please review the error and send me the details.\nMON DES EST LA"),+ ("p0411/segment_joined_tokens",+ "Please review the error and send me the details.\nNav/Com OS/2 ESA/UN C:\\DOS\\mode radio listing"),+ ("p0411/segment_two_same_word",+ "Please review the error and send me the details.\ncom com com com com com"),+ ("p0411/segment_call_syntax",+ "Please review the error and send me the details.\nround(el, 2) para que no tengo el"),+ ("p0411/segment_list_state",+ ["Please review the error and send me the details.", "Quiero cancelar mi pedido por favor ahora mismo"]),+ ("p0411/segment_nested_dict",+ {"ticket": {"note": "Please review the error and send me the details. " * 3,+ "customer": {"msg": "ich habe mein paket noch nicht bekommen"}}}),+ ("p0411/segment_too_deep",+ {"a": {"b": {"c": {"d": {"e": {"f": {"g": "ich habe mein paket noch nicht bekommen"}}}}}},+ "n": "Please review the error and send me the details. " * 3}),+ ("p0411/segment_unicode_strip", "Please review the error and send me the details.\n\u00a0\u00a0Eu nao consigo entrar na minha conta.\u00a0"),+ # leaf scan: names, short values, other scripts, diacritics without a named language+ ("p0411/leaf_name_field", {"note": "Please review the error and send me the details. " * 5, "name": "José"}),+ ("p0411/leaf_short_value", {"note": "Please review the error and send me the details. " * 5, "x": "žluť"}),+ ("p0411/leaf_nonlatin_short", {"note": "Please review the error and send me the details. " * 5, "x": "你好吗"}),+ ("p0411/leaf_nonlatin_ten", {"note": "Please review the error and send me the details. " * 5, "x": "我的账户无法登录请帮助我吧"}),+ ("p0411/leaf_cyrillic_name", {"note": "Please review the error and send me the details. " * 5, "x": "Дмитрий Петрович Савицкий"}),+ ("p0411/leaf_code_value", {"note": "Please review the error and send me the details. " * 5, "x": "x = żółć żółć żółć żółć"}),+ ("p0411/leaf_diacritics_four_words", {"note": "Please review the error and send me the details. " * 5, "x": "żółć gęślą jaźń łódź"}),+ ("p0411/leaf_two_candidates",+ {"note": "Please review the error and send me the details. " * 5,+ "a": "Quiero cancelar mi pedido por favor",+ "b": "Ich habe mein Paket noch nicht bekommen und möchte es zurück"}),+ ("p0411/leaf_shouted_foreign",+ {"note": "Please review the error and send me the details. " * 5, "x": "EU NAO CONSIGO ENTRAR NA MINHA CONTA"}),+ ("p0411/leaf_shouted_acronyms",+ {"note": "Please review the error and send me the details. " * 5, "x": "MON DES EST LA"}),+ ("p0411/leaf_number_and_none", {"n": 5, "m": None, "t": True, "s": "Please review the error and send me the details."}),+ ("p0411/leaf_long_second_field",+ {"a": "Please review the error and send me the details. " * 100, "b": "Quiero cancelar mi pedido por favor ahora mismo"}),+ ("p0411/empty_string", ""),+ ("p0411/whitespace_only", " \n\t "),+ ("p0411/digits_only", "12345 67890"),+ ("p0411/state_text_cap_multibyte", "ç" * 4100),+ ("p0411/roman_numerals", "Ⅳ Ⅴ Ⅵ Ⅶ please"),]@@ -262,6 +381,8 @@ def _td_questions(ids):return {i: {"type": "noul", "instructions": "x"} for i in ids}+_ENG6 = "Please review the error and send me the details of this and what you would like to do.\n" * 6+ROUTE_CASES = [("upstream/undetermined_latin_custom_default", {"default": "multilingual"}, {"state": "Hello"}),("upstream/mixed_foreign_line", {}, {"state":@@ -368,6 +489,90 @@ ROUTE_CASES = [("payload_shape_hindi", {}, {"state": {"body": "मुझसे दो बार शुल्क लिया गया"}, "questions": Q_GENERIC}),("payload_shape_armenian_non_latin_percent", {}, {"state": "Հայերեն abc"}),("payload_shape_unidentified_latin_percent", {}, {"state": "Müşteriden iki kez ücret alındı ve para iadesi istiyor"}),++ # ---- Python 0.4.11 behaviours neither SDK's upstream corpus pinned ----+ # codes that name no language fall through to detection (_english_from_code)+ ("p0411/lang_blank", {}, {"state": "Hello there", "lang": ""}),+ ("p0411/lang_blank_foreign_text", {}, {"state": "Hindi मुझसे दो बार शुल्क लिया गया", "lang": " "}),+ ("p0411/lang_C", {}, {"state": "Hello there my friend", "lang": "C"}),+ ("p0411/lang_C_UTF8", {}, {"state": "Hello there my friend", "lang": "C.UTF-8"}),+ ("p0411/lang_POSIX", {}, {"state": "二重に請求されました", "lang": "POSIX"}),+ ("p0411/lang_und", {}, {"state": "Hello there my friend", "lang": "und"}),+ ("p0411/lang_zxx", {}, {"state": "Hello there my friend", "lang": "zxx"}),+ ("p0411/lang_mul", {}, {"state": "Hello there my friend", "lang": "mul"}),+ ("p0411/lang_only_dot", {}, {"state": "Hello there my friend", "lang": "."}),+ ("p0411/lang_posix_underscore", {}, {"state": "hola", "lang": "en_US"}),+ ("p0411/lang_posix_encoding", {}, {"state": "hola", "lang": "en_US.UTF-8"}),+ ("p0411/lang_en_dot_encoding", {}, {"state": "hola", "lang": "en.UTF-8"}),+ ("p0411/lang_uppercase_spaces", {}, {"state": "hola", "lang": " EN "}),+ ("p0411/lang_zh_CN", {}, {"state": "hello", "lang": "zh_CN"}),+ ("p0411/lang_unknown_code", {}, {"state": "hello", "lang": "qq"}),+ ("p0411/lang_quote_in_code", {}, {"state": "hello", "lang": "it's"}),+ ("p0411/lang_nonprintable_code", {}, {"state": "hello", "lang": "x\u200by"}),+ ("p0411/lang_beats_lang_guess", {}, {"state": "hello", "lang": "en", "lang_guess": "ro"}),+ ("p0411/lang_blank_then_guess", {}, {"state": "hello", "lang": "", "lang_guess": "ro"}),+ ("p0411/lang_C_then_guess", {}, {"state": "hello", "lang": "C", "lang_guess": "ro"}),+ ("p0411/typed_workflow_beats_lang_guess",+ {"auto_task_detection": True}, {"state": "x", "questions": {i: {"type": "noul", "instructions": "x"} for i in _TD_IDS["customer_service"]}, "lang_guess": "ro"}),+ ("p0411/task_beats_lang_guess", {}, {"state": "x", "task": "typed_decisions", "lang_guess": "ro"}),+ ("p0411/model_beats_lang_guess", {}, {"state": "x", "model": "english", "lang_guess": "ro"}),+ # lang_guess: per-call and installed on the Router+ ("p0411/guess_ro", {}, {"state": "Care este ora in Tokyo?", "lang_guess": "ro"}),+ ("p0411/guess_en", {}, {"state": "Care este ora in Tokyo?", "lang_guess": "en"}),+ ("p0411/guess_en_US", {}, {"state": "Care este ora in Tokyo?", "lang_guess": "en_US"}),+ ("p0411/guess_en_UTF8", {}, {"state": "Care este ora in Tokyo?", "lang_guess": "en_US.UTF-8"}),+ ("p0411/guess_de_DE", {}, {"state": "Care este ora in Tokyo?", "lang_guess": "de-DE"}),+ ("p0411/guess_RO_upper", {}, {"state": "Care este ora in Tokyo?", "lang_guess": "RO"}),+ ("p0411/guess_spaces_en", {}, {"state": "Care este ora in Tokyo?", "lang_guess": " en "}),+ ("p0411/guess_unknown_code", {}, {"state": "Care este ora in Tokyo?", "lang_guess": "qq"}),+ ("p0411/guess_blank_abstains", {}, {"state": "Care este ora in Tokyo?", "lang_guess": ""}),+ ("p0411/guess_spaces_abstain", {}, {"state": "Care este ora in Tokyo?", "lang_guess": " "}),+ ("p0411/guess_C_abstains", {}, {"state": "Care este ora in Tokyo?", "lang_guess": "C"}),+ ("p0411/guess_CUTF8_abstains", {}, {"state": "Care este ora in Tokyo?", "lang_guess": "C.UTF-8"}),+ ("p0411/guess_POSIX_abstains", {}, {"state": "Care este ora in Tokyo?", "lang_guess": "POSIX"}),+ ("p0411/guess_und_abstains", {}, {"state": "Hindi मुझसे दो बार", "lang_guess": "und"}),+ ("p0411/guess_dot_abstains", {}, {"state": "Care este ora in Tokyo?", "lang_guess": "."}),+ ("p0411/guess_dot_utf8", {}, {"state": "Care este ora in Tokyo?", "lang_guess": ".UTF-8"}),+ ("p0411/guess_overrides_script", {}, {"state": "二重に請求されました", "lang_guess": "en"}),+ ("p0411/guess_overrides_default", {"default": "english"}, {"state": "", "lang_guess": "fr"}),+ ("p0411/installed_guess_ro", {"lang_guess": "ro"}, {"state": "Care este ora in Tokyo?"}),+ ("p0411/installed_guess_en_script", {"lang_guess": "en"}, {"state": "二重に請求されました"}),+ ("p0411/installed_guess_C_abstains", {"lang_guess": "C"}, {"state": "Care este ora in Tokyo?"}),+ ("p0411/installed_guess_overridden", {"lang_guess": "ro"}, {"state": "Care este ora in Tokyo?", "lang_guess": "en"}),+ ("p0411/installed_guess_after_abstaining_call",+ {"lang_guess": "ro"}, {"state": "Care este ora in Tokyo?", "lang_guess": "C"}),+ ("p0411/installed_guess_lang_wins", {"lang_guess": "ro"}, {"state": "hello", "lang": "en"}),+ # detection reasons+ ("p0411/reason_nordic_undecided", {}, {"state": "hej min vän"}),+ ("p0411/reason_undecided_english_default", {"default": "english"}, {"state": "Hello"}),+ ("p0411/reason_undecided_multilingual_default", {"default": "multilingual"}, {"state": "Hello"}),+ ("p0411/reason_unknown_default_english", {"default": "english"}, {"state": "12345"}),+ ("p0411/reason_unknown_default_typed", {"default": "typed-decisions"}, {"state": "12345"}),+ ("p0411/reason_shouted_acronyms", {}, {"state": "MON DES EST LA"}),+ ("p0411/reason_unidentified_latin", {}, {"state": "žluťoučký kůň úpěl ďábelské ódy"}),+ ("p0411/reason_percent_rounding", {}, {"state": "ok ㄅㄆㄇㄈㄉㄊㄋㄌㄍㄎ"}),+ ("p0411/reason_non_latin_promotion", {}, {"state": "Please " + "ありがとう" * 4}),+ ("p0411/mixed_segment_long_line",+ {}, {"state": _ENG6 + "Eu nao consigo entrar na minha conta " + "e depois eu nao sei mais o que fazer com isso tudo."}),+ ("p0411/mixed_segment_quote_char",+ {}, {"state": _ENG6 + "Eu nao consigo entrar na minha conta, don't ask me e depois"}),+ ("p0411/mixed_segment_double_quotes",+ {}, {"state": _ENG6 + "Eu nao consigo entrar na \"minha\" conta, don't ask me e depois"}),+ ("p0411/mixed_segment_nonprintable",+ {}, {"state": _ENG6 + "Eu nao consigo\u200b entrar na minha\u0007 conta e depois eu nao sei mais o que fazer com isso"}),+ ("p0411/mixed_segment_astral",+ {}, {"state": _ENG6 + "Eu nao consigo \U0001f600 entrar na minha conta e depois eu nao sei mais o que fazer com isso tudo"}),+ ("p0411/mixed_segment_field",+ {}, {"state": {"note": "Please review the error and send me the details. " * 10, "body": "Quiero cancelar mi pedido por favor ahora"}}),+ ("p0411/mixed_segment_shouted_rejected",+ {}, {"state": _ENG6 + "MON DES EST LA"}),+ ("p0411/mixed_segment_customer_default",+ {"default": "english"}, {"state": _ENG6 + "Eu nao consigo entrar na minha conta e depois eu nao sei mais o que fazer."}),+ ("p0411/leaf_flip_no_mixed_segment",+ {}, {"state": {"note": "Please review the error and send me the details. " * 5, "x": "žluťoučký kůň úpěl ďábelské ódy"}}),+ ("p0411/state_none", {}, {"state": None}),+ ("p0411/state_number", {}, {"state": 5}),+ ("p0411/state_empty_dict", {}, {"state": {}}),]@@ -508,6 +713,213 @@ EMAIL_CLEAN_CASES = ["On Tue, Sep 22, 2026 at 9:14 AM, Support <[email protected]> wrote:\n""> Hi Priya, thanks for reaching out, we are looking into it.\n""> - Support Team\n", 3000),++ # ---- Python 0.4.11 behaviours neither SDK's upstream corpus pinned ----+ # max_chars: budget, zero, negative, and the max_chars * 4 pre-cut+ ("p0411/budget_default_cut", "word " * 1000, 3000),+ ("p0411/budget_small", "Please refund the duplicate charge on invoice 4411 today.", 10),+ ("p0411/budget_one", "Hello", 1),+ ("p0411/budget_zero", "Please refund it.", 0),+ ("p0411/budget_zero_empty_body", "", 0),+ ("p0411/budget_negative_one", "Please refund it today and thanks for the help.", -1),+ ("p0411/budget_negative_ten", "Please refund it today and thanks for the help with the invoice.", -10),+ ("p0411/budget_negative_beyond_body", "Please refund it.", -100),+ ("p0411/budget_larger_than_body", "Please refund it.", 100000),+ ("p0411/budget_pre_cut_hides_footer",+ "Request line.\n" + ("filler line that is not a footer.\n" * 3) + "\n\nThis email is confidential and intended solely for the named addressee.",+ 25),+ ("p0411/budget_pre_cut_hides_quote",+ "Short request.\nMore detail here.\nOn Mon, Jan 1, 2024 at 9:00 AM Jane <[email protected]> wrote:\nOld text.", 12),+ ("p0411/budget_pre_cut_multibyte", "退款请求 " * 40, 30),+ ("p0411/budget_boundary_exact", "abcde", 5),+ ("p0411/budget_negative_multiline", "line one\nline two\nline three\nline four", -2),+ # quote headers+ ("p0411/quote_first_line_kept", "On Mon, Jan 1, 2024 Jane wrote:\nPlease refund it.", 3000),+ ("p0411/quote_en_wrote", "Please refund it.\nOn Mon, Jan 1, 2024 at 9:00 AM Jane <[email protected]> wrote:\nOld request.", 3000),+ ("p0411/quote_en_wrote_trailing_space", "Please refund it.\nON TUE, SEP 22 JANE WROTE: \nOld request.", 3000),+ ("p0411/quote_pt_no_date_is_body", "Preciso de ajuda.\nEm resposta ao que você escreveu:\nMais detalhes aqui.", 3000),+ ("p0411/quote_pt_date", "Preciso de ajuda.\nEm 10/09/2024 às 10:00, João escreveu:\nOld request.", 3000),+ ("p0411/quote_pt_unicode_digits", "Preciso de ajuda.\nEm ١٠/٠٩ João escreveu:\nOld request.", 3000),+ ("p0411/quote_es_no_date_is_body", "Necesito ayuda.\nComo usted escribió:\nEl resto del texto.", 3000),+ ("p0411/quote_es_no_date_el", "Necesito ayuda.\nEl cliente que escribió:\nEl resto del texto.", 3000),+ ("p0411/quote_es_date_ascii_o", "Necesito ayuda.\nEl 10/09 Juan escribio:\nOld request.", 3000),+ ("p0411/quote_fr_no_date_is_body", "Je demande de l'aide.\nLe rapport que vous avez écrit :\nLe reste du texte.", 3000),+ ("p0411/quote_fr_no_space_before_colon", "Je demande de l'aide.\nLe 10/09 Jean a écrit:\nOld request.", 3000),+ ("p0411/quote_fr_ecrit_ascii", "Je demande de l'aide.\nLe 10/09 Jean a ecrit :\nOld request.", 3000),+ ("p0411/quote_original_message", "Please refund it.\n-----Original Message-----\nOld request.", 3000),+ ("p0411/quote_forwarded_message", "Please refund it.\n---- Forwarded Message ----\nOld request.", 3000),+ ("p0411/quote_mensagem_original", "Preciso de ajuda.\n-----Mensagem original-----\nOld request.", 3000),+ ("p0411/quote_mensagem_encaminhada", "Preciso de ajuda.\n----- Mensagem encaminhada -----\nOld request.", 3000),+ ("p0411/quote_mensaje_original", "Necesito ayuda.\n-----Mensaje original-----\nOld request.", 3000),+ ("p0411/quote_mensaje_reenviado", "Necesito ayuda.\n-----Mensaje reenviado-----\nOld request.", 3000),+ ("p0411/quote_message_d_origine", "Je demande de l'aide.\n-----Message d'origine-----\nOld request.", 3000),+ ("p0411/quote_underscores", "Please refund it.\n________\nOld request.", 3000),+ ("p0411/quote_underscores_seven", "Please refund it.\n_______\nStill here.", 3000),+ ("p0411/quote_from_address", "Please refund it.\nFrom: Jane <[email protected]>\nOld request.", 3000),+ ("p0411/quote_from_prose_with_at", "Please refund it.\nFrom: the desk at 5pm we meet\nStill here.", 3000),+ ("p0411/quote_de_address", "Preciso de ajuda.\nDe: Maria <[email protected]>\nOld request.", 3000),+ ("p0411/quote_de_french_spaced", "Je demande de l'aide.\nDe : Marie <[email protected]>\nOld request.", 3000),+ ("p0411/quote_de_dates_is_body", "Quero folga.\nDe: 10/09 a 15/09\nPara: 20/09\nObrigada.", 3000),+ ("p0411/quote_de_name_enviado", "Preciso de ajuda.\nDe: Maria Souza\nEnviado: segunda-feira\nOld request.", 3000),+ ("p0411/quote_de_name_enviada_em", "Preciso de ajuda.\nDe: Maria Souza\nEnviada em: segunda\nOld request.", 3000),+ ("p0411/quote_de_name_fecha", "Necesito ayuda.\nDe: Maria Souza\nFecha: 10 de septiembre de 2024\nOld request.", 3000),+ ("p0411/quote_de_name_data_no_year", "Preciso de ajuda.\nDe: Maria Souza\nData: 10/09\nStill here.", 3000),+ ("p0411/quote_de_name_para_only", "Quero folga.\nDe: 10/09\nPara: 15/09\nObrigada.", 3000),+ ("p0411/quote_de_french_envoye", "Je demande de l'aide.\nDe : Marie Dupont\nEnvoyé : lundi 10 septembre\nOld request.", 3000),+ ("p0411/quote_from_name_date", "Please refund it.\nFrom: Jane Smith\nDate: Mon, 10 Sep 2018 10:00\nOld request.", 3000),+ ("p0411/quote_from_name_last_line", "Please refund it.\nFrom: Jane Smith", 3000),+ ("p0411/quote_from_name_first_line", "From: Jane Smith\nSent: Monday\nOld request.", 3000),+ ("p0411/quote_gt_lines_dropped", "Please refund it.\n> quoted one\n > quoted two\nStill mine.", 3000),+ ("p0411/quote_attribution_tail_en", "Please refund it.\nOn Mon, 10/09/2018, Jane Smith\[email protected]> wrote:\nOld request.", 3000),+ ("p0411/quote_attribution_tail_pt", "Preciso de ajuda.\nEm 10/09/2018, Fulano de Tal\[email protected]> escreveu:\nOld request.", 3000),+ ("p0411/quote_attribution_tail_es", "Necesito ayuda.\nEl 10/09/2018, Fulano de Tal\[email protected]> escribió:\nOld request.", 3000),+ ("p0411/quote_attribution_tail_fr", "Je demande de l'aide.\nLe 10/09/2018, Fulano\[email protected]> a écrit :\nOld request.", 3000),+ ("p0411/quote_attribution_tail_no_head_date", "Please refund it.\nOn the other hand\[email protected]> wrote:\nOld request.", 3000),+ ("p0411/quote_attribution_tail_unrelated_head", "Please refund it.\nWith thanks 10/09\[email protected]> wrote:\nOld request.", 3000),+ ("p0411/quote_attribution_tail_first_line", "[email protected]> wrote:\nOld request.", 3000),+ ("p0411/quote_attribution_tail_too_long", "Please refund it.\nOn 10/09\n" + "x" * 130 + "@y.com> wrote:\nOld request.", 3000),+ ("p0411/quote_crlf", "Please refund it.\r\nOn Mon, Jan 1, 2024 Jane wrote:\r\nOld request.", 3000),+ ("p0411/quote_lone_cr", "Please refund it.\rOn Mon, Jan 1, 2024 Jane wrote:\rOld request.", 3000),+ ("p0411/quote_literal_backslash_n", "Please refund it.\\nOn Mon, Jan 1, 2024 Jane wrote:\\nOld request.", 3000),+ # signatures+ ("p0411/sig_dashes", "Line one.\nLine two.\nLine three.\nLine four.\nLine five.\n--\nJohn Smith\nCEO", 3000),+ ("p0411/sig_dashes_trailing_space", "Line one.\nLine two.\nLine three.\nLine four.\nLine five.\n-- \nJohn Smith", 3000),+ ("p0411/sig_best_regards", "Please refund it.\nBest regards,\nJohn Smith", 3000),+ ("p0411/sig_kind_regards_name", "Please refund it.\nKind regards, John", 3000),+ ("p0411/sig_thanks_and_regards", "Please refund it.\nThanks and regards, José", 3000),+ ("p0411/sig_thanks_ampersand_regards", "Please refund it.\nThanks & regards,\nJosé", 3000),+ ("p0411/sig_warmest_wishes", "Please refund it.\nWarmest wishes, Pat", 3000),+ ("p0411/sig_many_thanks_in_advance", "Please refund it.\nMany thanks in advance!", 3000),+ ("p0411/sig_thank_you_so_much", "Please refund it.\nThank you so much, Priya Sharma", 3000),+ ("p0411/sig_three_name_tokens", "Please refund it.\nThanks, Anna Maria Lopez", 3000),+ ("p0411/sig_four_name_tokens_is_prose", "Please refund it.\nThanks, Anna Maria Lopez Garcia", 3000),+ ("p0411/sig_thanks_for_is_prose", "Please refund it.\nThanks for the quick reply", 3000),+ ("p0411/sig_thanks_digits", "Please refund it.\nThanks, Anna 2", 3000),+ ("p0411/sig_apostrophe_hyphen_name", "Please refund it.\nRegards, D'Arcy-Smith", 3000),+ ("p0411/sig_lowercase_name_is_prose", "Please refund it.\nThanks, anna", 3000),+ ("p0411/sig_caseless_name", "Please refund it.\nRegards, 山田", 3000),+ ("p0411/sig_cyrillic_name", "Please refund it.\nSincerely, Дмитрий", 3000),+ ("p0411/sig_name_with_semicolon_is_prose", "Please refund it.\nThanks; Anna", 3000),+ ("p0411/sig_cheers_bang", "Please refund it.\nCheers!!", 3000),+ ("p0411/sig_too_long_is_not_cut", "Please refund it.\nBest regards, " + "Averyveryverylongnamethatgoes", 3000),+ ("p0411/sig_pt_atenciosamente", "Preciso de ajuda.\nAtenciosamente,\nMaria", 3000),+ ("p0411/sig_pt_att", "Preciso de ajuda.\nAtt.\nMaria", 3000),+ ("p0411/sig_pt_abraco", "Preciso de ajuda.\nAbraços!\nMaria", 3000),+ ("p0411/sig_pt_um_abraco", "Preciso de ajuda.\nUm abraço\nMaria", 3000),+ ("p0411/sig_pt_obrigado_desde_ja", "Preciso de ajuda.\nObrigado desde já!\nMaria", 3000),+ ("p0411/sig_pt_muito_obrigada_atencao", "Preciso de ajuda.\nMuito obrigada pela atenção.\nMaria", 3000),+ ("p0411/sig_pt_obrigado_request_is_prose", "Preciso de ajuda.\nObrigado pelo retorno, mas ainda preciso do boleto.", 3000),+ ("p0411/sig_pt_cumprimentos", "Preciso de ajuda.\nCom os melhores cumprimentos,\nMaria", 3000),+ ("p0411/sig_es_saludos_cordiales", "Necesito ayuda.\nSaludos cordiales,\nJuan", 3000),+ ("p0411/sig_es_un_saludo", "Necesito ayuda.\nUn saludo\nJuan", 3000),+ ("p0411/sig_es_gracias_de_antemano", "Necesito ayuda.\nMuchas gracias de antemano.\nJuan", 3000),+ ("p0411/sig_es_atentamente", "Necesito ayuda.\nAtentamente,\nJuan", 3000),+ ("p0411/sig_fr_cordialement", "Je demande de l'aide.\nCordialement,\nMarie", 3000),+ ("p0411/sig_fr_bien_cordialement", "Je demande de l'aide.\nBien cordialement\nMarie", 3000),+ ("p0411/sig_fr_salutations_distinguees", "Je demande de l'aide.\nSalutations distinguées,\nMarie", 3000),+ ("p0411/sig_fr_bien_a_vous", "Je demande de l'aide.\nBien à vous\nMarie", 3000),+ ("p0411/sig_fr_merci_d_avance", "Je demande de l'aide.\nMerci d'avance !\nMarie", 3000),+ ("p0411/sig_fr_bonne_journee", "Je demande de l'aide.\nBonne journée.\nMarie", 3000),+ ("p0411/sig_fr_merci_request_is_prose", "Je demande de l'aide.\nMerci pour votre aide, mais il reste un problème.", 3000),+ ("p0411/sig_scan_starts_late",+ "Cheers,\nJohn\n" + "\n".join("Body line %d." % i for i in range(12)), 3000),+ ("p0411/sig_scan_window_ten_lines",+ "\n".join("Body line %d." % i for i in range(5)) + "\nThanks,\n" + "\n".join("More %d." % i for i in range(4)), 3000),+ ("p0411/sig_in_short_body_first_line_never_cut", "Thanks,\nPlease refund it.", 3000),+ ("p0411/sig_longer_than_forty_is_not_cut",+ "Line one.\nLine two.\nLine three.\nLine four.\nLine five.\nRegards, Someone With A Very Long Name Indeed Yes", 3000),+ # device footers+ ("p0411/device_sent_from_iphone", "Please refund it.\nSent from my iPhone", 3000),+ ("p0411/device_sent_from_galaxy", "Please refund it.\nSent from my Samsung Galaxy S21 Ultra 5G.", 3000),+ ("p0411/device_sent_from_using", "Please refund it.\nSent from Outlook using mail-client_1.2", 3000),+ ("p0411/device_samsung_pt", "Preciso de ajuda.\nEnviado do meu smartphone Samsung Galaxy.", 3000),+ ("p0411/device_pt_celular_request", "Preciso de ajuda.\nEnviado do meu celular o comprovante ontem.", 3000),+ ("p0411/device_get_outlook_ios", "Please refund it.\nGet Outlook for iOS", 3000),+ ("p0411/device_obter_outlook_android", "Preciso de ajuda.\nObter o Outlook para Android", 3000),+ ("p0411/device_fr_iphone", "Je demande de l'aide.\nEnvoyé depuis mon iPhone", 3000),+ ("p0411/device_fr_de_ipad", "Je demande de l'aide.\nEnvoyé de mon iPad.", 3000),+ ("p0411/device_fr_mentioned_in_prose", "Bonjour,\nEnvoyé depuis mon iPhone par erreur.\nMerci de rembourser la facture.", 3000),+ ("p0411/device_es_movil", "Necesito ayuda.\nEnviado desde mi móvil", 3000),+ ("p0411/device_pt_mi", "Preciso de ajuda.\nEnviado via mi Android", 3000),+ ("p0411/device_longer_than_sixty", "Please refund it.\nSent from my iPhone " + "phone " * 12, 3000),+ # disclaimers+ ("p0411/disclaimer_en_standard", "Please refund it.\n\n" + _DISCLAIMER_TEXT, 3000),+ ("p0411/disclaimer_en_wrapped", "Please refund it.\n\nThis email and its contents are\nconfidential and intended solely for the addressee.", 3000),+ ("p0411/disclaimer_en_privileged", "Please refund it.\n\nThis information is confidential and may be privileged.", 3000),+ ("p0411/disclaimer_en_received_in_error", "Please refund it.\n\nIf you have received this email in error please delete it.", 3000),+ ("p0411/disclaimer_en_question_is_body", "Is this confidential?", 3000),+ ("p0411/disclaimer_en_fused_lines", "My account is locked\nThis email is confidential and intended solely for the addressee.", 3000),+ ("p0411/disclaimer_en_inline", "My account is locked. This email is confidential and intended solely for the addressee. Please help me.", 3000),+ ("p0411/disclaimer_pt", "Preciso de ajuda.\n\nEsta mensagem é confidencial e destinada exclusivamente ao destinatário.", 3000),+ ("p0411/disclaimer_pt_request_confidencial", "Preciso do contrato confidencial.", 3000),+ ("p0411/disclaimer_pt_print", "Preciso de ajuda.\n\nAntes de imprimir, pense no meio ambiente.", 3000),+ ("p0411/disclaimer_pt_print_request", "Antes de imprimir o boleto, confira o valor.", 3000),+ ("p0411/disclaimer_pt_ambiente_first", "Preciso de ajuda.\n\nPense no meio ambiente antes de imprimir.", 3000),+ ("p0411/disclaimer_pt_recebeu_engano", "Preciso de ajuda.\n\nSe você recebeu esta mensagem por engano, apague-a.", 3000),+ ("p0411/disclaimer_es", "Necesito ayuda.\n\nEste mensaje es confidencial y de uso exclusivo del destinatario.", 3000),+ ("p0411/disclaimer_es_recibido_error", "Necesito ayuda.\n\nSi ha recibido este mensaje por error, bórrelo.", 3000),+ ("p0411/disclaimer_fr", "Je demande de l'aide.\n\nCe message est confidentiel et réservé au destinataire.", 3000),+ ("p0411/disclaimer_fr_recu_erreur", "Je demande de l'aide.\n\nSi vous avez reçu ce message par erreur, supprimez-le.", 3000),+ ("p0411/disclaimer_fr_request_confidentiel", "Je veux le contrat confidentiel.", 3000),+ ("p0411/disclaimer_fr_usage_exclusif", "Je demande de l'aide.\n\nUsage exclusif du destinataire.", 3000),+ ("p0411/disclaimer_mixed_paragraph", "Please help me. The email is confidential and intended solely for you. Thanks!", 3000),+ ("p0411/disclaimer_uppercase_split", "Locked out\nThis email is confidential and intended solely for you.\nMore text.", 3000),+ ("p0411/disclaimer_caseless_wrapped", "退款请求\n\n这封邮件是机密的 This email is\nconfidential and intended solely for you.", 3000),+ ("p0411/disclaimer_sentence_split_ws", "Hello there.\u2003\u2003This email is confidential and intended solely for you. Bye!", 3000),+ ("p0411/whitespace_collapse", "Please refund\t\tit.\n\n\n\nThanks a lot", 3000),+ ("p0411/empty_body", "", 3000),+ ("p0411/only_quoted", "> a\n> b", 3000),+ ("p0411/only_whitespace", " \n\t\n ", 3000),+ ("p0411/unit_separator_blank_line", "First\n\x1f\nSecond", 3000),+ ("p0411/unicode_whitespace_line", "First\n\u00a0\nSecond\n\u2003\n\nThird", 3000),+ # ---- Python `re` semantics the `regex` crate gets differently (\b over Python's \w, re.I folding) ----+ ("pyre/boundary_after_no_superscript",+ "Please help me.\n\nThis email² is confidential and intended solely for you.", 3000),+ ("pyre/boundary_before_no_superscript",+ "Please help me.\n\nThis ²email is confidential and intended solely for you.", 3000),+ ("pyre/boundary_after_combining_mark",+ "Please help me.\n\nThis emaiĺ is confidential and intended solely for you.", 3000),+ ("pyre/boundary_before_combining_mark",+ "Please help me.\n\nThis ́email is confidential and intended solely for you.", 3000),+ ("pyre/boundary_after_connector_punctuation",+ "Please help me.\n\nThis email‿ is confidential and intended solely for you.", 3000),+ ("pyre/boundary_after_zero_width_joiner",+ "Please help me.\n\nThis email is confidential and intended solely for you.", 3000),+ ("pyre/boundary_after_circled_letter",+ "Please help me.\n\nThis emailⒶ is confidential and intended solely for you.", 3000),+ ("pyre/boundary_after_roman_numeral",+ "Preciso de ajuda.\n\nEsta mensagemⅠ é confidencial.", 3000),+ ("pyre/ignorecase_dotted_capital_i_disclaimer",+ "Please help me.\n\nİf you received this email in error, delete it.", 3000),+ ("pyre/ignorecase_dotless_i_disclaimer",+ "Please help me.\n\nIf you received this email ın error, delete it.", 3000),+ ("pyre/ignorecase_dotless_i_device_footer", "Please help me.\nSent from my ıPhone", 3000),+ ("pyre/ignorecase_dotted_i_device_range",+ "Please help me.\nSent from my iPhone using İOS", 3000),+ ("pyre/ignorecase_dotless_i_signoff", "Please help me.\nKınd regards,\nJohn", 3000),+ ("pyre/ignorecase_dotless_i_pt_signoff", "Preciso de ajuda.\nCordıalmente,\nMaria", 3000),+ ("pyre/ignorecase_dotted_i_original_message",+ "Please help me.\n-----Orİginal Message-----\nOld request.", 3000),+ ("pyre/ignorecase_dotless_i_french_signoff", "Je demande de l'aide.\nBonne journée,\nMarıe", 3000),+ ("pyre/ignorecase_long_s_and_kelvin_sign",+ "Please help me.\n\nThis email is confidential and intended ſolely for you, thankK you.", 3000),+ # astral characters: `re` counts and bounds code points, a UTF-16 engine sees two units+ ("pyre/astral_boundary_after_letter",+ "Please help me.\n\nThis email\U0001d400 is confidential and intended solely for you.", 3000),+ ("pyre/astral_boundary_before_letter",+ "Please help me.\n\nThis \U0001d400email is confidential and intended solely for you.", 3000),+ ("pyre/astral_boundary_after_emoji",+ "Please help me.\n\nThis email\U0001f600 is confidential and intended solely for you.", 3000),+ ("pyre/astral_dot_run_within_bound",+ "Please help me.\n\nThis email " + "\U0001d400" * 40 + " is confidential and intended solely for you.", 3000),+ ("pyre/astral_dot_run_past_bound",+ "Please help me.\n\nThis email " + "\U0001d400" * 70 + " is confidential and intended solely for you.", 3000),+ ("pyre/astral_quote_header_within_bound",+ "Please help me.\nOn Monday 1 Jan, " + "\U0001d400" * 200 + " wrote:\n> old request", 3000),+ ("pyre/astral_quote_header_past_bound",+ "Please help me.\nOn Monday 1 Jan, " + "\U0001d400" * 301 + " wrote:\n> old request\nStill mine.", 3000),+ ("pyre/astral_digit_boundary",+ "Please help me.\n\nThis email\U0001d7ce is confidential and intended solely for you.", 3000),]EMAIL_STATE_CASES = [@@ -520,6 +932,21 @@ EMAIL_STATE_CASES = [("extra_none_values_dropped", "Ticket", "Body text.", None, True, {"nullable": None, "kept": "value"}),("empty_subject_and_body", "", "", None, True, {}),("hindi_subject_and_body", "सहायता चाहिए", "मुझसे दो बार शुल्क लिया गया, कृपया रिफंड करें।", None, True, {}),+ # ---- Python 0.4.11 behaviours neither SDK's upstream corpus pinned ----+ ("p0411/max_chars_extra_small", "Locked out", "My account is locked and I need help right now.", None, True, {"max_chars": 10}),+ ("p0411/max_chars_extra_zero", "Locked out", "My account is locked.", None, True, {"max_chars": 0}),+ ("p0411/max_chars_extra_negative", "Locked out", "My account is locked and I need help right now.", None, True, {"max_chars": -5}),+ ("p0411/max_chars_extra_large", "Locked out", "My account is locked. " * 400, None, True, {"max_chars": 100000}),+ ("p0411/max_chars_default_cut", "Locked out", "My account is locked. " * 400, None, True, {}),+ ("p0411/max_chars_ignored_when_not_clean", "Locked out", "My account is locked and I need help right now.", None, False, {"max_chars": 5}),+ ("p0411/max_chars_with_other_extra", "Locked out", "My account is locked and I need help.", "[email protected]", True, {"max_chars": 7, "priority": "high"}),+ ("p0411/sender_empty_dropped", "Locked out", "My account is locked.", "", True, {}),+ ("p0411/extra_overrides_from", "Locked out", "My account is locked.", "[email protected]", True, {"from": "[email protected]"}),+ ("p0411/extra_non_string_values", "Locked out", "My account is locked.", None, True, {"n": 1, "f": 1.5, "b": False, "l": [1, "x"], "d": {"k": "v"}}),+ ("p0411/subject_none_like_empty", "", "My account is locked.", None, True, {}),+ ("p0411/subject_unicode_whitespace", "\u00a0 Locked out \u2003", "My account is locked.", None, True, {}),+ ("p0411/state_quote_cleaned", "Locked out", "Please unlock it.\nOn Mon, Jan 1, 2024 Jane wrote:\nOld text.", None, True, {}),+ ("p0411/state_unclean_keeps_quote", "Locked out", "Please unlock it.\nOn Mon, Jan 1, 2024 Jane wrote:\nOld text.", None, False, {}),]