Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

test(dotnet): regenerate goldens from Python 0.3.24

kevin-gatimu4f22e901bd9d
diff --git a/laya-dotnet/CHANGELOG.md b/laya-dotnet/CHANGELOG.md
index 3eb0df09..88f395a6 100644
--- a/laya-dotnet/CHANGELOG.md
+++ b/laya-dotnet/CHANGELOG.md
@@ -6,7 +6,7 @@ All notable changes to the `Laya.Onnx` NuGet package are documented here. The fo
## [0.1.0] - unreleased
-First release: a port of the Laya Python SDK, tested against laya (Python) 0.3.21.
+First release: a port of the Laya Python SDK, tested against laya (Python) 0.3.24.
### Added
@@ -20,8 +20,23 @@ First release: a port of the Laya Python SDK, tested against laya (Python) 0.3.2
- Opt-in model download from Hugging Face.
- Samples: quickstart, routing and benchmark.
+### Fixed
+
+- Undetermined Latin text uses the router's configured default, including its route reason.
+- Language detection counts unlisted scripts, distinguishes mixed-language fields and lines
+ (`MixedSegment`), and applies current script, stopword and loanword rules.
+- Email cleaning preserves unpunctuated requests before disclaimers, distinguishes prose from
+ signatures and reply headers, and handles Portuguese, Spanish and French client markers.
+- Oversized questions report the surviving option markers and the actual sequence budget.
+- Score legends return the same rendered criterion text shown to the model, including JSON for
+ structured levels.
+- Golden fixtures regenerated from current Python for all three checkpoints and routing.
+
### Known gaps
-- Language detection and email handling follow Python 0.3.6 (`lang.py`, `email.py`).
-- Not yet ported from Python 0.3.21: abstention (`min_confidence`), `predict_long` and
- `lang_temperatures`.
+- Python-only opt-in features are not yet ported: abstention (`min_confidence`), `predict_long`,
+ `lang_temperatures`, hooks, batching APIs, question option ordering and digest controls.
+- `LayaEmail.State` does not expose Python's `max_chars` keyword; use `CleanBody` with a custom
+ budget and pass the cleaned body with `clean: false`.
+- New Python usage diagnostics (`state_tokens`, `truncated`, `state_tokens_dropped` and
+ `truncated_questions`) are not exposed by the .NET `Usage` record.
diff --git a/laya-dotnet/README.md b/laya-dotnet/README.md
index 9ff56336..2a576245 100644
--- a/laya-dotnet/README.md
+++ b/laya-dotnet/README.md
@@ -322,7 +322,7 @@ foreach (var id in result.Ids)
score.Score; // expected level: sum of i * p[i], fractional, 0..Legend.Count-1
score.MostLikelyLevel; // argmax level index
score.Probabilities; // IReadOnlyList<double>, one per level
- score.Legend; // the level descriptions you supplied
+ score.Legend; // rendered level descriptions (structured criteria become JSON text)
break;
case NoulAnswer noul:
@@ -860,19 +860,21 @@ with `--solution` rather than as a positional argument.
## Known gaps
-This SDK tracks the Python SDK at **0.3.21** for everything above, but two modules are pinned to
-an older Python release and have not been ported forward yet:
-
-- **Language detection** (`LanguageDetection`) follows Python's `lang.py` at **0.3.6**.
-- **Email handling** (`LayaEmail`) follows Python's `email.py` at **0.3.6**.
-
-Bringing both up to 0.3.21 is planned as a follow-up; until then, treat their behavior as frozen
-at 0.3.6 even though the rest of the SDK (sequence building, JSON dialects, calibration and
-answers) has moved to 0.3.21.
+Default predictions and routing are verified against goldens regenerated from Python
+**0.3.24**. Language detection uses current script and mixed-language rules; `MixedSegment`
+identifies the foreign line or field that overrides a mostly-English state. Undetermined
+Latin text uses the router's configured default. Email cleaning follows current request,
+disclaimer, signature and reply-header rules, including Portuguese, Spanish and French markers.
- **Opt-in features added in Python 0.3.21 are not ported yet:** abstention (`min_confidence`),
`predict_long`, and per-language temperatures (`lang_temperatures`). They are off by default in
Python, so default predictions are unaffected. They are planned as follow-ups.
+- **Other Python-only APIs are not ported:** hooks, batching APIs, question option ordering and
+ digest controls.
+- **Custom email-state body budgets:** `LayaEmail.State` does not expose Python's `max_chars`
+ keyword. Call `LayaEmail.CleanBody(body, maxChars)` and pass its result with `clean: false`.
+- **Usage diagnostics:** Python's `state_tokens`, `truncated`, `state_tokens_dropped` and
+ `truncated_questions` are not yet exposed by .NET's `Usage` record.
---
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_arabic.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_arabic.json
index 6cc59464..7f93f77a 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_arabic.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_arabic.json
@@ -889,18 +889,18 @@
"marker_axis_padded": false,
"logits": [
[
- 0.4125833809375763,
- -0.19337722659111023
+ 0.4125804305076599,
+ -0.1933775544166565
],
[
- 3.3612887859344482,
- 0.18131715059280396
+ 3.3612828254699707,
+ 0.18131807446479797
]
],
"act_logits": [
[
- 4117.9599609375,
- -3364.359375
+ 4117.95849609375,
+ -3364.35888671875
],
[
4817.98095703125,
@@ -935,7 +935,11 @@
},
"usage": {
"input_tokens": 252,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 69,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_choice_criteria_list.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_choice_criteria_list.json
index de8c70e0..fc8126bd 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_choice_criteria_list.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_choice_criteria_list.json
@@ -162,16 +162,16 @@
"marker_axis_padded": false,
"logits": [
[
- 6.577561855316162,
- -2.623333692550659,
- -3.148845672607422,
- -0.5583869218826294
+ 6.577563762664795,
+ -2.62333607673645,
+ -3.1488473415374756,
+ -0.5583876371383667
]
],
"act_logits": [
[
- 3876.216552734375,
- -3165.7255859375
+ 3876.214111328125,
+ -3165.72265625
]
],
"result": {
@@ -195,7 +195,11 @@
},
"usage": {
"input_tokens": 31,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 10,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_cjk.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_cjk.json
index 980e4f85..ce9e2c98 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_cjk.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_cjk.json
@@ -853,24 +853,24 @@
"marker_axis_padded": false,
"logits": [
[
- 1.6422864198684692,
- -0.7313228845596313,
- 0.32396093010902405
+ 1.6422786712646484,
+ -0.7313200235366821,
+ 0.32396090030670166
],
[
- -1.1324100494384766,
- 1.4276299476623535,
- 2.908419370651245
+ -1.1324177980422974,
+ 1.427627444267273,
+ 2.9084181785583496
]
],
"act_logits": [
[
- 4157.708984375,
- -3400.5478515625
+ 4157.71044921875,
+ -3400.548828125
],
[
- 3446.069580078125,
- -2831.5048828125
+ 3446.071533203125,
+ -2831.50634765625
]
],
"result": {
@@ -912,7 +912,11 @@
},
"usage": {
"input_tokens": 222,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 49,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_criterion_shapes.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_criterion_shapes.json
index dfec56cc..9ec61ac4 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_criterion_shapes.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_criterion_shapes.json
@@ -609,18 +609,18 @@
"marker_axis_padded": false,
"logits": [
[
- 4.096678733825684,
- -0.2300950288772583,
- -2.4581174850463867,
- -2.5262246131896973,
- -1.8533413410186768,
- -1.0468189716339111,
+ 4.096677780151367,
+ -0.23009303212165833,
+ -2.4581172466278076,
+ -2.5262258052825928,
+ -1.8533432483673096,
+ -1.0468206405639648,
-1.4776298999786377
],
[
- 0.10750558972358704,
- 2.0651817321777344,
- -0.057980358600616455,
+ 0.10751175880432129,
+ 2.0651803016662598,
+ -0.057982414960861206,
-10000.0,
-10000.0,
-10000.0,
@@ -629,12 +629,12 @@
],
"act_logits": [
[
- 4621.1904296875,
- -3779.87548828125
+ 4621.18994140625,
+ -3779.8759765625
],
[
- 4256.392578125,
- -3493.38330078125
+ 4256.39208984375,
+ -3493.3828125
]
],
"result": {
@@ -662,14 +662,8 @@
"type": "score",
"score": 0.9814,
"legend": {
- "0": {
- "label": "info",
- "cvss": 0.0
- },
- "1": {
- "label": "high",
- "cvss": 7.5
- },
+ "0": "{\"label\": \"info\", \"cvss\": 0.0}",
+ "1": "{\"label\": \"high\", \"cvss\": 7.5}",
"2": "critical"
},
"probabilities": {
@@ -686,7 +680,11 @@
},
"usage": {
"input_tokens": 144,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 11,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_dict_state.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_dict_state.json
index 40b0d65c..7dfc30d8 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_dict_state.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_dict_state.json
@@ -357,15 +357,15 @@
"marker_axis_padded": false,
"logits": [
[
- -0.2836790680885315,
- 2.304533004760742,
- 1.6238430738449097
+ -0.2836797535419464,
+ 2.3045341968536377,
+ 1.6238452196121216
]
],
"act_logits": [
[
- 3474.172607421875,
- -2870.3486328125
+ 3474.17578125,
+ -2870.3505859375
]
],
"result": {
@@ -393,7 +393,11 @@
},
"usage": {
"input_tokens": 93,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 64,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_json_escaping.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_json_escaping.json
index 587668f6..a27f182a 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_json_escaping.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_json_escaping.json
@@ -467,14 +467,14 @@
"marker_axis_padded": false,
"logits": [
[
- 2.0836143493652344,
- 0.5918171405792236
+ 2.0836265087127686,
+ 0.5918163061141968
]
],
"act_logits": [
[
- 4576.2177734375,
- -3738.3583984375
+ 4576.21875,
+ -3738.359375
]
],
"result": {
@@ -492,7 +492,11 @@
},
"usage": {
"input_tokens": 134,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 103,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_list_state.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_list_state.json
index 63cc620c..e2a15372 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_list_state.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_list_state.json
@@ -321,7 +321,7 @@
"logits": [
[
4.156437397003174,
- -0.3092767000198364
+ -0.30927616357803345
]
],
"act_logits": [
@@ -345,7 +345,11 @@
},
"usage": {
"input_tokens": 84,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 53,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_long_state.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_long_state.json
index fd5c9aa9..0ed50989 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_long_state.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_long_state.json
@@ -3165,22 +3165,22 @@
"marker_axis_padded": false,
"logits": [
[
- 2.5547876358032227,
- 0.21161600947380066
+ 2.5547900199890137,
+ 0.21161597967147827
],
[
- 1.9042129516601562,
- -0.07903063297271729
+ 1.9042150974273682,
+ -0.07902958989143372
]
],
"act_logits": [
[
- 4784.12890625,
+ 4784.12939453125,
-3916.51171875
],
[
- 4584.17822265625,
- -3742.8369140625
+ 4584.177734375,
+ -3742.83642578125
]
],
"result": {
@@ -3211,7 +3211,14 @@
},
"usage": {
"input_tokens": 1024,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 6804,
+ "state_tokens_dropped": 6323,
+ "truncated": true,
+ "truncated_questions": [
+ "frustrated",
+ "topic"
+ ]
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_many_options.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_many_options.json
index 52cfe818..8ac0cfc4 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_many_options.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_many_options.json
@@ -864,52 +864,52 @@
"marker_axis_padded": false,
"logits": [
[
- 0.6046832203865051,
- 0.7550267577171326,
- 0.9181180000305176,
- 0.8010208010673523,
+ 0.6046816110610962,
+ 0.7550269365310669,
+ 0.9181175231933594,
+ 0.801021158695221,
0.5008540153503418,
- 0.5897595286369324,
- 0.615754246711731,
+ 0.5897594094276428,
+ 0.6157547235488892,
0.6029906272888184,
0.5069749355316162,
- 0.6346789598464966,
- 0.7126632332801819,
- 0.5554916858673096,
- 0.319332480430603,
- 0.3918170928955078,
- 0.35489070415496826,
- 0.33871594071388245,
- 0.1896839439868927,
- 0.5009076595306396,
- 0.5126965641975403,
- 0.3750284016132355,
- 0.6199686527252197,
- 0.015123069286346436,
- -0.3031226396560669,
- 0.010186314582824707,
- 0.0962013304233551,
- -0.006642162799835205,
- 0.005327850580215454,
- -0.005197674036026001,
- 0.5291510820388794,
- 0.5070405602455139,
- 0.1432766318321228,
- 0.2855176329612732,
- -0.2885570228099823,
- 0.09954589605331421,
- 0.2411927580833435,
- -0.07459813356399536,
- -0.13390323519706726,
- 0.3199225664138794,
- 0.09801286458969116,
- -0.3789292573928833
+ 0.6346783638000488,
+ 0.7126637697219849,
+ 0.5554904937744141,
+ 0.31933215260505676,
+ 0.3918158710002899,
+ 0.35489052534103394,
+ 0.3387143909931183,
+ 0.18968331813812256,
+ 0.5009080171585083,
+ 0.5126953721046448,
+ 0.37502649426460266,
+ 0.6199690103530884,
+ 0.015121310949325562,
+ -0.30312448740005493,
+ 0.010184824466705322,
+ 0.09620040655136108,
+ -0.006642788648605347,
+ 0.005326271057128906,
+ -0.005198240280151367,
+ 0.52914959192276,
+ 0.5070400238037109,
+ 0.14327645301818848,
+ 0.285515159368515,
+ -0.28855693340301514,
+ 0.09954556822776794,
+ 0.24119195342063904,
+ -0.07459908723831177,
+ -0.13390490412712097,
+ 0.3199203908443451,
+ 0.09801268577575684,
+ -0.3789302408695221
]
],
"act_logits": [
[
- 4584.29931640625,
- -3744.84423828125
+ 4584.2998046875,
+ -3744.8447265625
]
],
"result": {
@@ -969,7 +969,11 @@
},
"usage": {
"input_tokens": 205,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 12,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_mask_injection.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_mask_injection.json
index 83862f15..4410f5a4 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_mask_injection.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_mask_injection.json
@@ -203,14 +203,14 @@
"marker_axis_padded": false,
"logits": [
[
- 2.7121849060058594,
- -1.4662480354309082
+ 2.7121903896331787,
+ -1.4662481546401978
]
],
"act_logits": [
[
4438.82373046875,
- -3625.92041015625
+ -3625.9208984375
]
],
"result": {
@@ -232,7 +232,11 @@
},
"usage": {
"input_tokens": 48,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 14,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_nonstring_instructions.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_nonstring_instructions.json
index 1f91c577..97945dcb 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_nonstring_instructions.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_nonstring_instructions.json
@@ -421,21 +421,21 @@
"marker_axis_padded": false,
"logits": [
[
- 0.8464744091033936,
- 0.35750165581703186
+ 0.8464770317077637,
+ 0.3575015962123871
],
[
- 3.689389228820801,
- -0.15227413177490234
+ 3.6893937587738037,
+ -0.15227362513542175
]
],
"act_logits": [
[
4315.3662109375,
- -3523.56884765625
+ -3523.5693359375
],
[
- 4536.421875,
+ 4536.42138671875,
-3702.72509765625
]
],
@@ -467,7 +467,11 @@
},
"usage": {
"input_tokens": 104,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 14,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_options_exceed_head.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_options_exceed_head.json
index f4ef3b08..d5cadfe3 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_options_exceed_head.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_options_exceed_head.json
@@ -310,5 +310,5 @@
}
},
"error_type": "ValueError",
- "error_message": "question 'too_wide' options exceed head_max_len=192"
+ "error_message": "question 'too_wide': only 126 of its 300 option markers fit in max_len=512 with head_max_len=192 spent on the question; lower head_max_len, raise max_len, or use fewer options"
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_quickstart.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_quickstart.json
index aa1b9af2..ecae5998 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_quickstart.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_quickstart.json
@@ -1397,46 +1397,46 @@
"marker_axis_padded": false,
"logits": [
[
- 4.488272190093994,
- -2.7296648025512695,
- -2.448031425476074,
+ 4.488276958465576,
+ -2.729666233062744,
+ -2.4480319023132324,
-10000.0
],
[
- -0.8349562287330627,
- 0.8832815289497375,
- 3.6844472885131836,
- 1.2054011821746826
+ -0.8349542617797852,
+ 0.8832842707633972,
+ 3.6844465732574463,
+ 1.2054039239883423
],
[
- -4.515445709228516,
- 2.2500112056732178,
+ -4.515446186065674,
+ 2.250016689300537,
-10000.0,
-10000.0
],
[
- 0.7536221742630005,
- 0.41879814863204956,
+ 0.7536231279373169,
+ 0.41879838705062866,
-10000.0,
-10000.0
]
],
"act_logits": [
[
- 4074.86767578125,
- -3334.5009765625
+ 4074.86572265625,
+ -3334.4990234375
],
[
4040.2958984375,
-3316.3017578125
],
[
- 2926.99853515625,
- -2398.7822265625
+ 2926.997314453125,
+ -2398.78125
],
[
- 4615.16162109375,
- -3778.9384765625
+ 4615.162109375,
+ -3778.93896484375
]
],
"result": {
@@ -1498,7 +1498,11 @@
},
"usage": {
"input_tokens": 379,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_sample_app_english.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_sample_app_english.json
index 151fd8fc..4e582196 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_sample_app_english.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_sample_app_english.json
@@ -1310,38 +1310,38 @@
"marker_axis_padded": false,
"logits": [
[
- 4.679883003234863,
- -2.7772488594055176,
- -3.344090700149536,
- -3.2511987686157227
+ 4.6798810958862305,
+ -2.7772486209869385,
+ -3.344090223312378,
+ -3.2511980533599854
],
[
- 0.3843749463558197,
- 1.676945686340332,
- 2.3419666290283203,
+ 0.3843739926815033,
+ 1.6769473552703857,
+ 2.341966152191162,
-10000.0
],
[
- -1.987611174583435,
- 1.084941029548645,
+ -1.9876132011413574,
+ 1.0849405527114868,
-10000.0,
-10000.0
],
[
- -2.25665283203125,
- 1.0766032934188843,
+ -2.256652355194092,
+ 1.076602816581726,
-10000.0,
-10000.0
]
],
"act_logits": [
[
- 4204.2607421875,
- -3440.07470703125
+ 4204.26220703125,
+ -3440.0751953125
],
[
- 4087.648681640625,
- -3356.3681640625
+ 4087.649169921875,
+ -3356.36865234375
],
[
4645.5791015625,
@@ -1349,7 +1349,7 @@
],
[
4762.5341796875,
- -3903.29736328125
+ -3903.2978515625
]
],
"result": {
@@ -1410,7 +1410,11 @@
},
"usage": {
"input_tokens": 348,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 50,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_sample_app_hindi.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_sample_app_hindi.json
index 14829cdf..b6125c22 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_sample_app_hindi.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_sample_app_hindi.json
@@ -1356,26 +1356,26 @@
"marker_axis_padded": false,
"logits": [
[
- 1.0225625038146973,
- 0.24040544033050537,
- -0.9230316877365112,
- -0.3869422674179077
+ 1.0225623846054077,
+ 0.24040594696998596,
+ -0.9230324029922485,
+ -0.38694146275520325
],
[
- 2.249563455581665,
- 2.5387072563171387,
- 1.690252423286438,
+ 2.2495646476745605,
+ 2.5387086868286133,
+ 1.6902523040771484,
-10000.0
],
[
- 4.11464786529541,
- -0.10430359840393066,
+ 4.114648818969727,
+ -0.1043037474155426,
-10000.0,
-10000.0
],
[
- 4.595251083374023,
- -0.24962911009788513,
+ 4.595254421234131,
+ -0.24962997436523438,
-10000.0,
-10000.0
]
@@ -1386,16 +1386,16 @@
-3653.57080078125
],
[
- 3890.695556640625,
- -3193.38427734375
+ 3890.696044921875,
+ -3193.384765625
],
[
- 4046.17333984375,
- -3316.2900390625
+ 4046.173828125,
+ -3316.29052734375
],
[
- 4189.326171875,
- -3421.08544921875
+ 4189.32421875,
+ -3421.083984375
]
],
"result": {
@@ -1456,7 +1456,11 @@
},
"usage": {
"input_tokens": 364,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 54,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_shortlist_many_options.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_shortlist_many_options.json
index 3c7537f5..eb219eb1 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_shortlist_many_options.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_shortlist_many_options.json
@@ -780,32 +780,32 @@
"marker_axis_padded": false,
"logits": [
[
- 3.5034117698669434,
- 0.46416565775871277,
- 2.564594268798828,
- 0.30703631043434143,
- -3.4262099266052246,
- -3.6496596336364746,
+ 3.5034213066101074,
+ 0.46416550874710083,
+ 2.564589023590088,
+ 0.30703526735305786,
+ -3.426210403442383,
+ -3.6496622562408447,
-1.637428641319275,
- -3.554462194442749,
- -0.9670939445495605,
- -1.1691794395446777,
- 2.3281009197235107,
- 1.4019513130187988,
- -1.1085056066513062,
- -2.09653902053833,
- -3.2419776916503906,
- -2.413379192352295,
- -3.6436588764190674,
+ -3.5544652938842773,
+ -0.9670940637588501,
+ -1.1691828966140747,
+ 2.3281021118164062,
+ 1.4019540548324585,
+ -1.1085090637207031,
+ -2.096541404724121,
+ -3.241981029510498,
+ -2.4133825302124023,
+ -3.643660545349121,
-0.12069264054298401,
- -1.7674641609191895,
- -2.5909695625305176
+ -1.7674661874771118,
+ -2.5909717082977295
]
],
"act_logits": [
[
- 3764.427490234375,
- -3073.59716796875
+ 3764.427001953125,
+ -3073.59619140625
]
],
"result": {
@@ -846,6 +846,10 @@
"usage": {
"input_tokens": 194,
"output_tokens": 0,
+ "state_tokens": 19,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": [],
"options": {
"intent": {
"total": 20,
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/case_single_option.json b/laya-dotnet/tests/Laya.Tests/golden/english/case_single_option.json
index 2e25b016..8f61dd56 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/case_single_option.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/case_single_option.json
@@ -125,14 +125,14 @@
"marker_axis_padded": true,
"logits": [
[
- 1.072884202003479,
+ 1.0728870630264282,
-10000.0
]
],
"act_logits": [
[
- 4239.0380859375,
- -3460.41748046875
+ 4239.0400390625,
+ -3460.41845703125
]
],
"result": {
@@ -153,7 +153,11 @@
},
"usage": {
"input_tokens": 23,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 4,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/english/meta.json b/laya-dotnet/tests/Laya.Tests/golden/english/meta.json
index 88e74919..0822aa10 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/english/meta.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/english/meta.json
@@ -1,7 +1,7 @@
{
"generator": "tools/dump_golden.py",
"onnx_dir": "english",
- "laya_version": "0.3.21",
+ "laya_version": "0.3.24",
"onnxruntime_version": "1.30.0",
"transformers_version": "5.17.0",
"config": {
@@ -70,12 +70,12 @@
},
"onnx_self_check": {
"single_noul": {
- "logits_maxdiff": 1.9073486328125e-06,
- "act_maxdiff": 0.001220703125
+ "logits_maxdiff": 9.953975677490234e-06,
+ "act_maxdiff": 0.0009765625
},
"triage_4q": {
- "logits_maxdiff": 2.288818359375e-05,
- "act_maxdiff": 0.00390625
+ "logits_maxdiff": 1.946091651916504e-05,
+ "act_maxdiff": 0.00146484375
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_arabic.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_arabic.json
index 558a4c88..6a51bd57 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_arabic.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_arabic.json
@@ -569,7 +569,11 @@
},
"usage": {
"input_tokens": 142,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 32,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_choice_criteria_list.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_choice_criteria_list.json
index 69510329..fb868590 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_choice_criteria_list.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_choice_criteria_list.json
@@ -192,7 +192,11 @@
},
"usage": {
"input_tokens": 30,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 10,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_cjk.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_cjk.json
index a9faf0e8..ffd5a829 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_cjk.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_cjk.json
@@ -571,7 +571,11 @@
},
"usage": {
"input_tokens": 129,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 27,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_criterion_shapes.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_criterion_shapes.json
index 86bc577e..f93493eb 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_criterion_shapes.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_criterion_shapes.json
@@ -666,14 +666,8 @@
"type": "score",
"score": 1.5365,
"legend": {
- "0": {
- "label": "info",
- "cvss": 0.0
- },
- "1": {
- "label": "high",
- "cvss": 7.5
- },
+ "0": "{\"label\": \"info\", \"cvss\": 0.0}",
+ "1": "{\"label\": \"high\", \"cvss\": 7.5}",
"2": "critical"
},
"probabilities": {
@@ -690,7 +684,11 @@
},
"usage": {
"input_tokens": 148,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 11,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_dict_state.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_dict_state.json
index 77f3c4d9..9f1cefa6 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_dict_state.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_dict_state.json
@@ -393,7 +393,11 @@
},
"usage": {
"input_tokens": 93,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 61,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_json_escaping.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_json_escaping.json
index 5b046da7..faedbc79 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_json_escaping.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_json_escaping.json
@@ -540,7 +540,11 @@
},
"usage": {
"input_tokens": 150,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 119,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_list_state.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_list_state.json
index ab45d4a4..0c02cb4e 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_list_state.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_list_state.json
@@ -342,7 +342,11 @@
},
"usage": {
"input_tokens": 83,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 52,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_long_state.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_long_state.json
index acc6a671..d9587269 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_long_state.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_long_state.json
@@ -6283,7 +6283,14 @@
},
"usage": {
"input_tokens": 2048,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 7894,
+ "state_tokens_dropped": 6901,
+ "truncated": true,
+ "truncated_questions": [
+ "frustrated",
+ "topic"
+ ]
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_many_options.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_many_options.json
index e01188c1..965c5a5c 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_many_options.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_many_options.json
@@ -1164,7 +1164,11 @@
},
"usage": {
"input_tokens": 270,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 10,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_mask_injection.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_mask_injection.json
index f7b403bf..f0fe7d5a 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_mask_injection.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_mask_injection.json
@@ -211,7 +211,11 @@
},
"usage": {
"input_tokens": 41,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 11,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_nonstring_instructions.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_nonstring_instructions.json
index fc23513c..cef51565 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_nonstring_instructions.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_nonstring_instructions.json
@@ -427,7 +427,11 @@
},
"usage": {
"input_tokens": 92,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 8,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_options_exceed_head.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_options_exceed_head.json
index 2336adc2..00334c9d 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_options_exceed_head.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_options_exceed_head.json
@@ -310,5 +310,5 @@
}
},
"error_type": "ValueError",
- "error_message": "question 'too_wide' options exceed head_max_len=256"
+ "error_message": "question 'too_wide': only 254 of its 300 option markers fit in max_len=1024 with head_max_len=256 spent on the question; lower head_max_len, raise max_len, or use fewer options"
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_quickstart.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_quickstart.json
index 6c55706c..d674c82b 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_quickstart.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_quickstart.json
@@ -1462,7 +1462,11 @@
},
"usage": {
"input_tokens": 375,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 56,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_sample_app_english.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_sample_app_english.json
index 57c1c11d..9ca2e154 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_sample_app_english.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_sample_app_english.json
@@ -1386,7 +1386,11 @@
},
"usage": {
"input_tokens": 348,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 50,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_sample_app_hindi.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_sample_app_hindi.json
index 3e43a5a7..0ca2f840 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_sample_app_hindi.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_sample_app_hindi.json
@@ -1048,7 +1048,11 @@
},
"usage": {
"input_tokens": 236,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 22,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_shortlist_many_options.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_shortlist_many_options.json
index b06be6f9..13e2deb0 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_shortlist_many_options.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_shortlist_many_options.json
@@ -1064,7 +1064,11 @@
},
"usage": {
"input_tokens": 267,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 19,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
},
"shortlist": {
"intent": {
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_single_option.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_single_option.json
index 77bddc6f..03042a99 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_single_option.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/case_single_option.json
@@ -153,7 +153,11 @@
},
"usage": {
"input_tokens": 23,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 4,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/multilingual/meta.json b/laya-dotnet/tests/Laya.Tests/golden/multilingual/meta.json
index 4079c81d..71b7b56a 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/multilingual/meta.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/multilingual/meta.json
@@ -1,7 +1,7 @@
{
"generator": "tools/dump_golden.py",
"onnx_dir": "multilingual",
- "laya_version": "0.3.21",
+ "laya_version": "0.3.24",
"onnxruntime_version": "1.30.0",
"transformers_version": "5.17.0",
"config": {
diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/email_probe.json b/laya-dotnet/tests/Laya.Tests/golden/routing/email_probe.json
index e2c1f27a..feb69962 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/routing/email_probe.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/email_probe.json
@@ -1,5 +1,119 @@
{
"clean": [
+ {
+ "label": "upstream/confidential_request",
+ "body": "Confidential: I need a refund.",
+ "max_chars": 3000,
+ "result": "Confidential: I need a refund."
+ },
+ {
+ "label": "upstream/from_prose",
+ "body": "Hello.\nFrom: my side the integration works, but please refund it.",
+ "max_chars": 3000,
+ "result": "Hello.\nFrom: my side the integration works, but please refund it."
+ },
+ {
+ "label": "upstream/from_header",
+ "body": "Please refund it.\nFrom: Jane Smith\nSent: Monday\nOld request.",
+ "max_chars": 3000,
+ "result": "Please refund it."
+ },
+ {
+ "label": "upstream/thanks_prose",
+ "body": "Hello.\nThanks for the quick reply, but I need a refund.",
+ "max_chars": 3000,
+ "result": "Hello.\nThanks for the quick reply, but I need a refund."
+ },
+ {
+ "label": "upstream/english_signoff",
+ "body": "Please refund it.\nWarmest regards, Łukasz",
+ "max_chars": 3000,
+ "result": "Please refund it."
+ },
+ {
+ "label": "upstream/lowercase_name_is_prose",
+ "body": "Please refund it.\nThanks, żaneta needs help.",
+ "max_chars": 3000,
+ "result": "Please refund it.\nThanks, żaneta needs help."
+ },
+ {
+ "label": "upstream/marked_name",
+ "body": "Please refund it.\nRegards, José",
+ "max_chars": 3000,
+ "result": "Please refund it."
+ },
+ {
+ "label": "upstream/astral_signoff_name",
+ "body": "Please refund it.\nRegards, 𐐀𐐨",
+ "max_chars": 3000,
+ "result": "Please refund it."
+ },
+ {
+ "label": "upstream/unattached_mark",
+ "body": "Please refund it.\nRegards, ́Jose",
+ "max_chars": 3000,
+ "result": "Please refund it.\nRegards, ́Jose"
+ },
+ {
+ "label": "upstream/portuguese_quote",
+ "body": "Preciso de ajuda.\nEm 10/09 João escreveu:\nOld request.",
+ "max_chars": 3000,
+ "result": "Preciso de ajuda."
+ },
+ {
+ "label": "upstream/spanish_quote",
+ "body": "Necesito ayuda.\nEl 10/09 Juan escribió:\nOld request.",
+ "max_chars": 3000,
+ "result": "Necesito ayuda."
+ },
+ {
+ "label": "upstream/french_quote",
+ "body": "Je demande un remboursement.\nLe 10/09 Jean a écrit :\nOld request.",
+ "max_chars": 3000,
+ "result": "Je demande un remboursement."
+ },
+ {
+ "label": "upstream/wrapped_attribution",
+ "body": "Please refund it.\nOn 10/09 Jane Smith\[email protected]> wrote:\nOld request.",
+ "max_chars": 3000,
+ "result": "Please refund it."
+ },
+ {
+ "label": "upstream/french_signoff",
+ "body": "Je demande un remboursement.\nCordialement,\nMarie",
+ "max_chars": 3000,
+ "result": "Je demande un remboursement."
+ },
+ {
+ "label": "upstream/portuguese_signoff",
+ "body": "Preciso de ajuda.\nAtenciosamente,\nMaria",
+ "max_chars": 3000,
+ "result": "Preciso de ajuda."
+ },
+ {
+ "label": "upstream/spanish_signoff",
+ "body": "Necesito ayuda.\nSaludos cordiales,\nJuan",
+ "max_chars": 3000,
+ "result": "Necesito ayuda."
+ },
+ {
+ "label": "upstream/device_footer",
+ "body": "Please refund it.\nSent from my Samsung Galaxy.",
+ "max_chars": 3000,
+ "result": "Please refund it."
+ },
+ {
+ "label": "upstream/device_mention_prose",
+ "body": "Please refund it.\nSent from my iPhone yesterday to support.",
+ "max_chars": 3000,
+ "result": "Please refund it.\nSent from my iPhone yesterday to support."
+ },
+ {
+ "label": "upstream/french_disclaimer",
+ "body": "Je demande un remboursement.\n\nCe message est confidentiel et réservé au destinataire.",
+ "max_chars": 3000,
+ "result": "Je demande un remboursement."
+ },
{
"label": "inline_footer_no_blank_line",
"body": "My account is locked.\nThis email is confidential and intended solely for the named addressee.\nPlease unlock it.",
@@ -10,7 +124,7 @@
"label": "inline_footer_no_terminal_punctuation",
"body": "My account is locked\nThis email is confidential and intended solely for the named addressee.\nPlease unlock it.",
"max_chars": 3000,
- "result": "Please unlock it."
+ "result": "My account is locked Please unlock it."
},
{
"label": "inline_footer_body_never_emptied",
diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/lang_probe.json b/laya-dotnet/tests/Laya.Tests/golden/routing/lang_probe.json
index 14c14a08..39a5351a 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/routing/lang_probe.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/lang_probe.json
@@ -1,4 +1,559 @@
[
+ {
+ "label": "upstream/swedish_short",
+ "state": "glömt lösenord",
+ "state_text": "glömt lösenord",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "sv",
+ "english_hits": 0,
+ "diacritic_rate": 0.14285714285714285,
+ "looks_non_english": true
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "sv",
+ "is_english": false,
+ "language_undecided": false,
+ "diacritic_rate": 0.1429,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/swedish_login",
+ "state": "kan inte logga in",
+ "state_text": "kan inte logga in",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "sv",
+ "english_hits": 1,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "sv",
+ "is_english": false,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/nordic_overlap",
+ "state": "hej min vän",
+ "state_text": "hej min vän",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": null,
+ "english_hits": 0,
+ "diacritic_rate": 0.09090909090909091,
+ "looks_non_english": true
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": null,
+ "is_english": false,
+ "language_undecided": true,
+ "diacritic_rate": 0.0909,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/banglish",
+ "state": "ami amar taka ferot chai",
+ "state_text": "ami amar taka ferot chai",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "bn",
+ "english_hits": 0,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "bn",
+ "is_english": false,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/azerbaijani",
+ "state": "mən bu hesab üçün kömək istəyirəm",
+ "state_text": "mən bu hesab üçün kömək istəyirəm",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "az",
+ "english_hits": 0,
+ "diacritic_rate": 0.24242424242424243,
+ "looks_non_english": true
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "az",
+ "is_english": false,
+ "language_undecided": false,
+ "diacritic_rate": 0.2424,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/german_ascii",
+ "state": "ich habe mein paket noch nicht",
+ "state_text": "ich habe mein paket noch nicht",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "de",
+ "english_hits": 0,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "de",
+ "is_english": false,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/portuguese_ascii",
+ "state": "eu nao consigo entrar na minha conta",
+ "state_text": "eu nao consigo entrar na minha conta",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "pt",
+ "english_hits": 0,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "pt",
+ "is_english": false,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/english_loanword",
+ "state": "Please send me the café receipt.",
+ "state_text": "Please send me the café receipt.",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "en",
+ "english_hits": 2,
+ "diacritic_rate": 0.03125,
+ "looks_non_english": true
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "en",
+ "is_english": true,
+ "language_undecided": false,
+ "diacritic_rate": 0.0312,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/repeated_collision",
+ "state": "come come come please send the receipt",
+ "state_text": "come come come please send the receipt",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "en",
+ "english_hits": 2,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "en",
+ "is_english": true,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/identifier_not_prose",
+ "state": "Please check my error at os.path and [email protected]",
+ "state_text": "Please check my error at os.path and [email protected]",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "en",
+ "english_hits": 3,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "en",
+ "is_english": true,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/fullwidth_latin",
+ "state": "ABCD hello",
+ "state_text": "ABCD hello",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": null,
+ "english_hits": 0,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": null,
+ "is_english": true,
+ "language_undecided": true,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/ipa_latin",
+ "state": "ɐɪɲ hello",
+ "state_text": "ɐɪɲ hello",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": null,
+ "english_hits": 0,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": null,
+ "is_english": true,
+ "language_undecided": true,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/cjk_brand",
+ "state": "iPhone 用户无法登录账户请帮助",
+ "state_text": "iPhone 用户无法登录账户请帮助",
+ "detect_script": "han",
+ "script_profile": {
+ "latin": 0.35294117647058826,
+ "han": 0.6470588235294118
+ },
+ "latin_profile": {
+ "language": null,
+ "english_hits": 0,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "han",
+ "script_profile": {
+ "latin": 0.35294117647058826,
+ "han": 0.6470588235294118
+ },
+ "language": null,
+ "is_english": false,
+ "language_undecided": true,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.6471,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/cyrillic_name",
+ "state": "Please send the receipt to Владимир Иванов today",
+ "state_text": "Please send the receipt to Владимир Иванов today",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 0.6585365853658537,
+ "cyrillic": 0.34146341463414637
+ },
+ "latin_profile": {
+ "language": "en",
+ "english_hits": 3,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 0.6585365853658537,
+ "cyrillic": 0.34146341463414637
+ },
+ "language": "en",
+ "is_english": true,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.3415,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/greek_symbol",
+ "state": "Please set α to the correct value",
+ "state_text": "Please set α to the correct value",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 0.9629629629629629,
+ "greek": 0.037037037037037035
+ },
+ "latin_profile": {
+ "language": "en",
+ "english_hits": 3,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 0.9629629629629629,
+ "greek": 0.037037037037037035
+ },
+ "language": "en",
+ "is_english": true,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.037,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/mixed_foreign_line",
+ "state": "Please review the error and send me the details.\nEu nao consigo entrar na minha conta.",
+ "state_text": "Please review the error and send me the details.\nEu nao consigo entrar na minha conta.",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "en",
+ "english_hits": 4,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "pt",
+ "is_english": false,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": "Eu nao consigo entrar na minha conta."
+ }
+ },
+ {
+ "label": "upstream/mixed_foreign_field",
+ "state": {
+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",
+ "body": "ich habe mein paket noch nicht"
+ },
+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and sen",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "en",
+ "english_hits": 327,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "de",
+ "is_english": false,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/mixed_unknown_latin_field",
+ "state": {
+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",
+ "body": "žluťoučký kůň úpěl ďábelské ódy"
+ },
+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and sen",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "en",
+ "english_hits": 327,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": null,
+ "is_english": false,
+ "language_undecided": true,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/mixed_nonlatin_field",
+ "state": {
+ "note": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. ",
+ "body": "我的账户无法登录请帮助我"
+ },
+ "state_text": "Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and send me the details. Please review the error and sen",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "en",
+ "english_hits": 327,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": null,
+ "is_english": false,
+ "language_undecided": true,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
+ {
+ "label": "upstream/mixed_code_line",
+ "state": "Please review the error and send me the details.\nos.path = nao consigo entrar na minha conta",
+ "state_text": "Please review the error and send me the details.\nos.path = nao consigo entrar na minha conta",
+ "detect_script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "latin_profile": {
+ "language": "en",
+ "english_hits": 4,
+ "diacritic_rate": 0.0,
+ "looks_non_english": false
+ },
+ "analyse": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "en",
+ "is_english": true,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ }
+ },
{
"label": "script/english",
"state": "The customer was charged twice and wants a refund.",
@@ -22,7 +577,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -48,7 +604,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -74,7 +631,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -96,7 +654,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -122,7 +681,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0484,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -148,7 +708,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -176,7 +737,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -202,7 +764,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -228,7 +791,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -254,7 +818,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -280,7 +845,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -306,7 +872,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -332,7 +899,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -358,7 +926,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -384,7 +953,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -406,7 +976,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -428,7 +999,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -454,7 +1026,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -480,7 +1053,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -506,7 +1080,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -532,7 +1107,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -560,7 +1136,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -586,7 +1163,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -612,7 +1190,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0515,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -638,7 +1217,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0229,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -664,7 +1244,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0962,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -690,7 +1271,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0282,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -716,7 +1298,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0735,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -742,7 +1325,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.2222,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -768,7 +1352,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.1029,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -794,7 +1379,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.1333,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -820,7 +1406,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -846,7 +1433,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0926,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -872,7 +1460,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -898,7 +1487,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.1333,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -924,7 +1514,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -950,7 +1541,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0435,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -976,7 +1568,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1002,7 +1595,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1028,7 +1622,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1054,7 +1649,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1080,7 +1676,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1106,7 +1703,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1132,7 +1730,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1161,7 +1760,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1193,7 +1793,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1224,7 +1825,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1246,7 +1848,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1275,7 +1878,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -1301,7 +1905,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -1329,7 +1934,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.7
+ "non_latin_fraction": 0.7,
+ "mixed_segment": null
}
},
{
@@ -1355,7 +1961,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1381,7 +1988,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1407,7 +2015,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1433,7 +2042,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1459,7 +2069,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1485,7 +2096,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1511,7 +2123,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1537,7 +2150,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1563,7 +2177,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1589,7 +2204,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1615,7 +2231,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0317,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1641,7 +2258,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0159,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1667,7 +2285,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0476,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1693,7 +2312,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1719,7 +2339,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1745,7 +2366,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1771,7 +2393,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1797,7 +2420,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1823,7 +2447,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1849,7 +2474,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1875,7 +2501,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0435,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1901,7 +2528,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1927,7 +2555,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -1953,7 +2582,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -1979,7 +2609,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2005,7 +2636,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2031,7 +2663,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2057,7 +2690,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2083,7 +2717,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2109,7 +2744,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2135,7 +2771,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2161,7 +2798,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2187,7 +2825,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2213,7 +2852,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2239,7 +2879,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2265,7 +2906,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2291,7 +2933,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2317,7 +2960,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2343,7 +2987,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2369,7 +3014,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2395,7 +3041,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2421,7 +3068,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2447,7 +3095,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2473,7 +3122,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2499,7 +3149,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2525,7 +3176,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2553,7 +3205,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2579,7 +3232,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2605,7 +3259,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -2633,7 +3288,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2661,7 +3317,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2689,7 +3346,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.5
+ "non_latin_fraction": 0.5,
+ "mixed_segment": null
}
},
{
@@ -2717,7 +3375,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.5
+ "non_latin_fraction": 0.5,
+ "mixed_segment": null
}
},
{
@@ -2747,7 +3406,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -2767,17 +3427,18 @@
"looks_non_english": false
},
"analyse": {
- "script": "latin",
+ "script": "arabic",
"script_profile": {
"latin": 0.6190476190476191,
"han": 0.09523809523809523,
"arabic": 0.2857142857142857
},
"language": null,
- "is_english": true,
+ "is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.381
+ "non_latin_fraction": 0.381,
+ "mixed_segment": null
}
},
{
@@ -2805,7 +3466,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.7143
+ "non_latin_fraction": 0.7143,
+ "mixed_segment": null
}
},
{
@@ -2833,7 +3495,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.125
+ "non_latin_fraction": 0.125,
+ "mixed_segment": null
}
},
{
@@ -2859,7 +3522,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0476,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -2885,7 +3549,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -2911,7 +3576,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.1071,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -2937,7 +3603,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0952,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -2959,7 +3626,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -2985,15 +3653,18 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
"label": "astral/deseret_letter_alone",
"state": "𐐀𐐁𐐂",
"state_text": "𐐀𐐁𐐂",
- "detect_script": "unknown",
- "script_profile": {},
+ "detect_script": "other",
+ "script_profile": {
+ "other": 1.0
+ },
"latin_profile": {
"language": null,
"english_hits": 0,
@@ -3001,13 +3672,16 @@
"looks_non_english": false
},
"analyse": {
- "script": "unknown",
- "script_profile": {},
+ "script": "other",
+ "script_profile": {
+ "other": 1.0
+ },
"language": null,
- "is_english": true,
+ "is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -3016,7 +3690,8 @@
"state_text": "refund 𐐀 today",
"detect_script": "latin",
"script_profile": {
- "latin": 1.0
+ "latin": 0.9166666666666666,
+ "other": 0.08333333333333333
},
"latin_profile": {
"language": null,
@@ -3027,21 +3702,25 @@
"analyse": {
"script": "latin",
"script_profile": {
- "latin": 1.0
+ "latin": 0.9166666666666666,
+ "other": 0.08333333333333333
},
"language": null,
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0833,
+ "mixed_segment": null
}
},
{
"label": "astral/mathematical_bold_letters",
"state": "𝐀𝐁𝐂",
"state_text": "𝐀𝐁𝐂",
- "detect_script": "unknown",
- "script_profile": {},
+ "detect_script": "other",
+ "script_profile": {
+ "other": 1.0
+ },
"latin_profile": {
"language": null,
"english_hits": 0,
@@ -3049,13 +3728,16 @@
"looks_non_english": false
},
"analyse": {
- "script": "unknown",
- "script_profile": {},
+ "script": "other",
+ "script_profile": {
+ "other": 1.0
+ },
"language": null,
- "is_english": true,
+ "is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -3081,7 +3763,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -3107,7 +3790,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3133,7 +3817,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3159,7 +3844,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.04,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3185,7 +3871,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.125,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3207,7 +3894,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3233,7 +3921,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3259,7 +3948,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3285,7 +3975,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3323,7 +4014,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3359,7 +4051,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3395,7 +4088,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3436,7 +4130,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3462,7 +4157,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -3488,7 +4184,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -3516,7 +4213,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0025
+ "non_latin_fraction": 0.0025,
+ "mixed_segment": null
}
},
{
@@ -3542,7 +4240,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
}
]
diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json b/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json
index 5cf9a90b..f21f9724 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/meta.json
@@ -1,15 +1,15 @@
{
"generator": "tools/dump_routing_golden.py",
- "laya_version": "0.3.6",
+ "laya_version": "0.3.24",
"default_shortlist_k": 20,
"notes": {
"route_probe": "RouteDecision is serialized as {model, reason, detection, workflow}; 'repo' is intentionally omitted (not part of the ported RouteDecision).",
"shortlist_probe": "Non-finite floats (NaN, +Inf, -Inf) inside a case's 'vectors' matrix are encoded as the JSON strings \"nan\", \"inf\", \"-inf\" because JSON has no native literal for them; a case with vectors=null is a k>=n passthrough case where embed_fn must never be called."
},
"counts": {
- "lang_probe": 134,
- "route_probe": 72,
- "email_probe_clean": 47,
+ "lang_probe": 154,
+ "route_probe": 74,
+ "email_probe_clean": 66,
"email_probe_state": 8,
"shortlist_probe": 22,
"hashing_embedder_probe": 22
diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/route_probe.json b/laya-dotnet/tests/Laya.Tests/golden/routing/route_probe.json
index 6d284165..f3798daf 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/routing/route_probe.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/route_probe.json
@@ -1,4 +1,54 @@
[
+ {
+ "label": "upstream/undetermined_latin_custom_default",
+ "router_kwargs": {
+ "default": "multilingual"
+ },
+ "route_kwargs": {
+ "state": "Hello"
+ },
+ "decision": {
+ "model": "multilingual",
+ "reason": "Latin script, language not identified and no non-English letters; using default (multilingual)",
+ "detection": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": null,
+ "is_english": true,
+ "language_undecided": true,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
+ },
+ "workflow": null
+ }
+ },
+ {
+ "label": "upstream/mixed_foreign_line",
+ "router_kwargs": {},
+ "route_kwargs": {
+ "state": "Please review the error and send me the details.\nEu nao consigo entrar na minha conta."
+ },
+ "decision": {
+ "model": "multilingual",
+ "reason": "Latin script, mostly English, but a line or field reads as 'pt' ('Eu nao consigo entrar na minha conta.'); the English checkpoint cannot read it",
+ "detection": {
+ "script": "latin",
+ "script_profile": {
+ "latin": 1.0
+ },
+ "language": "pt",
+ "is_english": false,
+ "language_undecided": false,
+ "diacritic_rate": 0.0,
+ "non_latin_fraction": 0.0,
+ "mixed_segment": "Eu nao consigo entrar na minha conta."
+ },
+ "workflow": null
+ }
+ },
{
"label": "english_text",
"router_kwargs": {},
@@ -29,7 +79,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -64,7 +115,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -125,7 +177,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -161,7 +214,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -196,7 +250,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -231,7 +286,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -266,7 +322,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -547,7 +604,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": "customer_service"
}
@@ -578,7 +636,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -609,7 +668,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -635,7 +695,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -828,7 +889,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -895,7 +957,7 @@
},
"decision": {
"model": "english",
- "reason": "English Latin text",
+ "reason": "Latin script, language not identified and no non-English letters; using default (english)",
"detection": {
"script": "latin",
"script_profile": {
@@ -905,7 +967,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -948,7 +1011,7 @@
},
"decision": {
"model": "english",
- "reason": "English Latin text",
+ "reason": "Latin script, language not identified and no non-English letters; using default (english)",
"detection": {
"script": "latin",
"script_profile": {
@@ -958,7 +1021,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -993,7 +1057,7 @@
},
"decision": {
"model": "english",
- "reason": "English Latin text",
+ "reason": "Latin script, language not identified and no non-English letters; using default (english)",
"detection": {
"script": "latin",
"script_profile": {
@@ -1003,7 +1067,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1042,7 +1107,7 @@
},
"decision": {
"model": "english",
- "reason": "English Latin text",
+ "reason": "Latin script, language not identified and no non-English letters; using default (english)",
"detection": {
"script": "latin",
"script_profile": {
@@ -1052,7 +1117,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1075,7 +1141,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0962,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1098,7 +1165,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0455,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1121,7 +1189,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0735,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1144,7 +1213,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.1029,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1167,7 +1237,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0926,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1190,7 +1261,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1203,7 +1275,7 @@
},
"decision": {
"model": "english",
- "reason": "English Latin text",
+ "reason": "Latin script, language not identified and no non-English letters; using default (english)",
"detection": {
"script": "latin",
"script_profile": {
@@ -1213,7 +1285,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1236,7 +1309,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1259,7 +1333,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1282,7 +1357,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1305,7 +1381,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1328,7 +1405,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0317,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1351,7 +1429,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0159,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1374,7 +1453,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0476,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1397,7 +1477,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1420,7 +1501,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1443,7 +1525,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0435,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1466,7 +1549,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1489,7 +1573,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1512,7 +1597,8 @@
"is_english": true,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1547,7 +1633,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1584,7 +1671,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1621,7 +1709,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1661,7 +1750,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1892,7 +1982,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1916,7 +2007,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.7
+ "non_latin_fraction": 0.7,
+ "mixed_segment": null
},
"workflow": null
}
@@ -1939,7 +2031,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0926,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/routing/sample_inputs.json b/laya-dotnet/tests/Laya.Tests/golden/routing/sample_inputs.json
index c3cb120f..ae591778 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/routing/sample_inputs.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/routing/sample_inputs.json
@@ -12,7 +12,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -27,7 +28,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -42,7 +44,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0484,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -57,7 +60,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0282,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
}
},
{
@@ -72,7 +76,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -87,7 +92,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -103,7 +109,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
}
},
{
@@ -122,7 +129,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.52
+ "non_latin_fraction": 0.52,
+ "mixed_segment": null
}
}
],
@@ -145,7 +153,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -168,7 +177,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 1.0
+ "non_latin_fraction": 1.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -191,7 +201,8 @@
"is_english": false,
"language_undecided": false,
"diacritic_rate": 0.0962,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
@@ -305,7 +316,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": "customer_service"
}
@@ -363,7 +375,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
},
@@ -380,7 +393,8 @@
"is_english": false,
"language_undecided": true,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.7381
+ "non_latin_fraction": 0.7381,
+ "mixed_segment": null
},
"workflow": null
}
@@ -407,7 +421,8 @@
"is_english": true,
"language_undecided": false,
"diacritic_rate": 0.0,
- "non_latin_fraction": 0.0
+ "non_latin_fraction": 0.0,
+ "mixed_segment": null
},
"workflow": null
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_arabic.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_arabic.json
index ff8e7ed3..23c9c5e3 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_arabic.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_arabic.json
@@ -935,7 +935,11 @@
},
"usage": {
"input_tokens": 252,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 69,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_choice_criteria_list.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_choice_criteria_list.json
index 7eaa2d73..38227450 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_choice_criteria_list.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_choice_criteria_list.json
@@ -195,7 +195,11 @@
},
"usage": {
"input_tokens": 31,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 10,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_cjk.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_cjk.json
index 4427f8b9..89ffd561 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_cjk.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_cjk.json
@@ -912,7 +912,11 @@
},
"usage": {
"input_tokens": 222,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 49,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_criterion_shapes.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_criterion_shapes.json
index 7d600ca7..abbe11d9 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_criterion_shapes.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_criterion_shapes.json
@@ -662,14 +662,8 @@
"type": "score",
"score": 1.1978,
"legend": {
- "0": {
- "label": "info",
- "cvss": 0.0
- },
- "1": {
- "label": "high",
- "cvss": 7.5
- },
+ "0": "{\"label\": \"info\", \"cvss\": 0.0}",
+ "1": "{\"label\": \"high\", \"cvss\": 7.5}",
"2": "critical"
},
"probabilities": {
@@ -686,7 +680,11 @@
},
"usage": {
"input_tokens": 144,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 11,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_dict_state.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_dict_state.json
index 8acbb75a..7a697c46 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_dict_state.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_dict_state.json
@@ -393,7 +393,11 @@
},
"usage": {
"input_tokens": 93,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 64,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_json_escaping.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_json_escaping.json
index ce900126..28da9ff4 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_json_escaping.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_json_escaping.json
@@ -492,7 +492,11 @@
},
"usage": {
"input_tokens": 134,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 103,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_list_state.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_list_state.json
index 40fdedf8..6d2789a7 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_list_state.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_list_state.json
@@ -345,7 +345,11 @@
},
"usage": {
"input_tokens": 84,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 53,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_long_state.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_long_state.json
index 8e468066..5a9a9a61 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_long_state.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_long_state.json
@@ -6283,7 +6283,14 @@
},
"usage": {
"input_tokens": 2048,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 6804,
+ "state_tokens_dropped": 5811,
+ "truncated": true,
+ "truncated_questions": [
+ "frustrated",
+ "topic"
+ ]
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_many_options.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_many_options.json
index 2c3b150d..6bc5024e 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_many_options.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_many_options.json
@@ -1170,7 +1170,11 @@
},
"usage": {
"input_tokens": 272,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 12,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_mask_injection.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_mask_injection.json
index dc307432..6e2a87a8 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_mask_injection.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_mask_injection.json
@@ -232,7 +232,11 @@
},
"usage": {
"input_tokens": 48,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 14,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_nonstring_instructions.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_nonstring_instructions.json
index 8159d9cc..a13d2823 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_nonstring_instructions.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_nonstring_instructions.json
@@ -467,7 +467,11 @@
},
"usage": {
"input_tokens": 104,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 14,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_options_exceed_head.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_options_exceed_head.json
index 2336adc2..00334c9d 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_options_exceed_head.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_options_exceed_head.json
@@ -310,5 +310,5 @@
}
},
"error_type": "ValueError",
- "error_message": "question 'too_wide' options exceed head_max_len=256"
+ "error_message": "question 'too_wide': only 254 of its 300 option markers fit in max_len=1024 with head_max_len=256 spent on the question; lower head_max_len, raise max_len, or use fewer options"
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_quickstart.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_quickstart.json
index 2661ad5a..f4d49024 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_quickstart.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_quickstart.json
@@ -1498,7 +1498,11 @@
},
"usage": {
"input_tokens": 379,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 57,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_sample_app_english.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_sample_app_english.json
index 14bf5a70..aee44581 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_sample_app_english.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_sample_app_english.json
@@ -1410,7 +1410,11 @@
},
"usage": {
"input_tokens": 348,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 50,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_sample_app_hindi.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_sample_app_hindi.json
index 3ffdfe50..4c052f0b 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_sample_app_hindi.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_sample_app_hindi.json
@@ -1456,7 +1456,11 @@
},
"usage": {
"input_tokens": 364,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 54,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_shortlist_many_options.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_shortlist_many_options.json
index 7eabc287..b53820e6 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_shortlist_many_options.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_shortlist_many_options.json
@@ -1067,7 +1067,11 @@
},
"usage": {
"input_tokens": 268,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 19,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
},
"shortlist": {
"intent": {
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_single_option.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_single_option.json
index d5c562f9..c04dd248 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_single_option.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_single_option.json
@@ -153,7 +153,11 @@
},
"usage": {
"input_tokens": 23,
- "output_tokens": 0
+ "output_tokens": 0,
+ "state_tokens": 4,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
}
}
}
diff --git a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/meta.json b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/meta.json
index 40d75c3d..079fcbe4 100644
--- a/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/meta.json
+++ b/laya-dotnet/tests/Laya.Tests/golden/typed-decisions/meta.json
@@ -1,7 +1,7 @@
{
"generator": "tools/dump_golden.py",
"onnx_dir": "typed-decisions",
- "laya_version": "0.3.21",
+ "laya_version": "0.3.24",
"onnxruntime_version": "1.30.0",
"transformers_version": "5.17.0",
"config": {
diff --git a/laya-dotnet/tools/routing_cases.py b/laya-dotnet/tools/routing_cases.py
index 068059e1..1644b7be 100644
--- a/laya-dotnet/tools/routing_cases.py
+++ b/laya-dotnet/tools/routing_cases.py
@@ -26,6 +26,34 @@ golden the same way the existing `Laya.Sample` is.
# not this module's -- these three take plain text, not a full state).
LANG_STATES = [
+ ("upstream/swedish_short", "glömt lösenord"),
+ ("upstream/swedish_login", "kan inte logga in"),
+ ("upstream/nordic_overlap", "hej min vän"),
+ ("upstream/banglish", "ami amar taka ferot chai"),
+ ("upstream/azerbaijani", "mən bu hesab üçün kömək istəyirəm"),
+ ("upstream/german_ascii", "ich habe mein paket noch nicht"),
+ ("upstream/portuguese_ascii", "eu nao consigo entrar na minha conta"),
+ ("upstream/english_loanword", "Please send me the café receipt."),
+ ("upstream/repeated_collision", "come come come please send the receipt"),
+ ("upstream/identifier_not_prose", "Please check my error at os.path and [email protected]"),
+ ("upstream/fullwidth_latin", "ABCD hello"),
+ ("upstream/ipa_latin", "ɐɪɲ hello"),
+ ("upstream/cjk_brand", "iPhone 用户无法登录账户请帮助"),
+ ("upstream/cyrillic_name", "Please send the receipt to Владимир Иванов today"),
+ ("upstream/greek_symbol", "Please set α to the correct value"),
+ ("upstream/mixed_foreign_line",
+ "Please review the error and send me the details.\nEu nao consigo entrar na minha conta."),
+ ("upstream/mixed_foreign_field",
+ {"note": "Please review the error and send me the details. " * 100,
+ "body": "ich habe mein paket noch nicht"}),
+ ("upstream/mixed_unknown_latin_field",
+ {"note": "Please review the error and send me the details. " * 100,
+ "body": "žluťoučký kůň úpěl ďábelské ódy"}),
+ ("upstream/mixed_nonlatin_field",
+ {"note": "Please review the error and send me the details. " * 100,
+ "body": "我的账户无法登录请帮助我"}),
+ ("upstream/mixed_code_line",
+ "Please review the error and send me the details.\nos.path = nao consigo entrar na minha conta"),
# ---- straight from tests/test_router.py SCRIPTS (script detection) ----
("script/english", "The customer was charged twice and wants a refund."),
("script/armenian", "Հայերեն"),
@@ -235,6 +263,9 @@ def _td_questions(ids):
ROUTE_CASES = [
+ ("upstream/undetermined_latin_custom_default", {"default": "multilingual"}, {"state": "Hello"}),
+ ("upstream/mixed_foreign_line", {}, {"state":
+ "Please review the error and send me the details.\nEu nao consigo entrar na minha conta."}),
# ---- default router, precedence ladder ----
("english_text", {}, {"state": {"body": "I was charged twice, please refund."}, "questions": Q_GENERIC}),
("armenian_text", {}, {"state": {"body": "Հայերեն"}, "questions": Q_GENERIC}),
@@ -345,6 +376,27 @@ ROUTE_CASES = [
_DISCLAIMER_TEXT = "This email is confidential and intended solely for the named addressee."
EMAIL_CLEAN_CASES = [
+ ("upstream/confidential_request", "Confidential: I need a refund.", 3000),
+ ("upstream/from_prose", "Hello.\nFrom: my side the integration works, but please refund it.", 3000),
+ ("upstream/from_header", "Please refund it.\nFrom: Jane Smith\nSent: Monday\nOld request.", 3000),
+ ("upstream/thanks_prose", "Hello.\nThanks for the quick reply, but I need a refund.", 3000),
+ ("upstream/english_signoff", "Please refund it.\nWarmest regards, Łukasz", 3000),
+ ("upstream/lowercase_name_is_prose", "Please refund it.\nThanks, żaneta needs help.", 3000),
+ ("upstream/marked_name", "Please refund it.\nRegards, Jose\u0301", 3000),
+ ("upstream/astral_signoff_name", "Please refund it.\nRegards, \U00010400\U00010428", 3000),
+ ("upstream/unattached_mark", "Please refund it.\nRegards, \u0301Jose", 3000),
+ ("upstream/portuguese_quote", "Preciso de ajuda.\nEm 10/09 João escreveu:\nOld request.", 3000),
+ ("upstream/spanish_quote", "Necesito ayuda.\nEl 10/09 Juan escribió:\nOld request.", 3000),
+ ("upstream/french_quote", "Je demande un remboursement.\nLe 10/09 Jean a écrit :\nOld request.", 3000),
+ ("upstream/wrapped_attribution",
+ "Please refund it.\nOn 10/09 Jane Smith\[email protected]> wrote:\nOld request.", 3000),
+ ("upstream/french_signoff", "Je demande un remboursement.\nCordialement,\nMarie", 3000),
+ ("upstream/portuguese_signoff", "Preciso de ajuda.\nAtenciosamente,\nMaria", 3000),
+ ("upstream/spanish_signoff", "Necesito ayuda.\nSaludos cordiales,\nJuan", 3000),
+ ("upstream/device_footer", "Please refund it.\nSent from my Samsung Galaxy.", 3000),
+ ("upstream/device_mention_prose", "Please refund it.\nSent from my iPhone yesterday to support.", 3000),
+ ("upstream/french_disclaimer",
+ "Je demande un remboursement.\n\nCe message est confidentiel et réservé au destinataire.", 3000),
# ---- straight from tests/test_email.py ----
("inline_footer_no_blank_line", "My account is locked.\n%s\nPlease unlock it." % _DISCLAIMER_TEXT, 3000),
("inline_footer_no_terminal_punctuation", "My account is locked\n%s\nPlease unlock it." % _DISCLAIMER_TEXT, 3000),