Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

release: 0.3.19, measured long-document table for laya-multilingual

Nandakishor332310f91941
diff --git a/README.md b/README.md
index 304c1162..4b54a7f8 100644
--- a/README.md
+++ b/README.md
@@ -29,15 +29,21 @@ python -m pip install laya
Python 3.10 or newer. Optional extras: `laya[serve]` (HTTP server), `laya[mcp]` (MCP server), `laya[langchain]` (LangChain and LangGraph), `laya[onnx]` (ONNX Runtime), `laya[fast]` (TileLang GPU fast path). Step-by-step setup for each platform, CPU-only or GPU PyTorch builds, and troubleshooting are in [Installation details](#installation-details).
+**Long documents.** `laya-multilingual` accepts up to 8,192 tokens with `max_len`. Measured accuracy and time by document length, reproducible with [`research/scripts/bench_long_context.py`](https://github.com/NandhaKishorM/laya/blob/main/research/scripts/bench_long_context.py):
+
+<p align="center">
+ <img src="https://raw.githubusercontent.com/NandhaKishorM/laya/main/assets/long_context_table.png" alt="laya-multilingual long-document accuracy: with the request after 2,000 to 4,000 tokens of other text, 17 to 18 of 20 correct at a higher limit against 7 of 20 at the default 1,024; unstable beyond about 4,000 tokens" width="100%" />
+</p>
+
## Quickstart
-> **Long documents: `laya-multilingual` reads up to 8,192 tokens.** It ships with a 1,024-token limit; raise it per call:
+> **Long documents: `laya-multilingual` accepts up to 8,192 tokens and is reliable up to about 4,000.** It ships with a 1,024-token limit, which cuts a long document off; raise it per call:
>
> ```python
-> result = router.predict(long_document, questions, model="multilingual", max_len=8192)
+> result = router.predict(long_document, questions, model="multilingual", max_len=4096)
> ```
>
-> **Accuracy does not drop.** Short inputs give identical answers at either limit. On long inputs it improves: with the request placed after about 6,300 tokens of other text, 8 of 10 test requests were answered correctly at 8,192, against 3 of 10 at the default limit, which cuts the request off. **Speed:** short inputs run at the same speed, because the cost follows the input's real length, not the limit. Long inputs take longer in proportion to their length: a 6,300-token input took 2.5 s instead of 0.18 s on an Apple GPU (8 s instead of 0.35 s on CPU). Name the checkpoint with `model="multilingual"`, since long mostly-English text would otherwise route to the English checkpoint.
+> In the table above, with the request placed after 2,000 to 4,000 tokens of other text, 17 to 18 of 20 requests were answered correctly at the higher limit, against 7 of 20 at the default. Beyond about 4,000 tokens results become unstable (8 to 17 of 20), probably because the checkpoint was trained with its 1,024-token limit, so treat `max_len=8192` as experimental. Short inputs give identical answers at any limit. Speed follows the input's real length, not the limit: short inputs are unchanged, and a 4,000-token input takes about 1.7 s on an Apple GPU. Name the checkpoint with `model="multilingual"`, since long mostly-English text would otherwise route to the English checkpoint.
```python
from laya import Router
@@ -84,9 +90,9 @@ The shipped checkpoints work zero-shot, but fine-tuning on decisions from your o
**[nandhakishorm.github.io/laya](https://nandhakishorm.github.io/laya/)**: guides for [prediction hooks](https://nandhakishorm.github.io/laya/hooks/), [schema-driven decisions](https://nandhakishorm.github.io/laya/structured/), [Docker](https://nandhakishorm.github.io/laya/docker/) and [LangChain and LangGraph](https://nandhakishorm.github.io/laya/langchain/), plus a full [API reference](https://nandhakishorm.github.io/laya/reference/).
-## What's new in 0.3.18
+## What's new in 0.3.19
-* **Long documents on `laya-multilingual`.** The Quickstart now shows how to read up to 8,192 tokens with `max_len=8192`, with measured speed and accuracy.
+* **Measured long-document table.** Accuracy and time for `laya-multilingual` by document length, from a reproducible benchmark: reliable up to about 4,000 tokens with a higher `max_len`, and experimental beyond that. This corrects the earlier note, which said accuracy held all the way to 8,192.
* **Installation, quickstart and documentation first.** This README now opens with how to install Laya, runnable English and multilingual examples, how fine-tuning improves accuracy, and where the docs are.
* **Documentation site** at [nandhakishorm.github.io/laya](https://nandhakishorm.github.io/laya/), with an API reference generated from the docstrings.
diff --git a/assets/long_context_table.png b/assets/long_context_table.png
new file mode 100644
index 00000000..d704be72
Binary files /dev/null and b/assets/long_context_table.png differ
diff --git a/laya/__init__.py b/laya/__init__.py
index e5dc80df..477c7b5e 100644
--- a/laya/__init__.py
+++ b/laya/__init__.py
@@ -14,7 +14,7 @@ from .presets import (
from .router import DEFAULT_MODELS, RouteDecision, Router
from .structured import DecisionResult, decide
-__version__ = "0.3.18"
+__version__ = "0.3.19"
# Routing, language detection and email cleaning are pure Python. The torch-backed names are
# resolved lazily so that `import laya` -- and therefore `from laya import Router` or
diff --git a/pyproject.toml b/pyproject.toml
index 2776fed6..55d9f3ea 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "laya"
-version = "0.3.18"
+version = "0.3.19"
description = "Fast, non-autoregressive System 1 decision engine with calibrated probabilities"
readme = "README.md"
requires-python = ">=3.10"
diff --git a/research/README.md b/research/README.md
index 44e9d04e..3da4ff3b 100644
--- a/research/README.md
+++ b/research/README.md
@@ -18,6 +18,8 @@ the numbers quoted in the main README — nothing here is imported by the `laya`
| `scripts/bench_latency.py` | inference speed including what routing costs: detection overhead, hot path, cold-swap, mixed-language throughput at several `max_loaded` settings |
| `scripts/bench_length_batching.py` | compare upstream contiguous batches with optional length sorting on synthetic tickets, including output consistency and optional fresh-process memory profiles (`psutil` required for memory mode) |
| `scripts/make_plots.py` | renders `assets/laya_benchmark.png` from the result JSONs |
+| `scripts/bench_long_context.py` | `laya-multilingual` on long documents: 20 support requests in 8 languages, each placed after 0 to 7,000 tokens of unrelated text, scored at the default limit and at `max_len=8192` |
+| `scripts/plot_long_context.py` | renders `assets/long_context_table.png` from `results/long_context_multilingual.json` |
Everything runs with `USE_TF=0` — `transformers` probes for TensorFlow at import, and when TF is
installed its abseil runtime can deadlock model construction on macOS/Python 3.9.
@@ -66,6 +68,7 @@ python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-si
| file | contents |
|---|---|
| `results/t4_colab_benchmark.json` | 17,416 questions on one T4, both checkpoints, identical questions per model |
+| `results/long_context_multilingual.json` | the long-document run behind `assets/long_context_table.png`: every prediction, with device and library versions |
| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options |
| `results/cpu_51_language_sweep_clamped.json` | the same 51 languages and 5,100 cases re-run after the temperature clamp, raw temperatures and served temperatures side by side ([#208](https://github.com/NandhaKishorM/laya/issues/208)) |
diff --git a/research/results/long_context_multilingual.json b/research/results/long_context_multilingual.json
new file mode 100644
index 00000000..dc16f3bd
--- /dev/null
+++ b/research/results/long_context_multilingual.json
@@ -0,0 +1,5289 @@
+{
+ "checkpoint": "convaiinnovations/laya (multilingual/)",
+ "default_limit": 1024,
+ "device": "mps",
+ "platform": "macOS-26.5.2-arm64-arm-64bit",
+ "torch": "2.14.0",
+ "transformers": "5.17.0",
+ "laya": "0.3.18",
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which department should handle this request?",
+ "criteria": {
+ "billing": "invoices, payments, refunds",
+ "technical": "bugs, outages, system errors",
+ "sales": "pricing, new contracts, plan upgrades",
+ "other": "everything else"
+ }
+ }
+ },
+ "rows": [
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "n": 20,
+ "correct": 19,
+ "accuracy": 0.95,
+ "median_latency_s": 0.017,
+ "median_input_tokens": 61
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "n": 20,
+ "correct": 19,
+ "accuracy": 0.95,
+ "median_latency_s": 0.016,
+ "median_input_tokens": 61
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "n": 20,
+ "correct": 13,
+ "accuracy": 0.65,
+ "median_latency_s": 0.224,
+ "median_input_tokens": 1024
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "n": 20,
+ "correct": 16,
+ "accuracy": 0.8,
+ "median_latency_s": 0.211,
+ "median_input_tokens": 1032
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "n": 20,
+ "correct": 7,
+ "accuracy": 0.35,
+ "median_latency_s": 0.198,
+ "median_input_tokens": 1024
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "n": 20,
+ "correct": 17,
+ "accuracy": 0.85,
+ "median_latency_s": 0.503,
+ "median_input_tokens": 2040
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "n": 20,
+ "correct": 7,
+ "accuracy": 0.35,
+ "median_latency_s": 0.208,
+ "median_input_tokens": 1024
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "n": 20,
+ "correct": 17,
+ "accuracy": 0.85,
+ "median_latency_s": 0.895,
+ "median_input_tokens": 3006
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "n": 20,
+ "correct": 7,
+ "accuracy": 0.35,
+ "median_latency_s": 0.216,
+ "median_input_tokens": 1024
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "n": 20,
+ "correct": 18,
+ "accuracy": 0.9,
+ "median_latency_s": 1.707,
+ "median_input_tokens": 3972
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "n": 20,
+ "correct": 7,
+ "accuracy": 0.35,
+ "median_latency_s": 0.262,
+ "median_input_tokens": 1024
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "n": 20,
+ "correct": 11,
+ "accuracy": 0.55,
+ "median_latency_s": 2.586,
+ "median_input_tokens": 4938
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "n": 20,
+ "correct": 7,
+ "accuracy": 0.35,
+ "median_latency_s": 0.253,
+ "median_input_tokens": 1024
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "n": 20,
+ "correct": 17,
+ "accuracy": 0.85,
+ "median_latency_s": 3.505,
+ "median_input_tokens": 5946
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "n": 20,
+ "correct": 7,
+ "accuracy": 0.35,
+ "median_latency_s": 0.264,
+ "median_input_tokens": 1024
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "n": 20,
+ "correct": 8,
+ "accuracy": 0.4,
+ "median_latency_s": 4.503,
+ "median_input_tokens": 6912
+ }
+ ],
+ "cases": [
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 1.0,
+ "technical": 0.0,
+ "sales": 0.0,
+ "other": 0.0
+ },
+ "correct": true,
+ "input_tokens": 64,
+ "latency_s": 0.0169
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0008,
+ "technical": 0.9948,
+ "sales": 0.0006,
+ "other": 0.0038
+ },
+ "correct": true,
+ "input_tokens": 60,
+ "latency_s": 0.019
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0483,
+ "technical": 0.012,
+ "sales": 0.9352,
+ "other": 0.0045
+ },
+ "correct": true,
+ "input_tokens": 61,
+ "latency_s": 0.0167
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 1.0,
+ "technical": 0.0,
+ "sales": 0.0,
+ "other": 0.0
+ },
+ "correct": true,
+ "input_tokens": 61,
+ "latency_s": 0.015
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0022,
+ "technical": 0.993,
+ "sales": 0.0016,
+ "other": 0.0033
+ },
+ "correct": true,
+ "input_tokens": 63,
+ "latency_s": 0.0148
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0002,
+ "technical": 0.0,
+ "sales": 0.9997,
+ "other": 0.0001
+ },
+ "correct": true,
+ "input_tokens": 60,
+ "latency_s": 0.0144
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9999,
+ "technical": 0.0,
+ "sales": 0.0,
+ "other": 0.0001
+ },
+ "correct": true,
+ "input_tokens": 66,
+ "latency_s": 0.025
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0003,
+ "technical": 0.9968,
+ "sales": 0.0005,
+ "other": 0.0024
+ },
+ "correct": true,
+ "input_tokens": 58,
+ "latency_s": 0.0186
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0018,
+ "technical": 0.0006,
+ "sales": 0.9973,
+ "other": 0.0003
+ },
+ "correct": true,
+ "input_tokens": 61,
+ "latency_s": 0.0171
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 1.0,
+ "technical": 0.0,
+ "sales": 0.0,
+ "other": 0.0
+ },
+ "correct": true,
+ "input_tokens": 68,
+ "latency_s": 0.0226
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0752,
+ "technical": 0.8002,
+ "sales": 0.0422,
+ "other": 0.0823
+ },
+ "correct": true,
+ "input_tokens": 59,
+ "latency_s": 0.0182
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9985,
+ "technical": 0.0005,
+ "sales": 0.0003,
+ "other": 0.0007
+ },
+ "correct": true,
+ "input_tokens": 67,
+ "latency_s": 0.0245
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0091,
+ "technical": 0.9108,
+ "sales": 0.0139,
+ "other": 0.0662
+ },
+ "correct": true,
+ "input_tokens": 60,
+ "latency_s": 0.0158
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9974,
+ "technical": 0.0011,
+ "sales": 0.0002,
+ "other": 0.0013
+ },
+ "correct": true,
+ "input_tokens": 61,
+ "latency_s": 0.0162
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0061,
+ "technical": 0.9238,
+ "sales": 0.0196,
+ "other": 0.0506
+ },
+ "correct": true,
+ "input_tokens": 61,
+ "latency_s": 0.0155
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9932,
+ "technical": 0.0005,
+ "sales": 0.0055,
+ "other": 0.0009
+ },
+ "correct": false,
+ "input_tokens": 58,
+ "latency_s": 0.0153
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.6315,
+ "technical": 0.0473,
+ "sales": 0.2452,
+ "other": 0.076
+ },
+ "correct": true,
+ "input_tokens": 71,
+ "latency_s": 0.0221
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0003,
+ "technical": 0.9949,
+ "sales": 0.0004,
+ "other": 0.0044
+ },
+ "correct": true,
+ "input_tokens": 62,
+ "latency_s": 0.0162
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.6582,
+ "technical": 0.0862,
+ "sales": 0.0667,
+ "other": 0.1889
+ },
+ "correct": true,
+ "input_tokens": 62,
+ "latency_s": 0.0155
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 1024,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9085,
+ "technical": 0.0131,
+ "sales": 0.0197,
+ "other": 0.0587
+ },
+ "correct": true,
+ "input_tokens": 70,
+ "latency_s": 0.0225
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 1.0,
+ "technical": 0.0,
+ "sales": 0.0,
+ "other": 0.0
+ },
+ "correct": true,
+ "input_tokens": 64,
+ "latency_s": 0.0165
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0008,
+ "technical": 0.9948,
+ "sales": 0.0006,
+ "other": 0.0038
+ },
+ "correct": true,
+ "input_tokens": 60,
+ "latency_s": 0.0148
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0483,
+ "technical": 0.012,
+ "sales": 0.9352,
+ "other": 0.0045
+ },
+ "correct": true,
+ "input_tokens": 61,
+ "latency_s": 0.0149
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 1.0,
+ "technical": 0.0,
+ "sales": 0.0,
+ "other": 0.0
+ },
+ "correct": true,
+ "input_tokens": 61,
+ "latency_s": 0.0155
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0022,
+ "technical": 0.993,
+ "sales": 0.0016,
+ "other": 0.0033
+ },
+ "correct": true,
+ "input_tokens": 63,
+ "latency_s": 0.0155
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0002,
+ "technical": 0.0,
+ "sales": 0.9997,
+ "other": 0.0001
+ },
+ "correct": true,
+ "input_tokens": 60,
+ "latency_s": 0.0158
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9999,
+ "technical": 0.0,
+ "sales": 0.0,
+ "other": 0.0001
+ },
+ "correct": true,
+ "input_tokens": 66,
+ "latency_s": 0.0202
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0003,
+ "technical": 0.9968,
+ "sales": 0.0005,
+ "other": 0.0024
+ },
+ "correct": true,
+ "input_tokens": 58,
+ "latency_s": 0.0152
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0018,
+ "technical": 0.0006,
+ "sales": 0.9973,
+ "other": 0.0003
+ },
+ "correct": true,
+ "input_tokens": 61,
+ "latency_s": 0.0156
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 1.0,
+ "technical": 0.0,
+ "sales": 0.0,
+ "other": 0.0
+ },
+ "correct": true,
+ "input_tokens": 68,
+ "latency_s": 0.02
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0752,
+ "technical": 0.8002,
+ "sales": 0.0422,
+ "other": 0.0823
+ },
+ "correct": true,
+ "input_tokens": 59,
+ "latency_s": 0.0154
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9985,
+ "technical": 0.0005,
+ "sales": 0.0003,
+ "other": 0.0007
+ },
+ "correct": true,
+ "input_tokens": 67,
+ "latency_s": 0.021
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0091,
+ "technical": 0.9108,
+ "sales": 0.0139,
+ "other": 0.0662
+ },
+ "correct": true,
+ "input_tokens": 60,
+ "latency_s": 0.0176
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9974,
+ "technical": 0.0011,
+ "sales": 0.0002,
+ "other": 0.0013
+ },
+ "correct": true,
+ "input_tokens": 61,
+ "latency_s": 0.0164
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0061,
+ "technical": 0.9238,
+ "sales": 0.0196,
+ "other": 0.0506
+ },
+ "correct": true,
+ "input_tokens": 61,
+ "latency_s": 0.0157
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9932,
+ "technical": 0.0005,
+ "sales": 0.0055,
+ "other": 0.0009
+ },
+ "correct": false,
+ "input_tokens": 58,
+ "latency_s": 0.0156
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.6315,
+ "technical": 0.0473,
+ "sales": 0.2452,
+ "other": 0.076
+ },
+ "correct": true,
+ "input_tokens": 71,
+ "latency_s": 0.0208
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0003,
+ "technical": 0.9949,
+ "sales": 0.0004,
+ "other": 0.0044
+ },
+ "correct": true,
+ "input_tokens": 62,
+ "latency_s": 0.0174
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.6582,
+ "technical": 0.0862,
+ "sales": 0.0667,
+ "other": 0.1889
+ },
+ "correct": true,
+ "input_tokens": 62,
+ "latency_s": 0.016
+ },
+ {
+ "pad_tokens": 0,
+ "limit": 8192,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9085,
+ "technical": 0.0131,
+ "sales": 0.0197,
+ "other": 0.0587
+ },
+ "correct": true,
+ "input_tokens": 70,
+ "latency_s": 0.0216
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9994,
+ "technical": 0.0003,
+ "sales": 0.0001,
+ "other": 0.0002
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2237
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0021,
+ "technical": 0.8932,
+ "sales": 0.0086,
+ "other": 0.096
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2309
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.1716,
+ "technical": 0.2876,
+ "sales": 0.3276,
+ "other": 0.2131
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.242
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.999,
+ "technical": 0.0006,
+ "sales": 0.0001,
+ "other": 0.0003
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2513
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.3547,
+ "technical": 0.3897,
+ "sales": 0.0574,
+ "other": 0.1982
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2546
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0755,
+ "technical": 0.0163,
+ "sales": 0.8911,
+ "other": 0.017
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2417
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.999,
+ "technical": 0.0005,
+ "sales": 0.0002,
+ "other": 0.0003
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2285
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.2345,
+ "technical": 0.4542,
+ "sales": 0.1093,
+ "other": 0.2021
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2219
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8553,
+ "technical": 0.0473,
+ "sales": 0.0927,
+ "other": 0.0047
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2463
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.4003,
+ "technical": 0.4326,
+ "sales": 0.0737,
+ "other": 0.0935
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2324
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.011,
+ "technical": 0.96,
+ "sales": 0.0145,
+ "other": 0.0145
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2215
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0924,
+ "technical": 0.8017,
+ "sales": 0.0259,
+ "other": 0.08
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2251
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "other",
+ "probabilities": {
+ "billing": 0.0185,
+ "technical": 0.0626,
+ "sales": 0.1542,
+ "other": 0.7647
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2255
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.2715,
+ "technical": 0.5501,
+ "sales": 0.0243,
+ "other": 0.154
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2231
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0295,
+ "technical": 0.8645,
+ "sales": 0.0453,
+ "other": 0.0608
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2159
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.7892,
+ "technical": 0.0635,
+ "sales": 0.0356,
+ "other": 0.1117
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2129
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1841,
+ "technical": 0.4925,
+ "sales": 0.0436,
+ "other": 0.2797
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2163
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0348,
+ "technical": 0.6652,
+ "sales": 0.2472,
+ "other": 0.0528
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.1966
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.5482,
+ "technical": 0.2297,
+ "sales": 0.1347,
+ "other": 0.0874
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2056
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 1024,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.7625,
+ "technical": 0.149,
+ "sales": 0.0447,
+ "other": 0.0437
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2207
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9995,
+ "technical": 0.0002,
+ "sales": 0.0002,
+ "other": 0.0001
+ },
+ "correct": true,
+ "input_tokens": 1035,
+ "latency_s": 0.2707
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0016,
+ "technical": 0.9252,
+ "sales": 0.0055,
+ "other": 0.0676
+ },
+ "correct": true,
+ "input_tokens": 1031,
+ "latency_s": 0.248
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0277,
+ "technical": 0.0961,
+ "sales": 0.6554,
+ "other": 0.2208
+ },
+ "correct": true,
+ "input_tokens": 1032,
+ "latency_s": 0.2541
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9925,
+ "technical": 0.0042,
+ "sales": 0.0008,
+ "other": 0.0025
+ },
+ "correct": true,
+ "input_tokens": 1032,
+ "latency_s": 0.2319
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0206,
+ "technical": 0.913,
+ "sales": 0.025,
+ "other": 0.0415
+ },
+ "correct": true,
+ "input_tokens": 1034,
+ "latency_s": 0.2191
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0122,
+ "technical": 0.0055,
+ "sales": 0.9779,
+ "other": 0.0045
+ },
+ "correct": true,
+ "input_tokens": 1031,
+ "latency_s": 0.2079
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9994,
+ "technical": 0.0004,
+ "sales": 0.0001,
+ "other": 0.0002
+ },
+ "correct": true,
+ "input_tokens": 1037,
+ "latency_s": 0.2065
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.037,
+ "technical": 0.7866,
+ "sales": 0.0766,
+ "other": 0.0998
+ },
+ "correct": true,
+ "input_tokens": 1029,
+ "latency_s": 0.2058
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.5915,
+ "technical": 0.0174,
+ "sales": 0.3814,
+ "other": 0.0097
+ },
+ "correct": false,
+ "input_tokens": 1032,
+ "latency_s": 0.2111
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8398,
+ "technical": 0.0667,
+ "sales": 0.0611,
+ "other": 0.0324
+ },
+ "correct": true,
+ "input_tokens": 1039,
+ "latency_s": 0.2418
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0373,
+ "technical": 0.8791,
+ "sales": 0.0384,
+ "other": 0.0453
+ },
+ "correct": true,
+ "input_tokens": 1030,
+ "latency_s": 0.2208
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.2352,
+ "technical": 0.7265,
+ "sales": 0.018,
+ "other": 0.0204
+ },
+ "correct": false,
+ "input_tokens": 1038,
+ "latency_s": 0.2126
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "other",
+ "probabilities": {
+ "billing": 0.0653,
+ "technical": 0.3102,
+ "sales": 0.1143,
+ "other": 0.5101
+ },
+ "correct": false,
+ "input_tokens": 1031,
+ "latency_s": 0.2052
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8862,
+ "technical": 0.0806,
+ "sales": 0.0098,
+ "other": 0.0234
+ },
+ "correct": true,
+ "input_tokens": 1032,
+ "latency_s": 0.2041
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0029,
+ "technical": 0.859,
+ "sales": 0.0364,
+ "other": 0.1017
+ },
+ "correct": true,
+ "input_tokens": 1032,
+ "latency_s": 0.2095
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.6815,
+ "technical": 0.0904,
+ "sales": 0.0345,
+ "other": 0.1936
+ },
+ "correct": false,
+ "input_tokens": 1029,
+ "latency_s": 0.2112
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8362,
+ "technical": 0.1059,
+ "sales": 0.0313,
+ "other": 0.0267
+ },
+ "correct": true,
+ "input_tokens": 1042,
+ "latency_s": 0.2167
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0017,
+ "technical": 0.979,
+ "sales": 0.0043,
+ "other": 0.015
+ },
+ "correct": true,
+ "input_tokens": 1033,
+ "latency_s": 0.2061
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8857,
+ "technical": 0.0726,
+ "sales": 0.0204,
+ "other": 0.0212
+ },
+ "correct": true,
+ "input_tokens": 1033,
+ "latency_s": 0.2024
+ },
+ {
+ "pad_tokens": 1000,
+ "limit": 8192,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8039,
+ "technical": 0.0569,
+ "sales": 0.0949,
+ "other": 0.0443
+ },
+ "correct": true,
+ "input_tokens": 1041,
+ "latency_s": 0.2084
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.198
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.1977
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.1983
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.1999
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.199
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.199
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.1981
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.1965
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.1985
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.1963
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.1974
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.1974
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.1972
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.1984
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2781
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.1969
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.1987
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2008
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2016
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 1024,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2132
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9973,
+ "technical": 0.0011,
+ "sales": 0.001,
+ "other": 0.0005
+ },
+ "correct": true,
+ "input_tokens": 2043,
+ "latency_s": 0.5077
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0029,
+ "technical": 0.9509,
+ "sales": 0.0081,
+ "other": 0.038
+ },
+ "correct": true,
+ "input_tokens": 2039,
+ "latency_s": 0.5169
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0725,
+ "technical": 0.2498,
+ "sales": 0.3589,
+ "other": 0.3188
+ },
+ "correct": true,
+ "input_tokens": 2040,
+ "latency_s": 0.5026
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9042,
+ "technical": 0.0074,
+ "sales": 0.0837,
+ "other": 0.0047
+ },
+ "correct": true,
+ "input_tokens": 2040,
+ "latency_s": 0.5074
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0469,
+ "technical": 0.8677,
+ "sales": 0.0461,
+ "other": 0.0393
+ },
+ "correct": true,
+ "input_tokens": 2042,
+ "latency_s": 0.5019
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.457,
+ "technical": 0.0193,
+ "sales": 0.52,
+ "other": 0.0037
+ },
+ "correct": true,
+ "input_tokens": 2039,
+ "latency_s": 0.5155
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9832,
+ "technical": 0.0072,
+ "sales": 0.0068,
+ "other": 0.0027
+ },
+ "correct": true,
+ "input_tokens": 2045,
+ "latency_s": 0.5087
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0431,
+ "technical": 0.6873,
+ "sales": 0.1057,
+ "other": 0.1639
+ },
+ "correct": true,
+ "input_tokens": 2037,
+ "latency_s": 0.501
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.6699,
+ "technical": 0.1689,
+ "sales": 0.0782,
+ "other": 0.083
+ },
+ "correct": false,
+ "input_tokens": 2040,
+ "latency_s": 0.4992
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8633,
+ "technical": 0.0656,
+ "sales": 0.0415,
+ "other": 0.0297
+ },
+ "correct": true,
+ "input_tokens": 2047,
+ "latency_s": 0.5033
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.22,
+ "technical": 0.486,
+ "sales": 0.0366,
+ "other": 0.2575
+ },
+ "correct": true,
+ "input_tokens": 2038,
+ "latency_s": 0.5015
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.7424,
+ "technical": 0.0671,
+ "sales": 0.1807,
+ "other": 0.0097
+ },
+ "correct": true,
+ "input_tokens": 2046,
+ "latency_s": 0.498
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0441,
+ "technical": 0.4859,
+ "sales": 0.1138,
+ "other": 0.3562
+ },
+ "correct": true,
+ "input_tokens": 2039,
+ "latency_s": 0.4997
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.787,
+ "technical": 0.1019,
+ "sales": 0.0328,
+ "other": 0.0783
+ },
+ "correct": true,
+ "input_tokens": 2040,
+ "latency_s": 0.5001
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "other",
+ "probabilities": {
+ "billing": 0.0108,
+ "technical": 0.3056,
+ "sales": 0.0663,
+ "other": 0.6173
+ },
+ "correct": false,
+ "input_tokens": 2040,
+ "latency_s": 0.497
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9807,
+ "technical": 0.0077,
+ "sales": 0.005,
+ "other": 0.0066
+ },
+ "correct": false,
+ "input_tokens": 2037,
+ "latency_s": 0.4953
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.7201,
+ "technical": 0.0542,
+ "sales": 0.1909,
+ "other": 0.0349
+ },
+ "correct": true,
+ "input_tokens": 2050,
+ "latency_s": 0.5115
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0257,
+ "technical": 0.5177,
+ "sales": 0.0868,
+ "other": 0.3698
+ },
+ "correct": true,
+ "input_tokens": 2041,
+ "latency_s": 0.5032
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.6403,
+ "technical": 0.1165,
+ "sales": 0.2153,
+ "other": 0.0279
+ },
+ "correct": true,
+ "input_tokens": 2041,
+ "latency_s": 0.5056
+ },
+ {
+ "pad_tokens": 2000,
+ "limit": 8192,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.6447,
+ "technical": 0.1926,
+ "sales": 0.1146,
+ "other": 0.0482
+ },
+ "correct": true,
+ "input_tokens": 2049,
+ "latency_s": 0.5245
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2081
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2075
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2052
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2044
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2034
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2034
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2028
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2038
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2089
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2244
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2136
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2077
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2081
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2047
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2071
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2084
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.21
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2116
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2118
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 1024,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2101
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8344,
+ "technical": 0.0978,
+ "sales": 0.0467,
+ "other": 0.0211
+ },
+ "correct": true,
+ "input_tokens": 3009,
+ "latency_s": 0.9089
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0053,
+ "technical": 0.9487,
+ "sales": 0.0137,
+ "other": 0.0323
+ },
+ "correct": true,
+ "input_tokens": 3005,
+ "latency_s": 0.8579
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0217,
+ "technical": 0.2806,
+ "sales": 0.6104,
+ "other": 0.0874
+ },
+ "correct": true,
+ "input_tokens": 3006,
+ "latency_s": 0.865
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9073,
+ "technical": 0.0269,
+ "sales": 0.0554,
+ "other": 0.0104
+ },
+ "correct": true,
+ "input_tokens": 3006,
+ "latency_s": 0.8836
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0406,
+ "technical": 0.8045,
+ "sales": 0.0824,
+ "other": 0.0725
+ },
+ "correct": true,
+ "input_tokens": 3008,
+ "latency_s": 0.8833
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.2608,
+ "technical": 0.0564,
+ "sales": 0.6712,
+ "other": 0.0116
+ },
+ "correct": true,
+ "input_tokens": 3005,
+ "latency_s": 0.8768
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.5647,
+ "technical": 0.2784,
+ "sales": 0.1396,
+ "other": 0.0172
+ },
+ "correct": true,
+ "input_tokens": 3011,
+ "latency_s": 0.8856
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1061,
+ "technical": 0.5795,
+ "sales": 0.1366,
+ "other": 0.1778
+ },
+ "correct": true,
+ "input_tokens": 3003,
+ "latency_s": 0.8867
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1063,
+ "technical": 0.6598,
+ "sales": 0.1899,
+ "other": 0.0439
+ },
+ "correct": false,
+ "input_tokens": 3006,
+ "latency_s": 0.9258
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.5451,
+ "technical": 0.2854,
+ "sales": 0.096,
+ "other": 0.0735
+ },
+ "correct": true,
+ "input_tokens": 3013,
+ "latency_s": 0.9323
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1199,
+ "technical": 0.6749,
+ "sales": 0.0714,
+ "other": 0.1338
+ },
+ "correct": true,
+ "input_tokens": 3004,
+ "latency_s": 0.8702
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.7753,
+ "technical": 0.1373,
+ "sales": 0.0709,
+ "other": 0.0165
+ },
+ "correct": true,
+ "input_tokens": 3012,
+ "latency_s": 0.9031
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0768,
+ "technical": 0.5993,
+ "sales": 0.1218,
+ "other": 0.2021
+ },
+ "correct": true,
+ "input_tokens": 3005,
+ "latency_s": 0.8948
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.7084,
+ "technical": 0.172,
+ "sales": 0.0791,
+ "other": 0.0405
+ },
+ "correct": true,
+ "input_tokens": 3006,
+ "latency_s": 0.895
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0124,
+ "technical": 0.8603,
+ "sales": 0.0501,
+ "other": 0.0772
+ },
+ "correct": true,
+ "input_tokens": 3006,
+ "latency_s": 0.9091
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.4222,
+ "technical": 0.1659,
+ "sales": 0.3523,
+ "other": 0.0596
+ },
+ "correct": false,
+ "input_tokens": 3003,
+ "latency_s": 0.9637
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9193,
+ "technical": 0.0162,
+ "sales": 0.054,
+ "other": 0.0104
+ },
+ "correct": true,
+ "input_tokens": 3016,
+ "latency_s": 1.0056
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0447,
+ "technical": 0.6479,
+ "sales": 0.0823,
+ "other": 0.2251
+ },
+ "correct": true,
+ "input_tokens": 3007,
+ "latency_s": 0.832
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.5546,
+ "technical": 0.2218,
+ "sales": 0.1686,
+ "other": 0.055
+ },
+ "correct": true,
+ "input_tokens": 3007,
+ "latency_s": 1.156
+ },
+ {
+ "pad_tokens": 3000,
+ "limit": 8192,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.2848,
+ "technical": 0.5622,
+ "sales": 0.0753,
+ "other": 0.0777
+ },
+ "correct": false,
+ "input_tokens": 3015,
+ "latency_s": 1.6724
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2412
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2259
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2215
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2217
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2172
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2118
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.214
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2105
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2095
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2137
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2146
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.217
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2182
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2155
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2077
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.1859
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2291
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2093
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2251
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 1024,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2358
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9421,
+ "technical": 0.0374,
+ "sales": 0.0126,
+ "other": 0.0079
+ },
+ "correct": true,
+ "input_tokens": 3975,
+ "latency_s": 1.8231
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0082,
+ "technical": 0.9427,
+ "sales": 0.0171,
+ "other": 0.032
+ },
+ "correct": true,
+ "input_tokens": 3971,
+ "latency_s": 1.4936
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0094,
+ "technical": 0.2885,
+ "sales": 0.5426,
+ "other": 0.1595
+ },
+ "correct": true,
+ "input_tokens": 3972,
+ "latency_s": 1.5718
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9196,
+ "technical": 0.0349,
+ "sales": 0.0304,
+ "other": 0.0151
+ },
+ "correct": true,
+ "input_tokens": 3972,
+ "latency_s": 1.5661
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.096,
+ "technical": 0.6866,
+ "sales": 0.1003,
+ "other": 0.1171
+ },
+ "correct": true,
+ "input_tokens": 3974,
+ "latency_s": 1.647
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.1409,
+ "technical": 0.2129,
+ "sales": 0.5327,
+ "other": 0.1135
+ },
+ "correct": true,
+ "input_tokens": 3971,
+ "latency_s": 1.5932
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8401,
+ "technical": 0.1191,
+ "sales": 0.0302,
+ "other": 0.0106
+ },
+ "correct": true,
+ "input_tokens": 3977,
+ "latency_s": 1.5528
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.099,
+ "technical": 0.58,
+ "sales": 0.1605,
+ "other": 0.1605
+ },
+ "correct": true,
+ "input_tokens": 3969,
+ "latency_s": 1.5736
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.3632,
+ "technical": 0.2684,
+ "sales": 0.3226,
+ "other": 0.0458
+ },
+ "correct": false,
+ "input_tokens": 3972,
+ "latency_s": 1.6369
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.5448,
+ "technical": 0.267,
+ "sales": 0.1085,
+ "other": 0.0797
+ },
+ "correct": true,
+ "input_tokens": 3979,
+ "latency_s": 1.704
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.2278,
+ "technical": 0.5519,
+ "sales": 0.0632,
+ "other": 0.157
+ },
+ "correct": true,
+ "input_tokens": 3970,
+ "latency_s": 1.7331
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8088,
+ "technical": 0.1081,
+ "sales": 0.0532,
+ "other": 0.0299
+ },
+ "correct": true,
+ "input_tokens": 3978,
+ "latency_s": 1.7456
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1165,
+ "technical": 0.5871,
+ "sales": 0.1633,
+ "other": 0.1331
+ },
+ "correct": true,
+ "input_tokens": 3971,
+ "latency_s": 1.6623
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.6844,
+ "technical": 0.1678,
+ "sales": 0.0759,
+ "other": 0.0719
+ },
+ "correct": true,
+ "input_tokens": 3972,
+ "latency_s": 1.7318
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0421,
+ "technical": 0.5851,
+ "sales": 0.1039,
+ "other": 0.2689
+ },
+ "correct": true,
+ "input_tokens": 3972,
+ "latency_s": 1.7957
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8505,
+ "technical": 0.0606,
+ "sales": 0.041,
+ "other": 0.0479
+ },
+ "correct": false,
+ "input_tokens": 3969,
+ "latency_s": 1.7339
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9029,
+ "technical": 0.0489,
+ "sales": 0.0285,
+ "other": 0.0197
+ },
+ "correct": true,
+ "input_tokens": 3982,
+ "latency_s": 1.7106
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0638,
+ "technical": 0.7184,
+ "sales": 0.0949,
+ "other": 0.1229
+ },
+ "correct": true,
+ "input_tokens": 3973,
+ "latency_s": 1.7793
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.6808,
+ "technical": 0.1638,
+ "sales": 0.1272,
+ "other": 0.0282
+ },
+ "correct": true,
+ "input_tokens": 3973,
+ "latency_s": 1.7993
+ },
+ {
+ "pad_tokens": 4000,
+ "limit": 8192,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.476,
+ "technical": 0.3463,
+ "sales": 0.0763,
+ "other": 0.1014
+ },
+ "correct": true,
+ "input_tokens": 3981,
+ "latency_s": 1.7964
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2659
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2645
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2605
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.257
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2525
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2479
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2483
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2558
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2578
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2582
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2591
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.268
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2661
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2609
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2655
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2635
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2697
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2694
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2682
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 1024,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2645
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.7983,
+ "technical": 0.111,
+ "sales": 0.0441,
+ "other": 0.0465
+ },
+ "correct": true,
+ "input_tokens": 4941,
+ "latency_s": 2.4755
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0358,
+ "technical": 0.8148,
+ "sales": 0.0518,
+ "other": 0.0975
+ },
+ "correct": true,
+ "input_tokens": 4937,
+ "latency_s": 2.4847
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1233,
+ "technical": 0.3924,
+ "sales": 0.1405,
+ "other": 0.3438
+ },
+ "correct": false,
+ "input_tokens": 4938,
+ "latency_s": 2.5142
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.7252,
+ "technical": 0.1433,
+ "sales": 0.053,
+ "other": 0.0785
+ },
+ "correct": true,
+ "input_tokens": 4938,
+ "latency_s": 2.4527
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0978,
+ "technical": 0.67,
+ "sales": 0.1211,
+ "other": 0.1112
+ },
+ "correct": true,
+ "input_tokens": 4940,
+ "latency_s": 2.5813
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0861,
+ "technical": 0.4159,
+ "sales": 0.2357,
+ "other": 0.2622
+ },
+ "correct": false,
+ "input_tokens": 4937,
+ "latency_s": 2.5552
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.3753,
+ "technical": 0.2793,
+ "sales": 0.1125,
+ "other": 0.2329
+ },
+ "correct": true,
+ "input_tokens": 4943,
+ "latency_s": 2.6104
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0947,
+ "technical": 0.6261,
+ "sales": 0.1284,
+ "other": 0.1507
+ },
+ "correct": true,
+ "input_tokens": 4935,
+ "latency_s": 2.6318
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.3306,
+ "technical": 0.3327,
+ "sales": 0.1268,
+ "other": 0.21
+ },
+ "correct": false,
+ "input_tokens": 4938,
+ "latency_s": 2.5965
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.2937,
+ "technical": 0.4376,
+ "sales": 0.1017,
+ "other": 0.167
+ },
+ "correct": false,
+ "input_tokens": 4945,
+ "latency_s": 2.6252
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0643,
+ "technical": 0.717,
+ "sales": 0.0694,
+ "other": 0.1494
+ },
+ "correct": true,
+ "input_tokens": 4936,
+ "latency_s": 2.9064
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.2413,
+ "technical": 0.4798,
+ "sales": 0.1772,
+ "other": 0.1017
+ },
+ "correct": false,
+ "input_tokens": 4944,
+ "latency_s": 2.5591
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0685,
+ "technical": 0.6586,
+ "sales": 0.1313,
+ "other": 0.1416
+ },
+ "correct": true,
+ "input_tokens": 4937,
+ "latency_s": 2.5917
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.3755,
+ "technical": 0.4073,
+ "sales": 0.086,
+ "other": 0.1312
+ },
+ "correct": false,
+ "input_tokens": 4938,
+ "latency_s": 2.615
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0492,
+ "technical": 0.705,
+ "sales": 0.0803,
+ "other": 0.1654
+ },
+ "correct": true,
+ "input_tokens": 4938,
+ "latency_s": 2.5753
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.2488,
+ "technical": 0.3229,
+ "sales": 0.1395,
+ "other": 0.2888
+ },
+ "correct": false,
+ "input_tokens": 4935,
+ "latency_s": 2.6237
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.5135,
+ "technical": 0.265,
+ "sales": 0.0999,
+ "other": 0.1216
+ },
+ "correct": true,
+ "input_tokens": 4948,
+ "latency_s": 2.5953
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0792,
+ "technical": 0.6006,
+ "sales": 0.1046,
+ "other": 0.2156
+ },
+ "correct": true,
+ "input_tokens": 4939,
+ "latency_s": 2.5982
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1595,
+ "technical": 0.4993,
+ "sales": 0.1308,
+ "other": 0.2104
+ },
+ "correct": false,
+ "input_tokens": 4939,
+ "latency_s": 2.4926
+ },
+ {
+ "pad_tokens": 5000,
+ "limit": 8192,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1279,
+ "technical": 0.592,
+ "sales": 0.1029,
+ "other": 0.1772
+ },
+ "correct": false,
+ "input_tokens": 4947,
+ "latency_s": 2.5465
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2928
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2931
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2832
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2632
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2464
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.244
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2331
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2346
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.239
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2412
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2564
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2718
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2711
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2801
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2699
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2521
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2535
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.244
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2373
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 1024,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2333
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9586,
+ "technical": 0.0275,
+ "sales": 0.0125,
+ "other": 0.0014
+ },
+ "correct": true,
+ "input_tokens": 5949,
+ "latency_s": 3.4755
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0208,
+ "technical": 0.9289,
+ "sales": 0.0183,
+ "other": 0.0321
+ },
+ "correct": true,
+ "input_tokens": 5945,
+ "latency_s": 3.456
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0408,
+ "technical": 0.2888,
+ "sales": 0.6214,
+ "other": 0.049
+ },
+ "correct": true,
+ "input_tokens": 5946,
+ "latency_s": 3.4117
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9639,
+ "technical": 0.013,
+ "sales": 0.0206,
+ "other": 0.0025
+ },
+ "correct": true,
+ "input_tokens": 5946,
+ "latency_s": 3.537
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0891,
+ "technical": 0.7582,
+ "sales": 0.1054,
+ "other": 0.0473
+ },
+ "correct": true,
+ "input_tokens": 5948,
+ "latency_s": 3.4511
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.3029,
+ "technical": 0.1147,
+ "sales": 0.5626,
+ "other": 0.0198
+ },
+ "correct": true,
+ "input_tokens": 5945,
+ "latency_s": 3.4526
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9563,
+ "technical": 0.012,
+ "sales": 0.0313,
+ "other": 0.0004
+ },
+ "correct": true,
+ "input_tokens": 5951,
+ "latency_s": 3.8351
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0801,
+ "technical": 0.7417,
+ "sales": 0.1079,
+ "other": 0.0704
+ },
+ "correct": true,
+ "input_tokens": 5943,
+ "latency_s": 3.8274
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.3855,
+ "technical": 0.3995,
+ "sales": 0.2063,
+ "other": 0.0087
+ },
+ "correct": false,
+ "input_tokens": 5946,
+ "latency_s": 3.376
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.6968,
+ "technical": 0.0637,
+ "sales": 0.2294,
+ "other": 0.0101
+ },
+ "correct": true,
+ "input_tokens": 5953,
+ "latency_s": 3.6162
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1299,
+ "technical": 0.7835,
+ "sales": 0.0591,
+ "other": 0.0275
+ },
+ "correct": true,
+ "input_tokens": 5944,
+ "latency_s": 3.5467
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.5716,
+ "technical": 0.2197,
+ "sales": 0.2029,
+ "other": 0.0058
+ },
+ "correct": true,
+ "input_tokens": 5952,
+ "latency_s": 3.5259
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0936,
+ "technical": 0.7615,
+ "sales": 0.0856,
+ "other": 0.0592
+ },
+ "correct": true,
+ "input_tokens": 5945,
+ "latency_s": 3.4499
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8161,
+ "technical": 0.1147,
+ "sales": 0.0615,
+ "other": 0.0077
+ },
+ "correct": true,
+ "input_tokens": 5946,
+ "latency_s": 3.495
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0186,
+ "technical": 0.9389,
+ "sales": 0.0198,
+ "other": 0.0227
+ },
+ "correct": true,
+ "input_tokens": 5946,
+ "latency_s": 3.5258
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9047,
+ "technical": 0.0315,
+ "sales": 0.0537,
+ "other": 0.0101
+ },
+ "correct": false,
+ "input_tokens": 5943,
+ "latency_s": 3.5337
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.9231,
+ "technical": 0.0114,
+ "sales": 0.064,
+ "other": 0.0015
+ },
+ "correct": true,
+ "input_tokens": 5956,
+ "latency_s": 3.5095
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0437,
+ "technical": 0.7959,
+ "sales": 0.074,
+ "other": 0.0864
+ },
+ "correct": true,
+ "input_tokens": 5947,
+ "latency_s": 3.5579
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.3501,
+ "technical": 0.0984,
+ "sales": 0.5258,
+ "other": 0.0256
+ },
+ "correct": false,
+ "input_tokens": 5947,
+ "latency_s": 3.5007
+ },
+ {
+ "pad_tokens": 6000,
+ "limit": 8192,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.5852,
+ "technical": 0.0822,
+ "sales": 0.3076,
+ "other": 0.025
+ },
+ "correct": true,
+ "input_tokens": 5955,
+ "latency_s": 3.3542
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2487
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2558
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2566
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2622
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2646
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2652
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2644
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2634
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2637
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2585
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2591
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2588
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2609
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.27
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2734
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.274
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2706
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": true,
+ "input_tokens": 1024,
+ "latency_s": 0.2694
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2702
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 1024,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.041,
+ "technical": 0.5046,
+ "sales": 0.099,
+ "other": 0.3554
+ },
+ "correct": false,
+ "input_tokens": 1024,
+ "latency_s": 0.2808
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "I was charged twice for invoice 4411, please refund the duplicate payment.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.2611,
+ "technical": 0.3307,
+ "sales": 0.2399,
+ "other": 0.1683
+ },
+ "correct": false,
+ "input_tokens": 6915,
+ "latency_s": 4.5987
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "The dashboard crashes with an error every time I open the reports page.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0334,
+ "technical": 0.8169,
+ "sales": 0.0672,
+ "other": 0.0825
+ },
+ "correct": true,
+ "input_tokens": 6911,
+ "latency_s": 4.5849
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "What would an enterprise contract for 200 seats cost per year?",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0826,
+ "technical": 0.3564,
+ "sales": 0.2319,
+ "other": 0.3291
+ },
+ "correct": false,
+ "input_tokens": 6912,
+ "latency_s": 4.4796
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "Please refund my subscription payment from last month, it was billed by mistake.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.2523,
+ "technical": 0.3692,
+ "sales": 0.1808,
+ "other": 0.1978
+ },
+ "correct": false,
+ "input_tokens": 6912,
+ "latency_s": 4.5683
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "Our API returns 500 errors since this morning and the service is down.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0876,
+ "technical": 0.5578,
+ "sales": 0.1794,
+ "other": 0.1752
+ },
+ "correct": true,
+ "input_tokens": 6914,
+ "latency_s": 4.5567
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "Can you send me pricing for upgrading our plan to the business tier?",
+ "gold": "sales",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.0997,
+ "technical": 0.2864,
+ "sales": 0.5092,
+ "other": 0.1047
+ },
+ "correct": true,
+ "input_tokens": 6911,
+ "latency_s": 4.5336
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.",
+ "gold": "billing",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.1247,
+ "technical": 0.2638,
+ "sales": 0.4353,
+ "other": 0.1763
+ },
+ "correct": false,
+ "input_tokens": 6917,
+ "latency_s": 4.6067
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "La aplicación se cierra cada vez que abro la configuración.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0919,
+ "technical": 0.5122,
+ "sales": 0.2155,
+ "other": 0.1805
+ },
+ "correct": true,
+ "input_tokens": 6909,
+ "latency_s": 4.483
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "Quisiera una cotización para un contrato anual de 50 licencias.",
+ "gold": "sales",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0737,
+ "technical": 0.3788,
+ "sales": 0.358,
+ "other": 0.1895
+ },
+ "correct": false,
+ "input_tokens": 6912,
+ "latency_s": 4.3589
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.171,
+ "technical": 0.3933,
+ "sales": 0.177,
+ "other": 0.2587
+ },
+ "correct": false,
+ "input_tokens": 6919,
+ "latency_s": 4.554
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "O sistema cai toda vez que tento gerar o relatório mensal.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0694,
+ "technical": 0.5839,
+ "sales": 0.1421,
+ "other": 0.2046
+ },
+ "correct": true,
+ "input_tokens": 6910,
+ "latency_s": 4.3722
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1123,
+ "technical": 0.3748,
+ "sales": 0.3121,
+ "other": 0.2008
+ },
+ "correct": false,
+ "input_tokens": 6918,
+ "latency_s": 4.5599
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "L'application plante dès que j'ouvre la page des paramètres.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0709,
+ "technical": 0.5296,
+ "sales": 0.2135,
+ "other": 0.186
+ },
+ "correct": true,
+ "input_tokens": 6911,
+ "latency_s": 4.52
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1701,
+ "technical": 0.4553,
+ "sales": 0.225,
+ "other": 0.1496
+ },
+ "correct": false,
+ "input_tokens": 6912,
+ "latency_s": 4.4045
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0845,
+ "technical": 0.6309,
+ "sales": 0.1467,
+ "other": 0.1378
+ },
+ "correct": true,
+ "input_tokens": 6912,
+ "latency_s": 4.32
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "Was kostet ein Jahresvertrag für 100 Nutzer?",
+ "gold": "sales",
+ "choice": "other",
+ "probabilities": {
+ "billing": 0.0659,
+ "technical": 0.3256,
+ "sales": 0.1924,
+ "other": 0.4161
+ },
+ "correct": false,
+ "input_tokens": 6909,
+ "latency_s": 4.2518
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
+ "gold": "billing",
+ "choice": "sales",
+ "probabilities": {
+ "billing": 0.2697,
+ "technical": 0.1259,
+ "sales": 0.4876,
+ "other": 0.1167
+ },
+ "correct": false,
+ "input_tokens": 6922,
+ "latency_s": 4.2862
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।",
+ "gold": "technical",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0623,
+ "technical": 0.4758,
+ "sales": 0.191,
+ "other": 0.271
+ },
+ "correct": true,
+ "input_tokens": 6913,
+ "latency_s": 4.5278
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "請求書が二重に請求されました。重複分を返金してください。",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.0997,
+ "technical": 0.3768,
+ "sales": 0.2603,
+ "other": 0.2632
+ },
+ "correct": false,
+ "input_tokens": 6913,
+ "latency_s": 4.4854
+ },
+ {
+ "pad_tokens": 7000,
+ "limit": 8192,
+ "request": "تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.",
+ "gold": "billing",
+ "choice": "technical",
+ "probabilities": {
+ "billing": 0.1085,
+ "technical": 0.4273,
+ "sales": 0.2063,
+ "other": 0.2579
+ },
+ "correct": false,
+ "input_tokens": 6921,
+ "latency_s": 4.4806
+ }
+ ]
+}
\ No newline at end of file
diff --git a/research/scripts/bench_long_context.py b/research/scripts/bench_long_context.py
new file mode 100644
index 00000000..8a8c2a35
--- /dev/null
+++ b/research/scripts/bench_long_context.py
@@ -0,0 +1,98 @@
+"""Long-document accuracy and latency on laya-multilingual at the default limit and at 8,192 tokens.
+
+Each case is a support request placed at the END of a document padded with unrelated meeting
+notes, so the request sits after `pad` tokens of other text. That is the case a 1,024-token limit
+gets wrong: `build_sequence` keeps the start of the state by default and cuts the rest.
+
+ python research/scripts/bench_long_context.py --device mps --out research/results/long_context_multilingual.json
+
+The JSON keeps every prediction, so the table can be re-derived without re-running the model.
+"""
+import argparse, json, platform, statistics, sys, time
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+import torch
+import transformers
+import laya
+
+QUESTIONS = {"department": {"type": "choice", "instructions": "Which department should handle this request?",
+ "criteria": {"billing": "invoices, payments, refunds",
+ "technical": "bugs, outages, system errors",
+ "sales": "pricing, new contracts, plan upgrades",
+ "other": "everything else"}}}
+
+# 20 requests: English, Spanish, Portuguese, French, German, Hindi, Japanese, Arabic.
+REQUESTS = [
+ ("I was charged twice for invoice 4411, please refund the duplicate payment.", "billing"),
+ ("The dashboard crashes with an error every time I open the reports page.", "technical"),
+ ("What would an enterprise contract for 200 seats cost per year?", "sales"),
+ ("Please refund my subscription payment from last month, it was billed by mistake.", "billing"),
+ ("Our API returns 500 errors since this morning and the service is down.", "technical"),
+ ("Can you send me pricing for upgrading our plan to the business tier?", "sales"),
+ ("Me cobraron dos veces la factura de marzo, devuélvanme el cargo duplicado.", "billing"),
+ ("La aplicación se cierra cada vez que abro la configuración.", "technical"),
+ ("Quisiera una cotización para un contrato anual de 50 licencias.", "sales"),
+ ("Fui cobrado duas vezes na fatura de março, quero o reembolso da cobrança duplicada.", "billing"),
+ ("O sistema cai toda vez que tento gerar o relatório mensal.", "technical"),
+ ("J'ai été facturé deux fois ce mois-ci, merci de rembourser le doublon.", "billing"),
+ ("L'application plante dès que j'ouvre la page des paramètres.", "technical"),
+ ("Ich wurde zweimal belastet, bitte erstatten Sie die doppelte Zahlung.", "billing"),
+ ("Die Anwendung stürzt jedes Mal ab, wenn ich die Einstellungen öffne.", "technical"),
+ ("Was kostet ein Jahresvertrag für 100 Nutzer?", "sales"),
+ ("मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।", "billing"),
+ ("ऐप हर बार सेटिंग्स खोलते ही बंद हो जाता है।", "technical"),
+ ("請求書が二重に請求されました。重複分を返金してください。", "billing"),
+ ("تم خصم المبلغ مرتين من بطاقتي، أرجو استرداد المبلغ المكرر.", "billing"),
+]
+FILLER = ("Thanks for the update on the quarterly planning meeting. We reviewed the roadmap slides, discussed "
+ "hiring for the design team, agreed on the offsite venue, and noted that the parking garage will be "
+ "closed next week. ")
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--device", default="cpu")
+ ap.add_argument("--pads", default="0,1000,2000,4000,7000", help="approximate filler tokens before the request")
+ ap.add_argument("--out", default="research/results/long_context_multilingual.json")
+ a = ap.parse_args()
+
+ agent = laya.load("convaiinnovations/laya", subfolder="multilingual", device=a.device)
+ per_rep = len(agent.tok(FILLER, add_special_tokens=False)["input_ids"])
+ default_limit = int(agent.cfg.get("max_len", 1024))
+ rows, cases = [], []
+ for pad in [int(p) for p in a.pads.split(",")]:
+ reps = round(pad / per_rep)
+ for limit in (default_limit, 8192):
+ correct, lats, toks = 0, [], []
+ for i, (text, gold) in enumerate(REQUESTS):
+ state = FILLER * reps + ("\n\nActual request: " if reps else "") + text
+ if i == 0:
+ agent.predict(state, QUESTIONS, max_len=limit) # warm-up
+ t0 = time.perf_counter()
+ r = agent.predict(state, QUESTIONS, max_len=limit)
+ lat = time.perf_counter() - t0
+ ans = r["answers"]["department"]
+ ok = ans["choice"] == gold
+ correct += ok; lats.append(lat); toks.append(r["usage"]["input_tokens"])
+ cases.append({"pad_tokens": pad, "limit": limit, "request": text, "gold": gold,
+ "choice": ans["choice"], "probabilities": ans["probabilities"],
+ "correct": ok, "input_tokens": r["usage"]["input_tokens"], "latency_s": round(lat, 4)})
+ row = {"pad_tokens": pad, "limit": limit, "n": len(REQUESTS), "correct": correct,
+ "accuracy": round(correct / len(REQUESTS), 3),
+ "median_latency_s": round(statistics.median(lats), 3),
+ "median_input_tokens": int(statistics.median(toks))}
+ rows.append(row)
+ print("pad %5d limit %5d accuracy %2d/%d median %.3fs tokens read %d"
+ % (pad, limit, correct, len(REQUESTS), row["median_latency_s"], row["median_input_tokens"]), flush=True)
+ out = {"checkpoint": "convaiinnovations/laya (multilingual/)", "default_limit": default_limit,
+ "device": a.device, "platform": platform.platform(), "torch": torch.__version__,
+ "transformers": transformers.__version__, "laya": laya.__version__,
+ "questions": QUESTIONS, "rows": rows, "cases": cases}
+ Path(a.out).parent.mkdir(parents=True, exist_ok=True)
+ Path(a.out).write_text(json.dumps(out, ensure_ascii=False, indent=1))
+ print("wrote", a.out)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/research/scripts/plot_long_context.py b/research/scripts/plot_long_context.py
new file mode 100644
index 00000000..2150819d
--- /dev/null
+++ b/research/scripts/plot_long_context.py
@@ -0,0 +1,46 @@
+"""Render research/results/long_context_multilingual.json as assets/long_context_table.png."""
+import json, sys
+from pathlib import Path
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+
+src = Path(sys.argv[1] if len(sys.argv) > 1 else "research/results/long_context_multilingual.json")
+out = Path(sys.argv[2] if len(sys.argv) > 2 else "assets/long_context_table.png")
+d = json.loads(src.read_text())
+rows = {}
+for r in d["rows"]:
+ rows.setdefault(r["pad_tokens"], {})[r["limit"]] = r
+limits = sorted({r["limit"] for r in d["rows"]})
+lo, hi = limits[0], limits[-1]
+
+header = ["Request placed after", "Default limit (%s)" % f"{lo:,}", "Limit %s" % f"{hi:,}", "Time at %s" % f"{hi:,}"]
+cells, colors = [], []
+GOOD, WARN, BAD, PLAIN = "#d9f2e0", "#fff1cc", "#fbdada", "#ffffff"
+for pad in sorted(rows):
+ a, b = rows[pad][lo], rows[pad][hi]
+ label = "short input" if pad == 0 else "~%s tokens" % f"{pad:,}"
+ cells.append([label, "%d / %d" % (a["correct"], a["n"]), "%d / %d" % (b["correct"], b["n"]), "%.2f s" % b["median_latency_s"]])
+ acc_b = b["correct"] / b["n"]
+ cb = GOOD if acc_b >= 0.8 else WARN if acc_b >= 0.5 else BAD
+ ca = GOOD if a["correct"] / a["n"] >= 0.8 else WARN if a["correct"] / a["n"] >= 0.5 else BAD
+ colors.append([PLAIN, ca, cb, PLAIN])
+
+fig, ax = plt.subplots(figsize=(10.5, 0.42 * (len(cells) + 1)), dpi=160)
+ax.axis("off")
+t = ax.table(cellText=cells, colLabels=header, cellColours=colors, cellLoc="center",
+ colColours=["#1f2937"] * 4, bbox=[0, 0, 1, 1])
+t.auto_set_font_size(False); t.set_fontsize(12)
+for (r, c), cell in t.get_celld().items():
+ cell.set_edgecolor("#d1d5db")
+ if r == 0:
+ cell.get_text().set_color("white"); cell.get_text().set_weight("bold")
+ax.set_title("laya-multilingual on long documents: accuracy on 20 support requests (8 languages)\n"
+ "with the request placed at the end of a document of unrelated text",
+ fontsize=13, weight="bold", pad=12)
+fig.text(0.5, -0.06,
+ "Green: 80%%+ correct. Amber: 50 to 79%%. Red: under 50%%. Time is the median per request on %s (%s). "
+ "Reproduce: research/scripts/bench_long_context.py" % ("an Apple GPU" if d["device"] == "mps" else d["device"], "laya " + d["laya"]),
+ ha="center", fontsize=9, color="#4b5563")
+fig.savefig(out, bbox_inches="tight", facecolor="white")
+print("wrote", out)