Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

test(shortlist): benchmark many-option strategies on three intent sets

Rishit Sharma4dee8f27335c
diff --git a/research/benchmarks/tournament/README.md b/research/benchmarks/tournament/README.md
new file mode 100644
index 00000000..ec493e73
--- /dev/null
+++ b/research/benchmarks/tournament/README.md
@@ -0,0 +1,108 @@
+# Many-option choice: one question, a wider budget, the shortlist, and a tournament
+
+A choice question's options share one `head_max_len` budget, so past roughly 20 labels each label
+is cut to a few tokens and some stop being distinguishable (README, "Honest limits"). This compares
+the ways around that on three public intent sets, each asked as one choice over its whole label set:
+
+| dataset | labels | test rows | development rows |
+|---|---:|---:|---|
+| BANKING77 (`mteb/banking77`) | 77 | 3,076 | 600 sampled from train |
+| CLINC150 (`clinc/clinc_oos`, `plus`, in-scope only) | 150 | 4,500 | 600 sampled from validation |
+| MASSIVE intent, en-US (`SetFit/amazon_massive_intent_en-US`) | 60 | 2,974 | 600 sampled from validation |
+
+The question is the one `research/scripts/build_benchmark_nb.py` asks of BANKING77: the state is
+`{"message": text}`, each label becomes an option with its underscores turned to spaces and no
+description, and the instruction is ``Which banking intent does `message` express?`` (``Which
+intent does `message` express?`` for the other two). Dataset files are pinned to the revisions in
+`benchmark.py`.
+
+Strategies, all on the English checkpoint at its defaults (`max_len=512`, `head_max_len=192`):
+
+- `single`: the whole label set in one `predict`.
+- `wide`: the same with `head_max_len=512, max_len=1024`, the first remedy the README lists.
+- `shortlist`: `predict_shortlist` with `k=20` and `embed_fn_from_agent`, the embedder that needs no
+ second model.
+- `tournament`: `predict_tournament` at its default `group_size=16`; `tournament-<n>` sets the
+ group size.
+
+Every row is a separate call, as a caller would make it, timed after three warm-up calls.
+
+```bash
+python research/benchmarks/tournament/benchmark.py --dataset banking77 --split test --out banking77.json
+python research/benchmarks/tournament/benchmark.py --dataset clinc150 --split test --out clinc150.json
+python research/benchmarks/tournament/benchmark.py --dataset massive --split test --out massive.json
+```
+
+The group size was chosen on the development rows only; no tournament ran on a test split before
+the default was fixed:
+
+```bash
+python research/benchmarks/tournament/benchmark.py --dataset banking77 --split train --limit 600 \
+ --strategies single,tournament-8,tournament-12,tournament-16,tournament-24,tournament-32 --out dev.json
+```
+
+`pyarrow` reads the parquet files; it is not a dependency of the package.
+
+## Recorded results
+
+Apple M4 Pro, MPS, Python 3.12, torch 2.14, Laya 0.3.27, English checkpoint. `results.json` holds
+every number below.
+
+### Test splits: one question against the tournament
+
+| dataset | labels | rows | one question | tournament | rows gained / lost | ECE, one question → tournament |
+|---|---:|---:|---:|---:|---:|---|
+| BANKING77 | 77 | 3,076 | 0.430 | **0.610** | 779 / 227 | 0.350 → 0.082 |
+| CLINC150 | 150 | 4,500 | 0.625 | **0.876** | 1,302 / 172 | 0.297 → 0.057 |
+| MASSIVE intent | 60 | 2,974 | 0.515 | **0.569** | 486 / 326 | 0.370 → 0.136 |
+
+The final call reads only the group winners, so its probabilities are over a few labels and its
+confidence is calibrated much closer to its accuracy than the one-question answer over every label.
+
+### The other remedies, on 500 test rows per dataset
+
+Accuracy, with ECE in brackets, every strategy on the same rows:
+
+| dataset | one question | `wide` | `shortlist` | tournament |
+|---|---:|---:|---:|---:|
+| BANKING77 | 0.416 (0.355) | 0.554 (0.308) | 0.224 (0.455) | **0.584** (0.111) |
+| CLINC150 | 0.604 (0.320) | 0.680 (0.266) | 0.188 (0.625) | **0.894** (0.054) |
+| MASSIVE intent | 0.542 (0.349) | **0.622** (0.270) | 0.184 (0.636) | 0.590 (0.125) |
+
+Raising the budget is the better choice when the labels fit uncut in the wider head: MASSIVE's 60
+short labels do at `head_max_len=512`. It falls behind as the label set outgrows the wider head, and
+on CLINC150's 150 labels it is still cut to 4 tokens per label. The shortlist lost to one question on
+all three sets with `embed_fn_from_agent`; its docstring already says a dedicated bi-encoder will
+usually shortlist better, and these numbers say the same about the built-in one.
+
+### Latency
+
+300 test rows per dataset (seed 1), one benchmark process on the GPU at a time:
+
+| dataset | one question, p50 / p95 | tournament, p50 / p95 | calls |
+|---|---:|---:|---:|
+| BANKING77 | 69 / 82 ms | 145 / 174 ms | 2 |
+| CLINC150 | 103 / 109 ms | 187 / 205 ms | 2 |
+| MASSIVE intent | 56 / 61 ms | 100 / 110 ms | 2 |
+
+The test and sample accuracy runs above had up to two benchmark processes sharing the GPU, so their
+recorded times are left out of `results.json`.
+
+### Group size, development rows only
+
+| group size | BANKING77 | CLINC150 | MASSIVE intent |
+|---|---:|---:|---:|
+| one question | 0.370 | 0.632 | 0.543 |
+| 8 | 0.570 | | |
+| 12 | 0.593 | 0.832 | 0.588 |
+| **16** | 0.583 | 0.855 | 0.580 |
+| 24 | 0.582 | 0.863 | 0.563 |
+| 32 | 0.610 | | 0.568 |
+
+Sizes 12 to 24 are within about a point of one another on average, so the default is 16: the largest
+size that settles up to 256 labels in a single round, and below the roughly 20 options at which the
+README says option text starts to be cut, so longer labels and descriptions keep their tokens too.
+Size 8 needs a second round for 77 labels and scored lower. Advancing the top two of each group of 16
+instead of one, measured with a local variant of this script, scored 0.602 on BANKING77 and 0.587
+on MASSIVE, within noise of one winner, so the library advances one. Size 8 ran on BANKING77 only, and
+size 32 did not finish on CLINC150.
diff --git a/research/benchmarks/tournament/benchmark.py b/research/benchmarks/tournament/benchmark.py
new file mode 100644
index 00000000..d9c21205
--- /dev/null
+++ b/research/benchmarks/tournament/benchmark.py
@@ -0,0 +1,154 @@
+"""Many-option choice on three public intent sets: one question, a wider budget, the embedding
+shortlist, and `predict_tournament`.
+
+Every strategy answers one row per call, as a caller would. Needs `pyarrow` for the parquet files.
+
+ python research/benchmarks/tournament/benchmark.py --dataset banking77 --split test \
+ --strategies single,wide,shortlist,tournament --out banking77.json
+"""
+import argparse
+import json
+import platform
+import random
+import sys
+import time
+from pathlib import Path
+
+from huggingface_hub import hf_hub_download
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
+
+import laya # noqa: E402
+from laya.evals import ece # noqa: E402
+
+REVISIONS = {
+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",
+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",
+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6",
+}
+
+
+def fetch(repo, filename):
+ return Path(hf_hub_download(repo, filename, repo_type="dataset", revision=REVISIONS[repo]))
+
+
+def banking77(split):
+ import pyarrow.parquet as pq
+
+ def read(name):
+ return pq.read_table(fetch("mteb/banking77", "data/%s-00000-of-00001.parquet" % name)).to_pylist()
+
+ names = dict((r["label"], r["label_text"]) for r in read("test"))
+ return [names[i] for i in sorted(names)], [(r["text"], r["label_text"]) for r in read(split)]
+
+
+def clinc150(split):
+ import pyarrow.parquet as pq
+
+ path = fetch("clinc/clinc_oos", "plus/%s-00000-of-00001.parquet" % split)
+ names = json.loads(pq.read_schema(path).metadata[b"huggingface"])["info"]["features"]["intent"]["names"]
+ rows = [(r["text"], names[r["intent"]]) for r in pq.read_table(path).to_pylist()]
+ # In-scope intents only: "oos" is not an intent a label can describe.
+ return [n for n in names if n != "oos"], [(text, gold) for text, gold in rows if gold != "oos"]
+
+
+def massive(split):
+ def read(name):
+ path = fetch("SetFit/amazon_massive_intent_en-US", "%s.jsonl" % name)
+ return [json.loads(line) for line in path.read_text().splitlines()]
+
+ names = dict((r["label"], r["label_text"]) for r in read("train"))
+ return [names[i] for i in sorted(names)], [(r["text"], r["label_text"]) for r in read(split)]
+
+
+DATASETS = {
+ "banking77": (banking77, "Which banking intent does `message` express?"),
+ "clinc150": (clinc150, "Which intent does `message` express?"),
+ "massive": (massive, "Which intent does `message` express?"),
+}
+
+
+def strategy(name, agent):
+ """`(state, question) -> (answer, model calls)` for one strategy name."""
+ if name in ("single", "wide"):
+ budget = {"head_max_len": 512, "max_len": 1024} if name == "wide" else {}
+ return lambda state, q: (agent.predict(state, {"q": q}, **budget)["answers"]["q"], 1)
+ if name == "shortlist":
+ embed_fn = laya.cached_embed_fn(laya.embed_fn_from_agent(agent))
+ return lambda state, q: (laya.predict_shortlist(agent, state, {"q": q}, embed_fn)["answers"]["q"], 1)
+ if name.startswith("tournament"):
+ size = {"group_size": int(name.split("-")[1])} if "-" in name else {}
+
+ def run(state, q):
+ out = laya.predict_tournament(agent, state, {"q": q}, **size)
+ return out["answers"]["q"], out["tournament"]["q"]["rounds"] + 1
+ return run
+ raise SystemExit("unknown strategy %r" % name)
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--dataset", required=True, choices=sorted(DATASETS))
+ ap.add_argument("--split", required=True)
+ ap.add_argument("--limit", type=int, help="score a seeded sample of this many rows")
+ ap.add_argument("--seed", type=int, default=0)
+ ap.add_argument("--strategies", default="single,wide,shortlist,tournament",
+ help="comma list of single, wide, shortlist, tournament, tournament-<group size>")
+ ap.add_argument("--model", default="english")
+ ap.add_argument("--device")
+ ap.add_argument("--out", required=True)
+ args = ap.parse_args()
+
+ load, instructions = DATASETS[args.dataset]
+ names, rows = load(args.split)
+ if args.limit and args.limit < len(rows):
+ rows = random.Random(args.seed).sample(rows, args.limit)
+ question = {"type": "choice", "instructions": instructions,
+ "criteria": {name.replace("_", " "): None for name in names}}
+ agent = laya.load(args.model, device=args.device)
+ few = dict(question, criteria=dict(list(question["criteria"].items())[:8]))
+ for text, _gold in rows[:3]: # warm up before anything is timed
+ agent.predict({"message": text}, {"q": few})
+
+ report = {"dataset": args.dataset, "split": args.split, "rows": len(rows), "options": len(names),
+ "revisions": REVISIONS, "model": args.model, "device": str(agent.device),
+ "laya": laya.__version__, "machine": platform.machine(), "strategies": {}}
+ first = None
+ for name in args.strategies.split(","):
+ answer_fn = strategy(name, agent)
+ correct, pairs, ms, calls, errors = [], [], [], [], []
+ for text, gold in rows:
+ started = time.perf_counter()
+ try:
+ answer, n_calls = answer_fn({"message": text}, question)
+ except ValueError as exc: # the question does not fit the window at all
+ errors.append(str(exc))
+ correct.append(0)
+ if len(errors) == len(correct) == 3:
+ break
+ continue
+ ms.append((time.perf_counter() - started) * 1000)
+ correct.append(int(answer["choice"] == gold.replace("_", " ")))
+ pairs.append((answer["answer_confidence"], bool(correct[-1])))
+ calls.append(n_calls)
+ entry = {"errors": len(errors), "first_error": errors[0] if errors else None}
+ if len(correct) == len(rows):
+ entry["accuracy"] = round(sum(correct) / len(rows), 4)
+ if ms:
+ ms.sort()
+ entry.update(ece=round(ece([c for c, _ in pairs], [ok for _, ok in pairs]), 4),
+ mean_ms=round(sum(ms) / len(ms), 1), p50_ms=round(ms[len(ms) // 2], 1),
+ p95_ms=round(ms[int(len(ms) * 0.95)], 1),
+ calls_per_row=round(sum(calls) / len(calls), 3))
+ if first is None:
+ first = (name, correct)
+ elif len(correct) == len(first[1]) == len(rows):
+ entry["vs_" + first[0]] = {"gained": sum(a > b for a, b in zip(correct, first[1])),
+ "lost": sum(a < b for a, b in zip(correct, first[1]))}
+ report["strategies"][name] = entry
+ print(args.dataset, name, json.dumps(entry), flush=True)
+ Path(args.out).write_text(json.dumps(report, indent=1) + "\n")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/research/benchmarks/tournament/results.json b/research/benchmarks/tournament/results.json
new file mode 100644
index 00000000..005c4de7
--- /dev/null
+++ b/research/benchmarks/tournament/results.json
@@ -0,0 +1,439 @@
+{
+ "note": "Accuracy runs (test, sample) ran with up to two benchmark processes on one GPU; latency comes from the separate latency runs, one process at a time.",
+ "test": {
+ "banking77": {
+ "split": "test",
+ "rows": 3076,
+ "options": 77,
+ "revisions": {
+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",
+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",
+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"
+ },
+ "model": "english",
+ "device": "mps",
+ "laya": "0.3.27",
+ "machine": "arm64",
+ "strategies": {
+ "single": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.4301,
+ "ece": 0.3499,
+ "calls_per_row": 1.0
+ },
+ "tournament": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.6096,
+ "ece": 0.0816,
+ "calls_per_row": 2.0,
+ "vs_single": {
+ "gained": 779,
+ "lost": 227
+ }
+ }
+ }
+ },
+ "clinc150": {
+ "split": "test",
+ "rows": 4500,
+ "options": 150,
+ "revisions": {
+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",
+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",
+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"
+ },
+ "model": "english",
+ "device": "mps",
+ "laya": "0.3.27",
+ "machine": "arm64",
+ "strategies": {
+ "single": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.6247,
+ "ece": 0.2972,
+ "calls_per_row": 1.0
+ },
+ "tournament": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.8758,
+ "ece": 0.057,
+ "calls_per_row": 2.0,
+ "vs_single": {
+ "gained": 1302,
+ "lost": 172
+ }
+ }
+ }
+ },
+ "massive": {
+ "split": "test",
+ "rows": 2974,
+ "options": 60,
+ "revisions": {
+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",
+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",
+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"
+ },
+ "model": "english",
+ "device": "mps",
+ "laya": "0.3.27",
+ "machine": "arm64",
+ "strategies": {
+ "single": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.5148,
+ "ece": 0.3696,
+ "calls_per_row": 1.0
+ },
+ "tournament": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.5686,
+ "ece": 0.1359,
+ "calls_per_row": 2.0,
+ "vs_single": {
+ "gained": 486,
+ "lost": 326
+ }
+ }
+ }
+ }
+ },
+ "sample": {
+ "banking77": {
+ "split": "test",
+ "rows": 500,
+ "options": 77,
+ "revisions": {
+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",
+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",
+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"
+ },
+ "model": "english",
+ "device": "mps",
+ "laya": "0.3.27",
+ "machine": "arm64",
+ "strategies": {
+ "single": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.416,
+ "ece": 0.3549,
+ "calls_per_row": 1.0
+ },
+ "wide": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.554,
+ "ece": 0.308,
+ "calls_per_row": 1.0,
+ "vs_single": {
+ "gained": 96,
+ "lost": 27
+ }
+ },
+ "shortlist": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.224,
+ "ece": 0.4552,
+ "calls_per_row": 1.0,
+ "vs_single": {
+ "gained": 60,
+ "lost": 156
+ }
+ },
+ "tournament": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.584,
+ "ece": 0.1114,
+ "calls_per_row": 2.0,
+ "vs_single": {
+ "gained": 126,
+ "lost": 42
+ }
+ }
+ }
+ },
+ "clinc150": {
+ "split": "test",
+ "rows": 500,
+ "options": 150,
+ "revisions": {
+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",
+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",
+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"
+ },
+ "model": "english",
+ "device": "mps",
+ "laya": "0.3.27",
+ "machine": "arm64",
+ "strategies": {
+ "single": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.604,
+ "ece": 0.3201,
+ "calls_per_row": 1.0
+ },
+ "wide": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.68,
+ "ece": 0.2659,
+ "calls_per_row": 1.0,
+ "vs_single": {
+ "gained": 53,
+ "lost": 15
+ }
+ },
+ "shortlist": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.188,
+ "ece": 0.6247,
+ "calls_per_row": 1.0,
+ "vs_single": {
+ "gained": 54,
+ "lost": 262
+ }
+ },
+ "tournament": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.894,
+ "ece": 0.0538,
+ "calls_per_row": 2.0,
+ "vs_single": {
+ "gained": 161,
+ "lost": 16
+ }
+ }
+ }
+ },
+ "massive": {
+ "split": "test",
+ "rows": 500,
+ "options": 60,
+ "revisions": {
+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",
+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",
+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"
+ },
+ "model": "english",
+ "device": "mps",
+ "laya": "0.3.27",
+ "machine": "arm64",
+ "strategies": {
+ "single": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.542,
+ "ece": 0.3492,
+ "calls_per_row": 1.0
+ },
+ "wide": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.622,
+ "ece": 0.2704,
+ "calls_per_row": 1.0,
+ "vs_single": {
+ "gained": 58,
+ "lost": 18
+ }
+ },
+ "shortlist": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.184,
+ "ece": 0.6364,
+ "calls_per_row": 1.0,
+ "vs_single": {
+ "gained": 23,
+ "lost": 202
+ }
+ },
+ "tournament": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.59,
+ "ece": 0.1253,
+ "calls_per_row": 2.0,
+ "vs_single": {
+ "gained": 85,
+ "lost": 61
+ }
+ }
+ }
+ }
+ },
+ "latency": {
+ "banking77": {
+ "split": "test",
+ "rows": 300,
+ "options": 77,
+ "revisions": {
+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",
+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",
+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"
+ },
+ "model": "english",
+ "device": "mps",
+ "laya": "0.3.27",
+ "machine": "arm64",
+ "strategies": {
+ "single": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.4267,
+ "ece": 0.3375,
+ "mean_ms": 71.7,
+ "p50_ms": 69.3,
+ "p95_ms": 81.7,
+ "calls_per_row": 1.0
+ },
+ "tournament": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.58,
+ "ece": 0.1254,
+ "mean_ms": 149.2,
+ "p50_ms": 145.4,
+ "p95_ms": 173.6,
+ "calls_per_row": 2.0,
+ "vs_single": {
+ "gained": 67,
+ "lost": 21
+ }
+ }
+ }
+ },
+ "clinc150": {
+ "split": "test",
+ "rows": 300,
+ "options": 150,
+ "revisions": {
+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",
+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",
+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"
+ },
+ "model": "english",
+ "device": "mps",
+ "laya": "0.3.27",
+ "machine": "arm64",
+ "strategies": {
+ "single": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.6133,
+ "ece": 0.3313,
+ "mean_ms": 103.9,
+ "p50_ms": 103.3,
+ "p95_ms": 108.7,
+ "calls_per_row": 1.0
+ },
+ "tournament": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.8767,
+ "ece": 0.0496,
+ "mean_ms": 187.3,
+ "p50_ms": 186.9,
+ "p95_ms": 205.3,
+ "calls_per_row": 2.0,
+ "vs_single": {
+ "gained": 90,
+ "lost": 11
+ }
+ }
+ }
+ },
+ "massive": {
+ "split": "test",
+ "rows": 300,
+ "options": 60,
+ "revisions": {
+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",
+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",
+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"
+ },
+ "model": "english",
+ "device": "mps",
+ "laya": "0.3.27",
+ "machine": "arm64",
+ "strategies": {
+ "single": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.4767,
+ "ece": 0.3974,
+ "mean_ms": 56.8,
+ "p50_ms": 56.1,
+ "p95_ms": 60.8,
+ "calls_per_row": 1.0
+ },
+ "tournament": {
+ "errors": 0,
+ "first_error": null,
+ "accuracy": 0.52,
+ "ece": 0.1738,
+ "mean_ms": 100.8,
+ "p50_ms": 99.5,
+ "p95_ms": 110.0,
+ "calls_per_row": 2.0,
+ "vs_single": {
+ "gained": 46,
+ "lost": 33
+ }
+ }
+ }
+ }
+ },
+ "development": {
+ "banking77": {
+ "split": "train",
+ "rows": 600,
+ "options": 77,
+ "accuracy": {
+ "single": 0.37,
+ "tournament-8": 0.57,
+ "tournament-12": 0.5933,
+ "tournament-16": 0.5833,
+ "tournament-24": 0.5817,
+ "tournament-32": 0.61,
+ "top2-16": 0.6017
+ }
+ },
+ "clinc150": {
+ "split": "validation",
+ "rows": 600,
+ "options": 150,
+ "accuracy": {
+ "single": 0.6317,
+ "tournament-12": 0.8317,
+ "tournament-16": 0.855,
+ "tournament-24": 0.8633
+ }
+ },
+ "massive": {
+ "split": "validation",
+ "rows": 600,
+ "options": 60,
+ "accuracy": {
+ "single": 0.5433,
+ "tournament-12": 0.5883,
+ "tournament-16": 0.58,
+ "tournament-24": 0.5633,
+ "tournament-32": 0.5683,
+ "top2-16": 0.5867
+ }
+ }
+ }
+}