Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
test(shortlist): benchmark many-option strategies on three intent sets
Rishit Sharma4dee8f27335c
diff --git a/research/benchmarks/tournament/README.md b/research/benchmarks/tournament/README.mdnew file mode 100644index 00000000..ec493e73--- /dev/null+++ b/research/benchmarks/tournament/README.md@@ -0,0 +1,108 @@+# Many-option choice: one question, a wider budget, the shortlist, and a tournament++A choice question's options share one `head_max_len` budget, so past roughly 20 labels each label+is cut to a few tokens and some stop being distinguishable (README, "Honest limits"). This compares+the ways around that on three public intent sets, each asked as one choice over its whole label set:++| dataset | labels | test rows | development rows |+|---|---:|---:|---|+| BANKING77 (`mteb/banking77`) | 77 | 3,076 | 600 sampled from train |+| CLINC150 (`clinc/clinc_oos`, `plus`, in-scope only) | 150 | 4,500 | 600 sampled from validation |+| MASSIVE intent, en-US (`SetFit/amazon_massive_intent_en-US`) | 60 | 2,974 | 600 sampled from validation |++The question is the one `research/scripts/build_benchmark_nb.py` asks of BANKING77: the state is+`{"message": text}`, each label becomes an option with its underscores turned to spaces and no+description, and the instruction is ``Which banking intent does `message` express?`` (``Which+intent does `message` express?`` for the other two). Dataset files are pinned to the revisions in+`benchmark.py`.++Strategies, all on the English checkpoint at its defaults (`max_len=512`, `head_max_len=192`):++- `single`: the whole label set in one `predict`.+- `wide`: the same with `head_max_len=512, max_len=1024`, the first remedy the README lists.+- `shortlist`: `predict_shortlist` with `k=20` and `embed_fn_from_agent`, the embedder that needs no+ second model.+- `tournament`: `predict_tournament` at its default `group_size=16`; `tournament-<n>` sets the+ group size.++Every row is a separate call, as a caller would make it, timed after three warm-up calls.++```bash+python research/benchmarks/tournament/benchmark.py --dataset banking77 --split test --out banking77.json+python research/benchmarks/tournament/benchmark.py --dataset clinc150 --split test --out clinc150.json+python research/benchmarks/tournament/benchmark.py --dataset massive --split test --out massive.json+```++The group size was chosen on the development rows only; no tournament ran on a test split before+the default was fixed:++```bash+python research/benchmarks/tournament/benchmark.py --dataset banking77 --split train --limit 600 \+ --strategies single,tournament-8,tournament-12,tournament-16,tournament-24,tournament-32 --out dev.json+```++`pyarrow` reads the parquet files; it is not a dependency of the package.++## Recorded results++Apple M4 Pro, MPS, Python 3.12, torch 2.14, Laya 0.3.27, English checkpoint. `results.json` holds+every number below.++### Test splits: one question against the tournament++| dataset | labels | rows | one question | tournament | rows gained / lost | ECE, one question → tournament |+|---|---:|---:|---:|---:|---:|---|+| BANKING77 | 77 | 3,076 | 0.430 | **0.610** | 779 / 227 | 0.350 → 0.082 |+| CLINC150 | 150 | 4,500 | 0.625 | **0.876** | 1,302 / 172 | 0.297 → 0.057 |+| MASSIVE intent | 60 | 2,974 | 0.515 | **0.569** | 486 / 326 | 0.370 → 0.136 |++The final call reads only the group winners, so its probabilities are over a few labels and its+confidence is calibrated much closer to its accuracy than the one-question answer over every label.++### The other remedies, on 500 test rows per dataset++Accuracy, with ECE in brackets, every strategy on the same rows:++| dataset | one question | `wide` | `shortlist` | tournament |+|---|---:|---:|---:|---:|+| BANKING77 | 0.416 (0.355) | 0.554 (0.308) | 0.224 (0.455) | **0.584** (0.111) |+| CLINC150 | 0.604 (0.320) | 0.680 (0.266) | 0.188 (0.625) | **0.894** (0.054) |+| MASSIVE intent | 0.542 (0.349) | **0.622** (0.270) | 0.184 (0.636) | 0.590 (0.125) |++Raising the budget is the better choice when the labels fit uncut in the wider head: MASSIVE's 60+short labels do at `head_max_len=512`. It falls behind as the label set outgrows the wider head, and+on CLINC150's 150 labels it is still cut to 4 tokens per label. The shortlist lost to one question on+all three sets with `embed_fn_from_agent`; its docstring already says a dedicated bi-encoder will+usually shortlist better, and these numbers say the same about the built-in one.++### Latency++300 test rows per dataset (seed 1), one benchmark process on the GPU at a time:++| dataset | one question, p50 / p95 | tournament, p50 / p95 | calls |+|---|---:|---:|---:|+| BANKING77 | 69 / 82 ms | 145 / 174 ms | 2 |+| CLINC150 | 103 / 109 ms | 187 / 205 ms | 2 |+| MASSIVE intent | 56 / 61 ms | 100 / 110 ms | 2 |++The test and sample accuracy runs above had up to two benchmark processes sharing the GPU, so their+recorded times are left out of `results.json`.++### Group size, development rows only++| group size | BANKING77 | CLINC150 | MASSIVE intent |+|---|---:|---:|---:|+| one question | 0.370 | 0.632 | 0.543 |+| 8 | 0.570 | | |+| 12 | 0.593 | 0.832 | 0.588 |+| **16** | 0.583 | 0.855 | 0.580 |+| 24 | 0.582 | 0.863 | 0.563 |+| 32 | 0.610 | | 0.568 |++Sizes 12 to 24 are within about a point of one another on average, so the default is 16: the largest+size that settles up to 256 labels in a single round, and below the roughly 20 options at which the+README says option text starts to be cut, so longer labels and descriptions keep their tokens too.+Size 8 needs a second round for 77 labels and scored lower. Advancing the top two of each group of 16+instead of one, measured with a local variant of this script, scored 0.602 on BANKING77 and 0.587+on MASSIVE, within noise of one winner, so the library advances one. Size 8 ran on BANKING77 only, and+size 32 did not finish on CLINC150.diff --git a/research/benchmarks/tournament/benchmark.py b/research/benchmarks/tournament/benchmark.pynew file mode 100644index 00000000..d9c21205--- /dev/null+++ b/research/benchmarks/tournament/benchmark.py@@ -0,0 +1,154 @@+"""Many-option choice on three public intent sets: one question, a wider budget, the embedding+shortlist, and `predict_tournament`.++Every strategy answers one row per call, as a caller would. Needs `pyarrow` for the parquet files.++ python research/benchmarks/tournament/benchmark.py --dataset banking77 --split test \+ --strategies single,wide,shortlist,tournament --out banking77.json+"""+import argparse+import json+import platform+import random+import sys+import time+from pathlib import Path++from huggingface_hub import hf_hub_download++sys.path.insert(0, str(Path(__file__).resolve().parents[3]))++import laya # noqa: E402+from laya.evals import ece # noqa: E402++REVISIONS = {+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6",+}+++def fetch(repo, filename):+ return Path(hf_hub_download(repo, filename, repo_type="dataset", revision=REVISIONS[repo]))+++def banking77(split):+ import pyarrow.parquet as pq++ def read(name):+ return pq.read_table(fetch("mteb/banking77", "data/%s-00000-of-00001.parquet" % name)).to_pylist()++ names = dict((r["label"], r["label_text"]) for r in read("test"))+ return [names[i] for i in sorted(names)], [(r["text"], r["label_text"]) for r in read(split)]+++def clinc150(split):+ import pyarrow.parquet as pq++ path = fetch("clinc/clinc_oos", "plus/%s-00000-of-00001.parquet" % split)+ names = json.loads(pq.read_schema(path).metadata[b"huggingface"])["info"]["features"]["intent"]["names"]+ rows = [(r["text"], names[r["intent"]]) for r in pq.read_table(path).to_pylist()]+ # In-scope intents only: "oos" is not an intent a label can describe.+ return [n for n in names if n != "oos"], [(text, gold) for text, gold in rows if gold != "oos"]+++def massive(split):+ def read(name):+ path = fetch("SetFit/amazon_massive_intent_en-US", "%s.jsonl" % name)+ return [json.loads(line) for line in path.read_text().splitlines()]++ names = dict((r["label"], r["label_text"]) for r in read("train"))+ return [names[i] for i in sorted(names)], [(r["text"], r["label_text"]) for r in read(split)]+++DATASETS = {+ "banking77": (banking77, "Which banking intent does `message` express?"),+ "clinc150": (clinc150, "Which intent does `message` express?"),+ "massive": (massive, "Which intent does `message` express?"),+}+++def strategy(name, agent):+ """`(state, question) -> (answer, model calls)` for one strategy name."""+ if name in ("single", "wide"):+ budget = {"head_max_len": 512, "max_len": 1024} if name == "wide" else {}+ return lambda state, q: (agent.predict(state, {"q": q}, **budget)["answers"]["q"], 1)+ if name == "shortlist":+ embed_fn = laya.cached_embed_fn(laya.embed_fn_from_agent(agent))+ return lambda state, q: (laya.predict_shortlist(agent, state, {"q": q}, embed_fn)["answers"]["q"], 1)+ if name.startswith("tournament"):+ size = {"group_size": int(name.split("-")[1])} if "-" in name else {}++ def run(state, q):+ out = laya.predict_tournament(agent, state, {"q": q}, **size)+ return out["answers"]["q"], out["tournament"]["q"]["rounds"] + 1+ return run+ raise SystemExit("unknown strategy %r" % name)+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--dataset", required=True, choices=sorted(DATASETS))+ ap.add_argument("--split", required=True)+ ap.add_argument("--limit", type=int, help="score a seeded sample of this many rows")+ ap.add_argument("--seed", type=int, default=0)+ ap.add_argument("--strategies", default="single,wide,shortlist,tournament",+ help="comma list of single, wide, shortlist, tournament, tournament-<group size>")+ ap.add_argument("--model", default="english")+ ap.add_argument("--device")+ ap.add_argument("--out", required=True)+ args = ap.parse_args()++ load, instructions = DATASETS[args.dataset]+ names, rows = load(args.split)+ if args.limit and args.limit < len(rows):+ rows = random.Random(args.seed).sample(rows, args.limit)+ question = {"type": "choice", "instructions": instructions,+ "criteria": {name.replace("_", " "): None for name in names}}+ agent = laya.load(args.model, device=args.device)+ few = dict(question, criteria=dict(list(question["criteria"].items())[:8]))+ for text, _gold in rows[:3]: # warm up before anything is timed+ agent.predict({"message": text}, {"q": few})++ report = {"dataset": args.dataset, "split": args.split, "rows": len(rows), "options": len(names),+ "revisions": REVISIONS, "model": args.model, "device": str(agent.device),+ "laya": laya.__version__, "machine": platform.machine(), "strategies": {}}+ first = None+ for name in args.strategies.split(","):+ answer_fn = strategy(name, agent)+ correct, pairs, ms, calls, errors = [], [], [], [], []+ for text, gold in rows:+ started = time.perf_counter()+ try:+ answer, n_calls = answer_fn({"message": text}, question)+ except ValueError as exc: # the question does not fit the window at all+ errors.append(str(exc))+ correct.append(0)+ if len(errors) == len(correct) == 3:+ break+ continue+ ms.append((time.perf_counter() - started) * 1000)+ correct.append(int(answer["choice"] == gold.replace("_", " ")))+ pairs.append((answer["answer_confidence"], bool(correct[-1])))+ calls.append(n_calls)+ entry = {"errors": len(errors), "first_error": errors[0] if errors else None}+ if len(correct) == len(rows):+ entry["accuracy"] = round(sum(correct) / len(rows), 4)+ if ms:+ ms.sort()+ entry.update(ece=round(ece([c for c, _ in pairs], [ok for _, ok in pairs]), 4),+ mean_ms=round(sum(ms) / len(ms), 1), p50_ms=round(ms[len(ms) // 2], 1),+ p95_ms=round(ms[int(len(ms) * 0.95)], 1),+ calls_per_row=round(sum(calls) / len(calls), 3))+ if first is None:+ first = (name, correct)+ elif len(correct) == len(first[1]) == len(rows):+ entry["vs_" + first[0]] = {"gained": sum(a > b for a, b in zip(correct, first[1])),+ "lost": sum(a < b for a, b in zip(correct, first[1]))}+ report["strategies"][name] = entry+ print(args.dataset, name, json.dumps(entry), flush=True)+ Path(args.out).write_text(json.dumps(report, indent=1) + "\n")+++if __name__ == "__main__":+ main()diff --git a/research/benchmarks/tournament/results.json b/research/benchmarks/tournament/results.jsonnew file mode 100644index 00000000..005c4de7--- /dev/null+++ b/research/benchmarks/tournament/results.json@@ -0,0 +1,439 @@+{+ "note": "Accuracy runs (test, sample) ran with up to two benchmark processes on one GPU; latency comes from the separate latency runs, one process at a time.",+ "test": {+ "banking77": {+ "split": "test",+ "rows": 3076,+ "options": 77,+ "revisions": {+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"+ },+ "model": "english",+ "device": "mps",+ "laya": "0.3.27",+ "machine": "arm64",+ "strategies": {+ "single": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.4301,+ "ece": 0.3499,+ "calls_per_row": 1.0+ },+ "tournament": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.6096,+ "ece": 0.0816,+ "calls_per_row": 2.0,+ "vs_single": {+ "gained": 779,+ "lost": 227+ }+ }+ }+ },+ "clinc150": {+ "split": "test",+ "rows": 4500,+ "options": 150,+ "revisions": {+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"+ },+ "model": "english",+ "device": "mps",+ "laya": "0.3.27",+ "machine": "arm64",+ "strategies": {+ "single": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.6247,+ "ece": 0.2972,+ "calls_per_row": 1.0+ },+ "tournament": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.8758,+ "ece": 0.057,+ "calls_per_row": 2.0,+ "vs_single": {+ "gained": 1302,+ "lost": 172+ }+ }+ }+ },+ "massive": {+ "split": "test",+ "rows": 2974,+ "options": 60,+ "revisions": {+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"+ },+ "model": "english",+ "device": "mps",+ "laya": "0.3.27",+ "machine": "arm64",+ "strategies": {+ "single": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.5148,+ "ece": 0.3696,+ "calls_per_row": 1.0+ },+ "tournament": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.5686,+ "ece": 0.1359,+ "calls_per_row": 2.0,+ "vs_single": {+ "gained": 486,+ "lost": 326+ }+ }+ }+ }+ },+ "sample": {+ "banking77": {+ "split": "test",+ "rows": 500,+ "options": 77,+ "revisions": {+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"+ },+ "model": "english",+ "device": "mps",+ "laya": "0.3.27",+ "machine": "arm64",+ "strategies": {+ "single": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.416,+ "ece": 0.3549,+ "calls_per_row": 1.0+ },+ "wide": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.554,+ "ece": 0.308,+ "calls_per_row": 1.0,+ "vs_single": {+ "gained": 96,+ "lost": 27+ }+ },+ "shortlist": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.224,+ "ece": 0.4552,+ "calls_per_row": 1.0,+ "vs_single": {+ "gained": 60,+ "lost": 156+ }+ },+ "tournament": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.584,+ "ece": 0.1114,+ "calls_per_row": 2.0,+ "vs_single": {+ "gained": 126,+ "lost": 42+ }+ }+ }+ },+ "clinc150": {+ "split": "test",+ "rows": 500,+ "options": 150,+ "revisions": {+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"+ },+ "model": "english",+ "device": "mps",+ "laya": "0.3.27",+ "machine": "arm64",+ "strategies": {+ "single": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.604,+ "ece": 0.3201,+ "calls_per_row": 1.0+ },+ "wide": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.68,+ "ece": 0.2659,+ "calls_per_row": 1.0,+ "vs_single": {+ "gained": 53,+ "lost": 15+ }+ },+ "shortlist": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.188,+ "ece": 0.6247,+ "calls_per_row": 1.0,+ "vs_single": {+ "gained": 54,+ "lost": 262+ }+ },+ "tournament": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.894,+ "ece": 0.0538,+ "calls_per_row": 2.0,+ "vs_single": {+ "gained": 161,+ "lost": 16+ }+ }+ }+ },+ "massive": {+ "split": "test",+ "rows": 500,+ "options": 60,+ "revisions": {+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"+ },+ "model": "english",+ "device": "mps",+ "laya": "0.3.27",+ "machine": "arm64",+ "strategies": {+ "single": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.542,+ "ece": 0.3492,+ "calls_per_row": 1.0+ },+ "wide": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.622,+ "ece": 0.2704,+ "calls_per_row": 1.0,+ "vs_single": {+ "gained": 58,+ "lost": 18+ }+ },+ "shortlist": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.184,+ "ece": 0.6364,+ "calls_per_row": 1.0,+ "vs_single": {+ "gained": 23,+ "lost": 202+ }+ },+ "tournament": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.59,+ "ece": 0.1253,+ "calls_per_row": 2.0,+ "vs_single": {+ "gained": 85,+ "lost": 61+ }+ }+ }+ }+ },+ "latency": {+ "banking77": {+ "split": "test",+ "rows": 300,+ "options": 77,+ "revisions": {+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"+ },+ "model": "english",+ "device": "mps",+ "laya": "0.3.27",+ "machine": "arm64",+ "strategies": {+ "single": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.4267,+ "ece": 0.3375,+ "mean_ms": 71.7,+ "p50_ms": 69.3,+ "p95_ms": 81.7,+ "calls_per_row": 1.0+ },+ "tournament": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.58,+ "ece": 0.1254,+ "mean_ms": 149.2,+ "p50_ms": 145.4,+ "p95_ms": 173.6,+ "calls_per_row": 2.0,+ "vs_single": {+ "gained": 67,+ "lost": 21+ }+ }+ }+ },+ "clinc150": {+ "split": "test",+ "rows": 300,+ "options": 150,+ "revisions": {+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"+ },+ "model": "english",+ "device": "mps",+ "laya": "0.3.27",+ "machine": "arm64",+ "strategies": {+ "single": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.6133,+ "ece": 0.3313,+ "mean_ms": 103.9,+ "p50_ms": 103.3,+ "p95_ms": 108.7,+ "calls_per_row": 1.0+ },+ "tournament": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.8767,+ "ece": 0.0496,+ "mean_ms": 187.3,+ "p50_ms": 186.9,+ "p95_ms": 205.3,+ "calls_per_row": 2.0,+ "vs_single": {+ "gained": 90,+ "lost": 11+ }+ }+ }+ },+ "massive": {+ "split": "test",+ "rows": 300,+ "options": 60,+ "revisions": {+ "mteb/banking77": "18072d2685ea682290f7b8924d94c62acc19c0b2",+ "clinc/clinc_oos": "155b9c710419136e17307b80d0a13e68cd46b4ec",+ "SetFit/amazon_massive_intent_en-US": "f7672a018e8ceb37fc0184dcfbb7e665155ffea6"+ },+ "model": "english",+ "device": "mps",+ "laya": "0.3.27",+ "machine": "arm64",+ "strategies": {+ "single": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.4767,+ "ece": 0.3974,+ "mean_ms": 56.8,+ "p50_ms": 56.1,+ "p95_ms": 60.8,+ "calls_per_row": 1.0+ },+ "tournament": {+ "errors": 0,+ "first_error": null,+ "accuracy": 0.52,+ "ece": 0.1738,+ "mean_ms": 100.8,+ "p50_ms": 99.5,+ "p95_ms": 110.0,+ "calls_per_row": 2.0,+ "vs_single": {+ "gained": 46,+ "lost": 33+ }+ }+ }+ }+ },+ "development": {+ "banking77": {+ "split": "train",+ "rows": 600,+ "options": 77,+ "accuracy": {+ "single": 0.37,+ "tournament-8": 0.57,+ "tournament-12": 0.5933,+ "tournament-16": 0.5833,+ "tournament-24": 0.5817,+ "tournament-32": 0.61,+ "top2-16": 0.6017+ }+ },+ "clinc150": {+ "split": "validation",+ "rows": 600,+ "options": 150,+ "accuracy": {+ "single": 0.6317,+ "tournament-12": 0.8317,+ "tournament-16": 0.855,+ "tournament-24": 0.8633+ }+ },+ "massive": {+ "split": "validation",+ "rows": 600,+ "options": 60,+ "accuracy": {+ "single": 0.5433,+ "tournament-12": 0.5883,+ "tournament-16": 0.58,+ "tournament-24": 0.5633,+ "tournament-32": 0.5683,+ "top2-16": 0.5867+ }+ }+ }+}