Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Laya vs Laya-Multilingual \u2014 extensive head-to-head benchmark\n",
"\n",
"Runs both open checkpoints over the same questions and writes a single\n",
"`laya_benchmark_results.json`.\n",
"\n",
"| | `convaiinnovations/laya` | `convaiinnovations/laya-multilingual` |\n",
"|---|---|---|\n",
"| encoder | ModernBERT-large (English) | mmBERT-base (100+ languages) |\n",
"| params | 421.3M (394.8M enc + 26.5M head) | 321.9M (306.9M enc + 15.0M head) |\n",
"| context | 512 tokens | 1024 tokens |\n",
"| shipped temperatures | fitted per type & option-count | **none (all 1.0)** |\n",
"\n",
"**What is measured**\n",
"\n",
"1. **typed-decisions** \u2014 400 cases / 2,000 decisions, 4 workflows (the benchmark behind the published 0.766 claim). Zero-shot for both of these checkpoints.\n",
"2. **Multilingual** \u2014 MASSIVE intent + scenario across 14 languages, XNLI across 15 languages.\n",
"3. **English zero-shot** \u2014 SST-5, emotion, prompt-injections, plus banking77 as a 77-option stress test.\n",
"4. **Latency** \u2014 p50/p95 at 1 / 5 / 10 / 50 questions per call.\n",
"5. **Option-order robustness** \u2014 how often the answer flips when the options are permuted (the independent Jev benchmark measured 13% for Jev; worst LLM 37%).\n",
"6. **Calibration repair** \u2014 ECE before and after refitting temperature, which matters because `laya-multilingual` shipped uncalibrated.\n",
"\n",
"**Setup:** Runtime \u2192 Change runtime type \u2192 **T4 GPU**. Then Run All (~25\u201340 min).\n",
"Both models see byte-identical questions (fixed seed), so every difference is the model."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. GPU and environment"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import subprocess, sys, os, json, time\n",
"print(subprocess.run([\"nvidia-smi\",\"--query-gpu=name,memory.total,compute_cap\",\"--format=csv,noheader\"],\n",
" capture_output=True, text=True).stdout.strip() or \"NO GPU - set Runtime > Change runtime type > T4 GPU\")\n",
"import torch\n",
"print(\"torch\", torch.__version__, \"| cuda\", torch.cuda.is_available(),\n",
" \"| device\", torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"cpu\")\n",
"assert torch.cuda.is_available(), \"Enable the T4 GPU runtime before running.\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Dependencies"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip -q install -U \"laya>=0.1.6\" \"transformers>=4.45\" \"datasets>=3.0\" safetensors huggingface_hub 2>&1 | tail -3\n",
"import laya, transformers, datasets\n",
"print(\"laya\", getattr(laya, \"__version__\", \"?\"), \"| transformers\", transformers.__version__, \"| datasets\", datasets.__version__)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Download both checkpoints\n",
"\n",
"`reference_compile` is forced off after loading: the `laya` package does not disable it (only\n",
"`rl_agent_api.py` does), and torch.compile is a loss on small batches / few SMs like a T4."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch, json, os, time\n",
"from huggingface_hub import snapshot_download\n",
"from safetensors import safe_open\n",
"\n",
"REPOS = {\"laya\": \"convaiinnovations/laya\", \"laya-multilingual\": \"convaiinnovations/laya-multilingual\"}\n",
"PATHS, MODEL_META = {}, {}\n",
"\n",
"def patch_tokenizer_config(model_dir):\n",
" '''laya-multilingual ships extra_special_tokens as a LIST (inherited from the mmBERT/Gemma\n",
" tokenizer); transformers expects a dict and raises\n",
" AttributeError: list object has no attribute keys. Without this, AutoTokenizer -- and so\n",
" laya.load() -- fails outright on that repo. Also normalises tokenizer_class for 4.x/5.x.'''\n",
" p = os.path.join(model_dir, \"tokenizer\", \"tokenizer_config.json\")\n",
" if not os.path.exists(p):\n",
" return []\n",
" c = json.load(open(p))\n",
" changed = []\n",
" est = c.get(\"extra_special_tokens\")\n",
" if isinstance(est, list):\n",
" c[\"extra_special_tokens\"] = {(\"extra_%d\" % i): t for i, t in enumerate(est)}\n",
" changed.append(\"extra_special_tokens list->dict\")\n",
" if c.get(\"tokenizer_class\") in (None, \"TokenizersBackend\"):\n",
" c[\"tokenizer_class\"] = \"PreTrainedTokenizerFast\"\n",
" c.pop(\"backend\", None); c.pop(\"is_local\", None)\n",
" changed.append(\"tokenizer_class\")\n",
" if changed:\n",
" json.dump(c, open(p, \"w\"), indent=2)\n",
" return changed\n",
"\n",
"for name, repo in REPOS.items():\n",
" t = time.time()\n",
" PATHS[name] = snapshot_download(repo)\n",
" fixed = patch_tokenizer_config(PATHS[name])\n",
" if fixed:\n",
" print(\" patched tokenizer config for %s: %s\" % (name, \", \".join(fixed)))\n",
" cfg = json.load(open(os.path.join(PATHS[name], \"rl_agent_config.json\")))\n",
" ecfg = json.load(open(os.path.join(PATHS[name], \"encoder\", \"config.json\")))\n",
" tot = enc = 0\n",
" with safe_open(os.path.join(PATHS[name], \"model.safetensors\"), \"pt\") as f:\n",
" for k in f.keys():\n",
" n = 1\n",
" for d in f.get_slice(k).get_shape():\n",
" n *= d\n",
" tot += n\n",
" enc += n if k.startswith(\"encoder.\") else 0\n",
" MODEL_META[name] = {\n",
" \"repo\": repo, \"params_total_m\": round(tot/1e6, 2), \"params_encoder_m\": round(enc/1e6, 2),\n",
" \"params_head_m\": round((tot-enc)/1e6, 2), \"hidden_size\": ecfg.get(\"hidden_size\"),\n",
" \"num_layers\": ecfg.get(\"num_hidden_layers\"), \"vocab_size\": ecfg.get(\"vocab_size\"),\n",
" \"max_len\": cfg.get(\"max_len\"), \"head_max_len\": cfg.get(\"head_max_len\"),\n",
" \"temperature\": cfg.get(\"temperature\"), \"temperature_by_options\": cfg.get(\"temperature_by_options\", {}),\n",
" \"training\": cfg.get(\"training\", {}), \"download_s\": round(time.time()-t, 1),\n",
" \"tokenizer_config_patched\": fixed,\n",
" }\n",
" print(name, \"->\", PATHS[name])\n",
" print(\" \", json.dumps({k: v for k, v in MODEL_META[name].items() if k != \"temperature_by_options\"}))\n",
"\n",
"def load_agent(name):\n",
" import laya\n",
" ag = laya.load(PATHS[name], device=\"cuda\")\n",
" try:\n",
" ag.model.encoder.config.reference_compile = False # torch.compile hurts on T4-class GPUs\n",
" except Exception as e:\n",
" print(\"could not disable reference_compile:\", e)\n",
" ag.model.eval()\n",
" return ag"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Benchmark engine \u2014 batched scoring and metrics"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%writefile bench_engine.py\n",
"import json, math, sys, time\n",
"import numpy as np\n",
"import torch\n",
"from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket\n",
"\n",
"def to_internal(qdef):\n",
" t = qdef[\"type\"]\n",
" crit = qdef.get(\"criteria\")\n",
" if t == \"choice\" and isinstance(crit, list):\n",
" crit = {c: None for c in crit}\n",
" ins = qdef[\"instructions\"]\n",
" if not isinstance(ins, str):\n",
" ins = json.dumps(ins)\n",
" return {\"t\": t, \"ins\": ins, \"crit\": crit}\n",
"\n",
"@torch.no_grad()\n",
"def score_cases(agent, cases, max_tokens=16384, max_seqs=128, progress=\"\"):\n",
" '''cases: [(state, {qid: qdef})] -> (raw_logits per question, index, seconds, n_dropped).\n",
"\n",
" Every question of every case is one sequence. Sequences are length-sorted and packed into\n",
" token-budgeted batches so padding stays small. Returns UNCALIBRATED logits; temperature is\n",
" applied later so raw/calibrated variants come from one forward pass.\n",
" '''\n",
" max_len = agent.cfg.get(\"max_len\", 512)\n",
" head_max_len = agent.cfg.get(\"head_max_len\", 192)\n",
" items, index, dropped = [], [], 0\n",
" for ci, (state, questions) in enumerate(cases):\n",
" for qid, qdef in questions.items():\n",
" q = to_internal(qdef)\n",
" try:\n",
" ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)\n",
" except Exception:\n",
" index.append((ci, qid, QTYPES[q[\"t\"]], 0)); items.append(None); dropped += 1\n",
" continue\n",
" if len(markers) != len(render_options(q)):\n",
" index.append((ci, qid, QTYPES[q[\"t\"]], 0)); items.append(None); dropped += 1\n",
" continue\n",
" items.append({\"ids\": ids, \"markers\": markers, \"qtype\": QTYPES[q[\"t\"]]})\n",
" index.append((ci, qid, QTYPES[q[\"t\"]], len(markers)))\n",
"\n",
" valid = [i for i, it in enumerate(items) if it is not None]\n",
" order = sorted(valid, key=lambda i: len(items[i][\"ids\"]))\n",
" out = [None] * len(items)\n",
" t0, done, i = time.time(), 0, 0\n",
" while i < len(order):\n",
" j, L = i, 0\n",
" while j < len(order) and j - i < max_seqs and max(L, len(items[order[j]][\"ids\"])) * (j - i + 1) <= max_tokens:\n",
" L = max(L, len(items[order[j]][\"ids\"])); j += 1\n",
" j = max(j, i + 1)\n",
" sel = [items[order[t]] for t in range(i, j)]\n",
" b = collate_items([sel], agent.tok.pad_token_id)\n",
" with torch.autocast(\"cuda\", dtype=agent.dtype, enabled=True):\n",
" logits, _ = agent.model(b[\"input_ids\"].cuda(), b[\"attention_mask\"].cuda(),\n",
" b[\"marker_pos\"].cuda(), b[\"marker_mask\"].cuda(), b[\"qtype\"].cuda())\n",
" logits = logits.float().cpu().numpy()\n",
" for r in range(j - i):\n",
" out[order[i + r]] = logits[r, :len(sel[r][\"markers\"])]\n",
" done += j - i\n",
" if progress:\n",
" el = time.time() - t0\n",
" msg = \"\\r [%s] %d/%d seq | %.0f seq/s | ETA %ds \" % (\n",
" progress, done, len(order), done/max(el, 1e-9),\n",
" int(el*(len(order)-done)/max(1, done)))\n",
" sys.stdout.write(msg)\n",
" sys.stdout.flush()\n",
" i = j\n",
" if progress:\n",
" print(\"\\r [%s] %d sequences in %.1fs (%d dropped: options did not fit)%s\" % (\n",
" progress, len(order), time.time()-t0, dropped, \" \"*20))\n",
" return out, index, time.time() - t0, dropped\n",
"\n",
"def softmax_t(z, t=1.0):\n",
" z = np.asarray(z, dtype=np.float64) / max(1e-3, float(t))\n",
" e = np.exp(z - z.max())\n",
" return e / e.sum()\n",
"\n",
"def temp_for(agent, qt, k, calibrated=True):\n",
" if not calibrated:\n",
" return 1.0\n",
" return float(agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt]))\n",
"\n",
"# ------------------------------------------------------------------ metrics\n",
"def ece_score(conf, correct, bins=15):\n",
" conf, correct = np.asarray(conf, float), np.asarray(correct, float)\n",
" if len(conf) == 0:\n",
" return float(\"nan\")\n",
" edges = np.linspace(0, 1, bins + 1)\n",
" e = 0.0\n",
" for lo, hi in zip(edges[:-1], edges[1:]):\n",
" sel = (conf > lo) & (conf <= hi)\n",
" if sel.any():\n",
" e += sel.mean() * abs(conf[sel].mean() - correct[sel].mean())\n",
" return float(e)\n",
"\n",
"def macro_f1(gold, pred):\n",
" gold, pred = np.asarray(gold), np.asarray(pred)\n",
" f1 = []\n",
" for c in sorted(set(gold.tolist()) | set(pred.tolist())):\n",
" tp = int(((pred == c) & (gold == c)).sum())\n",
" fp = int(((pred == c) & (gold != c)).sum())\n",
" fn = int(((pred != c) & (gold == c)).sum())\n",
" f1.append(2*tp / max(1, 2*tp + fp + fn))\n",
" return float(np.mean(f1))\n",
"\n",
"def aurc(conf, correct):\n",
" conf, correct = np.asarray(conf, float), np.asarray(correct, float)\n",
" if len(conf) == 0:\n",
" return float(\"nan\")\n",
" o = np.argsort(-conf)\n",
" return float((np.cumsum(1 - correct[o]) / np.arange(1, len(o)+1)).mean())\n",
"\n",
"def hard_metrics(rows):\n",
" '''rows: [(gold_idx, probs)] -> standard single-label classification + calibration metrics.'''\n",
" rows = [r for r in rows if r[1] is not None]\n",
" if not rows:\n",
" return {\"n\": 0}\n",
" gold = np.array([g for g, _ in rows])\n",
" pred = np.array([int(np.argmax(p)) for _, p in rows])\n",
" conf = np.array([float(np.max(p)) for _, p in rows])\n",
" corr = (pred == gold).astype(float)\n",
" m = {\"n\": len(rows),\n",
" \"accuracy\": float(corr.mean()),\n",
" \"macro_f1\": macro_f1(gold, pred),\n",
" \"ece\": ece_score(conf, corr),\n",
" \"brier\": float(np.mean([((np.asarray(p) - np.eye(len(p))[g])**2).sum() for g, p in rows])),\n",
" \"nll\": float(np.mean([-math.log(max(float(p[g]), 1e-12)) for g, p in rows])),\n",
" \"aurc\": aurc(conf, corr),\n",
" \"mean_confidence\": float(conf.mean())}\n",
" for cov in (0.5, 0.8):\n",
" k = max(1, int(len(conf)*cov))\n",
" m[\"acc_at_%d_coverage\" % int(cov*100)] = float(corr[np.argsort(-conf)[:k]].mean())\n",
" return m\n",
"\n",
"def fit_temperature(pairs, lo=0.2, hi=10.0, steps=160):\n",
" '''pairs: [(logits, gold_idx)] -> single temperature minimising NLL (grid search, no autograd).'''\n",
" pairs = [(np.asarray(z, float), int(g)) for z, g in pairs if z is not None]\n",
" if len(pairs) < 25:\n",
" return 1.0\n",
" best_t, best = 1.0, float(\"inf\")\n",
" for t in np.geomspace(lo, hi, steps):\n",
" tot = 0.0\n",
" for z, g in pairs:\n",
" zz = z/t\n",
" zz = zz - zz.max()\n",
" tot += -(zz[g] - math.log(np.exp(zz).sum()))\n",
" if tot < best:\n",
" best, best_t = tot, float(t)\n",
" return round(float(best_t), 4)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Task suites\n",
"\n",
"Every suite is built **once**, with a fixed seed, and reused for both models \u2014 so the two\n",
"checkpoints answer byte-identical questions. Option sets for many-label tasks use\n",
"`gold + N-1 distractors`, matching how the models were trained (the training recipe samples\n",
"5\u201320 options per choice question rather than showing the full label space)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import random, json\n",
"import numpy as np\n",
"from datasets import load_dataset\n",
"\n",
"SEED = 13\n",
"N_OPTS = 20 # options per many-label choice question (gold + 19 distractors)\n",
"PER_LANG = 300 # cases per language\n",
"SUITES = {} # name -> {\"cases\": [(state, questions)], \"gold\": [{qid: gold_idx}], \"meta\": {...}}\n",
"\n",
"def build_choice_questions(rng, gold, all_labels, n_opts, instructions, qid=\"label\"):\n",
" pool = [x for x in all_labels if x != gold]\n",
" keys = [gold] + rng.sample(pool, min(n_opts-1, len(pool)))\n",
" rng.shuffle(keys)\n",
" crit = {k: k.replace(\"_\", \" \").replace(\".\", \": \") for k in keys}\n",
" return {qid: {\"type\": \"choice\", \"instructions\": instructions, \"criteria\": crit}}, keys.index(gold)\n",
"\n",
"def register(name, cases, golds, **meta):\n",
" SUITES[name] = {\"cases\": cases, \"gold\": golds, \"meta\": meta}\n",
" nq = sum(len(c[1]) for c in cases)\n",
" print(\" %-34s %4d cases / %5d questions %s\" % (name, len(cases), nq, meta.get(\"note\", \"\")))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5a. typed-decisions (English, 400 cases / 2,000 decisions)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"td_cases, td_golds, td_wf = [], [], []\n",
"try:\n",
" td = load_dataset(\"LocalLLaMA/typed-decisions\", \"all\", split=\"test\")\n",
" for r in td:\n",
" questions = json.loads(r[\"questions\"]) if isinstance(r[\"questions\"], str) else r[\"questions\"]\n",
" gold = json.loads(r[\"gold\"]) if isinstance(r[\"gold\"], str) else r[\"gold\"]\n",
" state = r[\"state\"]\n",
" try:\n",
" state = json.loads(state)\n",
" except Exception:\n",
" pass\n",
" gmap = {}\n",
" for qid, qdef in questions.items():\n",
" g = gold[qid]\n",
" if qdef[\"type\"] == \"choice\":\n",
" keys = list(qdef[\"criteria\"].keys())\n",
" gmap[qid] = {\"idx\": keys.index(str(g[\"label\"])), \"keys\": keys,\n",
" \"soft\": [float(g.get(\"probabilities\", {}).get(k, 0.0)) for k in keys]}\n",
" elif qdef[\"type\"] == \"noul\":\n",
" idx = 1 if str(g[\"label\"]).lower() == \"true\" else 0\n",
" pt = float(g.get(\"probabilities\", {}).get(\"true\", g.get(\"noul\", 0.5)))\n",
" gmap[qid] = {\"idx\": idx, \"keys\": [\"false\", \"true\"], \"soft\": [1-pt, pt]}\n",
" else: # score\n",
" n = len(qdef[\"criteria\"])\n",
" gmap[qid] = {\"idx\": int(g[\"label\"]), \"keys\": [str(i) for i in range(n)],\n",
" \"soft\": [float(g.get(\"probabilities\", {}).get(str(i), 0.0)) for i in range(n)],\n",
" \"gold_score\": float(g.get(\"score\", float(g[\"label\"])))}\n",
" td_cases.append((state, questions))\n",
" td_golds.append(gmap)\n",
" td_wf.append(r[\"workflow\"])\n",
" register(\"typed_decisions\", td_cases, td_golds, note=\"4 workflows, soft teacher labels\",\n",
" workflows=td_wf, soft=True)\n",
"except Exception as e:\n",
" print(\"typed-decisions FAILED:\", type(e).__name__, e)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5b. Multilingual \u2014 MASSIVE intent & scenario (14 languages), XNLI (15 languages)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"MASSIVE_LANGS = [\"en\",\"de\",\"fr\",\"es\",\"pt\",\"ru\",\"tr\",\"ar\",\"hi\",\"ta\",\"zh-CN\",\"ja\",\"ko\",\"sw\"]\n",
"XNLI_LANGS = [\"en\",\"de\",\"fr\",\"es\",\"ru\",\"tr\",\"ar\",\"hi\",\"ur\",\"vi\",\"th\",\"el\",\"bg\",\"zh\",\"sw\"]\n",
"\n",
"for task, short, instr in [(\"mteb/amazon_massive_intent\", \"massive_intent\",\n",
" \"What is the user asking for in `utterance`?\"),\n",
" (\"mteb/amazon_massive_scenario\", \"massive_scenario\",\n",
" \"Which domain does `utterance` belong to?\")]:\n",
" for lg in MASSIVE_LANGS:\n",
" try:\n",
" d = load_dataset(task, lg, split=\"test\")\n",
" labels = sorted(set(d[\"label_text\"]))\n",
" rng = random.Random(SEED)\n",
" cases, golds = [], []\n",
" for r in list(d)[:PER_LANG]:\n",
" qs, gi = build_choice_questions(rng, r[\"label_text\"], labels, N_OPTS, instr)\n",
" cases.append(({\"utterance\": r[\"text\"]}, qs))\n",
" golds.append({\"label\": {\"idx\": gi}})\n",
" register(\"%s.%s\" % (short, lg), cases, golds, lang=lg, n_options=N_OPTS,\n",
" label_space=len(labels), family=short)\n",
" except Exception as e:\n",
" print(\" FAIL %s %s: %s\" % (short, lg, str(e)[:90]))\n",
"\n",
"NLI_CRIT = {\"entailment\": \"the premise implies the hypothesis is true\",\n",
" \"neutral\": \"the premise neither implies nor contradicts the hypothesis\",\n",
" \"contradiction\": \"the premise implies the hypothesis is false\"}\n",
"for lg in XNLI_LANGS:\n",
" try:\n",
" d = load_dataset(\"facebook/xnli\", lg, split=\"test\")\n",
" cases, golds = [], []\n",
" for r in list(d)[:PER_LANG]:\n",
" qs = {\"relation\": {\"type\": \"choice\",\n",
" \"instructions\": \"What is the relationship between `premise` and `hypothesis`?\",\n",
" \"criteria\": dict(NLI_CRIT)}}\n",
" cases.append(({\"premise\": r[\"premise\"], \"hypothesis\": r[\"hypothesis\"]}, qs))\n",
" golds.append({\"relation\": {\"idx\": int(r[\"label\"])}})\n",
" register(\"xnli.%s\" % lg, cases, golds, lang=lg, n_options=3, family=\"xnli\")\n",
" except Exception as e:\n",
" print(\" FAIL xnli %s: %s\" % (lg, str(e)[:90]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5c. English tasks\n",
"\n",
"`sst5`, `emotion` and `prompt_injections` were held out of Laya's training entirely (true\n",
"zero-shot). `ag_news` and `boolq` were in the training mix, so they measure retention rather\n",
"than generalisation. `banking77` is the many-option stress test \u2014 77 options at once, versus\n",
"Jev's documented hard cap at 255."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rng = random.Random(SEED)\n",
"\n",
"# SST-5: ordinal score\n",
"try:\n",
" d = load_dataset(\"SetFit/sst5\", split=\"test\")\n",
" crit = [\"very negative\", \"negative\", \"neutral\", \"positive\", \"very positive\"]\n",
" cases, golds = [], []\n",
" for r in list(d)[:600]:\n",
" cases.append(({\"text\": r[\"text\"]},\n",
" {\"sentiment\": {\"type\": \"score\", \"instructions\": \"How positive is the sentiment of `text`?\",\n",
" \"criteria\": crit}}))\n",
" golds.append({\"sentiment\": {\"idx\": int(r[\"label\"]), \"gold_score\": float(r[\"label\"])}})\n",
" register(\"en.sst5\", cases, golds, note=\"zero-shot, ordinal\", family=\"english\", held_out=True)\n",
"except Exception as e: print(\" FAIL sst5:\", str(e)[:90])\n",
"\n",
"# emotion: 6-way choice\n",
"try:\n",
" d = load_dataset(\"dair-ai/emotion\", \"split\", split=\"test\")\n",
" names = [\"sadness\",\"joy\",\"love\",\"anger\",\"fear\",\"surprise\"]\n",
" cases, golds = [], []\n",
" for r in list(d)[:600]:\n",
" cases.append(({\"text\": r[\"text\"]},\n",
" {\"emotion\": {\"type\": \"choice\",\n",
" \"instructions\": \"Which emotion is most strongly expressed in `text`?\",\n",
" \"criteria\": {n: None for n in names}}}))\n",
" golds.append({\"emotion\": {\"idx\": int(r[\"label\"])}})\n",
" register(\"en.emotion\", cases, golds, note=\"zero-shot\", family=\"english\", held_out=True)\n",
"except Exception as e: print(\" FAIL emotion:\", str(e)[:90])\n",
"\n",
"# prompt injections: noul\n",
"try:\n",
" d = load_dataset(\"deepset/prompt-injections\", split=\"test\")\n",
" cases, golds = [], []\n",
" for r in list(d):\n",
" cases.append(({\"text\": r[\"text\"]},\n",
" {\"injection\": {\"type\": \"noul\",\n",
" \"instructions\": \"Does `text` try to inject or override instructions given to an AI system?\"}}))\n",
" golds.append({\"injection\": {\"idx\": int(r[\"label\"])}})\n",
" register(\"en.prompt_injections\", cases, golds, note=\"zero-shot\", family=\"english\", held_out=True)\n",
"except Exception as e: print(\" FAIL prompt_injections:\", str(e)[:90])\n",
"\n",
"# banking77: 77-option stress test\n",
"try:\n",
" d = load_dataset(\"PolyAI/banking77\", split=\"test\")\n",
" names = d.features[\"label\"].names\n",
" cases, golds = [], []\n",
" for r in list(d)[:500]:\n",
" cases.append(({\"message\": r[\"text\"]},\n",
" {\"intent\": {\"type\": \"choice\", \"instructions\": \"Which banking intent does `message` express?\",\n",
" \"criteria\": {n.replace(\"_\", \" \"): None for n in names}}}))\n",
" golds.append({\"intent\": {\"idx\": int(r[\"label\"])}})\n",
" register(\"en.banking77_full\", cases, golds, note=\"77 options at once\", family=\"english\",\n",
" n_options=len(names), held_out=True)\n",
"except Exception as e: print(\" FAIL banking77:\", str(e)[:90])\n",
"\n",
"# ag_news + boolq: in-task retention\n",
"try:\n",
" d = load_dataset(\"fancyzhx/ag_news\", split=\"test\")\n",
" crit = {\"world\": \"world news and international politics\", \"sports\": \"sports\",\n",
" \"business\": \"business and economy\", \"sci_tech\": \"science and technology\"}\n",
" cases, golds = [], []\n",
" for r in list(d)[:600]:\n",
" cases.append(({\"article\": r[\"text\"]},\n",
" {\"topic\": {\"type\": \"choice\", \"instructions\": \"What is the topic of `article`?\",\n",
" \"criteria\": dict(crit)}}))\n",
" golds.append({\"topic\": {\"idx\": int(r[\"label\"])}})\n",
" register(\"en.ag_news\", cases, golds, note=\"in-task (trained on)\", family=\"english\", held_out=False)\n",
"except Exception as e: print(\" FAIL ag_news:\", str(e)[:90])\n",
"\n",
"try:\n",
" d = load_dataset(\"google/boolq\", split=\"validation\")\n",
" cases, golds = [], []\n",
" for r in list(d)[:600]:\n",
" cases.append(({\"passage\": r[\"passage\"], \"question\": r[\"question\"]},\n",
" {\"answer\": {\"type\": \"noul\",\n",
" \"instructions\": \"Based on `passage`, is the answer to `question` yes?\"}}))\n",
" golds.append({\"answer\": {\"idx\": int(bool(r[\"answer\"]))}})\n",
" register(\"en.boolq\", cases, golds, note=\"in-task (trained on)\", family=\"english\", held_out=False)\n",
"except Exception as e: print(\" FAIL boolq:\", str(e)[:90])\n",
"\n",
"print()\n",
"print(\"suites built:\", len(SUITES), \"| total questions:\",\n",
" sum(sum(len(c[1]) for c in s[\"cases\"]) for s in SUITES.values()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Run every suite on both models"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import bench_engine as BE\n",
"import numpy as np, gc, time, json\n",
"\n",
"RESULTS = {\"meta\": {\"seed\": SEED, \"n_opts\": N_OPTS, \"per_lang\": PER_LANG,\n",
" \"timestamp\": time.strftime(\"%Y-%m-%d %H:%M:%S\"),\n",
" \"gpu\": torch.cuda.get_device_name(0),\n",
" \"torch\": torch.__version__, \"transformers\": transformers.__version__,\n",
" \"models\": MODEL_META},\n",
" \"suites\": {}}\n",
"RAW = {} # model -> suite -> (logits, index) kept in RAM for the calibration-repair section\n",
"\n",
"def evaluate_suite(agent, model_name, suite_name):\n",
" S = SUITES[suite_name]\n",
" logits, index, secs, dropped = BE.score_cases(agent, S[\"cases\"], progress=\"%s/%s\" % (model_name, suite_name))\n",
" RAW[model_name][suite_name] = (logits, index)\n",
" soft = S[\"meta\"].get(\"soft\", False)\n",
" rows_cal, rows_raw, extra = [], [], {\"soft_acc\": [], \"brier_soft\": [], \"tv\": [], \"kl\": [],\n",
" \"score_mae\": [], \"within_1\": []}\n",
" per_q_type = {}\n",
" for (ci, qid, qt, k), z in zip(index, logits):\n",
" g = S[\"gold\"][ci].get(qid)\n",
" if g is None:\n",
" continue\n",
" if z is None:\n",
" rows_cal.append((g[\"idx\"], None)); rows_raw.append((g[\"idx\"], None)); continue\n",
" p_cal = BE.softmax_t(z, BE.temp_for(agent, qt, k, True))\n",
" p_raw = BE.softmax_t(z, 1.0)\n",
" rows_cal.append((g[\"idx\"], p_cal)); rows_raw.append((g[\"idx\"], p_raw))\n",
" per_q_type.setdefault(qt, []).append((g[\"idx\"], p_cal))\n",
" if soft and \"soft\" in g:\n",
" gp = np.asarray(g[\"soft\"], float)\n",
" if gp.sum() > 0:\n",
" gp = gp / gp.sum()\n",
" pp = p_cal[:len(gp)] if len(p_cal) >= len(gp) else np.pad(p_cal, (0, len(gp)-len(p_cal)))\n",
" pp = pp / max(pp.sum(), 1e-12)\n",
" extra[\"soft_acc\"].append(float((pp*gp).sum()))\n",
" extra[\"brier_soft\"].append(float(((pp-gp)**2).sum()))\n",
" extra[\"tv\"].append(float(0.5*np.abs(pp-gp).sum()))\n",
" extra[\"kl\"].append(float((gp*np.log(np.clip(gp/np.clip(pp,1e-12,None),1e-12,1e4))).sum()))\n",
" if \"gold_score\" in g:\n",
" exp = float((np.arange(len(p_cal))*p_cal).sum())\n",
" extra[\"score_mae\"].append(abs(exp - g[\"gold_score\"]))\n",
" extra[\"within_1\"].append(float(abs(exp - g[\"gold_score\"]) <= 1.0))\n",
"\n",
" out = {\"calibrated\": BE.hard_metrics(rows_cal), \"raw\": BE.hard_metrics(rows_raw),\n",
" \"seconds\": round(secs, 2), \"dropped_questions\": dropped,\n",
" \"questions_per_second\": round(len([r for r in rows_cal if r[1] is not None])/max(secs,1e-9), 1)}\n",
" for k2, v in extra.items():\n",
" if v:\n",
" out[k2] = float(np.mean(v))\n",
" if len(per_q_type) > 1:\n",
" names = {0: \"choice\", 1: \"score\", 2: \"noul\"}\n",
" out[\"by_question_type\"] = {names[qt]: BE.hard_metrics(rs) for qt, rs in sorted(per_q_type.items())}\n",
" return out\n",
"\n",
"for model_name in REPOS:\n",
" print(\"\\n\" + \"=\"*78 + \"\\n %s\\n\" % model_name + \"=\"*78)\n",
" agent = load_agent(model_name)\n",
" RAW[model_name] = {}\n",
" for suite_name in SUITES:\n",
" try:\n",
" r = evaluate_suite(agent, model_name, suite_name)\n",
" RESULTS[\"suites\"].setdefault(suite_name, {})[model_name] = r\n",
" c = r[\"calibrated\"]\n",
" print(\" %-32s acc %.3f f1 %.3f ECE %.3f n=%d\" %\n",
" (suite_name, c.get(\"accuracy\", float(\"nan\")), c.get(\"macro_f1\", float(\"nan\")),\n",
" c.get(\"ece\", float(\"nan\")), c.get(\"n\", 0)))\n",
" except Exception as e:\n",
" import traceback; traceback.print_exc()\n",
" RESULTS[\"suites\"].setdefault(suite_name, {})[model_name] = {\"error\": \"%s: %s\" % (type(e).__name__, e)}\n",
" globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")] = agent\n",
" gc.collect(); torch.cuda.empty_cache()\n",
"\n",
"json.dump(RESULTS, open(\"laya_benchmark_results.json\", \"w\"), indent=2)\n",
"print(\"\\ncheckpoint saved\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 6b. Control \u2014 matched context budget\n",
"\n",
"The two checkpoints ship different context windows (512 vs 1024 tokens), so on long states\n",
"`laya-multilingual` sees more of the input. This re-runs typed-decisions with **both** models\n",
"capped at 512 tokens, isolating the architecture from the context-window advantage."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import bench_engine as BE\n",
"import numpy as np\n",
"\n",
"RESULTS[\"typed_decisions_matched_512\"] = {}\n",
"if \"typed_decisions\" in SUITES:\n",
" S = SUITES[\"typed_decisions\"]\n",
" for model_name in REPOS:\n",
" agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
" orig = agent.cfg.get(\"max_len\")\n",
" agent.cfg[\"max_len\"] = 512\n",
" try:\n",
" logits, index, secs, dropped = BE.score_cases(agent, S[\"cases\"], progress=\"%s/td@512\" % model_name)\n",
" rows = []\n",
" for (ci, qid, qt, k), z in zip(index, logits):\n",
" g = S[\"gold\"][ci].get(qid)\n",
" if g is None or z is None:\n",
" continue\n",
" rows.append((g[\"idx\"], BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))\n",
" RESULTS[\"typed_decisions_matched_512\"][model_name] = BE.hard_metrics(rows)\n",
" print(\" %-22s max_len 512 -> acc %.3f ECE %.3f n=%d\"\n",
" % (model_name, RESULTS[\"typed_decisions_matched_512\"][model_name][\"accuracy\"],\n",
" RESULTS[\"typed_decisions_matched_512\"][model_name][\"ece\"],\n",
" RESULTS[\"typed_decisions_matched_512\"][model_name][\"n\"]))\n",
" finally:\n",
" agent.cfg[\"max_len\"] = orig"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Latency (p50 / p95 at 1, 5, 10, 50 questions per call)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np, time, torch\n",
"\n",
"LAT_STATE = {\"ticket\": {\"subject\": \"Payout failing\",\n",
" \"messages\": [{\"from\": \"customer\",\n",
" \"text\": \"Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. \" * 6}]}}\n",
"Q_NOUL = {\"type\": \"noul\", \"instructions\": \"Does `ticket.messages[0].text` express urgency?\"}\n",
"Q_CHOICE = {\"type\": \"choice\", \"instructions\": \"Which team should handle this?\",\n",
" \"criteria\": {\"billing\": \"payments\", \"technical\": \"bugs and integrations\", \"sales\": \"pricing\"}}\n",
"\n",
"RESULTS[\"latency\"] = {}\n",
"for model_name in REPOS:\n",
" agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
" res = {}\n",
" for nq in (1, 5, 10, 50):\n",
" qs = {(\"q%d\" % i): (Q_NOUL if i % 2 else Q_CHOICE) for i in range(nq)}\n",
" for _ in range(3):\n",
" agent.system_one(LAT_STATE, qs)\n",
" ts = []\n",
" for _ in range(20):\n",
" torch.cuda.synchronize(); t = time.perf_counter()\n",
" agent.system_one(LAT_STATE, qs)\n",
" torch.cuda.synchronize(); ts.append((time.perf_counter()-t)*1000)\n",
" res[\"%d_questions\" % nq] = {\"p50_ms\": round(float(np.percentile(ts, 50)), 1),\n",
" \"p95_ms\": round(float(np.percentile(ts, 95)), 1),\n",
" \"ms_per_question\": round(float(np.percentile(ts, 50))/nq, 2)}\n",
" RESULTS[\"latency\"][model_name] = res\n",
" print(model_name, json.dumps(res))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8. Option-order robustness\n",
"\n",
"Same question, options permuted. A well-behaved decision model should pick the same label.\n",
"The independent Jev benchmark measured a **13% flip rate for Jev**, with the worst LLM at 37%."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import random, numpy as np\n",
"import bench_engine as BE\n",
"\n",
"PB_SUITES = [s for s in (\"massive_intent.en\", \"en.emotion\", \"xnli.en\") if s in SUITES]\n",
"RESULTS[\"option_order_robustness\"] = {}\n",
"\n",
"for model_name in REPOS:\n",
" agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
" out = {}\n",
" for sname in PB_SUITES:\n",
" S = SUITES[sname]\n",
" base = S[\"cases\"][:200]\n",
" rng = random.Random(99)\n",
" perm_cases, perm_maps = [], []\n",
" for state, questions in base:\n",
" qid = list(questions)[0]\n",
" qdef = questions[qid]\n",
" keys = list(qdef[\"criteria\"].keys())\n",
" order = list(range(len(keys))); rng.shuffle(order)\n",
" newcrit = {keys[i]: qdef[\"criteria\"][keys[i]] for i in order}\n",
" perm_cases.append((state, {qid: {**qdef, \"criteria\": newcrit}}))\n",
" perm_maps.append((keys, [keys[i] for i in order]))\n",
" la, ia, _, _ = BE.score_cases(agent, base, progress=\"\")\n",
" lb, ib, _, _ = BE.score_cases(agent, perm_cases, progress=\"\")\n",
" flips = comp = 0\n",
" for (za, (_, _, _, ka)), (zb, (_, _, _, kb)), (keys, pkeys) in zip(zip(la, ia), zip(lb, ib), perm_maps):\n",
" if za is None or zb is None:\n",
" continue\n",
" comp += 1\n",
" if keys[int(np.argmax(za))] != pkeys[int(np.argmax(zb))]:\n",
" flips += 1\n",
" out[sname] = {\"n\": comp, \"flip_rate\": round(flips/max(1, comp), 4)}\n",
" print(\" %-22s %-24s flip rate %.3f (n=%d)\" % (model_name, sname, out[sname][\"flip_rate\"], comp))\n",
" RESULTS[\"option_order_robustness\"][model_name] = out"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 9. Calibration repair\n",
"\n",
"`laya-multilingual` ships with `temperature = [1.0, 1.0, 1.0]` and no per-option-count buckets \u2014\n",
"it was never calibrated. This refits one temperature per (question type, option-count bucket) on\n",
"**half** of each suite and reports ECE on the other half, so the improvement is measured\n",
"out-of-sample. It shows how much of any calibration gap is a missing post-processing step rather\n",
"than a property of the model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import bench_engine as BE\n",
"from laya.common import temp_bucket\n",
"\n",
"RESULTS[\"calibration_repair\"] = {}\n",
"for model_name in REPOS:\n",
" agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
" per_suite, fitted_all = {}, {}\n",
" for sname, S in SUITES.items():\n",
" if sname not in RAW[model_name]:\n",
" continue\n",
" logits, index = RAW[model_name][sname]\n",
" pairs = []\n",
" for (ci, qid, qt, k), z in zip(index, logits):\n",
" g = S[\"gold\"][ci].get(qid)\n",
" if g is not None and z is not None:\n",
" pairs.append((z, g[\"idx\"], qt, k))\n",
" if len(pairs) < 60:\n",
" continue\n",
" half = len(pairs)//2\n",
" fit_set, held = pairs[:half], pairs[half:]\n",
" buckets = {}\n",
" for z, gi, qt, k in fit_set:\n",
" buckets.setdefault(temp_bucket(qt, k), []).append((z, gi))\n",
" fitted = {b: BE.fit_temperature(v) for b, v in buckets.items() if len(v) >= 25}\n",
" rows_ship, rows_fit = [], []\n",
" for z, gi, qt, k in held:\n",
" rows_ship.append((gi, BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))\n",
" rows_fit.append((gi, BE.softmax_t(z, fitted.get(temp_bucket(qt, k),\n",
" BE.temp_for(agent, qt, k, True)))))\n",
" ms, mf = BE.hard_metrics(rows_ship), BE.hard_metrics(rows_fit)\n",
" per_suite[sname] = {\"n_heldout\": ms[\"n\"], \"fitted_temperatures\": fitted,\n",
" \"ece_shipped\": ms[\"ece\"], \"ece_refit\": mf[\"ece\"],\n",
" \"nll_shipped\": ms[\"nll\"], \"nll_refit\": mf[\"nll\"],\n",
" \"accuracy\": ms[\"accuracy\"]}\n",
" fitted_all.update(fitted)\n",
" RESULTS[\"calibration_repair\"][model_name] = {\"per_suite\": per_suite}\n",
" if per_suite:\n",
" a = float(np.mean([v[\"ece_shipped\"] for v in per_suite.values()]))\n",
" b = float(np.mean([v[\"ece_refit\"] for v in per_suite.values()]))\n",
" RESULTS[\"calibration_repair\"][model_name][\"mean_ece_shipped\"] = round(a, 4)\n",
" RESULTS[\"calibration_repair\"][model_name][\"mean_ece_refit\"] = round(b, 4)\n",
" print(\"%-22s mean ECE shipped %.4f -> refit %.4f\" % (model_name, a, b))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 10. Aggregate, save and download"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np, json\n",
"\n",
"def agg(prefix_or_family, key=\"family\"):\n",
" out = {}\n",
" for m in REPOS:\n",
" accs, eces, ns = [], [], 0\n",
" for sname, S in SUITES.items():\n",
" if S[\"meta\"].get(key) != prefix_or_family:\n",
" continue\n",
" r = RESULTS[\"suites\"].get(sname, {}).get(m, {})\n",
" c = r.get(\"calibrated\", {})\n",
" if c.get(\"n\"):\n",
" accs.append(c[\"accuracy\"]); eces.append(c[\"ece\"]); ns += c[\"n\"]\n",
" if accs:\n",
" out[m] = {\"macro_accuracy\": round(float(np.mean(accs)), 4),\n",
" \"macro_ece\": round(float(np.mean(eces)), 4),\n",
" \"n_questions\": ns, \"n_suites\": len(accs)}\n",
" return out\n",
"\n",
"RESULTS[\"caveats\"] = {\n",
" \"context_window\": \"laya max_len=512, laya-multilingual max_len=1024 (shipped configs). \"\n",
" \"See typed_decisions_matched_512 for the equal-budget control.\",\n",
" \"shipped_calibration\": \"laya ships fitted temperatures; laya-multilingual ships all-1.0 \"\n",
" \"(uncalibrated). See calibration_repair for the out-of-sample refit.\",\n",
" \"training_overlap\": \"ag_news and boolq were in Laya's training mix (retention, not generalisation). \"\n",
" \"sst5, emotion, prompt_injections, banking77 were held out. \"\n",
" \"Neither checkpoint trained on typed-decisions, MASSIVE or XNLI.\",\n",
"}\n",
"\n",
"RESULTS[\"summary\"] = {\n",
" \"typed_decisions\": {m: RESULTS[\"suites\"].get(\"typed_decisions\", {}).get(m, {}).get(\"calibrated\", {})\n",
" for m in REPOS},\n",
" \"massive_intent\": agg(\"massive_intent\"),\n",
" \"massive_scenario\": agg(\"massive_scenario\"),\n",
" \"xnli\": agg(\"xnli\"),\n",
" \"english\": agg(\"english\"),\n",
"}\n",
"\n",
"# English vs non-English split inside each multilingual family\n",
"for fam in (\"massive_intent\", \"massive_scenario\", \"xnli\"):\n",
" en, non = {}, {}\n",
" for m in REPOS:\n",
" ea, na = [], []\n",
" for sname, S in SUITES.items():\n",
" if S[\"meta\"].get(\"family\") != fam:\n",
" continue\n",
" c = RESULTS[\"suites\"].get(sname, {}).get(m, {}).get(\"calibrated\", {})\n",
" if not c.get(\"n\"):\n",
" continue\n",
" (ea if S[\"meta\"].get(\"lang\") in (\"en\",) else na).append(c[\"accuracy\"])\n",
" if ea: en[m] = round(float(np.mean(ea)), 4)\n",
" if na: non[m] = round(float(np.mean(na)), 4)\n",
" RESULTS[\"summary\"][fam + \"_english_vs_rest\"] = {\"english\": en, \"non_english\": non}\n",
"\n",
"json.dump(RESULTS, open(\"laya_benchmark_results.json\", \"w\"), indent=2)\n",
"print(json.dumps(RESULTS[\"summary\"], indent=2))\n",
"print(\"\\nwrote laya_benchmark_results.json (%.1f KB)\" % (os.path.getsize(\"laya_benchmark_results.json\")/1024))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Per-language table, printed for a quick eyeball before you download the JSON\n",
"import numpy as np\n",
"for fam in (\"massive_intent\", \"massive_scenario\", \"xnli\"):\n",
" langs = sorted({S[\"meta\"][\"lang\"] for S in SUITES.values() if S[\"meta\"].get(\"family\") == fam})\n",
" if not langs:\n",
" continue\n",
" print(\"\\n=== %s : accuracy by language ===\" % fam)\n",
" print(\"%-8s %12s %22s %10s\" % (\"lang\", \"laya\", \"laya-multilingual\", \"delta\"))\n",
" for lg in langs:\n",
" sname = \"%s.%s\" % (fam, lg)\n",
" a = RESULTS[\"suites\"].get(sname, {}).get(\"laya\", {}).get(\"calibrated\", {}).get(\"accuracy\")\n",
" b = RESULTS[\"suites\"].get(sname, {}).get(\"laya-multilingual\", {}).get(\"calibrated\", {}).get(\"accuracy\")\n",
" if a is None or b is None:\n",
" continue\n",
" print(\"%-8s %12.3f %22.3f %+10.3f\" % (lg, a, b, b-a))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" from google.colab import files\n",
" files.download(\"laya_benchmark_results.json\")\n",
"except Exception as e:\n",
" print(\"Not on Colab or download blocked (%s). Grab laya_benchmark_results.json from the file browser.\" % type(e).__name__)"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"provenance": [],
"gpuType": "T4",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}