Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

47.0 KB
Raw Blame
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Laya vs Laya-Multilingual \u2014 extensive head-to-head benchmark\n",
    "\n",
    "Runs both open checkpoints over the same questions and writes a single\n",
    "`laya_benchmark_results.json`.\n",
    "\n",
    "| | `convaiinnovations/laya` | `convaiinnovations/laya-multilingual` |\n",
    "|---|---|---|\n",
    "| encoder | ModernBERT-large (English) | mmBERT-base (100+ languages) |\n",
    "| params | 421.3M (394.8M enc + 26.5M head) | 321.9M (306.9M enc + 15.0M head) |\n",
    "| context | 512 tokens | 1024 tokens |\n",
    "| shipped temperatures | fitted per type & option-count | **none (all 1.0)** |\n",
    "\n",
    "**What is measured**\n",
    "\n",
    "1. **typed-decisions** \u2014 400 cases / 2,000 decisions, 4 workflows (the benchmark behind the published 0.766 claim). Zero-shot for both of these checkpoints.\n",
    "2. **Multilingual** \u2014 MASSIVE intent + scenario across 14 languages, XNLI across 15 languages.\n",
    "3. **English zero-shot** \u2014 SST-5, emotion, prompt-injections, plus banking77 as a 77-option stress test.\n",
    "4. **Latency** \u2014 p50/p95 at 1 / 5 / 10 / 50 questions per call.\n",
    "5. **Option-order robustness** \u2014 how often the answer flips when the options are permuted (the independent Jev benchmark measured 13% for Jev; worst LLM 37%).\n",
    "6. **Calibration repair** \u2014 ECE before and after refitting temperature, which matters because `laya-multilingual` shipped uncalibrated.\n",
    "\n",
    "**Setup:** Runtime \u2192 Change runtime type \u2192 **T4 GPU**. Then Run All (~25\u201340 min).\n",
    "Both models see byte-identical questions (fixed seed), so every difference is the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. GPU and environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess, sys, os, json, time\n",
    "print(subprocess.run([\"nvidia-smi\",\"--query-gpu=name,memory.total,compute_cap\",\"--format=csv,noheader\"],\n",
    "                     capture_output=True, text=True).stdout.strip() or \"NO GPU - set Runtime > Change runtime type > T4 GPU\")\n",
    "import torch\n",
    "print(\"torch\", torch.__version__, \"| cuda\", torch.cuda.is_available(),\n",
    "      \"| device\", torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"cpu\")\n",
    "assert torch.cuda.is_available(), \"Enable the T4 GPU runtime before running.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip -q install -U \"laya>=0.1.6\" \"transformers>=4.45\" \"datasets>=3.0\" safetensors huggingface_hub 2>&1 | tail -3\n",
    "import laya, transformers, datasets\n",
    "print(\"laya\", getattr(laya, \"__version__\", \"?\"), \"| transformers\", transformers.__version__, \"| datasets\", datasets.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Download both checkpoints\n",
    "\n",
    "`reference_compile` is forced off after loading: the `laya` package does not disable it (only\n",
    "`rl_agent_api.py` does), and torch.compile is a loss on small batches / few SMs like a T4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch, json, os, time\n",
    "from huggingface_hub import snapshot_download\n",
    "from safetensors import safe_open\n",
    "\n",
    "REPOS = {\"laya\": \"convaiinnovations/laya\", \"laya-multilingual\": \"convaiinnovations/laya-multilingual\"}\n",
    "PATHS, MODEL_META = {}, {}\n",
    "\n",
    "def patch_tokenizer_config(model_dir):\n",
    "    '''laya-multilingual ships extra_special_tokens as a LIST (inherited from the mmBERT/Gemma\n",
    "    tokenizer); transformers expects a dict and raises\n",
    "    AttributeError: list object has no attribute keys. Without this, AutoTokenizer -- and so\n",
    "    laya.load() -- fails outright on that repo. Also normalises tokenizer_class for 4.x/5.x.'''\n",
    "    p = os.path.join(model_dir, \"tokenizer\", \"tokenizer_config.json\")\n",
    "    if not os.path.exists(p):\n",
    "        return []\n",
    "    c = json.load(open(p))\n",
    "    changed = []\n",
    "    est = c.get(\"extra_special_tokens\")\n",
    "    if isinstance(est, list):\n",
    "        c[\"extra_special_tokens\"] = {(\"extra_%d\" % i): t for i, t in enumerate(est)}\n",
    "        changed.append(\"extra_special_tokens list->dict\")\n",
    "    if c.get(\"tokenizer_class\") in (None, \"TokenizersBackend\"):\n",
    "        c[\"tokenizer_class\"] = \"PreTrainedTokenizerFast\"\n",
    "        c.pop(\"backend\", None); c.pop(\"is_local\", None)\n",
    "        changed.append(\"tokenizer_class\")\n",
    "    if changed:\n",
    "        json.dump(c, open(p, \"w\"), indent=2)\n",
    "    return changed\n",
    "\n",
    "for name, repo in REPOS.items():\n",
    "    t = time.time()\n",
    "    PATHS[name] = snapshot_download(repo)\n",
    "    fixed = patch_tokenizer_config(PATHS[name])\n",
    "    if fixed:\n",
    "        print(\"   patched tokenizer config for %s: %s\" % (name, \", \".join(fixed)))\n",
    "    cfg = json.load(open(os.path.join(PATHS[name], \"rl_agent_config.json\")))\n",
    "    ecfg = json.load(open(os.path.join(PATHS[name], \"encoder\", \"config.json\")))\n",
    "    tot = enc = 0\n",
    "    with safe_open(os.path.join(PATHS[name], \"model.safetensors\"), \"pt\") as f:\n",
    "        for k in f.keys():\n",
    "            n = 1\n",
    "            for d in f.get_slice(k).get_shape():\n",
    "                n *= d\n",
    "            tot += n\n",
    "            enc += n if k.startswith(\"encoder.\") else 0\n",
    "    MODEL_META[name] = {\n",
    "        \"repo\": repo, \"params_total_m\": round(tot/1e6, 2), \"params_encoder_m\": round(enc/1e6, 2),\n",
    "        \"params_head_m\": round((tot-enc)/1e6, 2), \"hidden_size\": ecfg.get(\"hidden_size\"),\n",
    "        \"num_layers\": ecfg.get(\"num_hidden_layers\"), \"vocab_size\": ecfg.get(\"vocab_size\"),\n",
    "        \"max_len\": cfg.get(\"max_len\"), \"head_max_len\": cfg.get(\"head_max_len\"),\n",
    "        \"temperature\": cfg.get(\"temperature\"), \"temperature_by_options\": cfg.get(\"temperature_by_options\", {}),\n",
    "        \"training\": cfg.get(\"training\", {}), \"download_s\": round(time.time()-t, 1),\n",
    "        \"tokenizer_config_patched\": fixed,\n",
    "    }\n",
    "    print(name, \"->\", PATHS[name])\n",
    "    print(\"   \", json.dumps({k: v for k, v in MODEL_META[name].items() if k != \"temperature_by_options\"}))\n",
    "\n",
    "def load_agent(name):\n",
    "    import laya\n",
    "    ag = laya.load(PATHS[name], device=\"cuda\")\n",
    "    try:\n",
    "        ag.model.encoder.config.reference_compile = False   # torch.compile hurts on T4-class GPUs\n",
    "    except Exception as e:\n",
    "        print(\"could not disable reference_compile:\", e)\n",
    "    ag.model.eval()\n",
    "    return ag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Benchmark engine \u2014 batched scoring and metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile bench_engine.py\n",
    "import json, math, sys, time\n",
    "import numpy as np\n",
    "import torch\n",
    "from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket\n",
    "\n",
    "def to_internal(qdef):\n",
    "    t = qdef[\"type\"]\n",
    "    crit = qdef.get(\"criteria\")\n",
    "    if t == \"choice\" and isinstance(crit, list):\n",
    "        crit = {c: None for c in crit}\n",
    "    ins = qdef[\"instructions\"]\n",
    "    if not isinstance(ins, str):\n",
    "        ins = json.dumps(ins)\n",
    "    return {\"t\": t, \"ins\": ins, \"crit\": crit}\n",
    "\n",
    "@torch.no_grad()\n",
    "def score_cases(agent, cases, max_tokens=16384, max_seqs=128, progress=\"\"):\n",
    "    '''cases: [(state, {qid: qdef})] -> (raw_logits per question, index, seconds, n_dropped).\n",
    "\n",
    "    Every question of every case is one sequence. Sequences are length-sorted and packed into\n",
    "    token-budgeted batches so padding stays small. Returns UNCALIBRATED logits; temperature is\n",
    "    applied later so raw/calibrated variants come from one forward pass.\n",
    "    '''\n",
    "    max_len = agent.cfg.get(\"max_len\", 512)\n",
    "    head_max_len = agent.cfg.get(\"head_max_len\", 192)\n",
    "    items, index, dropped = [], [], 0\n",
    "    for ci, (state, questions) in enumerate(cases):\n",
    "        for qid, qdef in questions.items():\n",
    "            q = to_internal(qdef)\n",
    "            try:\n",
    "                ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)\n",
    "            except Exception:\n",
    "                index.append((ci, qid, QTYPES[q[\"t\"]], 0)); items.append(None); dropped += 1\n",
    "                continue\n",
    "            if len(markers) != len(render_options(q)):\n",
    "                index.append((ci, qid, QTYPES[q[\"t\"]], 0)); items.append(None); dropped += 1\n",
    "                continue\n",
    "            items.append({\"ids\": ids, \"markers\": markers, \"qtype\": QTYPES[q[\"t\"]]})\n",
    "            index.append((ci, qid, QTYPES[q[\"t\"]], len(markers)))\n",
    "\n",
    "    valid = [i for i, it in enumerate(items) if it is not None]\n",
    "    order = sorted(valid, key=lambda i: len(items[i][\"ids\"]))\n",
    "    out = [None] * len(items)\n",
    "    t0, done, i = time.time(), 0, 0\n",
    "    while i < len(order):\n",
    "        j, L = i, 0\n",
    "        while j < len(order) and j - i < max_seqs and max(L, len(items[order[j]][\"ids\"])) * (j - i + 1) <= max_tokens:\n",
    "            L = max(L, len(items[order[j]][\"ids\"])); j += 1\n",
    "        j = max(j, i + 1)\n",
    "        sel = [items[order[t]] for t in range(i, j)]\n",
    "        b = collate_items([sel], agent.tok.pad_token_id)\n",
    "        with torch.autocast(\"cuda\", dtype=agent.dtype, enabled=True):\n",
    "            logits, _ = agent.model(b[\"input_ids\"].cuda(), b[\"attention_mask\"].cuda(),\n",
    "                                    b[\"marker_pos\"].cuda(), b[\"marker_mask\"].cuda(), b[\"qtype\"].cuda())\n",
    "        logits = logits.float().cpu().numpy()\n",
    "        for r in range(j - i):\n",
    "            out[order[i + r]] = logits[r, :len(sel[r][\"markers\"])]\n",
    "        done += j - i\n",
    "        if progress:\n",
    "            el = time.time() - t0\n",
    "            msg = \"\\r  [%s] %d/%d seq | %.0f seq/s | ETA %ds    \" % (\n",
    "                progress, done, len(order), done/max(el, 1e-9),\n",
    "                int(el*(len(order)-done)/max(1, done)))\n",
    "            sys.stdout.write(msg)\n",
    "            sys.stdout.flush()\n",
    "        i = j\n",
    "    if progress:\n",
    "        print(\"\\r  [%s] %d sequences in %.1fs (%d dropped: options did not fit)%s\" % (\n",
    "            progress, len(order), time.time()-t0, dropped, \" \"*20))\n",
    "    return out, index, time.time() - t0, dropped\n",
    "\n",
    "def softmax_t(z, t=1.0):\n",
    "    z = np.asarray(z, dtype=np.float64) / max(1e-3, float(t))\n",
    "    e = np.exp(z - z.max())\n",
    "    return e / e.sum()\n",
    "\n",
    "def temp_for(agent, qt, k, calibrated=True):\n",
    "    if not calibrated:\n",
    "        return 1.0\n",
    "    return float(agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt]))\n",
    "\n",
    "# ------------------------------------------------------------------ metrics\n",
    "def ece_score(conf, correct, bins=15):\n",
    "    conf, correct = np.asarray(conf, float), np.asarray(correct, float)\n",
    "    if len(conf) == 0:\n",
    "        return float(\"nan\")\n",
    "    edges = np.linspace(0, 1, bins + 1)\n",
    "    e = 0.0\n",
    "    for lo, hi in zip(edges[:-1], edges[1:]):\n",
    "        sel = (conf > lo) & (conf <= hi)\n",
    "        if sel.any():\n",
    "            e += sel.mean() * abs(conf[sel].mean() - correct[sel].mean())\n",
    "    return float(e)\n",
    "\n",
    "def macro_f1(gold, pred):\n",
    "    gold, pred = np.asarray(gold), np.asarray(pred)\n",
    "    f1 = []\n",
    "    for c in sorted(set(gold.tolist()) | set(pred.tolist())):\n",
    "        tp = int(((pred == c) & (gold == c)).sum())\n",
    "        fp = int(((pred == c) & (gold != c)).sum())\n",
    "        fn = int(((pred != c) & (gold == c)).sum())\n",
    "        f1.append(2*tp / max(1, 2*tp + fp + fn))\n",
    "    return float(np.mean(f1))\n",
    "\n",
    "def aurc(conf, correct):\n",
    "    conf, correct = np.asarray(conf, float), np.asarray(correct, float)\n",
    "    if len(conf) == 0:\n",
    "        return float(\"nan\")\n",
    "    o = np.argsort(-conf)\n",
    "    return float((np.cumsum(1 - correct[o]) / np.arange(1, len(o)+1)).mean())\n",
    "\n",
    "def hard_metrics(rows):\n",
    "    '''rows: [(gold_idx, probs)] -> standard single-label classification + calibration metrics.'''\n",
    "    rows = [r for r in rows if r[1] is not None]\n",
    "    if not rows:\n",
    "        return {\"n\": 0}\n",
    "    gold = np.array([g for g, _ in rows])\n",
    "    pred = np.array([int(np.argmax(p)) for _, p in rows])\n",
    "    conf = np.array([float(np.max(p)) for _, p in rows])\n",
    "    corr = (pred == gold).astype(float)\n",
    "    m = {\"n\": len(rows),\n",
    "         \"accuracy\": float(corr.mean()),\n",
    "         \"macro_f1\": macro_f1(gold, pred),\n",
    "         \"ece\": ece_score(conf, corr),\n",
    "         \"brier\": float(np.mean([((np.asarray(p) - np.eye(len(p))[g])**2).sum() for g, p in rows])),\n",
    "         \"nll\": float(np.mean([-math.log(max(float(p[g]), 1e-12)) for g, p in rows])),\n",
    "         \"aurc\": aurc(conf, corr),\n",
    "         \"mean_confidence\": float(conf.mean())}\n",
    "    for cov in (0.5, 0.8):\n",
    "        k = max(1, int(len(conf)*cov))\n",
    "        m[\"acc_at_%d_coverage\" % int(cov*100)] = float(corr[np.argsort(-conf)[:k]].mean())\n",
    "    return m\n",
    "\n",
    "def fit_temperature(pairs, lo=0.2, hi=10.0, steps=160):\n",
    "    '''pairs: [(logits, gold_idx)] -> single temperature minimising NLL (grid search, no autograd).'''\n",
    "    pairs = [(np.asarray(z, float), int(g)) for z, g in pairs if z is not None]\n",
    "    if len(pairs) < 25:\n",
    "        return 1.0\n",
    "    best_t, best = 1.0, float(\"inf\")\n",
    "    for t in np.geomspace(lo, hi, steps):\n",
    "        tot = 0.0\n",
    "        for z, g in pairs:\n",
    "            zz = z/t\n",
    "            zz = zz - zz.max()\n",
    "            tot += -(zz[g] - math.log(np.exp(zz).sum()))\n",
    "        if tot < best:\n",
    "            best, best_t = tot, float(t)\n",
    "    return round(float(best_t), 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Task suites\n",
    "\n",
    "Every suite is built **once**, with a fixed seed, and reused for both models \u2014 so the two\n",
    "checkpoints answer byte-identical questions. Option sets for many-label tasks use\n",
    "`gold + N-1 distractors`, matching how the models were trained (the training recipe samples\n",
    "5\u201320 options per choice question rather than showing the full label space)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random, json\n",
    "import numpy as np\n",
    "from datasets import load_dataset\n",
    "\n",
    "SEED = 13\n",
    "N_OPTS = 20          # options per many-label choice question (gold + 19 distractors)\n",
    "PER_LANG = 300       # cases per language\n",
    "SUITES = {}          # name -> {\"cases\": [(state, questions)], \"gold\": [{qid: gold_idx}], \"meta\": {...}}\n",
    "\n",
    "def build_choice_questions(rng, gold, all_labels, n_opts, instructions, qid=\"label\"):\n",
    "    pool = [x for x in all_labels if x != gold]\n",
    "    keys = [gold] + rng.sample(pool, min(n_opts-1, len(pool)))\n",
    "    rng.shuffle(keys)\n",
    "    crit = {k: k.replace(\"_\", \" \").replace(\".\", \": \") for k in keys}\n",
    "    return {qid: {\"type\": \"choice\", \"instructions\": instructions, \"criteria\": crit}}, keys.index(gold)\n",
    "\n",
    "def register(name, cases, golds, **meta):\n",
    "    SUITES[name] = {\"cases\": cases, \"gold\": golds, \"meta\": meta}\n",
    "    nq = sum(len(c[1]) for c in cases)\n",
    "    print(\"  %-34s %4d cases / %5d questions   %s\" % (name, len(cases), nq, meta.get(\"note\", \"\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5a. typed-decisions (English, 400 cases / 2,000 decisions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "td_cases, td_golds, td_wf = [], [], []\n",
    "try:\n",
    "    td = load_dataset(\"LocalLLaMA/typed-decisions\", \"all\", split=\"test\")\n",
    "    for r in td:\n",
    "        questions = json.loads(r[\"questions\"]) if isinstance(r[\"questions\"], str) else r[\"questions\"]\n",
    "        gold = json.loads(r[\"gold\"]) if isinstance(r[\"gold\"], str) else r[\"gold\"]\n",
    "        state = r[\"state\"]\n",
    "        try:\n",
    "            state = json.loads(state)\n",
    "        except Exception:\n",
    "            pass\n",
    "        gmap = {}\n",
    "        for qid, qdef in questions.items():\n",
    "            g = gold[qid]\n",
    "            if qdef[\"type\"] == \"choice\":\n",
    "                keys = list(qdef[\"criteria\"].keys())\n",
    "                gmap[qid] = {\"idx\": keys.index(str(g[\"label\"])), \"keys\": keys,\n",
    "                             \"soft\": [float(g.get(\"probabilities\", {}).get(k, 0.0)) for k in keys]}\n",
    "            elif qdef[\"type\"] == \"noul\":\n",
    "                idx = 1 if str(g[\"label\"]).lower() == \"true\" else 0\n",
    "                pt = float(g.get(\"probabilities\", {}).get(\"true\", g.get(\"noul\", 0.5)))\n",
    "                gmap[qid] = {\"idx\": idx, \"keys\": [\"false\", \"true\"], \"soft\": [1-pt, pt]}\n",
    "            else:  # score\n",
    "                n = len(qdef[\"criteria\"])\n",
    "                gmap[qid] = {\"idx\": int(g[\"label\"]), \"keys\": [str(i) for i in range(n)],\n",
    "                             \"soft\": [float(g.get(\"probabilities\", {}).get(str(i), 0.0)) for i in range(n)],\n",
    "                             \"gold_score\": float(g.get(\"score\", float(g[\"label\"])))}\n",
    "        td_cases.append((state, questions))\n",
    "        td_golds.append(gmap)\n",
    "        td_wf.append(r[\"workflow\"])\n",
    "    register(\"typed_decisions\", td_cases, td_golds, note=\"4 workflows, soft teacher labels\",\n",
    "             workflows=td_wf, soft=True)\n",
    "except Exception as e:\n",
    "    print(\"typed-decisions FAILED:\", type(e).__name__, e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5b. Multilingual \u2014 MASSIVE intent & scenario (14 languages), XNLI (15 languages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MASSIVE_LANGS = [\"en\",\"de\",\"fr\",\"es\",\"pt\",\"ru\",\"tr\",\"ar\",\"hi\",\"ta\",\"zh-CN\",\"ja\",\"ko\",\"sw\"]\n",
    "XNLI_LANGS    = [\"en\",\"de\",\"fr\",\"es\",\"ru\",\"tr\",\"ar\",\"hi\",\"ur\",\"vi\",\"th\",\"el\",\"bg\",\"zh\",\"sw\"]\n",
    "\n",
    "for task, short, instr in [(\"mteb/amazon_massive_intent\", \"massive_intent\",\n",
    "                            \"What is the user asking for in `utterance`?\"),\n",
    "                           (\"mteb/amazon_massive_scenario\", \"massive_scenario\",\n",
    "                            \"Which domain does `utterance` belong to?\")]:\n",
    "    for lg in MASSIVE_LANGS:\n",
    "        try:\n",
    "            d = load_dataset(task, lg, split=\"test\")\n",
    "            labels = sorted(set(d[\"label_text\"]))\n",
    "            rng = random.Random(SEED)\n",
    "            cases, golds = [], []\n",
    "            for r in list(d)[:PER_LANG]:\n",
    "                qs, gi = build_choice_questions(rng, r[\"label_text\"], labels, N_OPTS, instr)\n",
    "                cases.append(({\"utterance\": r[\"text\"]}, qs))\n",
    "                golds.append({\"label\": {\"idx\": gi}})\n",
    "            register(\"%s.%s\" % (short, lg), cases, golds, lang=lg, n_options=N_OPTS,\n",
    "                     label_space=len(labels), family=short)\n",
    "        except Exception as e:\n",
    "            print(\"  FAIL %s %s: %s\" % (short, lg, str(e)[:90]))\n",
    "\n",
    "NLI_CRIT = {\"entailment\": \"the premise implies the hypothesis is true\",\n",
    "            \"neutral\": \"the premise neither implies nor contradicts the hypothesis\",\n",
    "            \"contradiction\": \"the premise implies the hypothesis is false\"}\n",
    "for lg in XNLI_LANGS:\n",
    "    try:\n",
    "        d = load_dataset(\"facebook/xnli\", lg, split=\"test\")\n",
    "        cases, golds = [], []\n",
    "        for r in list(d)[:PER_LANG]:\n",
    "            qs = {\"relation\": {\"type\": \"choice\",\n",
    "                               \"instructions\": \"What is the relationship between `premise` and `hypothesis`?\",\n",
    "                               \"criteria\": dict(NLI_CRIT)}}\n",
    "            cases.append(({\"premise\": r[\"premise\"], \"hypothesis\": r[\"hypothesis\"]}, qs))\n",
    "            golds.append({\"relation\": {\"idx\": int(r[\"label\"])}})\n",
    "        register(\"xnli.%s\" % lg, cases, golds, lang=lg, n_options=3, family=\"xnli\")\n",
    "    except Exception as e:\n",
    "        print(\"  FAIL xnli %s: %s\" % (lg, str(e)[:90]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5c. English tasks\n",
    "\n",
    "`sst5`, `emotion` and `prompt_injections` were held out of Laya's training entirely (true\n",
    "zero-shot). `ag_news` and `boolq` were in the training mix, so they measure retention rather\n",
    "than generalisation. `banking77` is the many-option stress test \u2014 77 options at once, versus\n",
    "Jev's documented hard cap at 255."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = random.Random(SEED)\n",
    "\n",
    "# SST-5: ordinal score\n",
    "try:\n",
    "    d = load_dataset(\"SetFit/sst5\", split=\"test\")\n",
    "    crit = [\"very negative\", \"negative\", \"neutral\", \"positive\", \"very positive\"]\n",
    "    cases, golds = [], []\n",
    "    for r in list(d)[:600]:\n",
    "        cases.append(({\"text\": r[\"text\"]},\n",
    "                      {\"sentiment\": {\"type\": \"score\", \"instructions\": \"How positive is the sentiment of `text`?\",\n",
    "                                     \"criteria\": crit}}))\n",
    "        golds.append({\"sentiment\": {\"idx\": int(r[\"label\"]), \"gold_score\": float(r[\"label\"])}})\n",
    "    register(\"en.sst5\", cases, golds, note=\"zero-shot, ordinal\", family=\"english\", held_out=True)\n",
    "except Exception as e: print(\"  FAIL sst5:\", str(e)[:90])\n",
    "\n",
    "# emotion: 6-way choice\n",
    "try:\n",
    "    d = load_dataset(\"dair-ai/emotion\", \"split\", split=\"test\")\n",
    "    names = [\"sadness\",\"joy\",\"love\",\"anger\",\"fear\",\"surprise\"]\n",
    "    cases, golds = [], []\n",
    "    for r in list(d)[:600]:\n",
    "        cases.append(({\"text\": r[\"text\"]},\n",
    "                      {\"emotion\": {\"type\": \"choice\",\n",
    "                                   \"instructions\": \"Which emotion is most strongly expressed in `text`?\",\n",
    "                                   \"criteria\": {n: None for n in names}}}))\n",
    "        golds.append({\"emotion\": {\"idx\": int(r[\"label\"])}})\n",
    "    register(\"en.emotion\", cases, golds, note=\"zero-shot\", family=\"english\", held_out=True)\n",
    "except Exception as e: print(\"  FAIL emotion:\", str(e)[:90])\n",
    "\n",
    "# prompt injections: noul\n",
    "try:\n",
    "    d = load_dataset(\"deepset/prompt-injections\", split=\"test\")\n",
    "    cases, golds = [], []\n",
    "    for r in list(d):\n",
    "        cases.append(({\"text\": r[\"text\"]},\n",
    "                      {\"injection\": {\"type\": \"noul\",\n",
    "                                     \"instructions\": \"Does `text` try to inject or override instructions given to an AI system?\"}}))\n",
    "        golds.append({\"injection\": {\"idx\": int(r[\"label\"])}})\n",
    "    register(\"en.prompt_injections\", cases, golds, note=\"zero-shot\", family=\"english\", held_out=True)\n",
    "except Exception as e: print(\"  FAIL prompt_injections:\", str(e)[:90])\n",
    "\n",
    "# banking77: 77-option stress test\n",
    "try:\n",
    "    d = load_dataset(\"PolyAI/banking77\", split=\"test\")\n",
    "    names = d.features[\"label\"].names\n",
    "    cases, golds = [], []\n",
    "    for r in list(d)[:500]:\n",
    "        cases.append(({\"message\": r[\"text\"]},\n",
    "                      {\"intent\": {\"type\": \"choice\", \"instructions\": \"Which banking intent does `message` express?\",\n",
    "                                  \"criteria\": {n.replace(\"_\", \" \"): None for n in names}}}))\n",
    "        golds.append({\"intent\": {\"idx\": int(r[\"label\"])}})\n",
    "    register(\"en.banking77_full\", cases, golds, note=\"77 options at once\", family=\"english\",\n",
    "             n_options=len(names), held_out=True)\n",
    "except Exception as e: print(\"  FAIL banking77:\", str(e)[:90])\n",
    "\n",
    "# ag_news + boolq: in-task retention\n",
    "try:\n",
    "    d = load_dataset(\"fancyzhx/ag_news\", split=\"test\")\n",
    "    crit = {\"world\": \"world news and international politics\", \"sports\": \"sports\",\n",
    "            \"business\": \"business and economy\", \"sci_tech\": \"science and technology\"}\n",
    "    cases, golds = [], []\n",
    "    for r in list(d)[:600]:\n",
    "        cases.append(({\"article\": r[\"text\"]},\n",
    "                      {\"topic\": {\"type\": \"choice\", \"instructions\": \"What is the topic of `article`?\",\n",
    "                                 \"criteria\": dict(crit)}}))\n",
    "        golds.append({\"topic\": {\"idx\": int(r[\"label\"])}})\n",
    "    register(\"en.ag_news\", cases, golds, note=\"in-task (trained on)\", family=\"english\", held_out=False)\n",
    "except Exception as e: print(\"  FAIL ag_news:\", str(e)[:90])\n",
    "\n",
    "try:\n",
    "    d = load_dataset(\"google/boolq\", split=\"validation\")\n",
    "    cases, golds = [], []\n",
    "    for r in list(d)[:600]:\n",
    "        cases.append(({\"passage\": r[\"passage\"], \"question\": r[\"question\"]},\n",
    "                      {\"answer\": {\"type\": \"noul\",\n",
    "                                  \"instructions\": \"Based on `passage`, is the answer to `question` yes?\"}}))\n",
    "        golds.append({\"answer\": {\"idx\": int(bool(r[\"answer\"]))}})\n",
    "    register(\"en.boolq\", cases, golds, note=\"in-task (trained on)\", family=\"english\", held_out=False)\n",
    "except Exception as e: print(\"  FAIL boolq:\", str(e)[:90])\n",
    "\n",
    "print()\n",
    "print(\"suites built:\", len(SUITES), \"| total questions:\",\n",
    "      sum(sum(len(c[1]) for c in s[\"cases\"]) for s in SUITES.values()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Run every suite on both models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bench_engine as BE\n",
    "import numpy as np, gc, time, json\n",
    "\n",
    "RESULTS = {\"meta\": {\"seed\": SEED, \"n_opts\": N_OPTS, \"per_lang\": PER_LANG,\n",
    "                    \"timestamp\": time.strftime(\"%Y-%m-%d %H:%M:%S\"),\n",
    "                    \"gpu\": torch.cuda.get_device_name(0),\n",
    "                    \"torch\": torch.__version__, \"transformers\": transformers.__version__,\n",
    "                    \"models\": MODEL_META},\n",
    "           \"suites\": {}}\n",
    "RAW = {}   # model -> suite -> (logits, index) kept in RAM for the calibration-repair section\n",
    "\n",
    "def evaluate_suite(agent, model_name, suite_name):\n",
    "    S = SUITES[suite_name]\n",
    "    logits, index, secs, dropped = BE.score_cases(agent, S[\"cases\"], progress=\"%s/%s\" % (model_name, suite_name))\n",
    "    RAW[model_name][suite_name] = (logits, index)\n",
    "    soft = S[\"meta\"].get(\"soft\", False)\n",
    "    rows_cal, rows_raw, extra = [], [], {\"soft_acc\": [], \"brier_soft\": [], \"tv\": [], \"kl\": [],\n",
    "                                         \"score_mae\": [], \"within_1\": []}\n",
    "    per_q_type = {}\n",
    "    for (ci, qid, qt, k), z in zip(index, logits):\n",
    "        g = S[\"gold\"][ci].get(qid)\n",
    "        if g is None:\n",
    "            continue\n",
    "        if z is None:\n",
    "            rows_cal.append((g[\"idx\"], None)); rows_raw.append((g[\"idx\"], None)); continue\n",
    "        p_cal = BE.softmax_t(z, BE.temp_for(agent, qt, k, True))\n",
    "        p_raw = BE.softmax_t(z, 1.0)\n",
    "        rows_cal.append((g[\"idx\"], p_cal)); rows_raw.append((g[\"idx\"], p_raw))\n",
    "        per_q_type.setdefault(qt, []).append((g[\"idx\"], p_cal))\n",
    "        if soft and \"soft\" in g:\n",
    "            gp = np.asarray(g[\"soft\"], float)\n",
    "            if gp.sum() > 0:\n",
    "                gp = gp / gp.sum()\n",
    "                pp = p_cal[:len(gp)] if len(p_cal) >= len(gp) else np.pad(p_cal, (0, len(gp)-len(p_cal)))\n",
    "                pp = pp / max(pp.sum(), 1e-12)\n",
    "                extra[\"soft_acc\"].append(float((pp*gp).sum()))\n",
    "                extra[\"brier_soft\"].append(float(((pp-gp)**2).sum()))\n",
    "                extra[\"tv\"].append(float(0.5*np.abs(pp-gp).sum()))\n",
    "                extra[\"kl\"].append(float((gp*np.log(np.clip(gp/np.clip(pp,1e-12,None),1e-12,1e4))).sum()))\n",
    "        if \"gold_score\" in g:\n",
    "            exp = float((np.arange(len(p_cal))*p_cal).sum())\n",
    "            extra[\"score_mae\"].append(abs(exp - g[\"gold_score\"]))\n",
    "            extra[\"within_1\"].append(float(abs(exp - g[\"gold_score\"]) <= 1.0))\n",
    "\n",
    "    out = {\"calibrated\": BE.hard_metrics(rows_cal), \"raw\": BE.hard_metrics(rows_raw),\n",
    "           \"seconds\": round(secs, 2), \"dropped_questions\": dropped,\n",
    "           \"questions_per_second\": round(len([r for r in rows_cal if r[1] is not None])/max(secs,1e-9), 1)}\n",
    "    for k2, v in extra.items():\n",
    "        if v:\n",
    "            out[k2] = float(np.mean(v))\n",
    "    if len(per_q_type) > 1:\n",
    "        names = {0: \"choice\", 1: \"score\", 2: \"noul\"}\n",
    "        out[\"by_question_type\"] = {names[qt]: BE.hard_metrics(rs) for qt, rs in sorted(per_q_type.items())}\n",
    "    return out\n",
    "\n",
    "for model_name in REPOS:\n",
    "    print(\"\\n\" + \"=\"*78 + \"\\n  %s\\n\" % model_name + \"=\"*78)\n",
    "    agent = load_agent(model_name)\n",
    "    RAW[model_name] = {}\n",
    "    for suite_name in SUITES:\n",
    "        try:\n",
    "            r = evaluate_suite(agent, model_name, suite_name)\n",
    "            RESULTS[\"suites\"].setdefault(suite_name, {})[model_name] = r\n",
    "            c = r[\"calibrated\"]\n",
    "            print(\"   %-32s acc %.3f  f1 %.3f  ECE %.3f  n=%d\" %\n",
    "                  (suite_name, c.get(\"accuracy\", float(\"nan\")), c.get(\"macro_f1\", float(\"nan\")),\n",
    "                   c.get(\"ece\", float(\"nan\")), c.get(\"n\", 0)))\n",
    "        except Exception as e:\n",
    "            import traceback; traceback.print_exc()\n",
    "            RESULTS[\"suites\"].setdefault(suite_name, {})[model_name] = {\"error\": \"%s: %s\" % (type(e).__name__, e)}\n",
    "    globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")] = agent\n",
    "    gc.collect(); torch.cuda.empty_cache()\n",
    "\n",
    "json.dump(RESULTS, open(\"laya_benchmark_results.json\", \"w\"), indent=2)\n",
    "print(\"\\ncheckpoint saved\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6b. Control \u2014 matched context budget\n",
    "\n",
    "The two checkpoints ship different context windows (512 vs 1024 tokens), so on long states\n",
    "`laya-multilingual` sees more of the input. This re-runs typed-decisions with **both** models\n",
    "capped at 512 tokens, isolating the architecture from the context-window advantage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bench_engine as BE\n",
    "import numpy as np\n",
    "\n",
    "RESULTS[\"typed_decisions_matched_512\"] = {}\n",
    "if \"typed_decisions\" in SUITES:\n",
    "    S = SUITES[\"typed_decisions\"]\n",
    "    for model_name in REPOS:\n",
    "        agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
    "        orig = agent.cfg.get(\"max_len\")\n",
    "        agent.cfg[\"max_len\"] = 512\n",
    "        try:\n",
    "            logits, index, secs, dropped = BE.score_cases(agent, S[\"cases\"], progress=\"%s/td@512\" % model_name)\n",
    "            rows = []\n",
    "            for (ci, qid, qt, k), z in zip(index, logits):\n",
    "                g = S[\"gold\"][ci].get(qid)\n",
    "                if g is None or z is None:\n",
    "                    continue\n",
    "                rows.append((g[\"idx\"], BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))\n",
    "            RESULTS[\"typed_decisions_matched_512\"][model_name] = BE.hard_metrics(rows)\n",
    "            print(\"  %-22s max_len 512 -> acc %.3f  ECE %.3f  n=%d\"\n",
    "                  % (model_name, RESULTS[\"typed_decisions_matched_512\"][model_name][\"accuracy\"],\n",
    "                     RESULTS[\"typed_decisions_matched_512\"][model_name][\"ece\"],\n",
    "                     RESULTS[\"typed_decisions_matched_512\"][model_name][\"n\"]))\n",
    "        finally:\n",
    "            agent.cfg[\"max_len\"] = orig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Latency (p50 / p95 at 1, 5, 10, 50 questions per call)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np, time, torch\n",
    "\n",
    "LAT_STATE = {\"ticket\": {\"subject\": \"Payout failing\",\n",
    "             \"messages\": [{\"from\": \"customer\",\n",
    "                           \"text\": \"Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. \" * 6}]}}\n",
    "Q_NOUL   = {\"type\": \"noul\", \"instructions\": \"Does `ticket.messages[0].text` express urgency?\"}\n",
    "Q_CHOICE = {\"type\": \"choice\", \"instructions\": \"Which team should handle this?\",\n",
    "            \"criteria\": {\"billing\": \"payments\", \"technical\": \"bugs and integrations\", \"sales\": \"pricing\"}}\n",
    "\n",
    "RESULTS[\"latency\"] = {}\n",
    "for model_name in REPOS:\n",
    "    agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
    "    res = {}\n",
    "    for nq in (1, 5, 10, 50):\n",
    "        qs = {(\"q%d\" % i): (Q_NOUL if i % 2 else Q_CHOICE) for i in range(nq)}\n",
    "        for _ in range(3):\n",
    "            agent.system_one(LAT_STATE, qs)\n",
    "        ts = []\n",
    "        for _ in range(20):\n",
    "            torch.cuda.synchronize(); t = time.perf_counter()\n",
    "            agent.system_one(LAT_STATE, qs)\n",
    "            torch.cuda.synchronize(); ts.append((time.perf_counter()-t)*1000)\n",
    "        res[\"%d_questions\" % nq] = {\"p50_ms\": round(float(np.percentile(ts, 50)), 1),\n",
    "                                    \"p95_ms\": round(float(np.percentile(ts, 95)), 1),\n",
    "                                    \"ms_per_question\": round(float(np.percentile(ts, 50))/nq, 2)}\n",
    "    RESULTS[\"latency\"][model_name] = res\n",
    "    print(model_name, json.dumps(res))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Option-order robustness\n",
    "\n",
    "Same question, options permuted. A well-behaved decision model should pick the same label.\n",
    "The independent Jev benchmark measured a **13% flip rate for Jev**, with the worst LLM at 37%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random, numpy as np\n",
    "import bench_engine as BE\n",
    "\n",
    "PB_SUITES = [s for s in (\"massive_intent.en\", \"en.emotion\", \"xnli.en\") if s in SUITES]\n",
    "RESULTS[\"option_order_robustness\"] = {}\n",
    "\n",
    "for model_name in REPOS:\n",
    "    agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
    "    out = {}\n",
    "    for sname in PB_SUITES:\n",
    "        S = SUITES[sname]\n",
    "        base = S[\"cases\"][:200]\n",
    "        rng = random.Random(99)\n",
    "        perm_cases, perm_maps = [], []\n",
    "        for state, questions in base:\n",
    "            qid = list(questions)[0]\n",
    "            qdef = questions[qid]\n",
    "            keys = list(qdef[\"criteria\"].keys())\n",
    "            order = list(range(len(keys))); rng.shuffle(order)\n",
    "            newcrit = {keys[i]: qdef[\"criteria\"][keys[i]] for i in order}\n",
    "            perm_cases.append((state, {qid: {**qdef, \"criteria\": newcrit}}))\n",
    "            perm_maps.append((keys, [keys[i] for i in order]))\n",
    "        la, ia, _, _ = BE.score_cases(agent, base, progress=\"\")\n",
    "        lb, ib, _, _ = BE.score_cases(agent, perm_cases, progress=\"\")\n",
    "        flips = comp = 0\n",
    "        for (za, (_, _, _, ka)), (zb, (_, _, _, kb)), (keys, pkeys) in zip(zip(la, ia), zip(lb, ib), perm_maps):\n",
    "            if za is None or zb is None:\n",
    "                continue\n",
    "            comp += 1\n",
    "            if keys[int(np.argmax(za))] != pkeys[int(np.argmax(zb))]:\n",
    "                flips += 1\n",
    "        out[sname] = {\"n\": comp, \"flip_rate\": round(flips/max(1, comp), 4)}\n",
    "        print(\"  %-22s %-24s flip rate %.3f (n=%d)\" % (model_name, sname, out[sname][\"flip_rate\"], comp))\n",
    "    RESULTS[\"option_order_robustness\"][model_name] = out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Calibration repair\n",
    "\n",
    "`laya-multilingual` ships with `temperature = [1.0, 1.0, 1.0]` and no per-option-count buckets \u2014\n",
    "it was never calibrated. This refits one temperature per (question type, option-count bucket) on\n",
    "**half** of each suite and reports ECE on the other half, so the improvement is measured\n",
    "out-of-sample. It shows how much of any calibration gap is a missing post-processing step rather\n",
    "than a property of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import bench_engine as BE\n",
    "from laya.common import temp_bucket\n",
    "\n",
    "RESULTS[\"calibration_repair\"] = {}\n",
    "for model_name in REPOS:\n",
    "    agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
    "    per_suite, fitted_all = {}, {}\n",
    "    for sname, S in SUITES.items():\n",
    "        if sname not in RAW[model_name]:\n",
    "            continue\n",
    "        logits, index = RAW[model_name][sname]\n",
    "        pairs = []\n",
    "        for (ci, qid, qt, k), z in zip(index, logits):\n",
    "            g = S[\"gold\"][ci].get(qid)\n",
    "            if g is not None and z is not None:\n",
    "                pairs.append((z, g[\"idx\"], qt, k))\n",
    "        if len(pairs) < 60:\n",
    "            continue\n",
    "        half = len(pairs)//2\n",
    "        fit_set, held = pairs[:half], pairs[half:]\n",
    "        buckets = {}\n",
    "        for z, gi, qt, k in fit_set:\n",
    "            buckets.setdefault(temp_bucket(qt, k), []).append((z, gi))\n",
    "        fitted = {b: BE.fit_temperature(v) for b, v in buckets.items() if len(v) >= 25}\n",
    "        rows_ship, rows_fit = [], []\n",
    "        for z, gi, qt, k in held:\n",
    "            rows_ship.append((gi, BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))\n",
    "            rows_fit.append((gi, BE.softmax_t(z, fitted.get(temp_bucket(qt, k),\n",
    "                                                            BE.temp_for(agent, qt, k, True)))))\n",
    "        ms, mf = BE.hard_metrics(rows_ship), BE.hard_metrics(rows_fit)\n",
    "        per_suite[sname] = {\"n_heldout\": ms[\"n\"], \"fitted_temperatures\": fitted,\n",
    "                            \"ece_shipped\": ms[\"ece\"], \"ece_refit\": mf[\"ece\"],\n",
    "                            \"nll_shipped\": ms[\"nll\"], \"nll_refit\": mf[\"nll\"],\n",
    "                            \"accuracy\": ms[\"accuracy\"]}\n",
    "        fitted_all.update(fitted)\n",
    "    RESULTS[\"calibration_repair\"][model_name] = {\"per_suite\": per_suite}\n",
    "    if per_suite:\n",
    "        a = float(np.mean([v[\"ece_shipped\"] for v in per_suite.values()]))\n",
    "        b = float(np.mean([v[\"ece_refit\"] for v in per_suite.values()]))\n",
    "        RESULTS[\"calibration_repair\"][model_name][\"mean_ece_shipped\"] = round(a, 4)\n",
    "        RESULTS[\"calibration_repair\"][model_name][\"mean_ece_refit\"] = round(b, 4)\n",
    "        print(\"%-22s mean ECE  shipped %.4f  ->  refit %.4f\" % (model_name, a, b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Aggregate, save and download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np, json\n",
    "\n",
    "def agg(prefix_or_family, key=\"family\"):\n",
    "    out = {}\n",
    "    for m in REPOS:\n",
    "        accs, eces, ns = [], [], 0\n",
    "        for sname, S in SUITES.items():\n",
    "            if S[\"meta\"].get(key) != prefix_or_family:\n",
    "                continue\n",
    "            r = RESULTS[\"suites\"].get(sname, {}).get(m, {})\n",
    "            c = r.get(\"calibrated\", {})\n",
    "            if c.get(\"n\"):\n",
    "                accs.append(c[\"accuracy\"]); eces.append(c[\"ece\"]); ns += c[\"n\"]\n",
    "        if accs:\n",
    "            out[m] = {\"macro_accuracy\": round(float(np.mean(accs)), 4),\n",
    "                      \"macro_ece\": round(float(np.mean(eces)), 4),\n",
    "                      \"n_questions\": ns, \"n_suites\": len(accs)}\n",
    "    return out\n",
    "\n",
    "RESULTS[\"caveats\"] = {\n",
    "    \"context_window\": \"laya max_len=512, laya-multilingual max_len=1024 (shipped configs). \"\n",
    "                      \"See typed_decisions_matched_512 for the equal-budget control.\",\n",
    "    \"shipped_calibration\": \"laya ships fitted temperatures; laya-multilingual ships all-1.0 \"\n",
    "                           \"(uncalibrated). See calibration_repair for the out-of-sample refit.\",\n",
    "    \"training_overlap\": \"ag_news and boolq were in Laya's training mix (retention, not generalisation). \"\n",
    "                        \"sst5, emotion, prompt_injections, banking77 were held out. \"\n",
    "                        \"Neither checkpoint trained on typed-decisions, MASSIVE or XNLI.\",\n",
    "}\n",
    "\n",
    "RESULTS[\"summary\"] = {\n",
    "    \"typed_decisions\": {m: RESULTS[\"suites\"].get(\"typed_decisions\", {}).get(m, {}).get(\"calibrated\", {})\n",
    "                        for m in REPOS},\n",
    "    \"massive_intent\": agg(\"massive_intent\"),\n",
    "    \"massive_scenario\": agg(\"massive_scenario\"),\n",
    "    \"xnli\": agg(\"xnli\"),\n",
    "    \"english\": agg(\"english\"),\n",
    "}\n",
    "\n",
    "# English vs non-English split inside each multilingual family\n",
    "for fam in (\"massive_intent\", \"massive_scenario\", \"xnli\"):\n",
    "    en, non = {}, {}\n",
    "    for m in REPOS:\n",
    "        ea, na = [], []\n",
    "        for sname, S in SUITES.items():\n",
    "            if S[\"meta\"].get(\"family\") != fam:\n",
    "                continue\n",
    "            c = RESULTS[\"suites\"].get(sname, {}).get(m, {}).get(\"calibrated\", {})\n",
    "            if not c.get(\"n\"):\n",
    "                continue\n",
    "            (ea if S[\"meta\"].get(\"lang\") in (\"en\",) else na).append(c[\"accuracy\"])\n",
    "        if ea: en[m] = round(float(np.mean(ea)), 4)\n",
    "        if na: non[m] = round(float(np.mean(na)), 4)\n",
    "    RESULTS[\"summary\"][fam + \"_english_vs_rest\"] = {\"english\": en, \"non_english\": non}\n",
    "\n",
    "json.dump(RESULTS, open(\"laya_benchmark_results.json\", \"w\"), indent=2)\n",
    "print(json.dumps(RESULTS[\"summary\"], indent=2))\n",
    "print(\"\\nwrote laya_benchmark_results.json  (%.1f KB)\" % (os.path.getsize(\"laya_benchmark_results.json\")/1024))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Per-language table, printed for a quick eyeball before you download the JSON\n",
    "import numpy as np\n",
    "for fam in (\"massive_intent\", \"massive_scenario\", \"xnli\"):\n",
    "    langs = sorted({S[\"meta\"][\"lang\"] for S in SUITES.values() if S[\"meta\"].get(\"family\") == fam})\n",
    "    if not langs:\n",
    "        continue\n",
    "    print(\"\\n=== %s : accuracy by language ===\" % fam)\n",
    "    print(\"%-8s %12s %22s %10s\" % (\"lang\", \"laya\", \"laya-multilingual\", \"delta\"))\n",
    "    for lg in langs:\n",
    "        sname = \"%s.%s\" % (fam, lg)\n",
    "        a = RESULTS[\"suites\"].get(sname, {}).get(\"laya\", {}).get(\"calibrated\", {}).get(\"accuracy\")\n",
    "        b = RESULTS[\"suites\"].get(sname, {}).get(\"laya-multilingual\", {}).get(\"calibrated\", {}).get(\"accuracy\")\n",
    "        if a is None or b is None:\n",
    "            continue\n",
    "        print(\"%-8s %12.3f %22.3f %+10.3f\" % (lg, a, b, b-a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from google.colab import files\n",
    "    files.download(\"laya_benchmark_results.json\")\n",
    "except Exception as e:\n",
    "    print(\"Not on Colab or download blocked (%s). Grab laya_benchmark_results.json from the file browser.\" % type(e).__name__)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": [],
   "gpuType": "T4",
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}