Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat: add Colab evaluation notebook for LocalLLaMA/typed-decisions benchmark
Nandakishor479248998428
diff --git a/notebooks/laya_eval_typed_decisions_colab.ipynb b/notebooks/laya_eval_typed_decisions_colab.ipynbnew file mode 100644index 00000000..c93f9e16--- /dev/null+++ b/notebooks/laya_eval_typed_decisions_colab.ipynb@@ -0,0 +1,386 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "# Evaluating Laya on `LocalLLaMA/typed-decisions` Benchmark (Single T4 GPU)\n",+ "\n",+ "[](https://colab.research.google.com/github/NandhaKishorM/laya/blob/main/notebooks/laya_eval_typed_decisions_colab.ipynb)\n",+ "[](https://pypi.org/project/laya/)\n",+ "[](https://huggingface.co/convaiinnovations/laya)\n",+ "[](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)\n",+ "\n",+ "This notebook evaluates **Laya** (`convaiinnovations/laya`) head-to-head against **TypeSafe Jev** on the independent [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) benchmark.\n",+ "\n",+ "### What is `typed-decisions`?\n",+ "It is the standard public benchmark for System One decision models, covering 4 production workflows:\n",+ "1. `agent_trace_observability`: Assess agent executions for human intervention and urgency.\n",+ "2. `customer_service`: Route assistant responses and actions from multi-turn customer dialogues.\n",+ "3. `invoice_processing`: Review vendor invoices against purchase orders and delivery receipts.\n",+ "4. `security_incidents`: Determine containment, investigation, or dismissal of security alerts.\n",+ "\n",+ "Each test case contains **5 typed questions** over a shared state. The entire benchmark consists of **400 cases (2,000 typed decisions)**.\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 1. GPU Check & Setup\n",+ "Ensure a T4 GPU is selected in `Runtime -> Change runtime type -> T4 GPU`.\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {},+ "source": [+ "!nvidia-smi\n",+ "import os, torch\n",+ "assert torch.cuda.is_available(), \"No GPU detected! Set Runtime -> Change runtime type -> T4 GPU\"\n",+ "props = torch.cuda.get_device_properties(0)\n",+ "print(f\"Connected to: {props.name} | Total VRAM: {props.total_memory / 1e9:.1f} GB\")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 2. Install Dependencies\n",+ "Install `laya`, `transformers`, `datasets`, and evaluation libraries.\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {},+ "source": [+ "!pip install -q -U \"laya>=0.1.6\" \"transformers>=4.48.0\" \"datasets>=3.0.0\" safetensors huggingface_hub pyarrow pandas scipy\n",+ "import laya, torch, datasets, pandas as pd\n",+ "print(\"Laya version:\", laya.__version__)\n",+ "print(\"PyTorch version:\", torch.__version__)\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 3. Load Laya Model\n",+ "Load the fine-tuned Laya model from Hugging Face on CUDA.\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {},+ "source": [+ "MODEL_ID = \"convaiinnovations/laya\"\n",+ "print(f\"Loading Laya model from {MODEL_ID}...\")\n",+ "\n",+ "agent = laya.load(MODEL_ID, device=\"cuda\")\n",+ "print(\"Model successfully loaded on GPU!\")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 4. Load the `LocalLLaMA/typed-decisions` Benchmark\n",+ "We load all 400 test cases across the four workflows.\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {},+ "source": [+ "from datasets import load_dataset\n",+ "import json\n",+ "\n",+ "print(\"Downloading LocalLLaMA/typed-decisions benchmark (split: test)...\")\n",+ "ds = load_dataset(\"LocalLLaMA/typed-decisions\", \"all\", split=\"test\")\n",+ "\n",+ "print(f\"Loaded {len(ds)} benchmark cases.\")\n",+ "print(\"Sample workflow breakdown:\")\n",+ "df_summary = pd.DataFrame(ds)\n",+ "print(df_summary[\"workflow\"].value_counts())\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 5. Run Laya Evaluation Loop\n",+ "Evaluate all 400 test cases (2,000 decisions) and measure per-case latency.\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {},+ "source": [+ "import time, math\n",+ "import numpy as np\n",+ "\n",+ "predictions = []\n",+ "latencies_ms = []\n",+ "\n",+ "print(\"Starting evaluation of 400 benchmark cases...\")\n",+ "t_start = time.time()\n",+ "\n",+ "for i, row in enumerate(ds):\n",+ " case_id = row[\"id\"]\n",+ " workflow = row[\"workflow\"]\n",+ " state = json.loads(row[\"state\"])\n",+ " questions = json.loads(row[\"questions\"])\n",+ " gold = json.loads(row[\"gold\"])\n",+ " \n",+ " t0 = time.perf_counter()\n",+ " res = agent.predict(state, questions)\n",+ " dt_ms = (time.perf_counter() - t0) * 1000\n",+ " latencies_ms.append(dt_ms)\n",+ " \n",+ " predictions.append({\n",+ " \"id\": case_id,\n",+ " \"workflow\": workflow,\n",+ " \"pred\": res[\"answers\"],\n",+ " \"gold\": gold,\n",+ " \"questions\": questions,\n",+ " \"latency_ms\": dt_ms\n",+ " })\n",+ " \n",+ " if (i + 1) % 50 == 0 or (i + 1) == len(ds):\n",+ " print(f\" Progress: {i + 1}/{len(ds)} cases evaluated | Average latency: {np.mean(latencies_ms):.1f} ms/case\")\n",+ "\n",+ "total_time = time.time() - t_start\n",+ "print(f\"\\nEvaluation complete in {total_time:.1f}s across {len(ds) * 5} decisions!\")\n",+ "print(f\"P50 Latency: {np.percentile(latencies_ms, 50):.1f} ms/case | P95 Latency: {np.percentile(latencies_ms, 95):.1f} ms/case\")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 6. Compute Official Benchmark Metrics\n",+ "We calculate the exact metrics defined by `LocalLLaMA/typed-decisions`: Accuracy, Soft Accuracy, Brier Score, ECE, KL Divergence, Total Variation (TV), and Score MAE.\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {},+ "source": [+ "from laya.common import ece_score\n",+ "\n",+ "accuracies = []\n",+ "soft_accuracies = []\n",+ "brier_scores = []\n",+ "kl_divs = []\n",+ "tv_distances = []\n",+ "score_maes = []\n",+ "all_confs = []\n",+ "all_corrects = []\n",+ "\n",+ "for item in predictions:\n",+ " pred_answers = item[\"pred\"]\n",+ " gold_answers = item[\"gold\"]\n",+ " questions = item[\"questions\"]\n",+ " \n",+ " for qid, qdef in questions.items():\n",+ " p_ans = pred_answers[qid]\n",+ " g_ans = gold_answers[qid]\n",+ " q_type = qdef[\"type\"]\n",+ " \n",+ " # 1. Choice / Multi-class\n",+ " if q_type == \"choice\":\n",+ " keys = list(qdef[\"criteria\"].keys())\n",+ " pred_choice = p_ans[\"choice\"]\n",+ " gold_label = g_ans[\"label\"]\n",+ " \n",+ " is_corr = float(pred_choice == gold_label)\n",+ " accuracies.append(is_corr)\n",+ " all_corrects.append(is_corr)\n",+ " all_confs.append(p_ans[\"confidence\"])\n",+ " \n",+ " p_probs = np.array([p_ans[\"probabilities\"].get(k, 1e-6) for k in keys])\n",+ " g_probs = np.array([g_ans[\"probabilities\"].get(k, 1e-6) for k in keys])\n",+ " \n",+ " p_probs /= p_probs.sum()\n",+ " g_probs /= g_probs.sum()\n",+ " \n",+ " # Soft accuracy: sum(p_i * g_i)\n",+ " soft_accuracies.append(float((p_probs * g_probs).sum()))\n",+ " # Brier score: sum((p - g)^2)\n",+ " brier_scores.append(float(((p_probs - g_probs) ** 2).sum()))\n",+ " # Total Variation: 0.5 * sum(|p - g|)\n",+ " tv_distances.append(float(0.5 * np.abs(p_probs - g_probs).sum()))\n",+ " # KL divergence: sum(g * log(g / p))\n",+ " kl_divs.append(float((g_probs * np.log(np.clip(g_probs / p_probs, 1e-12, 1e4))).sum()))\n",+ " \n",+ " # 2. Noul / Boolean\n",+ " elif q_type == \"noul\":\n",+ " p_val = p_ans[\"noul\"]\n",+ " g_val = g_ans.get(\"noul\", g_ans.get(\"probabilities\", {}).get(\"true\", 0.5))\n",+ " gold_label = g_ans[\"label\"]\n",+ " \n",+ " pred_label = \"true\" if p_val >= 0.5 else \"false\"\n",+ " is_corr = float(pred_label == str(gold_label).lower())\n",+ " accuracies.append(is_corr)\n",+ " all_corrects.append(is_corr)\n",+ " all_confs.append(max(p_val, 1.0 - p_val))\n",+ " \n",+ " p_dist = np.array([1.0 - p_val, p_val])\n",+ " g_dist = np.array([1.0 - g_val, g_val])\n",+ " \n",+ " soft_accuracies.append(float((p_dist * g_dist).sum()))\n",+ " brier_scores.append(float(((p_dist - g_dist) ** 2).sum()))\n",+ " tv_distances.append(float(0.5 * np.abs(p_dist - g_dist).sum()))\n",+ " kl_divs.append(float((g_dist * np.log(np.clip(g_dist / p_dist, 1e-12, 1e4))).sum()))\n",+ " \n",+ " # 3. Score / Ordinal Rubric\n",+ " elif q_type == \"score\":\n",+ " p_score = p_ans[\"score\"]\n",+ " g_score = g_ans.get(\"score\", 0.0)\n",+ " score_maes.append(abs(p_score - g_score))\n",+ " \n",+ " # Discrete level accuracy\n",+ " p_lvl = int(round(p_score))\n",+ " g_lvl = int(round(g_score))\n",+ " is_corr = float(p_lvl == g_lvl)\n",+ " accuracies.append(is_corr)\n",+ " all_corrects.append(is_corr)\n",+ " all_confs.append(p_ans.get(\"confidence\", 0.5))\n",+ "\n",+ "laya_acc = np.mean(accuracies)\n",+ "laya_soft_acc = np.mean(soft_accuracies)\n",+ "laya_brier = np.mean(brier_scores)\n",+ "laya_kl = np.mean(kl_divs)\n",+ "laya_tv = np.mean(tv_distances)\n",+ "laya_ece = ece_score(np.array(all_confs), np.array(all_corrects))\n",+ "laya_mae = np.mean(score_maes) if score_maes else 0.0\n",+ "laya_latency = np.percentile(latencies_ms, 50)\n",+ "\n",+ "print(\"--- Laya Benchmark Results on typed-decisions ---\")\n",+ "print(f\"Accuracy : {laya_acc:.3f}\")\n",+ "print(f\"Soft Accuracy : {laya_soft_acc:.3f}\")\n",+ "print(f\"Brier Score : {laya_brier:.3f}\")\n",+ "print(f\"ECE : {laya_ece:.3f}\")\n",+ "print(f\"Score MAE : {laya_mae:.3f}\")\n",+ "print(f\"KL Divergence : {laya_kl:.3f}\")\n",+ "print(f\"Latency (p50) : {laya_latency:.1f} ms/case\")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 7. Head-to-Head Comparison with TypeSafe Jev\n",+ "Compare Laya directly against the baseline numbers published on `LocalLLaMA/typed-decisions`.\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {},+ "source": [+ "comparison_data = [\n",+ " {\n",+ " \"Model\": \"TypeSafe Jev 1.13.0\",\n",+ " \"Kind\": \"general\",\n",+ " \"Accuracy\": 0.727,\n",+ " \"Soft Acc\": 0.580,\n",+ " \"Brier\": 0.148,\n",+ " \"ECE\": 0.144,\n",+ " \"Score MAE\": 0.391,\n",+ " \"ms/case\": 710,\n",+ " \"Cost/Case\": \"$0.0004 (API)\"\n",+ " },\n",+ " {\n",+ " \"Model\": \"Laya (Ours)\",\n",+ " \"Kind\": \"general\",\n",+ " \"Accuracy\": round(laya_acc, 3),\n",+ " \"Soft Acc\": round(laya_soft_acc, 3),\n",+ " \"Brier\": round(laya_brier, 3),\n",+ " \"ECE\": round(laya_ece, 3),\n",+ " \"Score MAE\": round(laya_mae, 3),\n",+ " \"ms/case\": round(laya_latency, 1),\n",+ " \"Cost/Case\": \"$0.00 (Self-Hosted)\"\n",+ " },\n",+ " {\n",+ " \"Model\": \"ModernBERT-base (149M)\",\n",+ " \"Kind\": \"specialist\",\n",+ " \"Accuracy\": 0.646,\n",+ " \"Soft Acc\": 0.542,\n",+ " \"Brier\": 0.119,\n",+ " \"ECE\": 0.179,\n",+ " \"Score MAE\": 0.444,\n",+ " \"ms/case\": 349,\n",+ " \"Cost/Case\": \"$0.00\"\n",+ " },\n",+ " {\n",+ " \"Model\": \"Teacher Self-Agreement\",\n",+ " \"Kind\": \"ceiling\",\n",+ " \"Accuracy\": 0.735,\n",+ " \"Soft Acc\": \"-\",\n",+ " \"Brier\": \"-\",\n",+ " \"ECE\": \"-\",\n",+ " \"Score MAE\": \"-\",\n",+ " \"ms/case\": \"-\",\n",+ " \"Cost/Case\": \"-\"\n",+ " }\n",+ "]\n",+ "\n",+ "df_comp = pd.DataFrame(comparison_data)\n",+ "print(\"=== HEAD-TO-HEAD BENCHMARK TABLE ===\\n\")\n",+ "print(df_comp.to_markdown(index=False))\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 8. Export Benchmark Report\n",+ "Save evaluation metrics as JSON and Markdown for reference.\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {},+ "source": [+ "report = {\n",+ " \"benchmark\": \"LocalLLaMA/typed-decisions\",\n",+ " \"model\": MODEL_ID,\n",+ " \"n_cases\": len(ds),\n",+ " \"n_decisions\": len(ds) * 5,\n",+ " \"metrics\": {\n",+ " \"accuracy\": float(laya_acc),\n",+ " \"soft_accuracy\": float(laya_soft_acc),\n",+ " \"brier_score\": float(laya_brier),\n",+ " \"ece\": float(laya_ece),\n",+ " \"score_mae\": float(laya_mae),\n",+ " \"kl_divergence\": float(laya_kl),\n",+ " \"total_variation\": float(laya_tv),\n",+ " \"latency_p50_ms\": float(laya_latency),\n",+ " \"latency_p95_ms\": float(np.percentile(latencies_ms, 95))\n",+ " }\n",+ "}\n",+ "\n",+ "with open(\"typed_decisions_eval_report.json\", \"w\") as f:\n",+ " json.dump(report, f, indent=2)\n",+ "\n",+ "print(\"Saved report to typed_decisions_eval_report.json!\")\n"+ ]+ }+ ],+ "metadata": {+ "accelerator": "GPU",+ "colab": {+ "name": "laya_eval_typed_decisions_colab.ipynb",+ "provenance": [],+ "gpuType": "T4"+ },+ "kernelspec": {+ "display_name": "Python 3",+ "name": "python3"+ },+ "language_info": {+ "name": "python"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 0+}\ No newline at end of file