Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat: add Colab evaluation notebook for LocalLLaMA/typed-decisions benchmark

Nandakishor479248998428
diff --git a/notebooks/laya_eval_typed_decisions_colab.ipynb b/notebooks/laya_eval_typed_decisions_colab.ipynb
new file mode 100644
index 00000000..c93f9e16
--- /dev/null
+++ b/notebooks/laya_eval_typed_decisions_colab.ipynb
@@ -0,0 +1,386 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Evaluating Laya on `LocalLLaMA/typed-decisions` Benchmark (Single T4 GPU)\n",
+ "\n",
+ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NandhaKishorM/laya/blob/main/notebooks/laya_eval_typed_decisions_colab.ipynb)\n",
+ "[![PyPI version](https://img.shields.io/pypi/v/laya.svg)](https://pypi.org/project/laya/)\n",
+ "[![Hugging Face Model](https://img.shields.io/badge/%F0%9F%A4%97%20Model-convaiinnovations%2Flaya-blue)](https://huggingface.co/convaiinnovations/laya)\n",
+ "[![Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-LocalLLaMA%2Ftyped--decisions-green)](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)\n",
+ "\n",
+ "This notebook evaluates **Laya** (`convaiinnovations/laya`) head-to-head against **TypeSafe Jev** on the independent [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) benchmark.\n",
+ "\n",
+ "### What is `typed-decisions`?\n",
+ "It is the standard public benchmark for System One decision models, covering 4 production workflows:\n",
+ "1. `agent_trace_observability`: Assess agent executions for human intervention and urgency.\n",
+ "2. `customer_service`: Route assistant responses and actions from multi-turn customer dialogues.\n",
+ "3. `invoice_processing`: Review vendor invoices against purchase orders and delivery receipts.\n",
+ "4. `security_incidents`: Determine containment, investigation, or dismissal of security alerts.\n",
+ "\n",
+ "Each test case contains **5 typed questions** over a shared state. The entire benchmark consists of **400 cases (2,000 typed decisions)**.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 1. GPU Check & Setup\n",
+ "Ensure a T4 GPU is selected in `Runtime -> Change runtime type -> T4 GPU`.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "source": [
+ "!nvidia-smi\n",
+ "import os, torch\n",
+ "assert torch.cuda.is_available(), \"No GPU detected! Set Runtime -> Change runtime type -> T4 GPU\"\n",
+ "props = torch.cuda.get_device_properties(0)\n",
+ "print(f\"Connected to: {props.name} | Total VRAM: {props.total_memory / 1e9:.1f} GB\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 2. Install Dependencies\n",
+ "Install `laya`, `transformers`, `datasets`, and evaluation libraries.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "source": [
+ "!pip install -q -U \"laya>=0.1.6\" \"transformers>=4.48.0\" \"datasets>=3.0.0\" safetensors huggingface_hub pyarrow pandas scipy\n",
+ "import laya, torch, datasets, pandas as pd\n",
+ "print(\"Laya version:\", laya.__version__)\n",
+ "print(\"PyTorch version:\", torch.__version__)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 3. Load Laya Model\n",
+ "Load the fine-tuned Laya model from Hugging Face on CUDA.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "source": [
+ "MODEL_ID = \"convaiinnovations/laya\"\n",
+ "print(f\"Loading Laya model from {MODEL_ID}...\")\n",
+ "\n",
+ "agent = laya.load(MODEL_ID, device=\"cuda\")\n",
+ "print(\"Model successfully loaded on GPU!\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 4. Load the `LocalLLaMA/typed-decisions` Benchmark\n",
+ "We load all 400 test cases across the four workflows.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "source": [
+ "from datasets import load_dataset\n",
+ "import json\n",
+ "\n",
+ "print(\"Downloading LocalLLaMA/typed-decisions benchmark (split: test)...\")\n",
+ "ds = load_dataset(\"LocalLLaMA/typed-decisions\", \"all\", split=\"test\")\n",
+ "\n",
+ "print(f\"Loaded {len(ds)} benchmark cases.\")\n",
+ "print(\"Sample workflow breakdown:\")\n",
+ "df_summary = pd.DataFrame(ds)\n",
+ "print(df_summary[\"workflow\"].value_counts())\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 5. Run Laya Evaluation Loop\n",
+ "Evaluate all 400 test cases (2,000 decisions) and measure per-case latency.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "source": [
+ "import time, math\n",
+ "import numpy as np\n",
+ "\n",
+ "predictions = []\n",
+ "latencies_ms = []\n",
+ "\n",
+ "print(\"Starting evaluation of 400 benchmark cases...\")\n",
+ "t_start = time.time()\n",
+ "\n",
+ "for i, row in enumerate(ds):\n",
+ " case_id = row[\"id\"]\n",
+ " workflow = row[\"workflow\"]\n",
+ " state = json.loads(row[\"state\"])\n",
+ " questions = json.loads(row[\"questions\"])\n",
+ " gold = json.loads(row[\"gold\"])\n",
+ " \n",
+ " t0 = time.perf_counter()\n",
+ " res = agent.predict(state, questions)\n",
+ " dt_ms = (time.perf_counter() - t0) * 1000\n",
+ " latencies_ms.append(dt_ms)\n",
+ " \n",
+ " predictions.append({\n",
+ " \"id\": case_id,\n",
+ " \"workflow\": workflow,\n",
+ " \"pred\": res[\"answers\"],\n",
+ " \"gold\": gold,\n",
+ " \"questions\": questions,\n",
+ " \"latency_ms\": dt_ms\n",
+ " })\n",
+ " \n",
+ " if (i + 1) % 50 == 0 or (i + 1) == len(ds):\n",
+ " print(f\" Progress: {i + 1}/{len(ds)} cases evaluated | Average latency: {np.mean(latencies_ms):.1f} ms/case\")\n",
+ "\n",
+ "total_time = time.time() - t_start\n",
+ "print(f\"\\nEvaluation complete in {total_time:.1f}s across {len(ds) * 5} decisions!\")\n",
+ "print(f\"P50 Latency: {np.percentile(latencies_ms, 50):.1f} ms/case | P95 Latency: {np.percentile(latencies_ms, 95):.1f} ms/case\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 6. Compute Official Benchmark Metrics\n",
+ "We calculate the exact metrics defined by `LocalLLaMA/typed-decisions`: Accuracy, Soft Accuracy, Brier Score, ECE, KL Divergence, Total Variation (TV), and Score MAE.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "source": [
+ "from laya.common import ece_score\n",
+ "\n",
+ "accuracies = []\n",
+ "soft_accuracies = []\n",
+ "brier_scores = []\n",
+ "kl_divs = []\n",
+ "tv_distances = []\n",
+ "score_maes = []\n",
+ "all_confs = []\n",
+ "all_corrects = []\n",
+ "\n",
+ "for item in predictions:\n",
+ " pred_answers = item[\"pred\"]\n",
+ " gold_answers = item[\"gold\"]\n",
+ " questions = item[\"questions\"]\n",
+ " \n",
+ " for qid, qdef in questions.items():\n",
+ " p_ans = pred_answers[qid]\n",
+ " g_ans = gold_answers[qid]\n",
+ " q_type = qdef[\"type\"]\n",
+ " \n",
+ " # 1. Choice / Multi-class\n",
+ " if q_type == \"choice\":\n",
+ " keys = list(qdef[\"criteria\"].keys())\n",
+ " pred_choice = p_ans[\"choice\"]\n",
+ " gold_label = g_ans[\"label\"]\n",
+ " \n",
+ " is_corr = float(pred_choice == gold_label)\n",
+ " accuracies.append(is_corr)\n",
+ " all_corrects.append(is_corr)\n",
+ " all_confs.append(p_ans[\"confidence\"])\n",
+ " \n",
+ " p_probs = np.array([p_ans[\"probabilities\"].get(k, 1e-6) for k in keys])\n",
+ " g_probs = np.array([g_ans[\"probabilities\"].get(k, 1e-6) for k in keys])\n",
+ " \n",
+ " p_probs /= p_probs.sum()\n",
+ " g_probs /= g_probs.sum()\n",
+ " \n",
+ " # Soft accuracy: sum(p_i * g_i)\n",
+ " soft_accuracies.append(float((p_probs * g_probs).sum()))\n",
+ " # Brier score: sum((p - g)^2)\n",
+ " brier_scores.append(float(((p_probs - g_probs) ** 2).sum()))\n",
+ " # Total Variation: 0.5 * sum(|p - g|)\n",
+ " tv_distances.append(float(0.5 * np.abs(p_probs - g_probs).sum()))\n",
+ " # KL divergence: sum(g * log(g / p))\n",
+ " kl_divs.append(float((g_probs * np.log(np.clip(g_probs / p_probs, 1e-12, 1e4))).sum()))\n",
+ " \n",
+ " # 2. Noul / Boolean\n",
+ " elif q_type == \"noul\":\n",
+ " p_val = p_ans[\"noul\"]\n",
+ " g_val = g_ans.get(\"noul\", g_ans.get(\"probabilities\", {}).get(\"true\", 0.5))\n",
+ " gold_label = g_ans[\"label\"]\n",
+ " \n",
+ " pred_label = \"true\" if p_val >= 0.5 else \"false\"\n",
+ " is_corr = float(pred_label == str(gold_label).lower())\n",
+ " accuracies.append(is_corr)\n",
+ " all_corrects.append(is_corr)\n",
+ " all_confs.append(max(p_val, 1.0 - p_val))\n",
+ " \n",
+ " p_dist = np.array([1.0 - p_val, p_val])\n",
+ " g_dist = np.array([1.0 - g_val, g_val])\n",
+ " \n",
+ " soft_accuracies.append(float((p_dist * g_dist).sum()))\n",
+ " brier_scores.append(float(((p_dist - g_dist) ** 2).sum()))\n",
+ " tv_distances.append(float(0.5 * np.abs(p_dist - g_dist).sum()))\n",
+ " kl_divs.append(float((g_dist * np.log(np.clip(g_dist / p_dist, 1e-12, 1e4))).sum()))\n",
+ " \n",
+ " # 3. Score / Ordinal Rubric\n",
+ " elif q_type == \"score\":\n",
+ " p_score = p_ans[\"score\"]\n",
+ " g_score = g_ans.get(\"score\", 0.0)\n",
+ " score_maes.append(abs(p_score - g_score))\n",
+ " \n",
+ " # Discrete level accuracy\n",
+ " p_lvl = int(round(p_score))\n",
+ " g_lvl = int(round(g_score))\n",
+ " is_corr = float(p_lvl == g_lvl)\n",
+ " accuracies.append(is_corr)\n",
+ " all_corrects.append(is_corr)\n",
+ " all_confs.append(p_ans.get(\"confidence\", 0.5))\n",
+ "\n",
+ "laya_acc = np.mean(accuracies)\n",
+ "laya_soft_acc = np.mean(soft_accuracies)\n",
+ "laya_brier = np.mean(brier_scores)\n",
+ "laya_kl = np.mean(kl_divs)\n",
+ "laya_tv = np.mean(tv_distances)\n",
+ "laya_ece = ece_score(np.array(all_confs), np.array(all_corrects))\n",
+ "laya_mae = np.mean(score_maes) if score_maes else 0.0\n",
+ "laya_latency = np.percentile(latencies_ms, 50)\n",
+ "\n",
+ "print(\"--- Laya Benchmark Results on typed-decisions ---\")\n",
+ "print(f\"Accuracy : {laya_acc:.3f}\")\n",
+ "print(f\"Soft Accuracy : {laya_soft_acc:.3f}\")\n",
+ "print(f\"Brier Score : {laya_brier:.3f}\")\n",
+ "print(f\"ECE : {laya_ece:.3f}\")\n",
+ "print(f\"Score MAE : {laya_mae:.3f}\")\n",
+ "print(f\"KL Divergence : {laya_kl:.3f}\")\n",
+ "print(f\"Latency (p50) : {laya_latency:.1f} ms/case\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 7. Head-to-Head Comparison with TypeSafe Jev\n",
+ "Compare Laya directly against the baseline numbers published on `LocalLLaMA/typed-decisions`.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "source": [
+ "comparison_data = [\n",
+ " {\n",
+ " \"Model\": \"TypeSafe Jev 1.13.0\",\n",
+ " \"Kind\": \"general\",\n",
+ " \"Accuracy\": 0.727,\n",
+ " \"Soft Acc\": 0.580,\n",
+ " \"Brier\": 0.148,\n",
+ " \"ECE\": 0.144,\n",
+ " \"Score MAE\": 0.391,\n",
+ " \"ms/case\": 710,\n",
+ " \"Cost/Case\": \"$0.0004 (API)\"\n",
+ " },\n",
+ " {\n",
+ " \"Model\": \"Laya (Ours)\",\n",
+ " \"Kind\": \"general\",\n",
+ " \"Accuracy\": round(laya_acc, 3),\n",
+ " \"Soft Acc\": round(laya_soft_acc, 3),\n",
+ " \"Brier\": round(laya_brier, 3),\n",
+ " \"ECE\": round(laya_ece, 3),\n",
+ " \"Score MAE\": round(laya_mae, 3),\n",
+ " \"ms/case\": round(laya_latency, 1),\n",
+ " \"Cost/Case\": \"$0.00 (Self-Hosted)\"\n",
+ " },\n",
+ " {\n",
+ " \"Model\": \"ModernBERT-base (149M)\",\n",
+ " \"Kind\": \"specialist\",\n",
+ " \"Accuracy\": 0.646,\n",
+ " \"Soft Acc\": 0.542,\n",
+ " \"Brier\": 0.119,\n",
+ " \"ECE\": 0.179,\n",
+ " \"Score MAE\": 0.444,\n",
+ " \"ms/case\": 349,\n",
+ " \"Cost/Case\": \"$0.00\"\n",
+ " },\n",
+ " {\n",
+ " \"Model\": \"Teacher Self-Agreement\",\n",
+ " \"Kind\": \"ceiling\",\n",
+ " \"Accuracy\": 0.735,\n",
+ " \"Soft Acc\": \"-\",\n",
+ " \"Brier\": \"-\",\n",
+ " \"ECE\": \"-\",\n",
+ " \"Score MAE\": \"-\",\n",
+ " \"ms/case\": \"-\",\n",
+ " \"Cost/Case\": \"-\"\n",
+ " }\n",
+ "]\n",
+ "\n",
+ "df_comp = pd.DataFrame(comparison_data)\n",
+ "print(\"=== HEAD-TO-HEAD BENCHMARK TABLE ===\\n\")\n",
+ "print(df_comp.to_markdown(index=False))\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 8. Export Benchmark Report\n",
+ "Save evaluation metrics as JSON and Markdown for reference.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "source": [
+ "report = {\n",
+ " \"benchmark\": \"LocalLLaMA/typed-decisions\",\n",
+ " \"model\": MODEL_ID,\n",
+ " \"n_cases\": len(ds),\n",
+ " \"n_decisions\": len(ds) * 5,\n",
+ " \"metrics\": {\n",
+ " \"accuracy\": float(laya_acc),\n",
+ " \"soft_accuracy\": float(laya_soft_acc),\n",
+ " \"brier_score\": float(laya_brier),\n",
+ " \"ece\": float(laya_ece),\n",
+ " \"score_mae\": float(laya_mae),\n",
+ " \"kl_divergence\": float(laya_kl),\n",
+ " \"total_variation\": float(laya_tv),\n",
+ " \"latency_p50_ms\": float(laya_latency),\n",
+ " \"latency_p95_ms\": float(np.percentile(latencies_ms, 95))\n",
+ " }\n",
+ "}\n",
+ "\n",
+ "with open(\"typed_decisions_eval_report.json\", \"w\") as f:\n",
+ " json.dump(report, f, indent=2)\n",
+ "\n",
+ "print(\"Saved report to typed_decisions_eval_report.json!\")\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "name": "laya_eval_typed_decisions_colab.ipynb",
+ "provenance": [],
+ "gpuType": "T4"
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file