Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
test(serve): record repeated MPS unload and shared-MCP measurements
Rishit Sharmabbaec3a16b1d
diff --git a/research/benchmarks/idle_unload/README.md b/research/benchmarks/idle_unload/README.mdnew file mode 100644index 00000000..ac391098--- /dev/null+++ b/research/benchmarks/idle_unload/README.md@@ -0,0 +1,63 @@+# Idle unload and shared-server MCP measurements++This script uses a checkpoint already on disk, sets `HF_HUB_OFFLINE=1`, and runs real MPS+inference. It needs the `serve` and `mcp` extras on an Apple Silicon Mac. Memory is in MiB;+process RSS comes from `ps`, Metal values from torch. Run every mode in a fresh process.++Set `CHECKPOINT` to a local English checkpoint containing `model.safetensors`, `encoder/`, and+`tokenizer/`. The default idle window in these measurements is two seconds so the check is quick.++```bash+python research/benchmarks/idle_unload/benchmark.py --checkpoint "$CHECKPOINT" --mode idle-off+python research/benchmarks/idle_unload/benchmark.py --checkpoint "$CHECKPOINT" --mode idle-on+python research/benchmarks/idle_unload/benchmark.py --checkpoint "$CHECKPOINT" --mode mcp-local+```++For the remote measurement, run this server in another terminal, then run the client:++```bash+python research/benchmarks/idle_unload/benchmark.py --checkpoint "$CHECKPOINT" --mode serve --port 8000+```++```bash+python research/benchmarks/idle_unload/benchmark.py --checkpoint "$CHECKPOINT" --mode mcp-remote --base-url http://127.0.0.1:8000+```++Repeat the four measurements twice. The idle modes assert that residency follows the setting,+that the next request reloads successfully, and that answers match before and after unloading.+The MCP modes exercise status, routing, prediction, batch prediction, structured decisions and+presets; remote mode also checks that shortlisting refuses the call and torch stays unimported.+Compare answer maps and decided values across local and remote runs, excluding timing and device+metadata. Remote-client RSS excludes the one shared HTTP server, which still holds the model.++The baseline is the feature disabled on the same implementation and checkpoint. Results do not+claim that idle unloading returns all process RAM to the OS: allocators can retain pages even+after Metal allocations and model references are freed.++## Recorded results++`results.json` records two fresh-process runs on an Apple M4 Pro, Python 3.12, using the English+checkpoint at revision `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`. Neither run downloaded weights.++| Measurement | Run 1 | Run 2 |+|---|---:|---:|+| Metal allocations, idle unloading disabled | 1610.387 MiB retained | 1610.387 MiB retained |+| Metal allocations, idle unloading enabled | 1610.387 → 0 MiB | 1610.387 → 0 MiB |+| Metal driver memory, idle unloading enabled | 2056.703 → 0.703 MiB | 2056.703 → 0.703 MiB |+| Warm request, idle unloading enabled | 188.836 ms | 193.897 ms |+| Request after idle unload | 727.361 ms | 836.043 ms |+| Local MCP process RSS after the tool sweep | 1074.453 MiB | 308.516 MiB |+| Remote MCP process RSS after the tool sweep | 71.453 MiB | 71.672 MiB |+| Torch imported by remote MCP | no | no |+| Local/remote answer parity | yes | yes |++Server RSS did not shrink consistently after unloading: it moved from 419.984 to 492.312 MiB in+the first enabled run and from 462.406 to 488.719 MiB in the second. Local MCP RSS also varied+substantially between processes. These are observed process snapshots, not a claim of a stable+percentage reduction or a total-system-memory measurement. Remote MCP RSS excludes the shared+server's model memory. The idle allocation reduction and answer parity reproduced in both runs.++`validation.json` records two complete runs of the shared CI/release suite list, the required lint+and compile gates, and strict documentation builds with no `griffe:` diagnostics. Run the same+checks with `python scripts/test_suites.py`, the commands in `AGENTS.md`, and+`zensical build --strict --clean`.diff --git a/research/benchmarks/idle_unload/benchmark.py b/research/benchmarks/idle_unload/benchmark.pynew file mode 100644index 00000000..b33d689b--- /dev/null+++ b/research/benchmarks/idle_unload/benchmark.py@@ -0,0 +1,153 @@+"""Measure #888 with a local checkpoint; never download weights.++Run each mode in a fresh process. See README.md beside this file for commands.+"""+import argparse+import json+import os+import subprocess+import sys+import time+from pathlib import Path++ROOT = Path(__file__).resolve().parents[3]+sys.path.insert(0, str(ROOT))+os.environ.setdefault("USE_TF", "0")+os.environ.setdefault("USE_TORCH", "1")+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")+os.environ["HF_HUB_OFFLINE"] = "1"++STATE = {"message": "I was billed twice. Please refund the duplicate payment."}+QUESTIONS = {"dept": {"type": "choice", "instructions": "Which team handles `message`?",+ "criteria": {"billing": "payments and refunds", "technical": "bugs and outages"}}}+SCHEMA = {"type": "object", "properties": {"dept": {"type": "string", "enum": ["billing", "technical"]}}}+++def rss_mb():+ value = subprocess.check_output(["ps", "-o", "rss=", "-p", str(os.getpid())], text=True)+ return round(int(value.strip()) / 1024, 3)+++def memory():+ import torch+ torch.mps.synchronize()+ return {"rss_mb": rss_mb(),+ "metal_mb": round(torch.mps.current_allocated_memory() / 1024**2, 3),+ "metal_driver_mb": round(torch.mps.driver_allocated_memory() / 1024**2, 3)}+++def make_router(checkpoint):+ from laya.router import Router+ router = Router(models={"english": str(checkpoint)}, device="mps")+ router.preload(["english"])+ return router+++def idle_benchmark(args):+ from fastapi.testclient import TestClient+ from laya.serve import create_app++ enabled = args.mode == "idle-on"+ os.environ["LAYA_IDLE_UNLOAD_SECONDS"] = str(args.idle_seconds if enabled else 0)+ started = time.perf_counter()+ router = make_router(args.checkpoint)+ load_seconds = time.perf_counter() - started+ with TestClient(create_app(router)) as client:+ body = {"state": STATE, "questions": QUESTIONS}++ def predict():+ start = time.perf_counter()+ response = client.post("/v1/systemone", json=body)+ response.raise_for_status()+ return response.json()["answers"], (time.perf_counter() - start) * 1000++ predict() # warm up Metal kernels+ answers, warm_ms = predict()+ resident = memory()+ if enabled:+ deadline = time.monotonic() + args.idle_seconds + 30+ while client.get("/health").json()["loaded"]:+ assert time.monotonic() < deadline, "idle unload did not complete"+ time.sleep(0.05)+ else:+ time.sleep(args.idle_seconds + 0.5)+ after_idle = memory()+ assert bool(router.loaded) != enabled+ reloaded_answers, next_ms = predict()+ assert answers == reloaded_answers, "reload changed the answer"+ assert router.loaded == ["english"]+ result = {"mode": args.mode, "load_seconds": round(load_seconds, 3),+ "warm_ms": round(warm_ms, 3), "next_request_ms": round(next_ms, 3),+ "resident": resident, "after_idle": after_idle, "after_next_request": memory(),+ "answers": answers, "answer_parity": True}+ router.unload()+ return result+++def mcp_benchmark(args):+ if args.mode == "mcp-remote":+ assert args.base_url, "--base-url is required in remote mode"+ os.environ["LAYA_BASE_URL"] = args.base_url+ else:+ os.environ.pop("LAYA_BASE_URL", None)+ from laya.mcp import server++ imported_rss = rss_mb()+ if args.mode == "mcp-local":+ server._ROUTER = make_router(args.checkpoint)+ outputs = {}+ outputs["status"] = json.loads(server.laya_status_tool())+ outputs["predict"] = json.loads(server.laya_predict_tool(STATE, QUESTIONS))+ items = [{"state": STATE, "questions": QUESTIONS}]+ outputs["predict_batch"] = json.loads(server.laya_predict_batch_tool(items))+ outputs["decide"] = json.loads(server.laya_decide_tool(STATE, SCHEMA))+ outputs["preset"] = json.loads(server.laya_preset_tool("triage", STATE))+ outputs["route"] = json.loads(server.laya_route_tool(STATE, QUESTIONS))+ outputs["route_batch"] = json.loads(server.laya_route_batch_tool(items))+ assert all("error" not in value for value in outputs.values()), outputs+ loaded = server._ensure_router().loaded+ result = {"mode": args.mode, "imported_rss_mb": imported_rss, "after_tools_rss_mb": rss_mb(),+ "torch_imported": "torch" in sys.modules, "loaded": loaded,+ "answers": outputs["predict"]["answers"],+ "batch_answers": outputs["predict_batch"]["requests"][0]["answers"],+ "decided_values": outputs["decide"]["values"], "preset_answers": outputs["preset"]["answers"]}+ assert result["torch_imported"] == (args.mode == "mcp-local")+ if args.mode == "mcp-local":+ server._ROUTER.unload()+ else:+ try:+ server.laya_shortlist_tool(STATE, QUESTIONS)+ except Exception as exc:+ assert "unsupported_remote" in str(exc), exc+ else:+ raise AssertionError("remote shortlist did not refuse the call")+ return result+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--checkpoint", type=Path, required=True)+ parser.add_argument("--mode", choices=["idle-off", "idle-on", "mcp-local", "mcp-remote", "serve"], required=True)+ parser.add_argument("--base-url")+ parser.add_argument("--port", type=int, default=8000)+ parser.add_argument("--idle-seconds", type=float, default=2.0)+ args = parser.parse_args()+ assert (args.checkpoint / "model.safetensors").is_file(), "local checkpoint is incomplete"+ os.environ.pop("LAYA_API_KEY", None)+ if args.mode == "serve":+ import uvicorn+ from laya.serve import create_app+ os.environ["LAYA_IDLE_UNLOAD_SECONDS"] = "0"+ router = make_router(args.checkpoint)+ try:+ uvicorn.run(create_app(router), host="127.0.0.1", port=args.port, log_level="warning")+ finally:+ router.unload()+ return+ result = idle_benchmark(args) if args.mode.startswith("idle-") else mcp_benchmark(args)+ result["checkpoint_revision"] = args.checkpoint.name+ print(json.dumps(result, sort_keys=True))+++if __name__ == "__main__":+ main()diff --git a/research/benchmarks/idle_unload/results.json b/research/benchmarks/idle_unload/results.jsonnew file mode 100644index 00000000..938cb27c--- /dev/null+++ b/research/benchmarks/idle_unload/results.json@@ -0,0 +1,619 @@+{+ "baseline": "same implementation with idle unload disabled; local MCP versus shared-server MCP",+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",+ "device": "mps",+ "hardware": "Apple M4 Pro",+ "implementation_commit": "c8ddefa4836a463f54d46accd19adfeefff82bbc",+ "memory_unit": "MiB",+ "packages": {+ "fastapi": "0.142.2",+ "mcp": "2.3.0",+ "torch": "2.14.1",+ "transformers": "5.18.0"+ },+ "platform": "macOS-27.0.1-arm64-arm-64bit",+ "python": "3.12.12",+ "runs": [+ {+ "idle-off": {+ "after_idle": {+ "metal_driver_mb": 2056.703,+ "metal_mb": 1610.387,+ "rss_mb": 307.984+ },+ "after_next_request": {+ "metal_driver_mb": 2056.703,+ "metal_mb": 1610.387,+ "rss_mb": 308.109+ },+ "answer_parity": true,+ "answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",+ "load_seconds": 3.461,+ "mode": "idle-off",+ "next_request_ms": 213.075,+ "resident": {+ "metal_driver_mb": 2056.703,+ "metal_mb": 1610.387,+ "rss_mb": 307.969+ },+ "warm_ms": 190.518+ },+ "idle-on": {+ "after_idle": {+ "metal_driver_mb": 0.703,+ "metal_mb": 0.0,+ "rss_mb": 492.312+ },+ "after_next_request": {+ "metal_driver_mb": 2056.703,+ "metal_mb": 1610.387,+ "rss_mb": 1159.766+ },+ "answer_parity": true,+ "answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",+ "load_seconds": 2.918,+ "mode": "idle-on",+ "next_request_ms": 727.361,+ "resident": {+ "metal_driver_mb": 2056.703,+ "metal_mb": 1610.387,+ "rss_mb": 419.984+ },+ "warm_ms": 188.836+ },+ "local_remote_answer_parity": true,+ "mcp-local": {+ "after_tools_rss_mb": 1074.453,+ "answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "batch_answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",+ "decided_values": {+ "dept": "billing"+ },+ "imported_rss_mb": 67.688,+ "loaded": [+ "english"+ ],+ "mode": "mcp-local",+ "preset_answers": {+ "churn_risk": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9997,+ "confidence": 0.9997,+ "noul": 0.0003,+ "type": "noul"+ },+ "frustration": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.5295,+ "confidence": 0.1579,+ "legend": {+ "0": "calm and neutral",+ "1": "concerned but civil",+ "2": "clearly annoyed",+ "3": "very angry or using strong language"+ },+ "probabilities": {+ "0": 0.114,+ "1": 0.2523,+ "2": 0.5295,+ "3": 0.1042+ },+ "score": 1.6239,+ "type": "score"+ },+ "intent": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9996,+ "choice": "refund",+ "confidence": 0.9978,+ "probabilities": {+ "billing_question": 0.0002,+ "cancellation": 0.0,+ "information": 0.0,+ "other": 0.0001,+ "refund": 0.9996,+ "technical_help": 0.0+ },+ "type": "choice"+ },+ "is_urgent": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.8974,+ "confidence": 0.8974,+ "noul": 0.1026,+ "type": "noul"+ },+ "refund_requested": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.8634,+ "confidence": 0.8634,+ "noul": 0.8634,+ "type": "noul"+ }+ },+ "torch_imported": true+ },+ "mcp-remote": {+ "after_tools_rss_mb": 71.453,+ "answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "batch_answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",+ "decided_values": {+ "dept": "billing"+ },+ "imported_rss_mb": 67.984,+ "loaded": [+ "english"+ ],+ "mode": "mcp-remote",+ "preset_answers": {+ "churn_risk": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9997,+ "confidence": 0.9997,+ "noul": 0.0003,+ "type": "noul"+ },+ "frustration": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.5295,+ "confidence": 0.1579,+ "legend": {+ "0": "calm and neutral",+ "1": "concerned but civil",+ "2": "clearly annoyed",+ "3": "very angry or using strong language"+ },+ "probabilities": {+ "0": 0.114,+ "1": 0.2523,+ "2": 0.5295,+ "3": 0.1042+ },+ "score": 1.6239,+ "type": "score"+ },+ "intent": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9996,+ "choice": "refund",+ "confidence": 0.9978,+ "probabilities": {+ "billing_question": 0.0002,+ "cancellation": 0.0,+ "information": 0.0,+ "other": 0.0001,+ "refund": 0.9996,+ "technical_help": 0.0+ },+ "type": "choice"+ },+ "is_urgent": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.8974,+ "confidence": 0.8974,+ "noul": 0.1026,+ "type": "noul"+ },+ "refund_requested": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.8634,+ "confidence": 0.8634,+ "noul": 0.8634,+ "type": "noul"+ }+ },+ "torch_imported": false+ },+ "repeat": 1+ },+ {+ "idle-off": {+ "after_idle": {+ "metal_driver_mb": 2056.703,+ "metal_mb": 1610.387,+ "rss_mb": 408.672+ },+ "after_next_request": {+ "metal_driver_mb": 2056.703,+ "metal_mb": 1610.387,+ "rss_mb": 408.703+ },+ "answer_parity": true,+ "answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",+ "load_seconds": 3.322,+ "mode": "idle-off",+ "next_request_ms": 168.304,+ "resident": {+ "metal_driver_mb": 2056.703,+ "metal_mb": 1610.387,+ "rss_mb": 408.672+ },+ "warm_ms": 98.172+ },+ "idle-on": {+ "after_idle": {+ "metal_driver_mb": 0.703,+ "metal_mb": 0.0,+ "rss_mb": 488.719+ },+ "after_next_request": {+ "metal_driver_mb": 2056.703,+ "metal_mb": 1610.387,+ "rss_mb": 1167.906+ },+ "answer_parity": true,+ "answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",+ "load_seconds": 3.072,+ "mode": "idle-on",+ "next_request_ms": 836.043,+ "resident": {+ "metal_driver_mb": 2056.703,+ "metal_mb": 1610.387,+ "rss_mb": 462.406+ },+ "warm_ms": 193.897+ },+ "local_remote_answer_parity": true,+ "mcp-local": {+ "after_tools_rss_mb": 308.516,+ "answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "batch_answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",+ "decided_values": {+ "dept": "billing"+ },+ "imported_rss_mb": 67.719,+ "loaded": [+ "english"+ ],+ "mode": "mcp-local",+ "preset_answers": {+ "churn_risk": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9997,+ "confidence": 0.9997,+ "noul": 0.0003,+ "type": "noul"+ },+ "frustration": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.5295,+ "confidence": 0.1579,+ "legend": {+ "0": "calm and neutral",+ "1": "concerned but civil",+ "2": "clearly annoyed",+ "3": "very angry or using strong language"+ },+ "probabilities": {+ "0": 0.114,+ "1": 0.2523,+ "2": 0.5295,+ "3": 0.1042+ },+ "score": 1.6239,+ "type": "score"+ },+ "intent": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9996,+ "choice": "refund",+ "confidence": 0.9978,+ "probabilities": {+ "billing_question": 0.0002,+ "cancellation": 0.0,+ "information": 0.0,+ "other": 0.0001,+ "refund": 0.9996,+ "technical_help": 0.0+ },+ "type": "choice"+ },+ "is_urgent": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.8974,+ "confidence": 0.8974,+ "noul": 0.1026,+ "type": "noul"+ },+ "refund_requested": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.8634,+ "confidence": 0.8634,+ "noul": 0.8634,+ "type": "noul"+ }+ },+ "torch_imported": true+ },+ "mcp-remote": {+ "after_tools_rss_mb": 71.672,+ "answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "batch_answers": {+ "dept": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9855,+ "choice": "billing",+ "confidence": 0.8908,+ "probabilities": {+ "billing": 0.9855,+ "technical": 0.0145+ },+ "type": "choice"+ }+ },+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",+ "decided_values": {+ "dept": "billing"+ },+ "imported_rss_mb": 68.094,+ "loaded": [+ "english"+ ],+ "mode": "mcp-remote",+ "preset_answers": {+ "churn_risk": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9997,+ "confidence": 0.9997,+ "noul": 0.0003,+ "type": "noul"+ },+ "frustration": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.5295,+ "confidence": 0.1579,+ "legend": {+ "0": "calm and neutral",+ "1": "concerned but civil",+ "2": "clearly annoyed",+ "3": "very angry or using strong language"+ },+ "probabilities": {+ "0": 0.114,+ "1": 0.2523,+ "2": 0.5295,+ "3": 0.1042+ },+ "score": 1.6239,+ "type": "score"+ },+ "intent": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.9996,+ "choice": "refund",+ "confidence": 0.9978,+ "probabilities": {+ "billing_question": 0.0002,+ "cancellation": 0.0,+ "information": 0.0,+ "other": 0.0001,+ "refund": 0.9996,+ "technical_help": 0.0+ },+ "type": "choice"+ },+ "is_urgent": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.8974,+ "confidence": 0.8974,+ "noul": 0.1026,+ "type": "noul"+ },+ "refund_requested": {+ "action": {+ "act_probability": 1.0+ },+ "answer_confidence": 0.8634,+ "confidence": 0.8634,+ "noul": 0.8634,+ "type": "noul"+ }+ },+ "torch_imported": false+ },+ "repeat": 2+ }+ ],+ "timestamp_utc": "2026-10-03T07:57:57.606565+00:00"+}diff --git a/research/benchmarks/idle_unload/validation.json b/research/benchmarks/idle_unload/validation.jsonnew file mode 100644index 00000000..dc83a63c--- /dev/null+++ b/research/benchmarks/idle_unload/validation.json@@ -0,0 +1,96 @@+[+ {+ "repeat": 1,+ "checks": [+ {+ "name": "lint",+ "command": ".venv/bin/ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120",+ "exit_code": 0,+ "seconds": 0.01,+ "summary": "All checks passed!"+ },+ {+ "name": "new-files-lint",+ "command": ".venv/bin/ruff check tests/test_mcp_remote.py research/benchmarks/idle_unload/benchmark.py --select=E9,F63,F7,F82,F401,F811 --line-length=120",+ "exit_code": 0,+ "seconds": 0.009,+ "summary": "All checks passed!"+ },+ {+ "name": "compile",+ "command": ".venv/bin/python -m compileall -q laya/ tests/ research/benchmarks/idle_unload/",+ "exit_code": 0,+ "seconds": 0.034,+ "summary": ""+ },+ {+ "name": "suites",+ "command": ".venv/bin/python scripts/test_suites.py",+ "exit_code": 0,+ "seconds": 81.638,+ "summary": "360 passed, 1 skipped, 5 warnings in 13.63s\n\nall 78 suites passed"+ },+ {+ "name": "docs",+ "command": ".venv/bin/zensical build --strict --clean",+ "exit_code": 0,+ "seconds": 1.656,+ "summary": "Build started\nNo issues found\nBuild finished in 1.41s"+ },+ {+ "name": "diff",+ "command": "git diff --check",+ "exit_code": 0,+ "seconds": 0.011,+ "summary": ""+ }+ ]+ },+ {+ "repeat": 2,+ "checks": [+ {+ "name": "lint",+ "command": ".venv/bin/ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120",+ "exit_code": 0,+ "seconds": 0.015,+ "summary": "All checks passed!"+ },+ {+ "name": "new-files-lint",+ "command": ".venv/bin/ruff check tests/test_mcp_remote.py research/benchmarks/idle_unload/benchmark.py --select=E9,F63,F7,F82,F401,F811 --line-length=120",+ "exit_code": 0,+ "seconds": 0.01,+ "summary": "All checks passed!"+ },+ {+ "name": "compile",+ "command": ".venv/bin/python -m compileall -q laya/ tests/ research/benchmarks/idle_unload/",+ "exit_code": 0,+ "seconds": 0.038,+ "summary": ""+ },+ {+ "name": "suites",+ "command": ".venv/bin/python scripts/test_suites.py",+ "exit_code": 0,+ "seconds": 77.435,+ "summary": "360 passed, 1 skipped, 5 warnings in 13.74s\n\nall 78 suites passed"+ },+ {+ "name": "docs",+ "command": ".venv/bin/zensical build --strict --clean",+ "exit_code": 0,+ "seconds": 1.54,+ "summary": "Build started\nNo issues found\nBuild finished in 1.37s"+ },+ {+ "name": "diff",+ "command": "git diff --check",+ "exit_code": 0,+ "seconds": 0.011,+ "summary": ""+ }+ ]+ }+]