Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

test(serve): record repeated MPS unload and shared-MCP measurements

Rishit Sharmabbaec3a16b1d
diff --git a/research/benchmarks/idle_unload/README.md b/research/benchmarks/idle_unload/README.md
new file mode 100644
index 00000000..ac391098
--- /dev/null
+++ b/research/benchmarks/idle_unload/README.md
@@ -0,0 +1,63 @@
+# Idle unload and shared-server MCP measurements
+
+This script uses a checkpoint already on disk, sets `HF_HUB_OFFLINE=1`, and runs real MPS
+inference. It needs the `serve` and `mcp` extras on an Apple Silicon Mac. Memory is in MiB;
+process RSS comes from `ps`, Metal values from torch. Run every mode in a fresh process.
+
+Set `CHECKPOINT` to a local English checkpoint containing `model.safetensors`, `encoder/`, and
+`tokenizer/`. The default idle window in these measurements is two seconds so the check is quick.
+
+```bash
+python research/benchmarks/idle_unload/benchmark.py --checkpoint "$CHECKPOINT" --mode idle-off
+python research/benchmarks/idle_unload/benchmark.py --checkpoint "$CHECKPOINT" --mode idle-on
+python research/benchmarks/idle_unload/benchmark.py --checkpoint "$CHECKPOINT" --mode mcp-local
+```
+
+For the remote measurement, run this server in another terminal, then run the client:
+
+```bash
+python research/benchmarks/idle_unload/benchmark.py --checkpoint "$CHECKPOINT" --mode serve --port 8000
+```
+
+```bash
+python research/benchmarks/idle_unload/benchmark.py --checkpoint "$CHECKPOINT" --mode mcp-remote --base-url http://127.0.0.1:8000
+```
+
+Repeat the four measurements twice. The idle modes assert that residency follows the setting,
+that the next request reloads successfully, and that answers match before and after unloading.
+The MCP modes exercise status, routing, prediction, batch prediction, structured decisions and
+presets; remote mode also checks that shortlisting refuses the call and torch stays unimported.
+Compare answer maps and decided values across local and remote runs, excluding timing and device
+metadata. Remote-client RSS excludes the one shared HTTP server, which still holds the model.
+
+The baseline is the feature disabled on the same implementation and checkpoint. Results do not
+claim that idle unloading returns all process RAM to the OS: allocators can retain pages even
+after Metal allocations and model references are freed.
+
+## Recorded results
+
+`results.json` records two fresh-process runs on an Apple M4 Pro, Python 3.12, using the English
+checkpoint at revision `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`. Neither run downloaded weights.
+
+| Measurement | Run 1 | Run 2 |
+|---|---:|---:|
+| Metal allocations, idle unloading disabled | 1610.387 MiB retained | 1610.387 MiB retained |
+| Metal allocations, idle unloading enabled | 1610.387 → 0 MiB | 1610.387 → 0 MiB |
+| Metal driver memory, idle unloading enabled | 2056.703 → 0.703 MiB | 2056.703 → 0.703 MiB |
+| Warm request, idle unloading enabled | 188.836 ms | 193.897 ms |
+| Request after idle unload | 727.361 ms | 836.043 ms |
+| Local MCP process RSS after the tool sweep | 1074.453 MiB | 308.516 MiB |
+| Remote MCP process RSS after the tool sweep | 71.453 MiB | 71.672 MiB |
+| Torch imported by remote MCP | no | no |
+| Local/remote answer parity | yes | yes |
+
+Server RSS did not shrink consistently after unloading: it moved from 419.984 to 492.312 MiB in
+the first enabled run and from 462.406 to 488.719 MiB in the second. Local MCP RSS also varied
+substantially between processes. These are observed process snapshots, not a claim of a stable
+percentage reduction or a total-system-memory measurement. Remote MCP RSS excludes the shared
+server's model memory. The idle allocation reduction and answer parity reproduced in both runs.
+
+`validation.json` records two complete runs of the shared CI/release suite list, the required lint
+and compile gates, and strict documentation builds with no `griffe:` diagnostics. Run the same
+checks with `python scripts/test_suites.py`, the commands in `AGENTS.md`, and
+`zensical build --strict --clean`.
diff --git a/research/benchmarks/idle_unload/benchmark.py b/research/benchmarks/idle_unload/benchmark.py
new file mode 100644
index 00000000..b33d689b
--- /dev/null
+++ b/research/benchmarks/idle_unload/benchmark.py
@@ -0,0 +1,153 @@
+"""Measure #888 with a local checkpoint; never download weights.
+
+Run each mode in a fresh process. See README.md beside this file for commands.
+"""
+import argparse
+import json
+import os
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[3]
+sys.path.insert(0, str(ROOT))
+os.environ.setdefault("USE_TF", "0")
+os.environ.setdefault("USE_TORCH", "1")
+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+os.environ["HF_HUB_OFFLINE"] = "1"
+
+STATE = {"message": "I was billed twice. Please refund the duplicate payment."}
+QUESTIONS = {"dept": {"type": "choice", "instructions": "Which team handles `message`?",
+ "criteria": {"billing": "payments and refunds", "technical": "bugs and outages"}}}
+SCHEMA = {"type": "object", "properties": {"dept": {"type": "string", "enum": ["billing", "technical"]}}}
+
+
+def rss_mb():
+ value = subprocess.check_output(["ps", "-o", "rss=", "-p", str(os.getpid())], text=True)
+ return round(int(value.strip()) / 1024, 3)
+
+
+def memory():
+ import torch
+ torch.mps.synchronize()
+ return {"rss_mb": rss_mb(),
+ "metal_mb": round(torch.mps.current_allocated_memory() / 1024**2, 3),
+ "metal_driver_mb": round(torch.mps.driver_allocated_memory() / 1024**2, 3)}
+
+
+def make_router(checkpoint):
+ from laya.router import Router
+ router = Router(models={"english": str(checkpoint)}, device="mps")
+ router.preload(["english"])
+ return router
+
+
+def idle_benchmark(args):
+ from fastapi.testclient import TestClient
+ from laya.serve import create_app
+
+ enabled = args.mode == "idle-on"
+ os.environ["LAYA_IDLE_UNLOAD_SECONDS"] = str(args.idle_seconds if enabled else 0)
+ started = time.perf_counter()
+ router = make_router(args.checkpoint)
+ load_seconds = time.perf_counter() - started
+ with TestClient(create_app(router)) as client:
+ body = {"state": STATE, "questions": QUESTIONS}
+
+ def predict():
+ start = time.perf_counter()
+ response = client.post("/v1/systemone", json=body)
+ response.raise_for_status()
+ return response.json()["answers"], (time.perf_counter() - start) * 1000
+
+ predict() # warm up Metal kernels
+ answers, warm_ms = predict()
+ resident = memory()
+ if enabled:
+ deadline = time.monotonic() + args.idle_seconds + 30
+ while client.get("/health").json()["loaded"]:
+ assert time.monotonic() < deadline, "idle unload did not complete"
+ time.sleep(0.05)
+ else:
+ time.sleep(args.idle_seconds + 0.5)
+ after_idle = memory()
+ assert bool(router.loaded) != enabled
+ reloaded_answers, next_ms = predict()
+ assert answers == reloaded_answers, "reload changed the answer"
+ assert router.loaded == ["english"]
+ result = {"mode": args.mode, "load_seconds": round(load_seconds, 3),
+ "warm_ms": round(warm_ms, 3), "next_request_ms": round(next_ms, 3),
+ "resident": resident, "after_idle": after_idle, "after_next_request": memory(),
+ "answers": answers, "answer_parity": True}
+ router.unload()
+ return result
+
+
+def mcp_benchmark(args):
+ if args.mode == "mcp-remote":
+ assert args.base_url, "--base-url is required in remote mode"
+ os.environ["LAYA_BASE_URL"] = args.base_url
+ else:
+ os.environ.pop("LAYA_BASE_URL", None)
+ from laya.mcp import server
+
+ imported_rss = rss_mb()
+ if args.mode == "mcp-local":
+ server._ROUTER = make_router(args.checkpoint)
+ outputs = {}
+ outputs["status"] = json.loads(server.laya_status_tool())
+ outputs["predict"] = json.loads(server.laya_predict_tool(STATE, QUESTIONS))
+ items = [{"state": STATE, "questions": QUESTIONS}]
+ outputs["predict_batch"] = json.loads(server.laya_predict_batch_tool(items))
+ outputs["decide"] = json.loads(server.laya_decide_tool(STATE, SCHEMA))
+ outputs["preset"] = json.loads(server.laya_preset_tool("triage", STATE))
+ outputs["route"] = json.loads(server.laya_route_tool(STATE, QUESTIONS))
+ outputs["route_batch"] = json.loads(server.laya_route_batch_tool(items))
+ assert all("error" not in value for value in outputs.values()), outputs
+ loaded = server._ensure_router().loaded
+ result = {"mode": args.mode, "imported_rss_mb": imported_rss, "after_tools_rss_mb": rss_mb(),
+ "torch_imported": "torch" in sys.modules, "loaded": loaded,
+ "answers": outputs["predict"]["answers"],
+ "batch_answers": outputs["predict_batch"]["requests"][0]["answers"],
+ "decided_values": outputs["decide"]["values"], "preset_answers": outputs["preset"]["answers"]}
+ assert result["torch_imported"] == (args.mode == "mcp-local")
+ if args.mode == "mcp-local":
+ server._ROUTER.unload()
+ else:
+ try:
+ server.laya_shortlist_tool(STATE, QUESTIONS)
+ except Exception as exc:
+ assert "unsupported_remote" in str(exc), exc
+ else:
+ raise AssertionError("remote shortlist did not refuse the call")
+ return result
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--checkpoint", type=Path, required=True)
+ parser.add_argument("--mode", choices=["idle-off", "idle-on", "mcp-local", "mcp-remote", "serve"], required=True)
+ parser.add_argument("--base-url")
+ parser.add_argument("--port", type=int, default=8000)
+ parser.add_argument("--idle-seconds", type=float, default=2.0)
+ args = parser.parse_args()
+ assert (args.checkpoint / "model.safetensors").is_file(), "local checkpoint is incomplete"
+ os.environ.pop("LAYA_API_KEY", None)
+ if args.mode == "serve":
+ import uvicorn
+ from laya.serve import create_app
+ os.environ["LAYA_IDLE_UNLOAD_SECONDS"] = "0"
+ router = make_router(args.checkpoint)
+ try:
+ uvicorn.run(create_app(router), host="127.0.0.1", port=args.port, log_level="warning")
+ finally:
+ router.unload()
+ return
+ result = idle_benchmark(args) if args.mode.startswith("idle-") else mcp_benchmark(args)
+ result["checkpoint_revision"] = args.checkpoint.name
+ print(json.dumps(result, sort_keys=True))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/research/benchmarks/idle_unload/results.json b/research/benchmarks/idle_unload/results.json
new file mode 100644
index 00000000..938cb27c
--- /dev/null
+++ b/research/benchmarks/idle_unload/results.json
@@ -0,0 +1,619 @@
+{
+ "baseline": "same implementation with idle unload disabled; local MCP versus shared-server MCP",
+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+ "device": "mps",
+ "hardware": "Apple M4 Pro",
+ "implementation_commit": "c8ddefa4836a463f54d46accd19adfeefff82bbc",
+ "memory_unit": "MiB",
+ "packages": {
+ "fastapi": "0.142.2",
+ "mcp": "2.3.0",
+ "torch": "2.14.1",
+ "transformers": "5.18.0"
+ },
+ "platform": "macOS-27.0.1-arm64-arm-64bit",
+ "python": "3.12.12",
+ "runs": [
+ {
+ "idle-off": {
+ "after_idle": {
+ "metal_driver_mb": 2056.703,
+ "metal_mb": 1610.387,
+ "rss_mb": 307.984
+ },
+ "after_next_request": {
+ "metal_driver_mb": 2056.703,
+ "metal_mb": 1610.387,
+ "rss_mb": 308.109
+ },
+ "answer_parity": true,
+ "answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+ "load_seconds": 3.461,
+ "mode": "idle-off",
+ "next_request_ms": 213.075,
+ "resident": {
+ "metal_driver_mb": 2056.703,
+ "metal_mb": 1610.387,
+ "rss_mb": 307.969
+ },
+ "warm_ms": 190.518
+ },
+ "idle-on": {
+ "after_idle": {
+ "metal_driver_mb": 0.703,
+ "metal_mb": 0.0,
+ "rss_mb": 492.312
+ },
+ "after_next_request": {
+ "metal_driver_mb": 2056.703,
+ "metal_mb": 1610.387,
+ "rss_mb": 1159.766
+ },
+ "answer_parity": true,
+ "answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+ "load_seconds": 2.918,
+ "mode": "idle-on",
+ "next_request_ms": 727.361,
+ "resident": {
+ "metal_driver_mb": 2056.703,
+ "metal_mb": 1610.387,
+ "rss_mb": 419.984
+ },
+ "warm_ms": 188.836
+ },
+ "local_remote_answer_parity": true,
+ "mcp-local": {
+ "after_tools_rss_mb": 1074.453,
+ "answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "batch_answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+ "decided_values": {
+ "dept": "billing"
+ },
+ "imported_rss_mb": 67.688,
+ "loaded": [
+ "english"
+ ],
+ "mode": "mcp-local",
+ "preset_answers": {
+ "churn_risk": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9997,
+ "confidence": 0.9997,
+ "noul": 0.0003,
+ "type": "noul"
+ },
+ "frustration": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.5295,
+ "confidence": 0.1579,
+ "legend": {
+ "0": "calm and neutral",
+ "1": "concerned but civil",
+ "2": "clearly annoyed",
+ "3": "very angry or using strong language"
+ },
+ "probabilities": {
+ "0": 0.114,
+ "1": 0.2523,
+ "2": 0.5295,
+ "3": 0.1042
+ },
+ "score": 1.6239,
+ "type": "score"
+ },
+ "intent": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9996,
+ "choice": "refund",
+ "confidence": 0.9978,
+ "probabilities": {
+ "billing_question": 0.0002,
+ "cancellation": 0.0,
+ "information": 0.0,
+ "other": 0.0001,
+ "refund": 0.9996,
+ "technical_help": 0.0
+ },
+ "type": "choice"
+ },
+ "is_urgent": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.8974,
+ "confidence": 0.8974,
+ "noul": 0.1026,
+ "type": "noul"
+ },
+ "refund_requested": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.8634,
+ "confidence": 0.8634,
+ "noul": 0.8634,
+ "type": "noul"
+ }
+ },
+ "torch_imported": true
+ },
+ "mcp-remote": {
+ "after_tools_rss_mb": 71.453,
+ "answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "batch_answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+ "decided_values": {
+ "dept": "billing"
+ },
+ "imported_rss_mb": 67.984,
+ "loaded": [
+ "english"
+ ],
+ "mode": "mcp-remote",
+ "preset_answers": {
+ "churn_risk": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9997,
+ "confidence": 0.9997,
+ "noul": 0.0003,
+ "type": "noul"
+ },
+ "frustration": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.5295,
+ "confidence": 0.1579,
+ "legend": {
+ "0": "calm and neutral",
+ "1": "concerned but civil",
+ "2": "clearly annoyed",
+ "3": "very angry or using strong language"
+ },
+ "probabilities": {
+ "0": 0.114,
+ "1": 0.2523,
+ "2": 0.5295,
+ "3": 0.1042
+ },
+ "score": 1.6239,
+ "type": "score"
+ },
+ "intent": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9996,
+ "choice": "refund",
+ "confidence": 0.9978,
+ "probabilities": {
+ "billing_question": 0.0002,
+ "cancellation": 0.0,
+ "information": 0.0,
+ "other": 0.0001,
+ "refund": 0.9996,
+ "technical_help": 0.0
+ },
+ "type": "choice"
+ },
+ "is_urgent": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.8974,
+ "confidence": 0.8974,
+ "noul": 0.1026,
+ "type": "noul"
+ },
+ "refund_requested": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.8634,
+ "confidence": 0.8634,
+ "noul": 0.8634,
+ "type": "noul"
+ }
+ },
+ "torch_imported": false
+ },
+ "repeat": 1
+ },
+ {
+ "idle-off": {
+ "after_idle": {
+ "metal_driver_mb": 2056.703,
+ "metal_mb": 1610.387,
+ "rss_mb": 408.672
+ },
+ "after_next_request": {
+ "metal_driver_mb": 2056.703,
+ "metal_mb": 1610.387,
+ "rss_mb": 408.703
+ },
+ "answer_parity": true,
+ "answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+ "load_seconds": 3.322,
+ "mode": "idle-off",
+ "next_request_ms": 168.304,
+ "resident": {
+ "metal_driver_mb": 2056.703,
+ "metal_mb": 1610.387,
+ "rss_mb": 408.672
+ },
+ "warm_ms": 98.172
+ },
+ "idle-on": {
+ "after_idle": {
+ "metal_driver_mb": 0.703,
+ "metal_mb": 0.0,
+ "rss_mb": 488.719
+ },
+ "after_next_request": {
+ "metal_driver_mb": 2056.703,
+ "metal_mb": 1610.387,
+ "rss_mb": 1167.906
+ },
+ "answer_parity": true,
+ "answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+ "load_seconds": 3.072,
+ "mode": "idle-on",
+ "next_request_ms": 836.043,
+ "resident": {
+ "metal_driver_mb": 2056.703,
+ "metal_mb": 1610.387,
+ "rss_mb": 462.406
+ },
+ "warm_ms": 193.897
+ },
+ "local_remote_answer_parity": true,
+ "mcp-local": {
+ "after_tools_rss_mb": 308.516,
+ "answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "batch_answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+ "decided_values": {
+ "dept": "billing"
+ },
+ "imported_rss_mb": 67.719,
+ "loaded": [
+ "english"
+ ],
+ "mode": "mcp-local",
+ "preset_answers": {
+ "churn_risk": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9997,
+ "confidence": 0.9997,
+ "noul": 0.0003,
+ "type": "noul"
+ },
+ "frustration": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.5295,
+ "confidence": 0.1579,
+ "legend": {
+ "0": "calm and neutral",
+ "1": "concerned but civil",
+ "2": "clearly annoyed",
+ "3": "very angry or using strong language"
+ },
+ "probabilities": {
+ "0": 0.114,
+ "1": 0.2523,
+ "2": 0.5295,
+ "3": 0.1042
+ },
+ "score": 1.6239,
+ "type": "score"
+ },
+ "intent": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9996,
+ "choice": "refund",
+ "confidence": 0.9978,
+ "probabilities": {
+ "billing_question": 0.0002,
+ "cancellation": 0.0,
+ "information": 0.0,
+ "other": 0.0001,
+ "refund": 0.9996,
+ "technical_help": 0.0
+ },
+ "type": "choice"
+ },
+ "is_urgent": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.8974,
+ "confidence": 0.8974,
+ "noul": 0.1026,
+ "type": "noul"
+ },
+ "refund_requested": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.8634,
+ "confidence": 0.8634,
+ "noul": 0.8634,
+ "type": "noul"
+ }
+ },
+ "torch_imported": true
+ },
+ "mcp-remote": {
+ "after_tools_rss_mb": 71.672,
+ "answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "batch_answers": {
+ "dept": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9855,
+ "choice": "billing",
+ "confidence": 0.8908,
+ "probabilities": {
+ "billing": 0.9855,
+ "technical": 0.0145
+ },
+ "type": "choice"
+ }
+ },
+ "checkpoint_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+ "decided_values": {
+ "dept": "billing"
+ },
+ "imported_rss_mb": 68.094,
+ "loaded": [
+ "english"
+ ],
+ "mode": "mcp-remote",
+ "preset_answers": {
+ "churn_risk": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9997,
+ "confidence": 0.9997,
+ "noul": 0.0003,
+ "type": "noul"
+ },
+ "frustration": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.5295,
+ "confidence": 0.1579,
+ "legend": {
+ "0": "calm and neutral",
+ "1": "concerned but civil",
+ "2": "clearly annoyed",
+ "3": "very angry or using strong language"
+ },
+ "probabilities": {
+ "0": 0.114,
+ "1": 0.2523,
+ "2": 0.5295,
+ "3": 0.1042
+ },
+ "score": 1.6239,
+ "type": "score"
+ },
+ "intent": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.9996,
+ "choice": "refund",
+ "confidence": 0.9978,
+ "probabilities": {
+ "billing_question": 0.0002,
+ "cancellation": 0.0,
+ "information": 0.0,
+ "other": 0.0001,
+ "refund": 0.9996,
+ "technical_help": 0.0
+ },
+ "type": "choice"
+ },
+ "is_urgent": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.8974,
+ "confidence": 0.8974,
+ "noul": 0.1026,
+ "type": "noul"
+ },
+ "refund_requested": {
+ "action": {
+ "act_probability": 1.0
+ },
+ "answer_confidence": 0.8634,
+ "confidence": 0.8634,
+ "noul": 0.8634,
+ "type": "noul"
+ }
+ },
+ "torch_imported": false
+ },
+ "repeat": 2
+ }
+ ],
+ "timestamp_utc": "2026-10-03T07:57:57.606565+00:00"
+}
diff --git a/research/benchmarks/idle_unload/validation.json b/research/benchmarks/idle_unload/validation.json
new file mode 100644
index 00000000..dc83a63c
--- /dev/null
+++ b/research/benchmarks/idle_unload/validation.json
@@ -0,0 +1,96 @@
+[
+ {
+ "repeat": 1,
+ "checks": [
+ {
+ "name": "lint",
+ "command": ".venv/bin/ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120",
+ "exit_code": 0,
+ "seconds": 0.01,
+ "summary": "All checks passed!"
+ },
+ {
+ "name": "new-files-lint",
+ "command": ".venv/bin/ruff check tests/test_mcp_remote.py research/benchmarks/idle_unload/benchmark.py --select=E9,F63,F7,F82,F401,F811 --line-length=120",
+ "exit_code": 0,
+ "seconds": 0.009,
+ "summary": "All checks passed!"
+ },
+ {
+ "name": "compile",
+ "command": ".venv/bin/python -m compileall -q laya/ tests/ research/benchmarks/idle_unload/",
+ "exit_code": 0,
+ "seconds": 0.034,
+ "summary": ""
+ },
+ {
+ "name": "suites",
+ "command": ".venv/bin/python scripts/test_suites.py",
+ "exit_code": 0,
+ "seconds": 81.638,
+ "summary": "360 passed, 1 skipped, 5 warnings in 13.63s\n\nall 78 suites passed"
+ },
+ {
+ "name": "docs",
+ "command": ".venv/bin/zensical build --strict --clean",
+ "exit_code": 0,
+ "seconds": 1.656,
+ "summary": "Build started\nNo issues found\nBuild finished in 1.41s"
+ },
+ {
+ "name": "diff",
+ "command": "git diff --check",
+ "exit_code": 0,
+ "seconds": 0.011,
+ "summary": ""
+ }
+ ]
+ },
+ {
+ "repeat": 2,
+ "checks": [
+ {
+ "name": "lint",
+ "command": ".venv/bin/ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120",
+ "exit_code": 0,
+ "seconds": 0.015,
+ "summary": "All checks passed!"
+ },
+ {
+ "name": "new-files-lint",
+ "command": ".venv/bin/ruff check tests/test_mcp_remote.py research/benchmarks/idle_unload/benchmark.py --select=E9,F63,F7,F82,F401,F811 --line-length=120",
+ "exit_code": 0,
+ "seconds": 0.01,
+ "summary": "All checks passed!"
+ },
+ {
+ "name": "compile",
+ "command": ".venv/bin/python -m compileall -q laya/ tests/ research/benchmarks/idle_unload/",
+ "exit_code": 0,
+ "seconds": 0.038,
+ "summary": ""
+ },
+ {
+ "name": "suites",
+ "command": ".venv/bin/python scripts/test_suites.py",
+ "exit_code": 0,
+ "seconds": 77.435,
+ "summary": "360 passed, 1 skipped, 5 warnings in 13.74s\n\nall 78 suites passed"
+ },
+ {
+ "name": "docs",
+ "command": ".venv/bin/zensical build --strict --clean",
+ "exit_code": 0,
+ "seconds": 1.54,
+ "summary": "Build started\nNo issues found\nBuild finished in 1.37s"
+ },
+ {
+ "name": "diff",
+ "command": "git diff --check",
+ "exit_code": 0,
+ "seconds": 0.011,
+ "summary": ""
+ }
+ ]
+ }
+]