Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
perf(rust): add benchmark sample
kevin-gatimu69e79b418f59
diff --git a/laya-rust/samples/laya-benchmark/.gitignore b/laya-rust/samples/laya-benchmark/.gitignorenew file mode 100644index 00000000..9cdfff78--- /dev/null+++ b/laya-rust/samples/laya-benchmark/.gitignore@@ -0,0 +1,8 @@+# Standalone builds (workspace builds go to laya-rust/target)+target/+**/*.rs.bk++# Downloaded model artifacts / run output+*.onnx+*.onnx_data+*.csvdiff --git a/laya-rust/samples/laya-benchmark/Cargo.toml b/laya-rust/samples/laya-benchmark/Cargo.tomlnew file mode 100644index 00000000..b90fd451--- /dev/null+++ b/laya-rust/samples/laya-benchmark/Cargo.toml@@ -0,0 +1,15 @@+[package]+name = "laya-benchmark"+version = "0.1.0"+edition = "2024"+description = "Benchmarks the Laya Rust SDK across one checkpoint."+license = "Apache-2.0"+publish = false++[[bin]]+name = "laya-benchmark"+path = "src/main.rs"++[dependencies]+laya = { package = "laya-onnx", path = "../.." }+serde_json = "1"diff --git a/laya-rust/samples/laya-benchmark/README.md b/laya-rust/samples/laya-benchmark/README.mdnew file mode 100644index 00000000..5a19af21--- /dev/null+++ b/laya-rust/samples/laya-benchmark/README.md@@ -0,0 +1,116 @@+# laya-benchmark++A small console app that measures the Laya Rust SDK on one checkpoint: how long the first call takes, how fast later calls are, and how much memory the process uses. It also records the answers, so they can be compared with the Python SDK.++It pairs with [`tools/bench_python.py`](../../tools/bench_python.py). Both use the same inputs, so their numbers and answers can be compared directly.++## What it measures++| Metric | How |+|---|---|+| Cold latency | `LayaEngine::create` plus the first `predict` call, in ms. This includes loading the model and ONNX Runtime's first-run setup. |+| Warm latency | Median and p95 over 30 timed `predict` calls, after 5 warm-up calls. The calls alternate between the two cases. |+| Peak memory | The process's peak working set in MB, via the Win32 `K32GetProcessMemoryInfo` API, read after the warm runs. On non-Windows targets this falls back to `VmHWM` from `/proc/self/status`, or `0` if that isn't available either. |+| Answers | Each question's answer for each case: choice, score or yes/no probability, plus the probabilities, confidence and act probability |++ONNX Runtime runs on the CPU with its default thread settings.++## Inputs++There are two cases, both asked the same four questions. The inputs match `laya-rust/tools/sample_cases.py` and the `laya-sample` app.++| Case | Input |+|---|---|+| `sample_app_english` | A billing email about a duplicate charge that asks for a refund and threatens to cancel |+| `sample_app_hindi` | A short Hindi message about being billed twice for March |++The four questions:+- `department`: choice of billing, technical, sales or other+- `urgency`: score from 0 to 2+- `churn_risk`: yes/no+- `refund_requested`: yes/no++## Prerequisites++- The Rust toolchain this workspace builds with (`rustc`/`cargo` 1.98, edition 2024)+- An exported ONNX folder for the checkpoint you want to test. See [`MODELS.md`](../../MODELS.md) for how to export them with `laya-rust/tools/export_onnx.py`.++## Usage++Run commands from `laya-rust`, and always build with `--release`: a debug build gives misleading timings.++```powershell+# Easiest: point LAYA_ONNX_ROOT at the folder that holds all three checkpoints+$env:LAYA_ONNX_ROOT = "C:/models/laya"++cargo run --release -p laya-benchmark -- --checkpoint multilingual+cargo run --release -p laya-benchmark -- --checkpoint english+cargo run --release -p laya-benchmark -- --checkpoint typed-decisions++# Or give one checkpoint's folder directly+cargo run --release -p laya-benchmark -- --checkpoint english --model-dir ../onnx/english+```++| Argument | Meaning |+|---|---|+| `--checkpoint` | `multilingual` (default), `english` or `typed-decisions` |+| `--model-dir` | Folder containing `model.onnx`, `model.onnx.data`, the tokenizer files and `rl_agent_config.json`. If you leave it out, the app uses `$LAYA_ONNX_ROOT/<checkpoint>` when that variable is set, and otherwise the SDK's normal resolution order. |++`--model-dir` takes one checkpoint's folder (`onnx/english`), not the export root (`onnx/`). To+point at the root, set `LAYA_ONNX_ROOT` as shown above. There is no `--model-root` flag here;+that one belongs to [`laya-routing`](../laya-routing/README.md) and is rejected with+`Unrecognised argument`. Keep the bare `--`: it separates `cargo run`'s options from the app's.++Progress lines go to stderr, each prefixed `[bench_rust]`. The results are printed to stdout as one JSON object, so you can redirect them to a file:++```powershell+cargo run --release -p laya-benchmark -- --checkpoint english > bench_rust_english.json+```++Exit codes:+- `0`: success+- `1`: the model files weren't found+- `2`: bad arguments++## Output++```jsonc+{+ "checkpoint": "english", // written exactly as the --checkpoint value's subdirectory name+ "runtime": "Rust + ONNX Runtime",+ "threads": "ORT default",+ "cold_ms": 0.0,+ "warm_times_ms": [ /* 30 sorted timings */ ],+ "warm_median_ms": 0.0,+ "warm_p95_ms": 0.0,+ "n_warmup": 5,+ "n_warm": 30,+ "peak_wset_mb": 0.0,+ "answers": {+ "sample_app_english": {+ "department": { "type": "choice", "choice": "...", "probabilities": { }, "confidence": 0.0, "act_probability": 0.0 },+ "urgency": { "type": "score", "score": 0.0, "probabilities": { }, "confidence": 0.0, "act_probability": 0.0 },+ "churn_risk": { "type": "noul", "noul": 0.0, "confidence": 0.0, "act_probability": 0.0 },+ "refund_requested": { "type": "noul", "noul": 0.0, "confidence": 0.0, "act_probability": 0.0 }+ },+ "sample_app_hindi": { /* same shape */ }+ }+}+```++## Comparing with Python++Run the Python counterpart with the repo's virtual environment. Set `HF_HOME` so the PyTorch weights download to D: instead of C:.++```powershell+$env:HF_HOME = "C:\hf_cache"+.venv/Scripts/python.exe laya-rust/tools/bench_python.py --checkpoint all --output bench_results.json+```++Python runs the PyTorch model directly on the CPU. Rust runs the exported ONNX graph through ONNX Runtime, so its answers should match Python within the parity tolerances. Expect the timings to differ, since the runtimes differ.++## Notes++- **Where the project sits:** it is a workspace member (`laya-rust/Cargo.toml`'s `[workspace] members`), a measurement tool rather than a shipped library artifact. `cargo build`/`cargo test` for the `laya-onnx` library crate alone (`-p laya-onnx`) does not build or run it; `cargo build --workspace` / `cargo test --workspace` does.+- **Running more than one:** run one checkpoint at a time. The ModernBERT-large checkpoints (english, typed-decisions) are about 1.7 GB each on disk.+- **When to measure:** cold latency depends on disk and OS file cache. Run twice and report the second cold number if you want warm file-cache timings.diff --git a/laya-rust/samples/laya-benchmark/src/main.rs b/laya-rust/samples/laya-benchmark/src/main.rsnew file mode 100644index 00000000..c5694542--- /dev/null+++ b/laya-rust/samples/laya-benchmark/src/main.rs@@ -0,0 +1,354 @@+//! Benchmark the Laya Rust SDK across one checkpoint.+//!+//! Measures:+//! cold latency : engine creation + first predict (ms)+//! warm latency : median and p95 over N_WARM timed calls (ms), after N_WARMUP warmups+//! peak working set (MB), via the Win32 `K32GetProcessMemoryInfo` API+//! answer snapshot for each bench case+//!+//! Outputs one JSON object to stdout.+//!+//! Usage:+//! cargo run --release -p laya-benchmark -- --checkpoint multilingual+//! cargo run --release -p laya-benchmark -- --checkpoint english+//! cargo run --release -p laya-benchmark -- --checkpoint typed-decisions+//!+//! Mirrors the methodology of `laya-rust/tools/bench_python.py` (arguments, output shape, exit+//! codes) so its output can be tabulated alongside the Python SDK's, with `[bench_rust]` on the+//! progress lines.++use laya::{Answer, LayaCheckpoint, LayaEngine, LayaError, LayaOptions, Question, QuestionSet};+use serde_json::{Map, Value, json};+use std::path::PathBuf;+use std::time::Instant;++const N_WARMUP: usize = 5;+const N_WARM: usize = 30;++fn main() {+ std::process::exit(run());+}++fn run() -> i32 {+ let args: Vec<String> = std::env::args().skip(1).collect();++ // ── parse args ────────────────────────────────────────────────────────────+ let mut checkpoint = LayaCheckpoint::Multilingual;+ let mut model_directory: Option<PathBuf> = None;+ let onnx_root = std::env::var("LAYA_ONNX_ROOT").ok();++ let mut i = 0;+ while i < args.len() {+ match args[i].as_str() {+ "--checkpoint" => {+ if i + 1 >= args.len() {+ return die("--checkpoint needs a value.");+ }+ i += 1;+ checkpoint = match args[i].as_str() {+ "multilingual" => LayaCheckpoint::Multilingual,+ "english" => LayaCheckpoint::English,+ "typed-decisions" => LayaCheckpoint::TypedDecisions,+ other => return die(&format!("Unknown checkpoint: {other}")),+ };+ }+ "--model-dir" => {+ if i + 1 >= args.len() {+ return die("--model-dir needs a value.");+ }+ i += 1;+ model_directory = Some(PathBuf::from(&args[i]));+ }+ other => return die(&format!("Unrecognised argument: {other}")),+ }+ i += 1;+ }++ let checkpoint_name = checkpoint.subdir();++ // If LAYA_ONNX_ROOT is set and no explicit --model-dir was given, derive the dir.+ if model_directory.is_none()+ && let Some(root) = &onnx_root+ && !root.trim().is_empty()+ {+ model_directory = Some(PathBuf::from(root).join(checkpoint_name));+ }++ // ── bench cases ───────────────────────────────────────────────────────────+ // Mirrors tools/bench_python.py: the same two states and question set.+ let english_state = json!({+ "from": "[email protected]",+ "subject": "Duplicate charge on invoice #4411",+ "body": "Hi, we were billed twice for March. Please refund the duplicate today \+ or we will cancel our plan.",+ });+ let hindi_state = json!({+ "body": "मार्च का बिल दो बार लिया गया, कृपया रिफंड करें।",+ });+ let questions = match build_questions() {+ Ok(q) => q,+ Err(e) => return die(&e.to_string()),+ };+ let bench_cases: [(&str, Value); 2] = [+ ("sample_app_english", english_state),+ ("sample_app_hindi", hindi_state),+ ];++ // ── cold measurement ──────────────────────────────────────────────────────+ let cold_clock = Instant::now();+ let mut options = LayaOptions::new();+ options.checkpoint = checkpoint;+ options.model_directory = model_directory;+ let engine = match LayaEngine::create(options) {+ Ok(engine) => engine,+ Err(+ e @ (LayaError::ArtifactsNotFound { .. }+ | LayaError::DirectoryNotFound { .. }+ | LayaError::MissingFile { .. }),+ ) => {+ eprintln!("{e}");+ return 1;+ }+ Err(e) => {+ eprintln!("{e}");+ return 1;+ }+ };++ // First call (still part of cold, as ORT JITs on the first run).+ let (_, first_state) = &bench_cases[0];+ if let Err(e) = engine.predict(first_state.clone(), &questions) {+ eprintln!("{e}");+ return 1;+ }+ let cold_ms = cold_clock.elapsed().as_secs_f64() * 1000.0;++ eprintln!("[bench_rust] cold={cold_ms:.0} ms, running warm-up...");++ // ── warmup ────────────────────────────────────────────────────────────────+ let mut call_idx: usize = 0;+ let mut next_call = || {+ let (_, state) = &bench_cases[call_idx % bench_cases.len()];+ call_idx += 1;+ state.clone()+ };++ for _ in 0..N_WARMUP {+ let state = next_call();+ engine+ .predict(state, &questions)+ .expect("predict failed during warm-up");+ }++ // ── warm measurement ──────────────────────────────────────────────────────+ let mut times_ms = Vec::with_capacity(N_WARM);+ for _ in 0..N_WARM {+ let state = next_call();+ let t0 = Instant::now();+ engine+ .predict(state, &questions)+ .expect("predict failed during warm run");+ times_ms.push(t0.elapsed().as_secs_f64() * 1000.0);+ }+ times_ms.sort_by(|a, b| a.partial_cmp(b).expect("timings are never NaN"));+ let median_ms = times_ms[times_ms.len() / 2];+ let p95_ms = times_ms[(times_ms.len() as f64 * 0.95) as usize];++ // ── peak working set ──────────────────────────────────────────────────────+ let peak_mb = win_mem::peak_working_set_bytes()+ .map(|bytes| bytes as f64 / (1024.0 * 1024.0))+ .unwrap_or(0.0);++ eprintln!(+ "[bench_rust] warm median={median_ms:.1} ms p95={p95_ms:.1} ms peak={peak_mb:.0} MB"+ );++ // ── answer snapshot ───────────────────────────────────────────────────────+ let mut answers = Map::new();+ for (name, state) in &bench_cases {+ let result = engine+ .predict(state.clone(), &questions)+ .expect("predict failed while snapshotting answers");+ let mut case_answers = Map::new();+ for id in result.ids() {+ case_answers.insert(id.clone(), answer_to_json(&result[id.as_str()]));+ }+ answers.insert((*name).to_string(), Value::Object(case_answers));+ }++ drop(engine);++ // ── emit JSON ─────────────────────────────────────────────────────────────+ let output = json!({+ "checkpoint": checkpoint_name,+ "runtime": "Rust + ONNX Runtime",+ "threads": "ORT default",+ "cold_ms": round_to(cold_ms, 1),+ "warm_times_ms": times_ms.iter().map(|t| round_to(*t, 2)).collect::<Vec<_>>(),+ "warm_median_ms": round_to(median_ms, 1),+ "warm_p95_ms": round_to(p95_ms, 1),+ "n_warmup": N_WARMUP,+ "n_warm": N_WARM,+ "peak_wset_mb": round_to(peak_mb, 1),+ "answers": Value::Object(answers),+ });++ println!(+ "{}",+ serde_json::to_string_pretty(&output).expect("benchmark output is always valid JSON")+ );+ 0+}++fn build_questions() -> laya::Result<QuestionSet> {+ QuestionSet::new()+ .with(+ "department",+ Question::choice(+ "Which department should handle this request?",+ [+ ("billing", "invoices, payments, refunds"),+ ("technical", "bugs, outages, system errors"),+ ("sales", "pricing, new contracts"),+ ("other", "everything else"),+ ],+ )?,+ )?+ .with(+ "urgency",+ Question::score(+ "How urgent is this request?",+ ["not urgent", "soon", "critical deadline or blocking issue"],+ )?,+ )?+ .with(+ "churn_risk",+ Question::noul(+ "Does the user threaten to cancel or leave?",+ Value::Null,+ Value::Null,+ ),+ )?+ .with(+ "refund_requested",+ Question::noul(+ "Does the user explicitly request a refund?",+ Value::Null,+ Value::Null,+ ),+ )+}++fn answer_to_json(answer: &Answer) -> Value {+ match answer {+ Answer::Choice(choice) => {+ let mut probabilities = Map::new();+ for (label, prob) in &choice.probabilities {+ probabilities.insert(label.clone(), json!(prob));+ }+ json!({+ "type": "choice",+ "choice": choice.choice,+ "probabilities": Value::Object(probabilities),+ "confidence": answer.confidence(),+ "act_probability": answer.action().act_probability,+ })+ }+ Answer::Score(score) => {+ let mut probabilities = Map::new();+ for (index, prob) in score.probabilities.iter().enumerate() {+ probabilities.insert(index.to_string(), json!(prob));+ }+ json!({+ "type": "score",+ "score": score.score,+ "probabilities": Value::Object(probabilities),+ "confidence": answer.confidence(),+ "act_probability": answer.action().act_probability,+ })+ }+ Answer::Noul(noul) => {+ json!({+ "type": "noul",+ "noul": noul.probability,+ "confidence": answer.confidence(),+ "act_probability": answer.action().act_probability,+ })+ }+ }+}++fn round_to(value: f64, decimals: i32) -> f64 {+ let factor = 10f64.powi(decimals);+ (value * factor).round() / factor+}++fn die(message: &str) -> i32 {+ eprintln!("{message}");+ 2+}++#[cfg(windows)]+mod win_mem {+ #![allow(non_snake_case)]++ use std::ffi::c_void;++ #[repr(C)]+ struct ProcessMemoryCountersEx {+ cb: u32,+ page_fault_count: u32,+ peak_working_set_size: usize,+ working_set_size: usize,+ quota_peak_paged_pool_usage: usize,+ quota_paged_pool_usage: usize,+ quota_peak_non_paged_pool_usage: usize,+ quota_non_paged_pool_usage: usize,+ pagefile_usage: usize,+ peak_pagefile_usage: usize,+ private_usage: usize,+ }++ #[link(name = "kernel32")]+ unsafe extern "system" {+ fn GetCurrentProcess() -> *mut c_void;+ }++ #[link(name = "psapi")]+ unsafe extern "system" {+ fn K32GetProcessMemoryInfo(+ process: *mut c_void,+ counters: *mut ProcessMemoryCountersEx,+ cb: u32,+ ) -> i32;+ }++ /// The process's peak working set, in bytes, via `K32GetProcessMemoryInfo`. `None` if the+ /// call fails (should not happen on a real process handle).+ pub fn peak_working_set_bytes() -> Option<u64> {+ let mut counters: ProcessMemoryCountersEx = unsafe { std::mem::zeroed() };+ counters.cb = std::mem::size_of::<ProcessMemoryCountersEx>() as u32;+ // SAFETY: `GetCurrentProcess` returns a valid pseudo-handle that needs no closing;+ // `counters` is a correctly-sized, zeroed buffer whose first field (`cb`) is set to that+ // size before the call, exactly as the Win32 API requires.+ let ok =+ unsafe { K32GetProcessMemoryInfo(GetCurrentProcess(), &mut counters, counters.cb) };+ (ok != 0).then_some(counters.peak_working_set_size as u64)+ }+}++#[cfg(not(windows))]+mod win_mem {+ /// Best-effort on non-Windows: read `VmHWM` (peak resident set) from `/proc/self/status`.+ /// `None` when that file's format doesn't match (e.g. a non-Linux Unix).+ pub fn peak_working_set_bytes() -> Option<u64> {+ let status = std::fs::read_to_string("/proc/self/status").ok()?;+ for line in status.lines() {+ if let Some(rest) = line.strip_prefix("VmHWM:") {+ let kb: u64 = rest.trim().trim_end_matches("kB").trim().parse().ok()?;+ return Some(kb * 1024);+ }+ }+ None+ }+}diff --git a/laya-rust/tools/bench_python.py b/laya-rust/tools/bench_python.pynew file mode 100644index 00000000..ce381812--- /dev/null+++ b/laya-rust/tools/bench_python.py@@ -0,0 +1,327 @@+"""Benchmark the Python laya SDK across all three checkpoints.++Measures:+ - Cold latency: model load + first Predict call (ms)+ - Warm latency: median and p95 over N_WARM timed calls (ms), after N_WARMUP warmups+ - Peak RSS/working-set (MB), via psutil if available+ - Answer snapshot for each case, for comparison with the Rust results++Outputs JSON to stdout so the caller can parse and tabulate.++By default this benchmarks the public SDK (`laya.load`, PyTorch eager on CPU). Pass+`--runtime onnx` to run the same SDK code path with the exported ONNX graph as the forward+pass, which separates runtime cost from language cost when comparing with the Rust SDK.++Usage:+ python laya-rust/tools/bench_python.py --checkpoint multilingual+ python laya-rust/tools/bench_python.py --checkpoint multilingual --runtime onnx+ python laya-rust/tools/bench_python.py --checkpoint english+ python laya-rust/tools/bench_python.py --checkpoint typed-decisions+ python laya-rust/tools/bench_python.py --checkpoint all # runs all three in sequence+ python laya-rust/tools/bench_python.py --checkpoint all --output bench_results.json+"""++import argparse+import json+import os+import sys+import time++TOOLS = os.path.dirname(os.path.abspath(__file__))+REPO = os.path.dirname(os.path.dirname(TOOLS))+sys.path.insert(0, REPO)+sys.path.insert(0, TOOLS)++from sample_cases import SAMPLE_ENGLISH_STATE, SAMPLE_HINDI_STATE, SAMPLE_QUESTIONS++N_WARMUP = 5+N_WARM = 30++# The benchmark cases: a subset of the full golden suite chosen to cover real behaviour+# without taking forever. These are the same cases the task brief specifies.+BENCH_CASES = [+ {+ "name": "sample_app_english",+ "state": SAMPLE_ENGLISH_STATE,+ "questions": SAMPLE_QUESTIONS,+ },+ {+ "name": "sample_app_hindi",+ "state": SAMPLE_HINDI_STATE,+ "questions": SAMPLE_QUESTIONS,+ },+]++ONNX_ROOT = os.path.join(REPO, "onnx")++CHECKPOINTS = ["english", "multilingual", "typed-decisions"]+++def _onnx_dir(checkpoint: str) -> str:+ return os.path.join(ONNX_ROOT, checkpoint)+++def _load_agent(onnx_dir: str):+ """Load an Agent using the ONNX forward pass, exactly as dump_golden.py does."""+ import numpy as np+ import onnxruntime as ort+ import torch+ from transformers import AutoTokenizer++ from laya.agent import Agent, _fix_tokenizer_config+ from laya.common import clamp_temperature++ NP_DTYPES = {+ "tensor(int64)": np.int64,+ "tensor(int32)": np.int32,+ "tensor(bool)": np.bool_,+ "tensor(float)": np.float32,+ }++ class OnnxForward:+ MIN_MARKERS = 2++ def __init__(self, session):+ self.sess = session+ self.dtypes = {i.name: NP_DTYPES[i.type] for i in session.get_inputs()}++ def _pad_markers(self, marker_pos, marker_mask):+ k = marker_pos.shape[1]+ if k >= self.MIN_MARKERS:+ return marker_pos, marker_mask+ pad = self.MIN_MARKERS - k+ marker_pos = torch.cat(+ [marker_pos, torch.zeros((marker_pos.shape[0], pad), dtype=marker_pos.dtype)], 1+ )+ marker_mask = torch.cat(+ [marker_mask, torch.zeros((marker_mask.shape[0], pad), dtype=torch.bool)], 1+ )+ return marker_pos, marker_mask++ def __call__(self, input_ids, attention_mask, marker_pos, marker_mask, qtype):+ marker_pos, marker_mask = self._pad_markers(marker_pos, marker_mask)+ feeds = {+ n: t.cpu().numpy().astype(self.dtypes[n])+ for n, t in [+ ("input_ids", input_ids),+ ("attention_mask", attention_mask),+ ("marker_pos", marker_pos),+ ("marker_mask", marker_mask),+ ("qtype", qtype),+ ]+ }+ logits, act_logits = self.sess.run(["logits", "act_logits"], feeds)+ return torch.from_numpy(logits), torch.from_numpy(act_logits)++ with open(os.path.join(onnx_dir, "rl_agent_config.json"), encoding="utf-8") as f:+ cfg = json.load(f)++ _fix_tokenizer_config(onnx_dir)+ tok = AutoTokenizer.from_pretrained(os.path.join(onnx_dir, "tokenizer"))++ sess_opts = ort.SessionOptions()+ # Single-thread to match default Python behaviour: the ONNX runtime defaults to+ # using all available cores; pin to 1 so the number is reproducible.+ sess_opts.intra_op_num_threads = 1+ sess_opts.inter_op_num_threads = 1+ sess = ort.InferenceSession(+ os.path.join(onnx_dir, "model.onnx"),+ sess_options=sess_opts,+ providers=["CPUExecutionProvider"],+ )++ agent = Agent.__new__(Agent)+ agent.cfg = cfg+ agent.tok = tok+ import torch as _torch+ agent.device = _torch.device("cpu")+ agent.dtype = _torch.float32+ agent.model = OnnxForward(sess)+ agent.temperature = [clamp_temperature(t) for t in cfg.get("temperature", [1.0, 1.0, 1.0])]+ agent.temperature_by_options = {+ k: clamp_temperature(v) for k, v in cfg.get("temperature_by_options", {}).items()+ }+ return agent+++def _load_sdk_agent(checkpoint: str):+ """Load the checkpoint through the public Python SDK (PyTorch eager, CPU)."""+ import laya++ subfolder = None if checkpoint == "english" else checkpoint+ return laya.load("convaiinnovations/laya", device="cpu", subfolder=subfolder)+++def _peak_wset_mb():+ """Return the process peak working set (Windows) or peak RSS (other) in MB."""+ try:+ import psutil+ except ImportError:+ return None+ mi = psutil.Process(os.getpid()).memory_info()+ peak = getattr(mi, "peak_wset", None)+ if peak is None:+ import resource # POSIX: ru_maxrss is KiB on Linux+ peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * 1024+ return round(peak / 1024 / 1024, 1)+++def bench_checkpoint(checkpoint: str, runtime: str) -> dict:+ """Run the full benchmark for one checkpoint. Returns a result dict."""+ print(f"[bench_python] {checkpoint} ({runtime}): loading...", flush=True)++ onnx_dir = _onnx_dir(checkpoint)+ if runtime == "onnx" and not os.path.isdir(onnx_dir):+ return {"checkpoint": checkpoint, "error": f"ONNX dir not found: {onnx_dir}"}++ # Imports are timed separately so cold latency matches the Rust number+ # (engine creation + first call, excluding runtime start-up).+ t_import = time.perf_counter()+ import torch # noqa: F401+ import laya # noqa: F401+ import_ms = (time.perf_counter() - t_import) * 1000.0++ # ── cold latency ─────────────────────────────────────────────────────────+ # Measure load + first predict together as the "cold" number.+ t_cold_start = time.perf_counter()+ agent = _load_sdk_agent(checkpoint) if runtime == "torch" else _load_agent(onnx_dir)++ # Use the first benchmark case for the cold call.+ first_case = BENCH_CASES[0]+ _ = agent.system_one(first_case["state"], first_case["questions"])+ cold_ms = (time.perf_counter() - t_cold_start) * 1000.0++ print(f"[bench_python] {checkpoint}: cold={cold_ms:.0f} ms, running warm-up...", flush=True)++ # ── warm latency ──────────────────────────────────────────────────────────+ # Cycle through all bench cases for variety; this also exercises both states.+ call_idx = 0++ def _next_call():+ nonlocal call_idx+ c = BENCH_CASES[call_idx % len(BENCH_CASES)]+ call_idx += 1+ return c["state"], c["questions"]++ # Warmup+ for _ in range(N_WARMUP):+ s, q = _next_call()+ agent.system_one(s, q)++ # Timed calls+ times_ms = []+ for _ in range(N_WARM):+ s, q = _next_call()+ t0 = time.perf_counter()+ agent.system_one(s, q)+ times_ms.append((time.perf_counter() - t0) * 1000.0)++ times_ms.sort()+ median_ms = times_ms[len(times_ms) // 2]+ p95_ms = times_ms[int(len(times_ms) * 0.95)]++ # ── peak memory ───────────────────────────────────────────────────────────+ peak_mb = _peak_wset_mb()++ print(+ f"[bench_python] {checkpoint}: warm median={median_ms:.1f} ms p95={p95_ms:.1f} ms"+ + (f" peak={peak_mb:.0f} MB" if peak_mb else ""),+ flush=True,+ )++ # ── answer snapshot ───────────────────────────────────────────────────────+ answers = {}+ for case in BENCH_CASES:+ result = agent.system_one(case["state"], case["questions"])+ answers[case["name"]] = result["answers"]++ import torch++ return {+ "checkpoint": checkpoint,+ "runtime": "Python SDK + PyTorch (CPU)" if runtime == "torch" else "Python + ONNX Runtime",+ "threads": f"torch default ({torch.get_num_threads()})" if runtime == "torch" else "intra=1 inter=1",+ "import_ms": round(import_ms, 1),+ "cold_ms": round(cold_ms, 1),+ "warm_times_ms": [round(t, 2) for t in times_ms],+ "warm_median_ms": round(median_ms, 1),+ "warm_p95_ms": round(p95_ms, 1),+ "n_warmup": N_WARMUP,+ "n_warm": N_WARM,+ "peak_wset_mb": peak_mb,+ "answers": answers,+ }+++def get_versions() -> dict:+ info = {}+ try:+ import onnxruntime as ort+ info["onnxruntime"] = ort.__version__+ except ImportError:+ info["onnxruntime"] = None+ try:+ import torch+ info["torch"] = torch.__version__+ except ImportError:+ info["torch"] = None+ try:+ import transformers+ info["transformers"] = transformers.__version__+ except ImportError:+ info["transformers"] = None+ try:+ import numpy as np+ info["numpy"] = np.__version__+ except ImportError:+ info["numpy"] = None+ try:+ import laya+ info["laya"] = laya.__version__+ except Exception:+ info["laya"] = None+ info["python"] = sys.version+ return info+++def main():+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument(+ "--checkpoint",+ choices=CHECKPOINTS + ["all"],+ default="multilingual",+ metavar="{multilingual,english,typed-decisions,all}",+ )+ ap.add_argument(+ "--runtime",+ choices=["torch", "onnx"],+ default="torch",+ help="torch: the public SDK (laya.load, PyTorch eager). onnx: the SDK code path with "+ "the exported ONNX graph as the forward pass, single-threaded.",+ )+ ap.add_argument("--output", default=None, help="Write JSON results to this file (default: stdout only)")+ args = ap.parse_args()++ checkpoints = CHECKPOINTS if args.checkpoint == "all" else [args.checkpoint]++ results = {"versions": None, "checkpoints": {}}++ # Note: "all" shares one process, so peak memory and cold latency of later checkpoints+ # are inflated. Run one checkpoint per process for publishable numbers.+ for cp in checkpoints:+ results["checkpoints"][cp] = bench_checkpoint(cp, args.runtime)+ # Collected after the runs so the imports it triggers are not charged to import_ms.+ results["versions"] = get_versions()++ out = json.dumps(results, indent=2, ensure_ascii=False)+ print("\n" + out)+ if args.output:+ with open(args.output, "w", encoding="utf-8") as f:+ f.write(out)+ print(f"\nResults written to {args.output}", flush=True)++ return 0+++if __name__ == "__main__":+ raise SystemExit(main())