Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

5.6 KB
Raw Blame

laya-benchmark

A small console app that measures the Laya Rust SDK on one checkpoint: how long the first call takes, how fast later calls are, and how much memory the process uses. It also records the answers, so they can be compared with the Python SDK.

It pairs with tools/bench_python.py. Both use the same inputs, so their numbers and answers can be compared directly.

What it measures

Metric How
Cold latency LayaEngine::create plus the first predict call, in ms. This includes loading the model and ONNX Runtime's first-run setup.
Warm latency Median and p95 over 30 timed predict calls, after 5 warm-up calls. The calls alternate between the two cases.
Peak memory The process's peak working set in MB, via the Win32 K32GetProcessMemoryInfo API, read after the warm runs. On non-Windows targets this falls back to VmHWM from /proc/self/status, or 0 if that isn't available either.
Answers Each question's answer for each case: choice, score or yes/no probability, plus the probabilities, confidence and act probability

ONNX Runtime runs on the CPU with its default thread settings.

Inputs

There are two cases, both asked the same four questions. The inputs match laya-rust/tools/sample_cases.py and the laya-sample app.

Case Input
sample_app_english A billing email about a duplicate charge that asks for a refund and threatens to cancel
sample_app_hindi A short Hindi message about being billed twice for March

The four questions:

  • department: choice of billing, technical, sales or other
  • urgency: score from 0 to 2
  • churn_risk: yes/no
  • refund_requested: yes/no

Prerequisites

  • The Rust toolchain this workspace builds with (rustc/cargo 1.98, edition 2024)
  • An exported ONNX folder for the checkpoint you want to test. See MODELS.md for how to export them with laya-rust/tools/export_onnx.py.

Usage

Run commands from laya-rust, and always build with --release: a debug build gives misleading timings.

# Easiest: point LAYA_ONNX_ROOT at the folder that holds all three checkpoints
$env:LAYA_ONNX_ROOT = "C:/models/laya"

cargo run --release -p laya-benchmark -- --checkpoint multilingual
cargo run --release -p laya-benchmark -- --checkpoint english
cargo run --release -p laya-benchmark -- --checkpoint typed-decisions

# Or give one checkpoint's folder directly
cargo run --release -p laya-benchmark -- --checkpoint english --model-dir ../onnx/english
Argument Meaning
--checkpoint multilingual (default), english or typed-decisions
--model-dir Folder containing model.onnx, model.onnx.data, the tokenizer files and rl_agent_config.json. If you leave it out, the app uses $LAYA_ONNX_ROOT/<checkpoint> when that variable is set, and otherwise the SDK's normal resolution order.

--model-dir takes one checkpoint's folder (onnx/english), not the export root (onnx/). To point at the root, set LAYA_ONNX_ROOT as shown above. There is no --model-root flag here; that one belongs to laya-routing and is rejected with Unrecognised argument. Keep the bare --: it separates cargo run's options from the app's.

Progress lines go to stderr, each prefixed [bench_rust]. The results are printed to stdout as one JSON object, so you can redirect them to a file:

cargo run --release -p laya-benchmark -- --checkpoint english > bench_rust_english.json

Exit codes:

  • 0: success
  • 1: the model files weren't found
  • 2: bad arguments

Output

{
  "checkpoint": "english",            // written exactly as the --checkpoint value's subdirectory name
  "runtime": "Rust + ONNX Runtime",
  "threads": "ORT default",
  "cold_ms": 0.0,
  "warm_times_ms": [ /* 30 sorted timings */ ],
  "warm_median_ms": 0.0,
  "warm_p95_ms": 0.0,
  "n_warmup": 5,
  "n_warm": 30,
  "peak_wset_mb": 0.0,
  "answers": {
    "sample_app_english": {
      "department":       { "type": "choice", "choice": "...", "probabilities": { }, "confidence": 0.0, "act_probability": 0.0 },
      "urgency":          { "type": "score",  "score": 0.0,    "probabilities": { }, "confidence": 0.0, "act_probability": 0.0 },
      "churn_risk":       { "type": "noul",   "noul": 0.0,     "confidence": 0.0, "act_probability": 0.0 },
      "refund_requested": { "type": "noul",   "noul": 0.0,     "confidence": 0.0, "act_probability": 0.0 }
    },
    "sample_app_hindi": { /* same shape */ }
  }
}

Comparing with Python

Run the Python counterpart with the repo's virtual environment. Set HF_HOME so the PyTorch weights download to D: instead of C:.

$env:HF_HOME = "C:\hf_cache"
.venv/Scripts/python.exe laya-rust/tools/bench_python.py --checkpoint all --output bench_results.json

Python runs the PyTorch model directly on the CPU. Rust runs the exported ONNX graph through ONNX Runtime, so its answers should match Python within the parity tolerances. Expect the timings to differ, since the runtimes differ.

Notes

  • Where the project sits: it is a workspace member (laya-rust/Cargo.toml's [workspace] members), a measurement tool rather than a shipped library artifact. cargo build/cargo test for the laya-onnx library crate alone (-p laya-onnx) does not build or run it; cargo build --workspace / cargo test --workspace does.
  • Running more than one: run one checkpoint at a time. The ModernBERT-large checkpoints (english, typed-decisions) are about 1.7 GB each on disk.
  • When to measure: cold latency depends on disk and OS file cache. Run twice and report the second cold number if you want warm file-cache timings.