Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(benchmarks): MPS fp16 autocast against fp32 on an Apple M1 Pro

cacheline9997446c6993ac0
diff --git a/BENCHMARKS.md b/BENCHMARKS.md
index 336fbfef..04c3367c 100644
--- a/BENCHMARKS.md
+++ b/BENCHMARKS.md
@@ -269,6 +269,40 @@ The best setting is the physical core count plus a little, not one thread per vC
CPU scales roughly linearly with question count (288.2 -> 2608.7 ms, 9.1x for 10x the questions), while XPU scales sub-linearly (29.7 -> 96.9 ms, 3.3x), so the speedup widens from ~10x to ~27x. The XPU p95 stays within ~6% of its p50 on every row (30.4, 46.8, 102.5). At one question the Arc B390 is slightly faster than the T4's 32.8 ms p50 above.
+### Apple M1 Pro, MPS — fp16 autocast against fp32
+
+On MPS, `Agent` autocasts a forward to fp16 at or above `mps_amp_min_rows` question rows (default 5, `LAYA_MPS_AMP_MIN_ROWS`). The default came from one M5 (#109), where fp16 won from four rows. On an M1 Pro (16 GB, macOS 26.1, torch 2.14.0, transformers 5.17.0, laya 0.3.26) it loses almost everywhere. `benchmarks/bench_mps_autocast.py` runs every request once in fp32 and once in fp16 on the same loaded agent, back to back with the order alternating, so both modes see the same load (the machine was not idle: 1-min load 6 to 10). 72 pairs per row: 24 states, three passes. A short state is one message, a long one a thread of six. Results in `benchmarks/results/mps_autocast_*_m1pro.json`.
+
+`english` (ModernBERT-large), median ms, and fp16 minus fp32 per request:
+
+| state | rows | fp32 | fp16 | fp16 − fp32 | max probability change |
+|---|---|---|---|---|---|
+| short | 1 | 58.2 | 79.4 | +21.0 | 0.0011 |
+| short | 4 | 147.6 | 170.4 | +23.6 | 0.0061 |
+| short | 5 | 190.4 | 213.6 | +22.9 | 0.0061 |
+| short | 8 | 290.6 | 316.4 | +28.6 | 0.0061 |
+| long | 1 | 133.7 | 157.3 | +24.2 | 0.0081 |
+| long | 4 | 458.1 | 492.6 | +23.8 | 0.0081 |
+| long | 5 | 394.4 | 403.3 | +9.1 | 0.0081 |
+| long | 8 | 585.5 | 566.6 | **−13.7** | 0.0081 |
+
+`multilingual` (mmBERT-base):
+
+| state | rows | fp32 | fp16 | fp16 − fp32 | max probability change |
+|---|---|---|---|---|---|
+| short | 1 | 25.5 | 34.7 | +9.8 | 0.0038 |
+| short | 4 | 45.9 | 55.7 | +9.9 | 0.0038 |
+| short | 5 | 56.4 | 66.5 | +10.3 | 0.0038 |
+| short | 8 | 78.1 | 90.9 | +11.8 | 0.0038 |
+| long | 1 | 44.0 | 53.7 | +9.3 | 0.0107 |
+| long | 4 | 145.4 | 157.1 | +11.7 | 0.0106 |
+| long | 5 | 183.6 | 195.1 | +12.9 | 0.0106 |
+| long | 8 | 267.6 | 283.6 | +17.2 | 0.0206 |
+
+fp16 costs about 20 to 30 ms per request on `english` and about 10 ms on `multilingual`, whatever the row count; it only wins on long `english` states with eight rows. So at the default of 5 an M1 Pro pays for autocast on every request of five or more rows. `LAYA_MPS_AMP_MIN_ROWS=1000000` keeps fp32 throughout.
+
+Of 1,248 decisions per checkpoint, fp16 changed none on `english` and one on `multilingual` (a `noul` next to 0.5). A decision near the boundary can flip.
+
### Calibration on a routing task runs the other way
On laya_router's 180 requests (zero-shot, one 3-tier `choice`), nearly every configuration we measured was **under**-confident (the few exceptions were +0.01 to +0.06, and among the least accurate). Mean P(chosen) (the chosen option's probability, not the entropy-based `confidence` field) sat below accuracy, by −0.18 on the root checkpoint with example-led tier descriptions (0.562 vs 0.744) and by −0.19 on `typed-decisions` (0.501 vs 0.694). This is one task and one set of labels, so it does not contradict the over-confidence reported above. It does mean the direction of the miscalibration depends on the task, and a temperature fit on your own data is the right fix either way.
diff --git a/benchmarks/bench_mps_autocast.py b/benchmarks/bench_mps_autocast.py
new file mode 100644
index 00000000..dbcc3fbf
--- /dev/null
+++ b/benchmarks/bench_mps_autocast.py
@@ -0,0 +1,162 @@
+#!/usr/bin/env python
+"""MPS fp16 autocast against fp32, request by request, on Apple silicon.
+
+ python benchmarks/bench_mps_autocast.py --checkpoint english
+ python benchmarks/bench_mps_autocast.py --checkpoint multilingual --out results.json
+
+`Agent` autocasts an MPS forward to fp16 at or above `mps_amp_min_rows` question rows
+(default 5, `LAYA_MPS_AMP_MIN_ROWS`). This script measures what that gate decides, for one
+checkpoint, offline after the checkpoint is cached:
+
+* latency: every request runs once in fp32 and once in fp16 on the same loaded agent, back
+ to back, with the order alternating. The two modes then see the same machine load, so the
+ paired difference holds on a machine that is not idle. Reported per (state length, rows):
+ median of each mode, and the median of the per-request difference.
+* answers: for the same requests, the largest and the 95th-percentile probability change
+ between the two modes, and how many decisions changed (choice: the option; score: the
+ most likely level; noul: the side of 0.5).
+
+The mode is switched through `agent.mps_amp_min_rows` (1 = fp16 for every request, above the
+row count = fp32), so nothing but the gate differs between the two runs of a request.
+"""
+import argparse
+import json
+import os
+import platform
+import statistics
+import subprocess
+import time
+
+os.environ.setdefault("USE_TF", "0")
+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+os.environ.setdefault("HF_HUB_DISABLE_TELEMETRY", "1")
+
+BUNDLE = "convaiinnovations/laya"
+QUESTIONS = [
+ {"type": "choice", "instructions": "Which department should handle this request?",
+ "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors",
+ "logistics": "delivery tracking and delivery problems", "other": "everything else"}},
+ {"type": "noul", "instructions": "Does the user explicitly request a refund?"},
+ {"type": "score", "instructions": "How urgent is this request?",
+ "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
+ {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
+ {"type": "noul", "instructions": "Is the user angry?"},
+ {"type": "choice", "instructions": "What should happen next?",
+ "criteria": {"reply": "answer the user directly", "escalate": "hand over to a human",
+ "wait": "nothing can be done yet"}},
+ {"type": "noul", "instructions": "Does the user ask to speak to a human?"},
+ {"type": "noul", "instructions": "Is a delivery late?"},
+]
+SUBJECTS = ["my order #4411", "the March invoice", "the export feature", "my account login"]
+PROBLEMS = [
+ "was charged twice and I want the duplicate back",
+ "has not arrived although tracking says it was delivered last week",
+ "stopped working this morning and the whole team is blocked",
+ "may or may not be fine, I honestly cannot tell from the page",
+ "is wrong, but before anything else I would like to know my options",
+ "was fine, thank you, though the refund you promised has not shown up yet",
+]
+SHORT = [{"subject": "About " + s, "body": "Hello, %s %s." % (s, p)} for s in SUBJECTS for p in PROBLEMS]
+# A long state is a thread: the request plus the five that follow it in the list.
+LONG = [{"thread": [SHORT[(i + k) % len(SHORT)] for k in range(6)]} for i in range(len(SHORT))]
+
+
+def chip():
+ try:
+ return subprocess.run(["sysctl", "-n", "machdep.cpu.brand_string"], capture_output=True, text=True,
+ timeout=5, check=True).stdout.strip()
+ except (OSError, subprocess.SubprocessError):
+ return platform.processor()
+
+
+def call(agent, state, questions, min_rows):
+ agent.mps_amp_min_rows = min_rows
+ t0 = time.perf_counter()
+ out = agent.system_one(state, questions)
+ return (time.perf_counter() - t0) * 1000, out
+
+
+def compare(a, b):
+ """(probability changes, decisions changed) between two answers to the same questions."""
+ deltas, changed = [], 0
+ for qid, x in a.items():
+ y = b[qid]
+ if x["type"] == "noul":
+ deltas.append(abs(x["noul"] - y["noul"]))
+ changed += (x["noul"] >= 0.5) != (y["noul"] >= 0.5)
+ else:
+ deltas += [abs(p - y["probabilities"][k]) for k, p in x["probabilities"].items()]
+ changed += max(x["probabilities"], key=x["probabilities"].get) != max(
+ y["probabilities"], key=y["probabilities"].get)
+ return deltas, changed
+
+
+def cell(agent, states, rows, repeats, warmup):
+ questions = {"q%d" % i: q for i, q in enumerate(QUESTIONS[:rows])}
+ fp32, fp16 = rows + 1, 1
+ for state in states[:warmup]: # both paths, on this shape
+ call(agent, state, questions, fp32)
+ call(agent, state, questions, fp16)
+ ms32, ms16, deltas, changed, decisions, tokens = [], [], [], 0, 0, 0
+ for rep in range(repeats):
+ for i, state in enumerate(states):
+ first, second = (fp32, fp16) if (i + rep) % 2 else (fp16, fp32)
+ results = {first: call(agent, state, questions, first), second: call(agent, state, questions, second)}
+ ms32.append(results[fp32][0])
+ ms16.append(results[fp16][0])
+ tokens = max(tokens, results[fp32][1]["usage"]["input_tokens"])
+ if rep == 0:
+ d, c = compare(results[fp32][1]["answers"], results[fp16][1]["answers"])
+ deltas += d
+ changed += c
+ decisions += rows
+ deltas.sort()
+ return {"rows": rows, "pairs": len(ms32), "max_input_tokens": tokens,
+ "fp32_ms": round(statistics.median(ms32), 1), "fp16_ms": round(statistics.median(ms16), 1),
+ "paired_ms": round(statistics.median(b - a for a, b in zip(ms32, ms16)), 1),
+ "max_delta": round(deltas[-1], 4), "p95_delta": round(deltas[int(0.95 * (len(deltas) - 1))], 4),
+ "decisions_changed": changed, "decisions": decisions}
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+ parser.add_argument("--checkpoint", default="english", choices=["english", "multilingual", "typed-decisions"])
+ parser.add_argument("--rows", default="1,2,4,5,6,8", help="question rows per request, comma separated")
+ parser.add_argument("--repeats", type=int, default=3, help="passes over the %d states" % len(SHORT))
+ parser.add_argument("--warmup", type=int, default=3, help="states run in both modes before timing")
+ parser.add_argument("--out", help="write the results as JSON")
+ args = parser.parse_args()
+
+ import torch
+ import transformers
+
+ import laya
+
+ if not torch.backends.mps.is_available():
+ raise SystemExit("this benchmark needs an MPS device")
+ agent = laya.load(BUNDLE, device="mps", subfolder=None if args.checkpoint == "english" else args.checkpoint)
+ default_rows = agent.mps_amp_min_rows
+ result = {"checkpoint": args.checkpoint, "chip": chip(), "os": "macOS " + platform.mac_ver()[0],
+ "torch": torch.__version__, "transformers": transformers.__version__, "laya": laya.__version__,
+ "default_mps_amp_min_rows": default_rows, "load_1m": round(os.getloadavg()[0], 1), "cells": []}
+ print("%s on %s, laya %s, torch %s (1-min load %.1f)"
+ % (args.checkpoint, result["chip"], laya.__version__, torch.__version__, result["load_1m"]))
+ print("| state | rows | input tokens | fp32 ms | fp16 ms | fp16 - fp32 (paired) | max delta | p95 delta "
+ "| decisions changed |")
+ print("|---|---|---|---|---|---|---|---|---|")
+ for name, states in (("short", SHORT), ("long", LONG)):
+ for rows in [int(r) for r in args.rows.split(",")]:
+ row = {"state": name, **cell(agent, states, rows, args.repeats, args.warmup)}
+ result["cells"].append(row)
+ print("| %s | %d | %d | %.1f | %.1f | %+.1f | %.4f | %.4f | %d / %d |"
+ % (name, rows, row["max_input_tokens"], row["fp32_ms"], row["fp16_ms"], row["paired_ms"],
+ row["max_delta"], row["p95_delta"], row["decisions_changed"], row["decisions"]))
+ agent.mps_amp_min_rows = default_rows
+ if args.out:
+ with open(args.out, "w") as f:
+ json.dump(result, f, indent=1)
+ f.write("\n")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/results/mps_autocast_english_m1pro.json b/benchmarks/results/mps_autocast_english_m1pro.json
new file mode 100644
index 00000000..e0be38d8
--- /dev/null
+++ b/benchmarks/results/mps_autocast_english_m1pro.json
@@ -0,0 +1,168 @@
+{
+ "checkpoint": "english",
+ "chip": "Apple M1 Pro",
+ "os": "macOS 26.1",
+ "torch": "2.14.0",
+ "transformers": "5.17.0",
+ "laya": "0.3.26",
+ "default_mps_amp_min_rows": 5,
+ "load_1m": 6.0,
+ "cells": [
+ {
+ "state": "short",
+ "rows": 1,
+ "pairs": 72,
+ "max_input_tokens": 87,
+ "fp32_ms": 58.2,
+ "fp16_ms": 79.4,
+ "paired_ms": 21.0,
+ "max_delta": 0.0011,
+ "p95_delta": 0.0006,
+ "decisions_changed": 0,
+ "decisions": 24
+ },
+ {
+ "state": "short",
+ "rows": 2,
+ "pairs": 72,
+ "max_input_tokens": 161,
+ "fp32_ms": 77.0,
+ "fp16_ms": 99.6,
+ "paired_ms": 22.5,
+ "max_delta": 0.0061,
+ "p95_delta": 0.0007,
+ "decisions_changed": 0,
+ "decisions": 48
+ },
+ {
+ "state": "short",
+ "rows": 4,
+ "pairs": 72,
+ "max_input_tokens": 309,
+ "fp32_ms": 147.6,
+ "fp16_ms": 170.4,
+ "paired_ms": 23.6,
+ "max_delta": 0.0061,
+ "p95_delta": 0.0008,
+ "decisions_changed": 0,
+ "decisions": 96
+ },
+ {
+ "state": "short",
+ "rows": 5,
+ "pairs": 72,
+ "max_input_tokens": 380,
+ "fp32_ms": 190.4,
+ "fp16_ms": 213.6,
+ "paired_ms": 22.9,
+ "max_delta": 0.0061,
+ "p95_delta": 0.0008,
+ "decisions_changed": 0,
+ "decisions": 120
+ },
+ {
+ "state": "short",
+ "rows": 6,
+ "pairs": 72,
+ "max_input_tokens": 456,
+ "fp32_ms": 233.5,
+ "fp16_ms": 260.4,
+ "paired_ms": 27.0,
+ "max_delta": 0.0061,
+ "p95_delta": 0.0009,
+ "decisions_changed": 0,
+ "decisions": 144
+ },
+ {
+ "state": "short",
+ "rows": 8,
+ "pairs": 72,
+ "max_input_tokens": 603,
+ "fp32_ms": 290.6,
+ "fp16_ms": 316.4,
+ "paired_ms": 28.6,
+ "max_delta": 0.0061,
+ "p95_delta": 0.001,
+ "decisions_changed": 0,
+ "decisions": 192
+ },
+ {
+ "state": "long",
+ "rows": 1,
+ "pairs": 72,
+ "max_input_tokens": 270,
+ "fp32_ms": 133.7,
+ "fp16_ms": 157.3,
+ "paired_ms": 24.2,
+ "max_delta": 0.0081,
+ "p95_delta": 0.0029,
+ "decisions_changed": 0,
+ "decisions": 24
+ },
+ {
+ "state": "long",
+ "rows": 2,
+ "pairs": 72,
+ "max_input_tokens": 527,
+ "fp32_ms": 259.6,
+ "fp16_ms": 292.1,
+ "paired_ms": 32.1,
+ "max_delta": 0.0081,
+ "p95_delta": 0.0029,
+ "decisions_changed": 0,
+ "decisions": 48
+ },
+ {
+ "state": "long",
+ "rows": 4,
+ "pairs": 72,
+ "max_input_tokens": 1041,
+ "fp32_ms": 458.1,
+ "fp16_ms": 492.6,
+ "paired_ms": 23.8,
+ "max_delta": 0.0081,
+ "p95_delta": 0.0029,
+ "decisions_changed": 0,
+ "decisions": 96
+ },
+ {
+ "state": "long",
+ "rows": 5,
+ "pairs": 72,
+ "max_input_tokens": 1295,
+ "fp32_ms": 394.4,
+ "fp16_ms": 403.3,
+ "paired_ms": 9.1,
+ "max_delta": 0.0081,
+ "p95_delta": 0.0028,
+ "decisions_changed": 0,
+ "decisions": 120
+ },
+ {
+ "state": "long",
+ "rows": 6,
+ "pairs": 72,
+ "max_input_tokens": 1554,
+ "fp32_ms": 417.4,
+ "fp16_ms": 421.9,
+ "paired_ms": 4.6,
+ "max_delta": 0.0081,
+ "p95_delta": 0.0027,
+ "decisions_changed": 0,
+ "decisions": 144
+ },
+ {
+ "state": "long",
+ "rows": 8,
+ "pairs": 72,
+ "max_input_tokens": 2067,
+ "fp32_ms": 585.5,
+ "fp16_ms": 566.6,
+ "paired_ms": -13.7,
+ "max_delta": 0.0081,
+ "p95_delta": 0.0028,
+ "decisions_changed": 0,
+ "decisions": 192
+ }
+ ]
+}
diff --git a/benchmarks/results/mps_autocast_multilingual_m1pro.json b/benchmarks/results/mps_autocast_multilingual_m1pro.json
new file mode 100644
index 00000000..475e2ead
--- /dev/null
+++ b/benchmarks/results/mps_autocast_multilingual_m1pro.json
@@ -0,0 +1,168 @@
+{
+ "checkpoint": "multilingual",
+ "chip": "Apple M1 Pro",
+ "os": "macOS 26.1",
+ "torch": "2.14.0",
+ "transformers": "5.17.0",
+ "laya": "0.3.26",
+ "default_mps_amp_min_rows": 5,
+ "load_1m": 10.3,
+ "cells": [
+ {
+ "state": "short",
+ "rows": 1,
+ "pairs": 72,
+ "max_input_tokens": 88,
+ "fp32_ms": 25.5,
+ "fp16_ms": 34.7,
+ "paired_ms": 9.8,
+ "max_delta": 0.0038,
+ "p95_delta": 0.0022,
+ "decisions_changed": 0,
+ "decisions": 24
+ },
+ {
+ "state": "short",
+ "rows": 2,
+ "pairs": 72,
+ "max_input_tokens": 166,
+ "fp32_ms": 31.3,
+ "fp16_ms": 40.6,
+ "paired_ms": 9.2,
+ "max_delta": 0.0038,
+ "p95_delta": 0.0018,
+ "decisions_changed": 0,
+ "decisions": 48
+ },
+ {
+ "state": "short",
+ "rows": 4,
+ "pairs": 72,
+ "max_input_tokens": 325,
+ "fp32_ms": 45.9,
+ "fp16_ms": 55.7,
+ "paired_ms": 9.9,
+ "max_delta": 0.0038,
+ "p95_delta": 0.0011,
+ "decisions_changed": 0,
+ "decisions": 96
+ },
+ {
+ "state": "short",
+ "rows": 5,
+ "pairs": 72,
+ "max_input_tokens": 400,
+ "fp32_ms": 56.4,
+ "fp16_ms": 66.5,
+ "paired_ms": 10.3,
+ "max_delta": 0.0038,
+ "p95_delta": 0.001,
+ "decisions_changed": 0,
+ "decisions": 120
+ },
+ {
+ "state": "short",
+ "rows": 6,
+ "pairs": 72,
+ "max_input_tokens": 479,
+ "fp32_ms": 63.8,
+ "fp16_ms": 72.1,
+ "paired_ms": 7.7,
+ "max_delta": 0.0038,
+ "p95_delta": 0.0012,
+ "decisions_changed": 0,
+ "decisions": 144
+ },
+ {
+ "state": "short",
+ "rows": 8,
+ "pairs": 72,
+ "max_input_tokens": 634,
+ "fp32_ms": 78.1,
+ "fp16_ms": 90.9,
+ "paired_ms": 11.8,
+ "max_delta": 0.0038,
+ "p95_delta": 0.0012,
+ "decisions_changed": 1,
+ "decisions": 192
+ },
+ {
+ "state": "long",
+ "rows": 1,
+ "pairs": 72,
+ "max_input_tokens": 291,
+ "fp32_ms": 44.0,
+ "fp16_ms": 53.7,
+ "paired_ms": 9.3,
+ "max_delta": 0.0107,
+ "p95_delta": 0.0058,
+ "decisions_changed": 0,
+ "decisions": 24
+ },
+ {
+ "state": "long",
+ "rows": 2,
+ "pairs": 72,
+ "max_input_tokens": 572,
+ "fp32_ms": 78.5,
+ "fp16_ms": 86.0,
+ "paired_ms": 8.9,
+ "max_delta": 0.0106,
+ "p95_delta": 0.0058,
+ "decisions_changed": 0,
+ "decisions": 48
+ },
+ {
+ "state": "long",
+ "rows": 4,
+ "pairs": 72,
+ "max_input_tokens": 1137,
+ "fp32_ms": 145.4,
+ "fp16_ms": 157.1,
+ "paired_ms": 11.7,
+ "max_delta": 0.0106,
+ "p95_delta": 0.0047,
+ "decisions_changed": 0,
+ "decisions": 96
+ },
+ {
+ "state": "long",
+ "rows": 5,
+ "pairs": 72,
+ "max_input_tokens": 1415,
+ "fp32_ms": 183.6,
+ "fp16_ms": 195.1,
+ "paired_ms": 12.9,
+ "max_delta": 0.0106,
+ "p95_delta": 0.0048,
+ "decisions_changed": 0,
+ "decisions": 120
+ },
+ {
+ "state": "long",
+ "rows": 6,
+ "pairs": 72,
+ "max_input_tokens": 1697,
+ "fp32_ms": 217.0,
+ "fp16_ms": 233.4,
+ "paired_ms": 14.3,
+ "max_delta": 0.0206,
+ "p95_delta": 0.0082,
+ "decisions_changed": 0,
+ "decisions": 144
+ },
+ {
+ "state": "long",
+ "rows": 8,
+ "pairs": 72,
+ "max_input_tokens": 2258,
+ "fp32_ms": 267.6,
+ "fp16_ms": 283.6,
+ "paired_ms": 17.2,
+ "max_delta": 0.0206,
+ "p95_delta": 0.0091,
+ "decisions_changed": 0,
+ "decisions": 192
+ }
+ ]
+}