Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(benchmarks): MPS fp16 autocast against fp32 on an Apple M5 Pro

johnseongefbe194ba542
diff --git a/BENCHMARKS.md b/BENCHMARKS.md
index 148fa523..5a484f6d 100644
--- a/BENCHMARKS.md
+++ b/BENCHMARKS.md
@@ -310,6 +310,50 @@ fp16 costs about 20 to 30 ms per request on `english` and about 10 ms on `multil
Of 1,248 decisions per checkpoint, fp16 changed none on `english` and one on `multilingual` (a `noul` next to 0.5). A decision near the boundary can flip.
+### Apple M5 Pro, MPS: fp16 autocast against fp32
+
+The same script, unchanged, on an Apple M5 Pro (24 GB, macOS 27.0.1, Python 3.12, torch 2.14.1, transformers 5.19.0, laya 0.4.6). Same protocol: 72 pairs per row, 24 states, three passes, fp32 and fp16 back to back on one loaded agent with the order alternating. The machine was lightly loaded (1-min load 4.3 during `english`, 0.8 during `multilingual`). Results in `benchmarks/results/mps_autocast_*_m5pro.json`.
+
+Here the gate goes the way #109 measured on its M5, and further: fp16 wins from two rows on short states and from one row on long states, on both checkpoints.
+
+`english` (ModernBERT-large), median ms, and fp16 minus fp32 per request:
+
+| state | rows | fp32 | fp16 | fp16 - fp32 | max probability change |
+|---|---|---|---|---|---|
+| short | 1 | 25.6 | 25.5 | -0.3 | 0.0018 |
+| short | 2 | 36.6 | 30.3 | -5.8 | 0.0018 |
+| short | 4 | 64.4 | 40.2 | -23.7 | 0.0031 |
+| short | 5 | 74.3 | 44.7 | -29.4 | 0.0031 |
+| short | 6 | 84.5 | 49.1 | -35.3 | 0.0031 |
+| short | 8 | 110.3 | 59.0 | -51.1 | 0.0126 |
+| long | 1 | 48.8 | 34.3 | -15.0 | 0.0012 |
+| long | 2 | 86.6 | 50.8 | -36.7 | 0.0012 |
+| long | 4 | 165.1 | 79.0 | -86.5 | 0.0014 |
+| long | 5 | 203.5 | 93.3 | -110.6 | 0.0014 |
+| long | 6 | 241.4 | 108.1 | -133.7 | 0.0014 |
+| long | 8 | 312.6 | 135.7 | -177.4 | 0.0021 |
+
+`multilingual` (mmBERT-base):
+
+| state | rows | fp32 | fp16 | fp16 - fp32 | max probability change |
+|---|---|---|---|---|---|
+| short | 1 | 12.1 | 13.4 | +1.3 | 0.0022 |
+| short | 2 | 17.2 | 16.3 | -1.1 | 0.0022 |
+| short | 4 | 25.5 | 19.7 | -5.8 | 0.0022 |
+| short | 5 | 30.7 | 22.2 | -8.4 | 0.0022 |
+| short | 6 | 34.9 | 24.5 | -10.5 | 0.0022 |
+| short | 8 | 44.3 | 29.6 | -14.6 | 0.0022 |
+| long | 1 | 21.5 | 17.3 | -4.3 | 0.0020 |
+| long | 2 | 36.4 | 25.0 | -11.7 | 0.0021 |
+| long | 4 | 69.0 | 40.4 | -27.8 | 0.0021 |
+| long | 5 | 79.8 | 45.0 | -35.0 | 0.0021 |
+| long | 6 | 95.1 | 52.3 | -43.0 | 0.0050 |
+| long | 8 | 123.3 | 65.2 | -58.7 | 0.0050 |
+
+The saving grows with the work in the request: on `english` long states fp16 is 2.3x faster at eight rows (312.6 to 135.7 ms), and the only cell where fp32 wins is a single short `multilingual` row, by 1.3 ms. So at the default of 5 an M5 Pro runs rows 2 to 4 (and every long single-row request) in fp32 when fp16 would have been faster, while an M1 Pro pays for fp16 from row 5 up. The two tables bracket the current default; which way it should move, or whether it should depend on the chip, is a separate decision. `LAYA_MPS_AMP_MIN_ROWS=1` takes fp16 everywhere on an M5 Pro.
+
+Of 1,248 decisions per checkpoint, fp16 changed one on `english` (a short state at eight rows, where the largest probability change was 0.0126) and none on `multilingual`.
+
### Calibration on a routing task runs the other way
On laya_router's 180 requests (zero-shot, one 3-tier `choice`), nearly every configuration we measured was **under**-confident (the few exceptions were +0.01 to +0.06, and among the least accurate). Mean P(chosen) (the chosen option's probability, not the entropy-based `confidence` field) sat below accuracy, by −0.18 on the root checkpoint with example-led tier descriptions (0.562 vs 0.744) and by −0.19 on `typed-decisions` (0.501 vs 0.694). This is one task and one set of labels, so it does not contradict the over-confidence reported above. It does mean the direction of the miscalibration depends on the task, and a temperature fit on your own data is the right fix either way.
diff --git a/benchmarks/results/mps_autocast_english_m5pro.json b/benchmarks/results/mps_autocast_english_m5pro.json
new file mode 100644
index 00000000..148359f8
--- /dev/null
+++ b/benchmarks/results/mps_autocast_english_m5pro.json
@@ -0,0 +1,168 @@
+{
+ "checkpoint": "english",
+ "chip": "Apple M5 Pro",
+ "os": "macOS 27.0.1",
+ "torch": "2.14.1",
+ "transformers": "5.19.0",
+ "laya": "0.4.6",
+ "default_mps_amp_min_rows": 5,
+ "load_1m": 4.3,
+ "cells": [
+ {
+ "state": "short",
+ "rows": 1,
+ "pairs": 72,
+ "max_input_tokens": 87,
+ "fp32_ms": 25.6,
+ "fp16_ms": 25.5,
+ "paired_ms": -0.3,
+ "max_delta": 0.0018,
+ "p95_delta": 0.0007,
+ "decisions_changed": 0,
+ "decisions": 24
+ },
+ {
+ "state": "short",
+ "rows": 2,
+ "pairs": 72,
+ "max_input_tokens": 161,
+ "fp32_ms": 36.6,
+ "fp16_ms": 30.3,
+ "paired_ms": -5.8,
+ "max_delta": 0.0018,
+ "p95_delta": 0.0009,
+ "decisions_changed": 0,
+ "decisions": 48
+ },
+ {
+ "state": "short",
+ "rows": 4,
+ "pairs": 72,
+ "max_input_tokens": 309,
+ "fp32_ms": 64.4,
+ "fp16_ms": 40.2,
+ "paired_ms": -23.7,
+ "max_delta": 0.0031,
+ "p95_delta": 0.0008,
+ "decisions_changed": 0,
+ "decisions": 96
+ },
+ {
+ "state": "short",
+ "rows": 5,
+ "pairs": 72,
+ "max_input_tokens": 380,
+ "fp32_ms": 74.3,
+ "fp16_ms": 44.7,
+ "paired_ms": -29.4,
+ "max_delta": 0.0031,
+ "p95_delta": 0.0008,
+ "decisions_changed": 0,
+ "decisions": 120
+ },
+ {
+ "state": "short",
+ "rows": 6,
+ "pairs": 72,
+ "max_input_tokens": 456,
+ "fp32_ms": 84.5,
+ "fp16_ms": 49.1,
+ "paired_ms": -35.3,
+ "max_delta": 0.0031,
+ "p95_delta": 0.0009,
+ "decisions_changed": 0,
+ "decisions": 144
+ },
+ {
+ "state": "short",
+ "rows": 8,
+ "pairs": 72,
+ "max_input_tokens": 603,
+ "fp32_ms": 110.3,
+ "fp16_ms": 59.0,
+ "paired_ms": -51.1,
+ "max_delta": 0.0126,
+ "p95_delta": 0.0009,
+ "decisions_changed": 1,
+ "decisions": 192
+ },
+ {
+ "state": "long",
+ "rows": 1,
+ "pairs": 72,
+ "max_input_tokens": 270,
+ "fp32_ms": 48.8,
+ "fp16_ms": 34.3,
+ "paired_ms": -15.0,
+ "max_delta": 0.0012,
+ "p95_delta": 0.0006,
+ "decisions_changed": 0,
+ "decisions": 24
+ },
+ {
+ "state": "long",
+ "rows": 2,
+ "pairs": 72,
+ "max_input_tokens": 527,
+ "fp32_ms": 86.6,
+ "fp16_ms": 50.8,
+ "paired_ms": -36.7,
+ "max_delta": 0.0012,
+ "p95_delta": 0.0006,
+ "decisions_changed": 0,
+ "decisions": 48
+ },
+ {
+ "state": "long",
+ "rows": 4,
+ "pairs": 72,
+ "max_input_tokens": 1041,
+ "fp32_ms": 165.1,
+ "fp16_ms": 79.0,
+ "paired_ms": -86.5,
+ "max_delta": 0.0014,
+ "p95_delta": 0.0006,
+ "decisions_changed": 0,
+ "decisions": 96
+ },
+ {
+ "state": "long",
+ "rows": 5,
+ "pairs": 72,
+ "max_input_tokens": 1295,
+ "fp32_ms": 203.5,
+ "fp16_ms": 93.3,
+ "paired_ms": -110.6,
+ "max_delta": 0.0014,
+ "p95_delta": 0.0006,
+ "decisions_changed": 0,
+ "decisions": 120
+ },
+ {
+ "state": "long",
+ "rows": 6,
+ "pairs": 72,
+ "max_input_tokens": 1554,
+ "fp32_ms": 241.4,
+ "fp16_ms": 108.1,
+ "paired_ms": -133.7,
+ "max_delta": 0.0014,
+ "p95_delta": 0.0006,
+ "decisions_changed": 0,
+ "decisions": 144
+ },
+ {
+ "state": "long",
+ "rows": 8,
+ "pairs": 72,
+ "max_input_tokens": 2067,
+ "fp32_ms": 312.6,
+ "fp16_ms": 135.7,
+ "paired_ms": -177.4,
+ "max_delta": 0.0021,
+ "p95_delta": 0.0007,
+ "decisions_changed": 0,
+ "decisions": 192
+ }
+ ]
+}
diff --git a/benchmarks/results/mps_autocast_multilingual_m5pro.json b/benchmarks/results/mps_autocast_multilingual_m5pro.json
new file mode 100644
index 00000000..1e12ad89
--- /dev/null
+++ b/benchmarks/results/mps_autocast_multilingual_m5pro.json
@@ -0,0 +1,168 @@
+{
+ "checkpoint": "multilingual",
+ "chip": "Apple M5 Pro",
+ "os": "macOS 27.0.1",
+ "torch": "2.14.1",
+ "transformers": "5.19.0",
+ "laya": "0.4.6",
+ "default_mps_amp_min_rows": 5,
+ "load_1m": 0.8,
+ "cells": [
+ {
+ "state": "short",
+ "rows": 1,
+ "pairs": 72,
+ "max_input_tokens": 88,
+ "fp32_ms": 12.1,
+ "fp16_ms": 13.4,
+ "paired_ms": 1.3,
+ "max_delta": 0.0022,
+ "p95_delta": 0.0007,
+ "decisions_changed": 0,
+ "decisions": 24
+ },
+ {
+ "state": "short",
+ "rows": 2,
+ "pairs": 72,
+ "max_input_tokens": 166,
+ "fp32_ms": 17.2,
+ "fp16_ms": 16.3,
+ "paired_ms": -1.1,
+ "max_delta": 0.0022,
+ "p95_delta": 0.0008,
+ "decisions_changed": 0,
+ "decisions": 48
+ },
+ {
+ "state": "short",
+ "rows": 4,
+ "pairs": 72,
+ "max_input_tokens": 325,
+ "fp32_ms": 25.5,
+ "fp16_ms": 19.7,
+ "paired_ms": -5.8,
+ "max_delta": 0.0022,
+ "p95_delta": 0.0004,
+ "decisions_changed": 0,
+ "decisions": 96
+ },
+ {
+ "state": "short",
+ "rows": 5,
+ "pairs": 72,
+ "max_input_tokens": 400,
+ "fp32_ms": 30.7,
+ "fp16_ms": 22.2,
+ "paired_ms": -8.4,
+ "max_delta": 0.0022,
+ "p95_delta": 0.0004,
+ "decisions_changed": 0,
+ "decisions": 120
+ },
+ {
+ "state": "short",
+ "rows": 6,
+ "pairs": 72,
+ "max_input_tokens": 479,
+ "fp32_ms": 34.9,
+ "fp16_ms": 24.5,
+ "paired_ms": -10.5,
+ "max_delta": 0.0022,
+ "p95_delta": 0.0006,
+ "decisions_changed": 0,
+ "decisions": 144
+ },
+ {
+ "state": "short",
+ "rows": 8,
+ "pairs": 72,
+ "max_input_tokens": 634,
+ "fp32_ms": 44.3,
+ "fp16_ms": 29.6,
+ "paired_ms": -14.6,
+ "max_delta": 0.0022,
+ "p95_delta": 0.0006,
+ "decisions_changed": 0,
+ "decisions": 192
+ },
+ {
+ "state": "long",
+ "rows": 1,
+ "pairs": 72,
+ "max_input_tokens": 291,
+ "fp32_ms": 21.5,
+ "fp16_ms": 17.3,
+ "paired_ms": -4.3,
+ "max_delta": 0.002,
+ "p95_delta": 0.0014,
+ "decisions_changed": 0,
+ "decisions": 24
+ },
+ {
+ "state": "long",
+ "rows": 2,
+ "pairs": 72,
+ "max_input_tokens": 572,
+ "fp32_ms": 36.4,
+ "fp16_ms": 25.0,
+ "paired_ms": -11.7,
+ "max_delta": 0.0021,
+ "p95_delta": 0.0014,
+ "decisions_changed": 0,
+ "decisions": 48
+ },
+ {
+ "state": "long",
+ "rows": 4,
+ "pairs": 72,
+ "max_input_tokens": 1137,
+ "fp32_ms": 69.0,
+ "fp16_ms": 40.4,
+ "paired_ms": -27.8,
+ "max_delta": 0.0021,
+ "p95_delta": 0.0009,
+ "decisions_changed": 0,
+ "decisions": 96
+ },
+ {
+ "state": "long",
+ "rows": 5,
+ "pairs": 72,
+ "max_input_tokens": 1415,
+ "fp32_ms": 79.8,
+ "fp16_ms": 45.0,
+ "paired_ms": -35.0,
+ "max_delta": 0.0021,
+ "p95_delta": 0.0008,
+ "decisions_changed": 0,
+ "decisions": 120
+ },
+ {
+ "state": "long",
+ "rows": 6,
+ "pairs": 72,
+ "max_input_tokens": 1697,
+ "fp32_ms": 95.1,
+ "fp16_ms": 52.3,
+ "paired_ms": -43.0,
+ "max_delta": 0.005,
+ "p95_delta": 0.0015,
+ "decisions_changed": 0,
+ "decisions": 144
+ },
+ {
+ "state": "long",
+ "rows": 8,
+ "pairs": 72,
+ "max_input_tokens": 2258,
+ "fp32_ms": 123.3,
+ "fp16_ms": 65.2,
+ "paired_ms": -58.7,
+ "max_delta": 0.005,
+ "p95_delta": 0.0015,
+ "decisions_changed": 0,
+ "decisions": 192
+ }
+ ]
+}