Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(benchmarks): MPS fp16 autocast against fp32 on an Apple M5 Pro
johnseongefbe194ba542
diff --git a/BENCHMARKS.md b/BENCHMARKS.mdindex 148fa523..5a484f6d 100644--- a/BENCHMARKS.md+++ b/BENCHMARKS.md@@ -310,6 +310,50 @@ fp16 costs about 20 to 30 ms per request on `english` and about 10 ms on `multilOf 1,248 decisions per checkpoint, fp16 changed none on `english` and one on `multilingual` (a `noul` next to 0.5). A decision near the boundary can flip.+### Apple M5 Pro, MPS: fp16 autocast against fp32++The same script, unchanged, on an Apple M5 Pro (24 GB, macOS 27.0.1, Python 3.12, torch 2.14.1, transformers 5.19.0, laya 0.4.6). Same protocol: 72 pairs per row, 24 states, three passes, fp32 and fp16 back to back on one loaded agent with the order alternating. The machine was lightly loaded (1-min load 4.3 during `english`, 0.8 during `multilingual`). Results in `benchmarks/results/mps_autocast_*_m5pro.json`.++Here the gate goes the way #109 measured on its M5, and further: fp16 wins from two rows on short states and from one row on long states, on both checkpoints.++`english` (ModernBERT-large), median ms, and fp16 minus fp32 per request:++| state | rows | fp32 | fp16 | fp16 - fp32 | max probability change |+|---|---|---|---|---|---|+| short | 1 | 25.6 | 25.5 | -0.3 | 0.0018 |+| short | 2 | 36.6 | 30.3 | -5.8 | 0.0018 |+| short | 4 | 64.4 | 40.2 | -23.7 | 0.0031 |+| short | 5 | 74.3 | 44.7 | -29.4 | 0.0031 |+| short | 6 | 84.5 | 49.1 | -35.3 | 0.0031 |+| short | 8 | 110.3 | 59.0 | -51.1 | 0.0126 |+| long | 1 | 48.8 | 34.3 | -15.0 | 0.0012 |+| long | 2 | 86.6 | 50.8 | -36.7 | 0.0012 |+| long | 4 | 165.1 | 79.0 | -86.5 | 0.0014 |+| long | 5 | 203.5 | 93.3 | -110.6 | 0.0014 |+| long | 6 | 241.4 | 108.1 | -133.7 | 0.0014 |+| long | 8 | 312.6 | 135.7 | -177.4 | 0.0021 |++`multilingual` (mmBERT-base):++| state | rows | fp32 | fp16 | fp16 - fp32 | max probability change |+|---|---|---|---|---|---|+| short | 1 | 12.1 | 13.4 | +1.3 | 0.0022 |+| short | 2 | 17.2 | 16.3 | -1.1 | 0.0022 |+| short | 4 | 25.5 | 19.7 | -5.8 | 0.0022 |+| short | 5 | 30.7 | 22.2 | -8.4 | 0.0022 |+| short | 6 | 34.9 | 24.5 | -10.5 | 0.0022 |+| short | 8 | 44.3 | 29.6 | -14.6 | 0.0022 |+| long | 1 | 21.5 | 17.3 | -4.3 | 0.0020 |+| long | 2 | 36.4 | 25.0 | -11.7 | 0.0021 |+| long | 4 | 69.0 | 40.4 | -27.8 | 0.0021 |+| long | 5 | 79.8 | 45.0 | -35.0 | 0.0021 |+| long | 6 | 95.1 | 52.3 | -43.0 | 0.0050 |+| long | 8 | 123.3 | 65.2 | -58.7 | 0.0050 |++The saving grows with the work in the request: on `english` long states fp16 is 2.3x faster at eight rows (312.6 to 135.7 ms), and the only cell where fp32 wins is a single short `multilingual` row, by 1.3 ms. So at the default of 5 an M5 Pro runs rows 2 to 4 (and every long single-row request) in fp32 when fp16 would have been faster, while an M1 Pro pays for fp16 from row 5 up. The two tables bracket the current default; which way it should move, or whether it should depend on the chip, is a separate decision. `LAYA_MPS_AMP_MIN_ROWS=1` takes fp16 everywhere on an M5 Pro.++Of 1,248 decisions per checkpoint, fp16 changed one on `english` (a short state at eight rows, where the largest probability change was 0.0126) and none on `multilingual`.+### Calibration on a routing task runs the other wayOn laya_router's 180 requests (zero-shot, one 3-tier `choice`), nearly every configuration we measured was **under**-confident (the few exceptions were +0.01 to +0.06, and among the least accurate). Mean P(chosen) (the chosen option's probability, not the entropy-based `confidence` field) sat below accuracy, by −0.18 on the root checkpoint with example-led tier descriptions (0.562 vs 0.744) and by −0.19 on `typed-decisions` (0.501 vs 0.694). This is one task and one set of labels, so it does not contradict the over-confidence reported above. It does mean the direction of the miscalibration depends on the task, and a temperature fit on your own data is the right fix either way.diff --git a/benchmarks/results/mps_autocast_english_m5pro.json b/benchmarks/results/mps_autocast_english_m5pro.jsonnew file mode 100644index 00000000..148359f8--- /dev/null+++ b/benchmarks/results/mps_autocast_english_m5pro.json@@ -0,0 +1,168 @@+{+ "checkpoint": "english",+ "chip": "Apple M5 Pro",+ "os": "macOS 27.0.1",+ "torch": "2.14.1",+ "transformers": "5.19.0",+ "laya": "0.4.6",+ "default_mps_amp_min_rows": 5,+ "load_1m": 4.3,+ "cells": [+ {+ "state": "short",+ "rows": 1,+ "pairs": 72,+ "max_input_tokens": 87,+ "fp32_ms": 25.6,+ "fp16_ms": 25.5,+ "paired_ms": -0.3,+ "max_delta": 0.0018,+ "p95_delta": 0.0007,+ "decisions_changed": 0,+ "decisions": 24+ },+ {+ "state": "short",+ "rows": 2,+ "pairs": 72,+ "max_input_tokens": 161,+ "fp32_ms": 36.6,+ "fp16_ms": 30.3,+ "paired_ms": -5.8,+ "max_delta": 0.0018,+ "p95_delta": 0.0009,+ "decisions_changed": 0,+ "decisions": 48+ },+ {+ "state": "short",+ "rows": 4,+ "pairs": 72,+ "max_input_tokens": 309,+ "fp32_ms": 64.4,+ "fp16_ms": 40.2,+ "paired_ms": -23.7,+ "max_delta": 0.0031,+ "p95_delta": 0.0008,+ "decisions_changed": 0,+ "decisions": 96+ },+ {+ "state": "short",+ "rows": 5,+ "pairs": 72,+ "max_input_tokens": 380,+ "fp32_ms": 74.3,+ "fp16_ms": 44.7,+ "paired_ms": -29.4,+ "max_delta": 0.0031,+ "p95_delta": 0.0008,+ "decisions_changed": 0,+ "decisions": 120+ },+ {+ "state": "short",+ "rows": 6,+ "pairs": 72,+ "max_input_tokens": 456,+ "fp32_ms": 84.5,+ "fp16_ms": 49.1,+ "paired_ms": -35.3,+ "max_delta": 0.0031,+ "p95_delta": 0.0009,+ "decisions_changed": 0,+ "decisions": 144+ },+ {+ "state": "short",+ "rows": 8,+ "pairs": 72,+ "max_input_tokens": 603,+ "fp32_ms": 110.3,+ "fp16_ms": 59.0,+ "paired_ms": -51.1,+ "max_delta": 0.0126,+ "p95_delta": 0.0009,+ "decisions_changed": 1,+ "decisions": 192+ },+ {+ "state": "long",+ "rows": 1,+ "pairs": 72,+ "max_input_tokens": 270,+ "fp32_ms": 48.8,+ "fp16_ms": 34.3,+ "paired_ms": -15.0,+ "max_delta": 0.0012,+ "p95_delta": 0.0006,+ "decisions_changed": 0,+ "decisions": 24+ },+ {+ "state": "long",+ "rows": 2,+ "pairs": 72,+ "max_input_tokens": 527,+ "fp32_ms": 86.6,+ "fp16_ms": 50.8,+ "paired_ms": -36.7,+ "max_delta": 0.0012,+ "p95_delta": 0.0006,+ "decisions_changed": 0,+ "decisions": 48+ },+ {+ "state": "long",+ "rows": 4,+ "pairs": 72,+ "max_input_tokens": 1041,+ "fp32_ms": 165.1,+ "fp16_ms": 79.0,+ "paired_ms": -86.5,+ "max_delta": 0.0014,+ "p95_delta": 0.0006,+ "decisions_changed": 0,+ "decisions": 96+ },+ {+ "state": "long",+ "rows": 5,+ "pairs": 72,+ "max_input_tokens": 1295,+ "fp32_ms": 203.5,+ "fp16_ms": 93.3,+ "paired_ms": -110.6,+ "max_delta": 0.0014,+ "p95_delta": 0.0006,+ "decisions_changed": 0,+ "decisions": 120+ },+ {+ "state": "long",+ "rows": 6,+ "pairs": 72,+ "max_input_tokens": 1554,+ "fp32_ms": 241.4,+ "fp16_ms": 108.1,+ "paired_ms": -133.7,+ "max_delta": 0.0014,+ "p95_delta": 0.0006,+ "decisions_changed": 0,+ "decisions": 144+ },+ {+ "state": "long",+ "rows": 8,+ "pairs": 72,+ "max_input_tokens": 2067,+ "fp32_ms": 312.6,+ "fp16_ms": 135.7,+ "paired_ms": -177.4,+ "max_delta": 0.0021,+ "p95_delta": 0.0007,+ "decisions_changed": 0,+ "decisions": 192+ }+ ]+}diff --git a/benchmarks/results/mps_autocast_multilingual_m5pro.json b/benchmarks/results/mps_autocast_multilingual_m5pro.jsonnew file mode 100644index 00000000..1e12ad89--- /dev/null+++ b/benchmarks/results/mps_autocast_multilingual_m5pro.json@@ -0,0 +1,168 @@+{+ "checkpoint": "multilingual",+ "chip": "Apple M5 Pro",+ "os": "macOS 27.0.1",+ "torch": "2.14.1",+ "transformers": "5.19.0",+ "laya": "0.4.6",+ "default_mps_amp_min_rows": 5,+ "load_1m": 0.8,+ "cells": [+ {+ "state": "short",+ "rows": 1,+ "pairs": 72,+ "max_input_tokens": 88,+ "fp32_ms": 12.1,+ "fp16_ms": 13.4,+ "paired_ms": 1.3,+ "max_delta": 0.0022,+ "p95_delta": 0.0007,+ "decisions_changed": 0,+ "decisions": 24+ },+ {+ "state": "short",+ "rows": 2,+ "pairs": 72,+ "max_input_tokens": 166,+ "fp32_ms": 17.2,+ "fp16_ms": 16.3,+ "paired_ms": -1.1,+ "max_delta": 0.0022,+ "p95_delta": 0.0008,+ "decisions_changed": 0,+ "decisions": 48+ },+ {+ "state": "short",+ "rows": 4,+ "pairs": 72,+ "max_input_tokens": 325,+ "fp32_ms": 25.5,+ "fp16_ms": 19.7,+ "paired_ms": -5.8,+ "max_delta": 0.0022,+ "p95_delta": 0.0004,+ "decisions_changed": 0,+ "decisions": 96+ },+ {+ "state": "short",+ "rows": 5,+ "pairs": 72,+ "max_input_tokens": 400,+ "fp32_ms": 30.7,+ "fp16_ms": 22.2,+ "paired_ms": -8.4,+ "max_delta": 0.0022,+ "p95_delta": 0.0004,+ "decisions_changed": 0,+ "decisions": 120+ },+ {+ "state": "short",+ "rows": 6,+ "pairs": 72,+ "max_input_tokens": 479,+ "fp32_ms": 34.9,+ "fp16_ms": 24.5,+ "paired_ms": -10.5,+ "max_delta": 0.0022,+ "p95_delta": 0.0006,+ "decisions_changed": 0,+ "decisions": 144+ },+ {+ "state": "short",+ "rows": 8,+ "pairs": 72,+ "max_input_tokens": 634,+ "fp32_ms": 44.3,+ "fp16_ms": 29.6,+ "paired_ms": -14.6,+ "max_delta": 0.0022,+ "p95_delta": 0.0006,+ "decisions_changed": 0,+ "decisions": 192+ },+ {+ "state": "long",+ "rows": 1,+ "pairs": 72,+ "max_input_tokens": 291,+ "fp32_ms": 21.5,+ "fp16_ms": 17.3,+ "paired_ms": -4.3,+ "max_delta": 0.002,+ "p95_delta": 0.0014,+ "decisions_changed": 0,+ "decisions": 24+ },+ {+ "state": "long",+ "rows": 2,+ "pairs": 72,+ "max_input_tokens": 572,+ "fp32_ms": 36.4,+ "fp16_ms": 25.0,+ "paired_ms": -11.7,+ "max_delta": 0.0021,+ "p95_delta": 0.0014,+ "decisions_changed": 0,+ "decisions": 48+ },+ {+ "state": "long",+ "rows": 4,+ "pairs": 72,+ "max_input_tokens": 1137,+ "fp32_ms": 69.0,+ "fp16_ms": 40.4,+ "paired_ms": -27.8,+ "max_delta": 0.0021,+ "p95_delta": 0.0009,+ "decisions_changed": 0,+ "decisions": 96+ },+ {+ "state": "long",+ "rows": 5,+ "pairs": 72,+ "max_input_tokens": 1415,+ "fp32_ms": 79.8,+ "fp16_ms": 45.0,+ "paired_ms": -35.0,+ "max_delta": 0.0021,+ "p95_delta": 0.0008,+ "decisions_changed": 0,+ "decisions": 120+ },+ {+ "state": "long",+ "rows": 6,+ "pairs": 72,+ "max_input_tokens": 1697,+ "fp32_ms": 95.1,+ "fp16_ms": 52.3,+ "paired_ms": -43.0,+ "max_delta": 0.005,+ "p95_delta": 0.0015,+ "decisions_changed": 0,+ "decisions": 144+ },+ {+ "state": "long",+ "rows": 8,+ "pairs": 72,+ "max_input_tokens": 2258,+ "fp32_ms": 123.3,+ "fp16_ms": 65.2,+ "paired_ms": -58.7,+ "max_delta": 0.005,+ "p95_delta": 0.0015,+ "decisions_changed": 0,+ "decisions": 192+ }+ ]+}