Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(hooks): say what OptionStabilityHook costs on a CPU

Nandakishordf8fe7c1ae75
diff --git a/docs/hooks/api.md b/docs/hooks/api.md
index 3bf910d9..39805925 100644
--- a/docs/hooks/api.md
+++ b/docs/hooks/api.md
@@ -172,6 +172,23 @@ It attaches a `reliability` diagnostic block to each choice question's answer:
- `variant_choices`: Mapping of variant name to decoded choice.
- `variant_support`: Mapping of variant name to probability mass on the primary choice.
+### What it costs
+
+One forward pass, not several, but a wider one: the variants are extra rows in the same batch.
+A four-option `choice` costs 3 rows under `variants="fast"` and 7 under `"full"`, against 1 with
+no hook. On a GPU or MPS those extra rows are largely absorbed by the batch; on a CPU they are
+not, and the cost is close to linear in rows. Measured here on one 4-option choice, CPU, median
+of 15 calls after 4 warmups:
+
+| | rows | median | against baseline |
+|---|---:|---:|---:|
+| no hook | 1 | 62.5 ms | 1.00x |
+| `variants="fast"` | 3 | 123.9 ms | 1.98x |
+| `variants="full"` | 7 | 230.5 ms | 3.69x |
+
+So on CPU this is a diagnostic to enable deliberately, per request or on a sample of traffic,
+rather than everywhere by default. Scale the row counts by option count from the table in #908.
+
## Convenience types
```python