Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs: add the Questions and answers guide

PerryLink2b1f6c4e22f5
diff --git a/docs/questions-and-answers.md b/docs/questions-and-answers.md
new file mode 100644
index 00000000..69937629
--- /dev/null
+++ b/docs/questions-and-answers.md
@@ -0,0 +1,236 @@
+# Questions and answers
+
+A Laya question is a typed decision, and the type decides both what you ask and what you get
+back. There are three, and one state can carry all of them in a single forward pass:
+
+```python
+import laya
+
+agent = laya.load("convaiinnovations/laya")
+
+questions = {
+ "dept": {"type": "choice", "instructions": "Which team should handle this?",
+ "criteria": {"billing": "money and invoices",
+ "technical": "bugs and outages",
+ "sales": "pricing and contracts"}},
+ "urgent": {"type": "noul", "instructions": "Is this urgent?"},
+ "severity": {"type": "score", "instructions": "How severe is this?",
+ "criteria": ["trivial", "minor", "moderate", "serious", "critical"]},
+}
+
+result = agent.system_one({"text": "I was charged twice and nobody has replied for a week. "
+ "Please refund me."}, questions)
+result["answers"]["dept"]["choice"] # 'billing'
+result["answers"]["urgent"]["noul"] # 0.8727
+result["answers"]["severity"]["score"] # 2.9046
+```
+
+One forward pass answers all three. That is the point of the typed interface: a `noul` question
+is not a `choice` with two options that happen to be "yes" and "no" — it is a different head
+with a different output shape, and the type tells Laya which to use.
+
+## The three types
+
+### `choice` — pick one of a set
+
+```python
+{"type": "choice",
+ "instructions": "Which team should handle this?",
+ "criteria": {"billing": "money and invoices", "technical": "bugs and outages"}}
+```
+
+`criteria` is an ordered mapping of label to description. **Order is positional** in the rendered
+question, so two questions with the same labels in a different order are different questions.
+A description is optional; `{"billing": None}` renders the label alone. Labels are returned
+exactly as you wrote them, so a non-string label comes back as itself in `choice` and as the key
+in `probabilities`.
+
+The description is worth writing. It is not decoration: the rendered question text is what the
+model reads, so a bare `{"a": None, "b": None}` gives it nothing to distinguish the options.
+
+```python
+{"type": "choice", "choice": "billing",
+ "probabilities": {"billing": 0.9881, "technical": 0.0057, "sales": 0.0062},
+ "confidence": 0.9339, "answer_confidence": 0.9881,
+ "action": {"act_probability": 1.0}}
+```
+
+### `noul` — yes or no
+
+```python
+{"type": "noul", "instructions": "Is this urgent?"}
+```
+
+`noul` is [the project's name](https://en.wikipedia.org/wiki/Yes%E2%80%93no_question) for a
+two-way decision, and its answer is **the probability of true**, not a thresholded boolean:
+
+```python
+{"type": "noul", "noul": 0.8727, "confidence": 0.8727, "answer_confidence": 0.8727,
+ "action": {"act_probability": 1.0}}
+```
+
+The threshold is yours to choose, because it depends on what a false positive costs you. There is
+no `bool` field to mistake for a decision.
+
+You can relabel the two options when the decision is not naturally a yes/no — `labels` takes
+exactly the keys `false` and `true`, and the polarity is unchanged: `noul` is still P(true).
+
+```python
+{"type": "noul", "instructions": "Does this need a human?",
+ "labels": {"false": "automatic", "true": "escalate"}}
+```
+
+Labels are not a way to fix a question the model gets wrong. `noul` follows its own option
+labels, most strongly on the English checkpoint, so a label pair that reads as a decision
+("approve" / "reject") can pull the answer toward the label rather than the state. Validate any
+relabelling on your own data before relying on it.
+
+### `score` — an ordered level
+
+```python
+{"type": "score", "instructions": "How severe is this?",
+ "criteria": ["trivial", "minor", "moderate", "serious", "critical"]}
+```
+
+`criteria` is an ordered list, and it must be ordered ascending — position *is* the scale.
+
+```python
+{"type": "score", "score": 2.9046,
+ "legend": {"0": "trivial", "1": "minor", "2": "moderate", "3": "serious", "4": "critical"},
+ "probabilities": {"0": 0.0134, "1": 0.05, "2": 0.0518, "3": 0.7881, "4": 0.0967},
+ "confidence": 0.5187, "answer_confidence": 0.7881,
+ "action": {"act_probability": 1.0}}
+```
+
+**`score` is an expected value, not the most likely level.** Above, `score` is 2.90 while the
+single most likely level is `serious` (3) at 0.788. Both are useful and they answer different
+questions: the expected value minimises squared error over the scale, the argmax minimises
+disagreement with the model. If you want the label, take the argmax of `probabilities` or read
+`answer_confidence`'s counterpart from the legend — do not round `score` and assume it is the
+label. `legend` exists so you never have to guess which index means what.
+
+## Reading confidence
+
+Every answer carries two confidence numbers, and they measure different things.
+
+| field | what it is | gating on it? |
+|---|---|---|
+| `answer_confidence` | `max(p)` — the probability of the answer being reported | **yes, after fitting** |
+| `confidence` | `1 - H(p) / log(k)` — how concentrated the whole distribution is | no |
+| `probabilities` | the full distribution (`choice`, `score`) | — |
+
+`answer_confidence` is the quantity temperature scaling fits and the quantity every calibration
+figure in the repository is computed on, which is what makes it the one to gate on. It is **not
+calibrated as shipped**: the property usually attributed to it — that of the answers returned at
+confidence *c*, about *c* of them are right — holds only once temperatures have been fitted and
+validated on **held-out data for your checkpoint at your option count**. The shipped checkpoints
+are over-confident and how far depends on the option count, so an untuned threshold can select
+below the model's own accuracy ([#394](https://github.com/NandhaKishorM/laya/issues/394)).
+
+```python
+# THRESHOLD is a number you measured on your own held-out data, not one the model ships.
+# Fit and validate the temperatures first — the fine-tuning notebook has the loop:
+# notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb
+ans = result["answers"]["dept"]
+if ans["answer_confidence"] >= THRESHOLD:
+ ...
+```
+
+`confidence` is normalized entropy: high when the distribution is peaked, low when it is spread
+out, regardless of whether the top answer is correct. It is a useful signal and it is **not** on
+the same scale, so the two must not be gated against one number:
+
+```python
+# the same three answers, and the two numbers are not the same
+dept confidence 0.9339 answer_confidence 0.9881
+urgent confidence 0.8727 answer_confidence 0.8727
+severity confidence 0.5187 answer_confidence 0.7881
+```
+
+For `noul` they are equal by construction — over two options, `max(p, 1-p)` is `max(p)` — so a
+`noul` answer cannot tell you which one you have been reading. Both keys are present on every
+type so the choice is explicit rather than implied.
+
+**A threshold is a policy, not a property of the model.** Both checkpoints ship over-confident,
+and how over-confident depends on the option count, so a number measured on a 3-option question
+does not transfer to a 20-option one. Measure it on your own data; the Calibration section of
+[`BENCHMARKS.md`](https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md) has the fitting
+loop and the fitted values.
+
+### `action` and `act_probability`
+
+`action.act_probability` is a separate head's score for "should an agent act on this at all",
+distinct from the answer's own confidence. It is reported for every question type. Nothing in
+the library thresholds it for you.
+
+## Presets
+
+Three ready-made question sets, so the common cases do not need hand-written criteria:
+
+```python
+from laya import triage_questions, guard_questions, moderation_questions
+
+agent.system_one(ticket, triage_questions())
+```
+
+Use them as a starting point rather than a contract — read the questions they produce with
+`render_options` and check the labels fit your domain before shipping them.
+
+## Reading the options back
+
+Because option order is positional and the option text is what the model reads, it is worth being
+able to see exactly what was sent:
+
+```python
+from laya import render_options
+
+render_options({"t": "choice", "crit": {"billing": None, "sales": "pricing"}})
+# ['billing', 'sales: pricing']
+
+render_options({"t": "score", "crit": ["low", "high"]})
+# ['level 0: low', 'level 1: high']
+```
+
+The same labels in a different order render in that order, which is why order is part of the
+question's identity:
+
+```python
+render_options({"t": "choice", "crit": {"x": "first", "y": "second"}})
+# ['x: first', 'y: second']
+render_options({"t": "choice", "crit": {"y": "second", "x": "first"}})
+# ['y: second', 'x: first']
+```
+
+**Note the keys.** `render_options` takes the *internal* short-key form `{"t": ..., "crit": ...}`,
+not the `{"type": ..., "criteria": ...}` form you write in a question — passing the public shape
+raises `KeyError: 't'`.
+
+The conversion is a short, stable mapping you can inline, which avoids reaching into a private
+helper — `Agent._to_internal` is internal and may change:
+
+```python
+def as_internal(q):
+ """The short-key shape `render_options` reads, from a question as you wrote it."""
+ crit = q.get("criteria")
+ if q["type"] == "choice" and isinstance(crit, list):
+ crit = {c: None for c in crit}
+ return {"t": q["type"], "ins": q["instructions"], "crit": crit}
+
+render_options(as_internal(question))
+```
+
+That mirrors what the library does for a `choice` question written as a list of labels; a
+`criteria` dict and a `score` list pass through unchanged.
+
+## Limits worth knowing before you design around this
+
+- **Confidence is not a correctness guarantee at high option counts.** On a 20-option question the
+ distributions for right and wrong answers overlap heavily, and a threshold can end up selecting
+ *below* the model's own accuracy. See [#394](https://github.com/NandhaKishorM/laya/issues/394).
+- **Negation is not reliably handled in forced-choice questions.** A cancellation question can
+ return the cancellation label for a state that says *not* to cancel, with high confidence, on
+ both checkpoints. See [#377](https://github.com/NandhaKishorM/laya/issues/377).
+- **`noul` can follow its labels rather than the state**, so validate any relabelling.
+- **More options is not free.** Past roughly 20 the model degrades quickly; use the shortlist
+ helper to reduce a large label space before asking, or split it into a coarse and a fine
+ question.