Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(adoption): cover adversarial decision context
Aditya Kushwahaab0c3f0eefe2
diff --git a/docs/staged-adoption.md b/docs/staged-adoption.mdindex 7ea2b303..cbb660e1 100644--- a/docs/staged-adoption.md+++ b/docs/staged-adoption.md@@ -1,145 +1,162 @@-# Staged adoption--Laya returns a typed decision, not permission to execute it. Adopt a decision engine in stages so-that the application keeps control of the real action while evidence accumulates.--This guide describes the application-side rollout around Laya. Laya supplies the decision,-probabilities, confidence and hook events; the application owns the incumbent action, rollout-policy, review boundary and rollback.--## 1. Shadow: observe without side effects--Run Laya on representative real traffic, but keep the incumbent action authoritative. A shadow-record should contain enough context to reproduce a comparison later:--- the request class and the Laya question schema;-- Laya's answer, per-option probabilities, confidence, checkpoint and `run_id`;-- the incumbent action and the eventual reviewed or ground-truth outcome;-- latency, errors and any fallback or review decision.--Use the existing prediction hooks for Laya-side evidence. The logging functions below are-application-owned placeholders; they are not Laya APIs:--```python-from laya import Router--class ShadowLog:- def on_predict_end(self, ctx):- result = ctx.results[0] if ctx.results else None- write_shadow_record({- "run_id": ctx.run_id,- "model": ctx.model,- "decision": ctx.decision,- "result": result,- "elapsed_ms": ctx.elapsed_ms,- "error": None if ctx.error is None else repr(ctx.error),- })--router = Router(hooks=[ShadowLog()])--def handle(request):- try:- laya_result = router.predict(request.state, request.questions)- except Exception as exc:- record_laya_failure(request, exc)- return run_incumbent_action(request)- # The shadow result is recorded by the hook; do not execute it here.- return run_incumbent_action(request)-```--Keep sensitive fields redacted according to the application's policy. Catch and log exceptions-around `router.predict(...)` at the application boundary. The prediction hook covers the prediction-lifecycle, but failures before that lifecycle require application-level capture; do not assume-`on_predict_end` saw them. A shadow logger must not turn logging into a new user-facing action.--See [Prediction hooks](hooks/index.md), the [hook lifecycle](hooks/lifecycle.md), and-[Tracing](hooks/tracing.md) for the event order and `run_id` correlation.--## 2. Compare: disagreement is a signal, not a verdict--Compare Laya with the incumbent on the same request and question meaning. A disagreement is not-automatically an error: the incumbent may be wrong, the cases may be ambiguous, or the action may-require human judgment. Use reviewed labels or ground-truth outcomes where available, and keep an-explicit `unknown` or review bucket instead of forcing every disagreement into a binary score.--Review comparisons by checkpoint, language or route, question schema, action type and risk class.-Record coverage and disagreement alongside accuracy. A high agreement rate on an easy subset does-not justify promotion for a different language, action or question shape.--## 3. Choose a policy from held-out evidence--A confidence threshold is an application policy, not a property supplied by Laya. Fit or calibrate-the decision scores on representative held-out data, then choose a threshold from the measured-accuracy and error cost at the coverage your application can tolerate. There is no universal number-that transfers across checkpoints, question types, languages or action risks.--Record the checkpoint and question-schema version, calibration method, threshold, evaluation set and-owner with the policy. Re-evaluate it when those inputs change. Confidence orders decisions; it does-not establish that a decision is correct, and high confidence is never execution permission by-itself.--Four of those six come from the evaluation harness: a `laya-evals run --json` report records the-checkpoint commit that answered (`config.revisions`), the dataset's bytes and question schema-(`config.dataset_sha256`, `config.questions_sha256`) and the gate it was scored under-(`config.thresholds`). A checkpoint given as a local directory has no commit to name, so its entry in-that block is `null`. Calibration method and owner are the application's to record. See-[Evaluation harness](evals.md) for the run identity and the comparability check between a baseline-and a candidate.--The README's [Automated Confidence Gating](https://github.com/NandhaKishorM/laya#automated-confidence-gating),-[Calibration](https://github.com/NandhaKishorM/laya#calibration), and [Honest limits](https://github.com/NandhaKishorM/laya#honest-limits) sections-give the existing calibration and confidence context. Keep irreversible or high-cost actions behind-an explicit review boundary even when their confidence is high.--## 4. Promote a bounded slice--Promotion should be a measured, reversible change rather than a global on/off switch. Define an-eligibility boundary before enabling automation, for example:--- the checkpoint, language/route and question schema are in the evaluated set;-- the action is reversible or has an explicit human review path;-- the request is not missing required context and has no Laya error;-- the slice has a size or traffic cap and a named rollback condition.--Start with a small canary. Keep review or fallback for ineligible, ambiguous and failed cases. Continue-sampling promoted decisions, compare them with the incumbent and reviewed outcomes, and monitor-disagreement, coverage, fallback rate, errors and latency. Roll back when the agreed guardrail is-breached; promotion is a bounded step, not a permanent declaration that the model is correct.--## Application/Laya boundary--Laya provides the decision evidence and exposes it through the existing API and hooks. The-application owns the incumbent result, action execution, eligibility rules, threshold, review,-fallback and rollback. Hooks can log or annotate evidence, but they do not make a high-impact action-safe to execute.--A practical rollout is therefore:--```text-real request- ├─ incumbent action (authoritative)- └─ Laya shadow decision ──> log, compare, evaluate- └─ bounded eligible slice- └─ review / fallback / rollback-```--## Rollout checklist--- [ ] Shadow logging is side-effect free and correlated by `run_id`.-- [ ] Comparison data includes the incumbent outcome and reviewed or ground-truth labels where- available.-- [ ] Thresholds are fitted and validated on representative held-out data.-- [ ] Irreversible or high-cost actions have an explicit review boundary.-- [ ] Promotion is bounded, sampled and reversible, with a named fallback and rollback path.-- [ ] The policy owner and re-evaluation trigger are recorded.--## See also--- [Prediction hooks](hooks/index.md) — the extension seam for audit, metrics and gating.-- [Evaluation harness](evals.md) — the run identity a baseline carries and the comparability check- between a baseline and a candidate.-- [Hook API reference](hooks/api.md) — `PredictContext` fields and lifecycle events.-- [Tracing](hooks/tracing.md) — `run_id` and span correlation.-- [README: Automated Confidence Gating](https://github.com/NandhaKishorM/laya#automated-confidence-gating) — confidence is a- policy input, not a correctness guarantee.-- [README: Calibration](https://github.com/NandhaKishorM/laya#calibration) and [Honest limits](https://github.com/NandhaKishorM/laya#honest-limits).+# Staged adoption++Laya returns a typed decision, not permission to execute it. Adopt a decision engine in stages so+that the application keeps control of the real action while evidence accumulates.++This guide describes the application-side rollout around Laya. Laya supplies the decision,+probabilities, confidence and hook events; the application owns the incumbent action, rollout+policy, review boundary and rollback.++## 1. Shadow: observe without side effects++Run Laya on representative real traffic, but keep the incumbent action authoritative. A shadow+record should contain enough context to reproduce a comparison later:++- the request class and the Laya question schema;+- Laya's answer, per-option probabilities, confidence, checkpoint and `run_id`;+- the incumbent action and the eventual reviewed or ground-truth outcome;+- latency, errors and any fallback or review decision.++Use the existing prediction hooks for Laya-side evidence. The logging functions below are+application-owned placeholders; they are not Laya APIs:++```python+from laya import Router++class ShadowLog:+ def on_predict_end(self, ctx):+ result = ctx.results[0] if ctx.results else None+ write_shadow_record({+ "run_id": ctx.run_id,+ "model": ctx.model,+ "decision": ctx.decision,+ "result": result,+ "elapsed_ms": ctx.elapsed_ms,+ "error": None if ctx.error is None else repr(ctx.error),+ })++router = Router(hooks=[ShadowLog()])++def handle(request):+ try:+ laya_result = router.predict(request.state, request.questions)+ except Exception as exc:+ record_laya_failure(request, exc)+ return run_incumbent_action(request)+ # The shadow result is recorded by the hook; do not execute it here.+ return run_incumbent_action(request)+```++Keep sensitive fields redacted according to the application's policy. Catch and log exceptions+around `router.predict(...)` at the application boundary. The prediction hook covers the prediction+lifecycle, but failures before that lifecycle require application-level capture; do not assume+`on_predict_end` saw them. A shadow logger must not turn logging into a new user-facing action.++See [Prediction hooks](hooks/index.md), the [hook lifecycle](hooks/lifecycle.md), and+[Tracing](hooks/tracing.md) for the event order and `run_id` correlation.++## 2. Compare: disagreement is a signal, not a verdict++Compare Laya with the incumbent on the same request and question meaning. A disagreement is not+automatically an error: the incumbent may be wrong, the cases may be ambiguous, or the action may+require human judgment. Use reviewed labels or ground-truth outcomes where available, and keep an+explicit `unknown` or review bucket instead of forcing every disagreement into a binary score.++Review comparisons by checkpoint, language or route, question schema, action type and risk class.+Record coverage and disagreement alongside accuracy. A high agreement rate on an easy subset does+not justify promotion for a different language, action or question shape.++## 3. Choose a policy from held-out evidence++A confidence threshold is an application policy, not a property supplied by Laya. Fit or calibrate+the decision scores on representative held-out data, then choose a threshold from the measured+accuracy and error cost at the coverage your application can tolerate. There is no universal number+that transfers across checkpoints, question types, languages or action risks.++Record the checkpoint and question-schema version, calibration method, threshold, evaluation set and+owner with the policy. Re-evaluate it when those inputs change. Confidence orders decisions; it does+not establish that a decision is correct, and high confidence is never execution permission by+itself.++### Attacker-controlled context++When a request includes text an attacker can influence, evaluate whether that text can steer the+decision as well as whether the model is accurate on ordinary traffic. Keep paired cases with the+same task and options: a clean state, then a version containing an instruction aimed at changing the+answer. Include document-shaped and indirect instructions, and compare both the selected option and+its confidence for each checkpoint and language you deploy. A confidence threshold fitted on clean+data is not an injection detector; an attacker-steered answer can still be highly confident.++The small probe reported in [#1229](https://github.com/NandhaKishorM/laya/issues/1229) found that+confidence gating behaved differently across the English and multilingual checkpoints, and that the+`guard` preset missed some document-shaped cases. Its eight requests per style establish a concrete+failure mode, not a general attack rate or a reliable threshold. Use representative, reviewed cases+from your own workflow, and keep high-impact actions behind an independent authorization or review+boundary even when the decision and guard both pass.++Four of those six come from the evaluation harness: a `laya-evals run --json` report records the+checkpoint commit that answered (`config.revisions`), the dataset's bytes and question schema+(`config.dataset_sha256`, `config.questions_sha256`) and the gate it was scored under+(`config.thresholds`). A checkpoint given as a local directory has no commit to name, so its entry in+that block is `null`. Calibration method and owner are the application's to record. See+[Evaluation harness](evals.md) for the run identity and the comparability check between a baseline+and a candidate.++The README's [Automated Confidence Gating](https://github.com/NandhaKishorM/laya#automated-confidence-gating),+[Calibration](https://github.com/NandhaKishorM/laya#calibration), and [Honest limits](https://github.com/NandhaKishorM/laya#honest-limits) sections+give the existing calibration and confidence context. Keep irreversible or high-cost actions behind+an explicit review boundary even when their confidence is high.++## 4. Promote a bounded slice++Promotion should be a measured, reversible change rather than a global on/off switch. Define an+eligibility boundary before enabling automation, for example:++- the checkpoint, language/route and question schema are in the evaluated set;+- the action is reversible or has an explicit human review path;+- the request is not missing required context and has no Laya error;+- the slice has a size or traffic cap and a named rollback condition.++Start with a small canary. Keep review or fallback for ineligible, ambiguous and failed cases. Continue+sampling promoted decisions, compare them with the incumbent and reviewed outcomes, and monitor+disagreement, coverage, fallback rate, errors and latency. Roll back when the agreed guardrail is+breached; promotion is a bounded step, not a permanent declaration that the model is correct.++## Application/Laya boundary++Laya provides the decision evidence and exposes it through the existing API and hooks. The+application owns the incumbent result, action execution, eligibility rules, threshold, review,+fallback and rollback. Hooks can log or annotate evidence, but they do not make a high-impact action+safe to execute.++A practical rollout is therefore:++```text+real request+ ├─ incumbent action (authoritative)+ └─ Laya shadow decision ──> log, compare, evaluate+ └─ bounded eligible slice+ └─ review / fallback / rollback+```++## Rollout checklist++- [ ] Shadow logging is side-effect free and correlated by `run_id`.+- [ ] Comparison data includes the incumbent outcome and reviewed or ground-truth labels where+ available.+- [ ] Thresholds are fitted and validated on representative held-out data.+- [ ] Irreversible or high-cost actions have an explicit review boundary.+- [ ] Promotion is bounded, sampled and reversible, with a named fallback and rollback path.+- [ ] The policy owner and re-evaluation trigger are recorded.++## See also++- [Prediction hooks](hooks/index.md) — the extension seam for audit, metrics and gating.+- [Evaluation harness](evals.md) — the run identity a baseline carries and the comparability check+ between a baseline and a candidate.+- [Hook API reference](hooks/api.md) — `PredictContext` fields and lifecycle events.+- [Tracing](hooks/tracing.md) — `run_id` and span correlation.+- [README: Automated Confidence Gating](https://github.com/NandhaKishorM/laya#automated-confidence-gating) — confidence is a+ policy input, not a correctness guarantee.+- [README: Calibration](https://github.com/NandhaKishorM/laya#calibration) and [Honest limits](https://github.com/NandhaKishorM/laya#honest-limits).+