Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(cli): make core's abstention gate reachable from the command line
Aashish681024928994
diff --git a/README.md b/README.mdindex 328a6194..98785b20 100644--- a/README.md+++ b/README.md@@ -224,6 +224,7 @@ laya "My payment failed twice" --preset triage # answer a ready-made preselaya --batch tickets.txt --predict # score a file of requests, one per line, in one batchcat tickets.txt | laya --batch - --predict --json # stdin; one JSON line of answers per requestlaya "Where is my card" --questions intents.json # answer your own questions, written in a JSON file+laya "Refund my card" --predict --min-confidence 0.9 # mark an answer the model is unsure oflaya # interactive mode```@@ -815,6 +816,24 @@ else:The threshold reads `answer_confidence` (`max(p)`) — the calibrated quantity, invariant to the number of options — never the entropy `confidence`. With `decide(..., min_confidence=...)` a low-confidence field comes back as `None` in the schema output, while `return_details=True` keeps the answer and its confidence. [LangChain `LayaRouter`](docs/langchain.md)'s `confidence_threshold` reads the same value: `answer_confidence` when the answer carries it, `confidence` otherwise. Left unset, `min_confidence` changes nothing.+The same gate is a flag on the `laya` command:++```bash+laya "Is it the blue one or the green one" --preset triage --min-confidence 0.9+```++```text+intent : other (p=0.992)+frustration : 1.06 [low-confidence]+churn_risk : 0.158 [low-confidence]+```++Each marked answer gets `[low-confidence]` on its printed line and the answer itself is still+printed, so a run at a terminal reads the same way the dict does; `--json` carries the raw+`low_confidence` key. `--batch FILE` applies one threshold to the whole file, which is the shape+`predict_batch` takes. Routing has no answer to gate, so `laya --min-confidence 0.9 "..."` without+`--predict`, `--preset` or `--questions` is refused rather than ignored.+---## Prediction Hooksdiff --git a/laya/cli.py b/laya/cli.pyindex 6b5a7955..89315c37 100644--- a/laya/cli.py+++ b/laya/cli.py@@ -10,6 +10,7 @@cat tickets.txt | laya --batch - --predict --json # stdin, one JSON line of answers eachlaya "Where is my card" --questions intents.json # your own questions, from a JSON filelaya "Where is my card" --questions intents.json --head-max-len 384+ laya "Where is my card" --questions intents.json --min-confidence 0.7Routing (the default) never downloads a checkpoint, so it works offline andreturns in milliseconds. --predict loads the routed checkpoint on first use,@@ -19,7 +20,9 @@ ready-made question presets from laya.presets and implies --predict.process, so the states share checkpoint loads and forward passes; --json prints JSONL.ready-made question presets from laya.presets and implies --predict. --questionsanswers the question set in a JSON file instead; --max-len and --head-max-len-raise its token budget for the request, which is what a many-label question needs.+raise its token budget for the request, which is what a many-label question needs,+and --min-confidence marks an answer whose confidence falls below the threshold+instead of changing it (it needs an answering mode: routing has no confidence to gate)."""import argparse@@ -69,6 +72,27 @@ def model_name(value):raise argparse.ArgumentTypeError("%s, or 'auto'" % error) from None+def min_confidence_value(value):+ """Resolve a `--min-confidence` argument through core's own validator, so the range is core's.++ ``laya.confidence.check_min_confidence`` is what ``Router.predict`` and ``Router.predict_batch``+ run this argument through, so a threshold this flag accepts is exactly one the call accepts --+ including the boolean and the infinite value that ``float()`` alone waves through+ (``--min-confidence inf`` would ask for a gate no answer can clear).+ """+ from laya.confidence import check_min_confidence++ try:+ number = float(value)+ except ValueError:+ raise argparse.ArgumentTypeError(+ "expected a number in [0.0, 1.0], got %r" % value) from None+ try:+ return check_min_confidence(number)+ except ValueError as error:+ raise argparse.ArgumentTypeError(str(error)) from None++def build_parser():parser = argparse.ArgumentParser(prog="laya",@@ -94,6 +118,11 @@ def build_parser():parser.add_argument("--head-max-len", type=int, dest="head_max_len", metavar="N",help="token budget the choice options share; raise it when a question has many"" labels, so each keeps enough tokens to stay distinct")+ parser.add_argument("--min-confidence", type=min_confidence_value, dest="min_confidence",+ metavar="P",+ help="flag an answer whose calibrated confidence falls below P (0.0-1.0) as"+ " low-confidence; the answer itself is kept, so this asks for a warning"+ " rather than a refusal. Needs --predict, --preset or --questions.")parser.add_argument("--device", help="torch device, e.g. cpu or cuda")parser.add_argument("--json", action="store_true", help="print the raw result as JSON")parser.add_argument("--batch", metavar="FILE",@@ -171,6 +200,10 @@ def show_answers(result):detail = "%.3f" % answer["noul"]else:detail = json.dumps(answer, ensure_ascii=False)+ # The gate marks an unsure answer and keeps it; without this line the mark was only ever+ # readable through --json, so a gated run at a terminal looked exactly like an ungated one.+ if answer.get("low_confidence"):+ detail += " [low-confidence]"print("%-12s: %s" % (qid, detail))@@ -188,6 +221,19 @@ def budget_overrides(args):return overrides+def abstention_override(args):+ """`--min-confidence` as a `predict` / `predict_batch` keyword argument, absent when unset.++ Separate from ``budget_overrides`` because the two reach the batch call differently: a token+ budget is per request, so it goes inside each request dict, while ``predict_batch`` validates+ one threshold for the whole batch and takes it as its own argument. Unset has to stay unsent+ for the same reason as the budgets -- a ``None`` here would overwrite a Router built with its+ own ``min_confidence``, and ``0.0`` is a real threshold, not an absence.+ """+ value = getattr(args, "min_confidence", None)+ return {} if value is None else {"min_confidence": value}++def resolve_questions(args):"""The question set to answer and the state field it reads, from the flags given."""if args.questions and args.preset:@@ -216,7 +262,7 @@ def run(text, args, router=None):state = {state_key: text}result = router.predict(state, questions,model=args.model, task=args.task, lang=args.lang,- **budget_overrides(args))+ **budget_overrides(args), **abstention_override(args))if args.json:print(json.dumps(result, ensure_ascii=False, indent=2, default=str))else:@@ -263,7 +309,7 @@ def run_batch(lines, args, router=None):overrides.update(budget_overrides(args))requests = [{"state": {key: line}, "questions": questions, **overrides}for line in lines]- kwargs = {"batch_size": args.batch_size}+ kwargs = {"batch_size": args.batch_size, **abstention_override(args)}if args.sort_by_length:# Sent only when asked for, as Router.predict_batch itself forwards it: a stub or# an attached agent that predates the knob keeps working for every run that does@@ -327,6 +373,13 @@ def main(argv=None):return eval_main(argv[1:])args = build_parser().parse_args(argv)text = " ".join(args.text).strip()+ # Routing answers nothing, so there is no confidence to gate: `route` does not take the+ # threshold at all. Accepting the flag there would let `laya --min-confidence 0.9 "..."` exit 0+ # having ignored it, which is the thing this flag exists to avoid.+ if args.min_confidence is not None and not (args.predict or args.preset or args.questions):+ print("laya: --min-confidence gates answers, so it needs --predict, --preset or --questions.",+ file=sys.stderr)+ return 2if args.batch:if text:print("laya: pass either a text or --batch FILE, not both.", file=sys.stderr)diff --git a/tests/test_cli.py b/tests/test_cli.pyindex b895e3c7..44d5c5c0 100644--- a/tests/test_cli.py+++ b/tests/test_cli.py@@ -531,6 +531,166 @@ for value in ("abc", ""):check("budget %r: rejected by argparse" % value, code == 2 and "invalid int value" in err,"code %r, err %r" % (code, err))+# --------------------------------------------------------------- --min-confidence (#361's gate)+# The gate itself is core's: `Router.predict` and `Router.predict_batch` both run the argument+# through `laya.confidence.check_min_confidence` and then `flag_low_confidence`, which marks an+# answer and keeps it. What this section covers is the CLI's half -- the flag reaches the call, an+# unset flag is not sent as `None`, a value outside core's range is refused in core's words,+# routing refuses the flag instead of exiting 0 having ignored it, and the mark a gated run+# produces is printed to a terminal rather than only reachable through --json.+import inspect # noqa: E402++from laya import Router as _Router # noqa: E402+from laya.confidence import check_min_confidence as _core_check # noqa: E402+++class GatedRouter(QuestionRecorder):+ """Returns one answer, optionally with the mark `flag_low_confidence` adds."""++ def __init__(self, mark=True, kind="choice"):+ super().__init__()+ self.mark = mark+ self.kind = kind++ def predict(self, state, questions, **kwargs):+ self.kwargs = kwargs+ answer = {"probabilities": {"refund": 0.31, "billing": 0.4}}+ if self.kind == "choice":+ answer["choice"] = "refund"+ else:+ answer["score"] = 1.0+ answer["answer_confidence"] = 0.31+ if self.mark:+ answer["low_confidence"] = True+ return {"answers": {"intent": answer}, "routing": dict(StubDecision())}+++code, out, err, stub = run_cli(["--predict", "--min-confidence", "0.5", "hello"],+ router=GatedRouter(mark=False))+check("threshold: forwarded to predict", stub.kwargs.get("min_confidence") == 0.5, str(stub.kwargs))+check("threshold: exit code", code == 0, "got %r %s" % (code, err))++code, out, err, stub = run_cli(["--predict", "hello"], router=GatedRouter(mark=False))+check("threshold: unset is not sent as None", "min_confidence" not in stub.kwargs, str(stub.kwargs))++# `0.0` is the one threshold core itself treats as a no-op (`flag_low_confidence` returns early on+# it), but it is still a value a caller chose rather than an absence; `0` and `1` are core's+# inclusive bounds.+for raw, want in (("0", 0.0), ("0.0", 0.0), ("1", 1.0)):+ code, out, err, stub = run_cli(["--predict", "--min-confidence", raw, "hello"],+ router=GatedRouter(mark=False))+ check("threshold %r: still a threshold, not an absence" % raw,+ code == 0 and stub.kwargs.get("min_confidence") == want,+ "code %r, got %r" % (code, stub.kwargs.get("min_confidence", "<absent>")))++# The accepted range is core's, so the CLI's verdict on a value has to match core's own -- the+# same check `predict` runs, not a copy of its bounds.+for raw in ("0", "0.5", "1", "0.9", "1.5", "-0.1", "nan", "inf", "1e999", "true", "abc", ""):+ try:+ _core_check(float(raw))+ accepted_by_core = True+ except (TypeError, ValueError):+ accepted_by_core = False+ code, out, err, stub = run_cli(["--predict", "--min-confidence", raw, "hello"],+ router=GatedRouter(mark=False))+ check("threshold %r: same verdict as core" % raw,+ (code == 0) == accepted_by_core,+ "code %r, core accepts %r, err %r" % (code, accepted_by_core, err))+ check("threshold %r: never a traceback" % raw, "Traceback" not in err, err)++code, out, err, stub = run_cli(["--predict", "--min-confidence", "1.5", "hello"],+ router=GatedRouter(mark=False))+check("threshold: an out-of-range value is refused in core's words",+ code == 2 and "min_confidence must be a float in [0.0, 1.0], got 1.5" in err, err)+code, out, err, stub = run_cli(["--predict", "--min-confidence", "inf", "hello"],+ router=GatedRouter(mark=False))+check("threshold: infinity is refused, not accepted as a gate",+ code == 2 and "min_confidence must be a float in [0.0, 1.0]" in err, err)++# Routing answers nothing, so there is no confidence to gate: `route` does not take the argument.+# Exiting 0 here would be the silent-ignore this flag exists to avoid.+code, out, err, stub = run_cli(["--min-confidence", "0.5", "hello"], router=GatedRouter())+check("threshold: routing mode refuses it", code == 2, "got %r" % code)+check("threshold: the refusal names the flag and an answering mode",+ "--min-confidence" in err and "--predict" in err, err)+check("threshold: nothing was routed", stub.route_calls == [] and stub.kwargs == {}, str(stub.kwargs))++# The refusal is about routing, not about the flag being unwelcome: every answering mode takes it,+# including the two that reach answering by implying `--predict` rather than being told.+for argv, label in ((["--predict", "hello"], "--predict"),+ (["--preset", "triage", "hello"], "--preset"),+ (["--questions", INTENTS_PATH, "hello"], "--questions")):+ code, out, err, stub = run_cli(argv + ["--min-confidence", "0.5"], router=GatedRouter(mark=False))+ check("threshold: %s mode takes it" % label,+ code == 0 and stub.kwargs.get("min_confidence") == 0.5,+ "code %r, kwargs %r, err %r" % (code, stub.kwargs, err))++code, out, err, stub = run_batch_cli(["--batch", path, "--min-confidence", "0.5"])+check("threshold: batch routing refuses it too", code == 2 and stub.predict_batch_calls == [],+ "code %r, calls %r" % (code, stub.predict_batch_calls))+++class ThresholdBatchRouter(BatchRouter):+ """Records the call-level keyword arguments of `predict_batch` separately from the requests."""++ def __init__(self):+ super().__init__()+ self.call_kwargs = []++ def predict_batch(self, requests, batch_size=None, **call_kwargs):+ self.call_kwargs.append(call_kwargs)+ return super().predict_batch(requests, batch_size=batch_size)+++code, out, err, stub = run_batch_cli(["--batch", path, "--predict", "--min-confidence", "0.5"],+ router=ThresholdBatchRouter())+requests = stub.predict_batch_calls[0][0]+check("threshold batch: sent to predict_batch as a call argument",+ code == 0 and stub.call_kwargs == [{"min_confidence": 0.5}],+ "code %r, kwargs %r" % (code, stub.call_kwargs))+# `predict_batch` validates one threshold for the whole batch and has no per-request form, so a+# key inside the request dicts would be read by nothing.+check("threshold batch: not stuffed into each request",+ all("min_confidence" not in r for r in requests), str(requests))++code, out, err, stub = run_batch_cli(["--batch", path, "--predict"], router=ThresholdBatchRouter())+check("threshold batch: unset sends nothing", stub.call_kwargs == [{}], str(stub.call_kwargs))++# Rendering: the mark has to reach the terminal, or --json is the only way to learn the model was+# unsure -- which is how the gate reads as a no-op to anyone not parsing JSON.+code, out, err, stub = run_cli(["--predict", "--min-confidence", "0.5", "hello"],+ router=GatedRouter(mark=True))+check("threshold: the printed line carries the mark", "[low-confidence]" in out, out)+check("threshold: the answer itself is still printed", "refund" in out, out)+code, out, err, stub = run_cli(["--predict", "hello"], router=GatedRouter(mark=False))+check("threshold: an unmarked answer prints no mark", "[low-confidence]" not in out, out)+code, out, err, stub = run_cli(["--predict", "--min-confidence", "0.5", "hello"],+ router=GatedRouter(mark=True, kind="score"))+check("threshold: a score answer is marked too", "[low-confidence]" in out and "1.00" in out, out)+code, out, err, stub = run_cli(["--predict", "--min-confidence", "0.5", "--json", "hello"],+ router=GatedRouter(mark=True))+check("threshold: --json still carries the raw key",+ '"low_confidence": true' in out, out)++# Drift pin, the other direction: every control `predict` takes that the parser can name must+# arrive in the call, and the ones it cannot name must be the accepted three. A control added to+# `predict` lands in the second check and has to be placed -- flagged and forwarded, or written+# down as something this command cannot mean -- before the suite passes again.+PREDICT_CONTROLS = {p for p in inspect.signature(_Router.predict).parameters+ if p not in ("self", "state", "questions")}+NAMED = set(vars(cli.build_parser().parse_args(["hi"])))+code, out, err, stub = run_cli(["--predict", "--model", "english", "--task", "multi",+ "--lang", "de", "--max-len", "1024", "--head-max-len", "512",+ "--min-confidence", "0.5", "hello"], router=GatedRouter(mark=False))+check("drift: every control the parser names a flag for arrives in the call",+ all(control in stub.kwargs for control in (PREDICT_CONTROLS & NAMED)),+ "missing %s" % sorted((PREDICT_CONTROLS & NAMED) - set(stub.kwargs)))+# `lang_guess` is the weaker form of a flag the CLI already has, and both are call-level, so+# `--lang` dominates it for a command line; `hooks_raise` / `hooks_timeout` govern hooks, and+# `make_router` installs none, so there is nothing in this process for them to change.+check("drift: the controls with no flag are exactly the accepted three",+ sorted(PREDICT_CONTROLS - NAMED), ["hooks_raise", "hooks_timeout", "lang_guess"])+# --------------------------------------------------------------------- reportprint("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL: