Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(cli): make core's abstention gate reachable from the command line

Aashish681024928994
diff --git a/README.md b/README.md
index 328a6194..98785b20 100644
--- a/README.md
+++ b/README.md
@@ -224,6 +224,7 @@ laya "My payment failed twice" --preset triage # answer a ready-made prese
laya --batch tickets.txt --predict # score a file of requests, one per line, in one batch
cat tickets.txt | laya --batch - --predict --json # stdin; one JSON line of answers per request
laya "Where is my card" --questions intents.json # answer your own questions, written in a JSON file
+laya "Refund my card" --predict --min-confidence 0.9 # mark an answer the model is unsure of
laya # interactive mode
```
@@ -815,6 +816,24 @@ else:
The threshold reads `answer_confidence` (`max(p)`) — the calibrated quantity, invariant to the number of options — never the entropy `confidence`. With `decide(..., min_confidence=...)` a low-confidence field comes back as `None` in the schema output, while `return_details=True` keeps the answer and its confidence. [LangChain `LayaRouter`](docs/langchain.md)'s `confidence_threshold` reads the same value: `answer_confidence` when the answer carries it, `confidence` otherwise. Left unset, `min_confidence` changes nothing.
+The same gate is a flag on the `laya` command:
+
+```bash
+laya "Is it the blue one or the green one" --preset triage --min-confidence 0.9
+```
+
+```text
+intent : other (p=0.992)
+frustration : 1.06 [low-confidence]
+churn_risk : 0.158 [low-confidence]
+```
+
+Each marked answer gets `[low-confidence]` on its printed line and the answer itself is still
+printed, so a run at a terminal reads the same way the dict does; `--json` carries the raw
+`low_confidence` key. `--batch FILE` applies one threshold to the whole file, which is the shape
+`predict_batch` takes. Routing has no answer to gate, so `laya --min-confidence 0.9 "..."` without
+`--predict`, `--preset` or `--questions` is refused rather than ignored.
+
---
## Prediction Hooks
diff --git a/laya/cli.py b/laya/cli.py
index 6b5a7955..89315c37 100644
--- a/laya/cli.py
+++ b/laya/cli.py
@@ -10,6 +10,7 @@
cat tickets.txt | laya --batch - --predict --json # stdin, one JSON line of answers each
laya "Where is my card" --questions intents.json # your own questions, from a JSON file
laya "Where is my card" --questions intents.json --head-max-len 384
+ laya "Where is my card" --questions intents.json --min-confidence 0.7
Routing (the default) never downloads a checkpoint, so it works offline and
returns in milliseconds. --predict loads the routed checkpoint on first use,
@@ -19,7 +20,9 @@ ready-made question presets from laya.presets and implies --predict.
process, so the states share checkpoint loads and forward passes; --json prints JSONL.
ready-made question presets from laya.presets and implies --predict. --questions
answers the question set in a JSON file instead; --max-len and --head-max-len
-raise its token budget for the request, which is what a many-label question needs.
+raise its token budget for the request, which is what a many-label question needs,
+and --min-confidence marks an answer whose confidence falls below the threshold
+instead of changing it (it needs an answering mode: routing has no confidence to gate).
"""
import argparse
@@ -69,6 +72,27 @@ def model_name(value):
raise argparse.ArgumentTypeError("%s, or 'auto'" % error) from None
+def min_confidence_value(value):
+ """Resolve a `--min-confidence` argument through core's own validator, so the range is core's.
+
+ ``laya.confidence.check_min_confidence`` is what ``Router.predict`` and ``Router.predict_batch``
+ run this argument through, so a threshold this flag accepts is exactly one the call accepts --
+ including the boolean and the infinite value that ``float()`` alone waves through
+ (``--min-confidence inf`` would ask for a gate no answer can clear).
+ """
+ from laya.confidence import check_min_confidence
+
+ try:
+ number = float(value)
+ except ValueError:
+ raise argparse.ArgumentTypeError(
+ "expected a number in [0.0, 1.0], got %r" % value) from None
+ try:
+ return check_min_confidence(number)
+ except ValueError as error:
+ raise argparse.ArgumentTypeError(str(error)) from None
+
+
def build_parser():
parser = argparse.ArgumentParser(
prog="laya",
@@ -94,6 +118,11 @@ def build_parser():
parser.add_argument("--head-max-len", type=int, dest="head_max_len", metavar="N",
help="token budget the choice options share; raise it when a question has many"
" labels, so each keeps enough tokens to stay distinct")
+ parser.add_argument("--min-confidence", type=min_confidence_value, dest="min_confidence",
+ metavar="P",
+ help="flag an answer whose calibrated confidence falls below P (0.0-1.0) as"
+ " low-confidence; the answer itself is kept, so this asks for a warning"
+ " rather than a refusal. Needs --predict, --preset or --questions.")
parser.add_argument("--device", help="torch device, e.g. cpu or cuda")
parser.add_argument("--json", action="store_true", help="print the raw result as JSON")
parser.add_argument("--batch", metavar="FILE",
@@ -171,6 +200,10 @@ def show_answers(result):
detail = "%.3f" % answer["noul"]
else:
detail = json.dumps(answer, ensure_ascii=False)
+ # The gate marks an unsure answer and keeps it; without this line the mark was only ever
+ # readable through --json, so a gated run at a terminal looked exactly like an ungated one.
+ if answer.get("low_confidence"):
+ detail += " [low-confidence]"
print("%-12s: %s" % (qid, detail))
@@ -188,6 +221,19 @@ def budget_overrides(args):
return overrides
+def abstention_override(args):
+ """`--min-confidence` as a `predict` / `predict_batch` keyword argument, absent when unset.
+
+ Separate from ``budget_overrides`` because the two reach the batch call differently: a token
+ budget is per request, so it goes inside each request dict, while ``predict_batch`` validates
+ one threshold for the whole batch and takes it as its own argument. Unset has to stay unsent
+ for the same reason as the budgets -- a ``None`` here would overwrite a Router built with its
+ own ``min_confidence``, and ``0.0`` is a real threshold, not an absence.
+ """
+ value = getattr(args, "min_confidence", None)
+ return {} if value is None else {"min_confidence": value}
+
+
def resolve_questions(args):
"""The question set to answer and the state field it reads, from the flags given."""
if args.questions and args.preset:
@@ -216,7 +262,7 @@ def run(text, args, router=None):
state = {state_key: text}
result = router.predict(state, questions,
model=args.model, task=args.task, lang=args.lang,
- **budget_overrides(args))
+ **budget_overrides(args), **abstention_override(args))
if args.json:
print(json.dumps(result, ensure_ascii=False, indent=2, default=str))
else:
@@ -263,7 +309,7 @@ def run_batch(lines, args, router=None):
overrides.update(budget_overrides(args))
requests = [{"state": {key: line}, "questions": questions, **overrides}
for line in lines]
- kwargs = {"batch_size": args.batch_size}
+ kwargs = {"batch_size": args.batch_size, **abstention_override(args)}
if args.sort_by_length:
# Sent only when asked for, as Router.predict_batch itself forwards it: a stub or
# an attached agent that predates the knob keeps working for every run that does
@@ -327,6 +373,13 @@ def main(argv=None):
return eval_main(argv[1:])
args = build_parser().parse_args(argv)
text = " ".join(args.text).strip()
+ # Routing answers nothing, so there is no confidence to gate: `route` does not take the
+ # threshold at all. Accepting the flag there would let `laya --min-confidence 0.9 "..."` exit 0
+ # having ignored it, which is the thing this flag exists to avoid.
+ if args.min_confidence is not None and not (args.predict or args.preset or args.questions):
+ print("laya: --min-confidence gates answers, so it needs --predict, --preset or --questions.",
+ file=sys.stderr)
+ return 2
if args.batch:
if text:
print("laya: pass either a text or --batch FILE, not both.", file=sys.stderr)
diff --git a/tests/test_cli.py b/tests/test_cli.py
index b895e3c7..44d5c5c0 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -531,6 +531,166 @@ for value in ("abc", ""):
check("budget %r: rejected by argparse" % value, code == 2 and "invalid int value" in err,
"code %r, err %r" % (code, err))
+# --------------------------------------------------------------- --min-confidence (#361's gate)
+# The gate itself is core's: `Router.predict` and `Router.predict_batch` both run the argument
+# through `laya.confidence.check_min_confidence` and then `flag_low_confidence`, which marks an
+# answer and keeps it. What this section covers is the CLI's half -- the flag reaches the call, an
+# unset flag is not sent as `None`, a value outside core's range is refused in core's words,
+# routing refuses the flag instead of exiting 0 having ignored it, and the mark a gated run
+# produces is printed to a terminal rather than only reachable through --json.
+import inspect # noqa: E402
+
+from laya import Router as _Router # noqa: E402
+from laya.confidence import check_min_confidence as _core_check # noqa: E402
+
+
+class GatedRouter(QuestionRecorder):
+ """Returns one answer, optionally with the mark `flag_low_confidence` adds."""
+
+ def __init__(self, mark=True, kind="choice"):
+ super().__init__()
+ self.mark = mark
+ self.kind = kind
+
+ def predict(self, state, questions, **kwargs):
+ self.kwargs = kwargs
+ answer = {"probabilities": {"refund": 0.31, "billing": 0.4}}
+ if self.kind == "choice":
+ answer["choice"] = "refund"
+ else:
+ answer["score"] = 1.0
+ answer["answer_confidence"] = 0.31
+ if self.mark:
+ answer["low_confidence"] = True
+ return {"answers": {"intent": answer}, "routing": dict(StubDecision())}
+
+
+code, out, err, stub = run_cli(["--predict", "--min-confidence", "0.5", "hello"],
+ router=GatedRouter(mark=False))
+check("threshold: forwarded to predict", stub.kwargs.get("min_confidence") == 0.5, str(stub.kwargs))
+check("threshold: exit code", code == 0, "got %r %s" % (code, err))
+
+code, out, err, stub = run_cli(["--predict", "hello"], router=GatedRouter(mark=False))
+check("threshold: unset is not sent as None", "min_confidence" not in stub.kwargs, str(stub.kwargs))
+
+# `0.0` is the one threshold core itself treats as a no-op (`flag_low_confidence` returns early on
+# it), but it is still a value a caller chose rather than an absence; `0` and `1` are core's
+# inclusive bounds.
+for raw, want in (("0", 0.0), ("0.0", 0.0), ("1", 1.0)):
+ code, out, err, stub = run_cli(["--predict", "--min-confidence", raw, "hello"],
+ router=GatedRouter(mark=False))
+ check("threshold %r: still a threshold, not an absence" % raw,
+ code == 0 and stub.kwargs.get("min_confidence") == want,
+ "code %r, got %r" % (code, stub.kwargs.get("min_confidence", "<absent>")))
+
+# The accepted range is core's, so the CLI's verdict on a value has to match core's own -- the
+# same check `predict` runs, not a copy of its bounds.
+for raw in ("0", "0.5", "1", "0.9", "1.5", "-0.1", "nan", "inf", "1e999", "true", "abc", ""):
+ try:
+ _core_check(float(raw))
+ accepted_by_core = True
+ except (TypeError, ValueError):
+ accepted_by_core = False
+ code, out, err, stub = run_cli(["--predict", "--min-confidence", raw, "hello"],
+ router=GatedRouter(mark=False))
+ check("threshold %r: same verdict as core" % raw,
+ (code == 0) == accepted_by_core,
+ "code %r, core accepts %r, err %r" % (code, accepted_by_core, err))
+ check("threshold %r: never a traceback" % raw, "Traceback" not in err, err)
+
+code, out, err, stub = run_cli(["--predict", "--min-confidence", "1.5", "hello"],
+ router=GatedRouter(mark=False))
+check("threshold: an out-of-range value is refused in core's words",
+ code == 2 and "min_confidence must be a float in [0.0, 1.0], got 1.5" in err, err)
+code, out, err, stub = run_cli(["--predict", "--min-confidence", "inf", "hello"],
+ router=GatedRouter(mark=False))
+check("threshold: infinity is refused, not accepted as a gate",
+ code == 2 and "min_confidence must be a float in [0.0, 1.0]" in err, err)
+
+# Routing answers nothing, so there is no confidence to gate: `route` does not take the argument.
+# Exiting 0 here would be the silent-ignore this flag exists to avoid.
+code, out, err, stub = run_cli(["--min-confidence", "0.5", "hello"], router=GatedRouter())
+check("threshold: routing mode refuses it", code == 2, "got %r" % code)
+check("threshold: the refusal names the flag and an answering mode",
+ "--min-confidence" in err and "--predict" in err, err)
+check("threshold: nothing was routed", stub.route_calls == [] and stub.kwargs == {}, str(stub.kwargs))
+
+# The refusal is about routing, not about the flag being unwelcome: every answering mode takes it,
+# including the two that reach answering by implying `--predict` rather than being told.
+for argv, label in ((["--predict", "hello"], "--predict"),
+ (["--preset", "triage", "hello"], "--preset"),
+ (["--questions", INTENTS_PATH, "hello"], "--questions")):
+ code, out, err, stub = run_cli(argv + ["--min-confidence", "0.5"], router=GatedRouter(mark=False))
+ check("threshold: %s mode takes it" % label,
+ code == 0 and stub.kwargs.get("min_confidence") == 0.5,
+ "code %r, kwargs %r, err %r" % (code, stub.kwargs, err))
+
+code, out, err, stub = run_batch_cli(["--batch", path, "--min-confidence", "0.5"])
+check("threshold: batch routing refuses it too", code == 2 and stub.predict_batch_calls == [],
+ "code %r, calls %r" % (code, stub.predict_batch_calls))
+
+
+class ThresholdBatchRouter(BatchRouter):
+ """Records the call-level keyword arguments of `predict_batch` separately from the requests."""
+
+ def __init__(self):
+ super().__init__()
+ self.call_kwargs = []
+
+ def predict_batch(self, requests, batch_size=None, **call_kwargs):
+ self.call_kwargs.append(call_kwargs)
+ return super().predict_batch(requests, batch_size=batch_size)
+
+
+code, out, err, stub = run_batch_cli(["--batch", path, "--predict", "--min-confidence", "0.5"],
+ router=ThresholdBatchRouter())
+requests = stub.predict_batch_calls[0][0]
+check("threshold batch: sent to predict_batch as a call argument",
+ code == 0 and stub.call_kwargs == [{"min_confidence": 0.5}],
+ "code %r, kwargs %r" % (code, stub.call_kwargs))
+# `predict_batch` validates one threshold for the whole batch and has no per-request form, so a
+# key inside the request dicts would be read by nothing.
+check("threshold batch: not stuffed into each request",
+ all("min_confidence" not in r for r in requests), str(requests))
+
+code, out, err, stub = run_batch_cli(["--batch", path, "--predict"], router=ThresholdBatchRouter())
+check("threshold batch: unset sends nothing", stub.call_kwargs == [{}], str(stub.call_kwargs))
+
+# Rendering: the mark has to reach the terminal, or --json is the only way to learn the model was
+# unsure -- which is how the gate reads as a no-op to anyone not parsing JSON.
+code, out, err, stub = run_cli(["--predict", "--min-confidence", "0.5", "hello"],
+ router=GatedRouter(mark=True))
+check("threshold: the printed line carries the mark", "[low-confidence]" in out, out)
+check("threshold: the answer itself is still printed", "refund" in out, out)
+code, out, err, stub = run_cli(["--predict", "hello"], router=GatedRouter(mark=False))
+check("threshold: an unmarked answer prints no mark", "[low-confidence]" not in out, out)
+code, out, err, stub = run_cli(["--predict", "--min-confidence", "0.5", "hello"],
+ router=GatedRouter(mark=True, kind="score"))
+check("threshold: a score answer is marked too", "[low-confidence]" in out and "1.00" in out, out)
+code, out, err, stub = run_cli(["--predict", "--min-confidence", "0.5", "--json", "hello"],
+ router=GatedRouter(mark=True))
+check("threshold: --json still carries the raw key",
+ '"low_confidence": true' in out, out)
+
+# Drift pin, the other direction: every control `predict` takes that the parser can name must
+# arrive in the call, and the ones it cannot name must be the accepted three. A control added to
+# `predict` lands in the second check and has to be placed -- flagged and forwarded, or written
+# down as something this command cannot mean -- before the suite passes again.
+PREDICT_CONTROLS = {p for p in inspect.signature(_Router.predict).parameters
+ if p not in ("self", "state", "questions")}
+NAMED = set(vars(cli.build_parser().parse_args(["hi"])))
+code, out, err, stub = run_cli(["--predict", "--model", "english", "--task", "multi",
+ "--lang", "de", "--max-len", "1024", "--head-max-len", "512",
+ "--min-confidence", "0.5", "hello"], router=GatedRouter(mark=False))
+check("drift: every control the parser names a flag for arrives in the call",
+ all(control in stub.kwargs for control in (PREDICT_CONTROLS & NAMED)),
+ "missing %s" % sorted((PREDICT_CONTROLS & NAMED) - set(stub.kwargs)))
+# `lang_guess` is the weaker form of a flag the CLI already has, and both are call-level, so
+# `--lang` dominates it for a command line; `hooks_raise` / `hooks_timeout` govern hooks, and
+# `make_router` installs none, so there is nothing in this process for them to change.
+check("drift: the controls with no flag are exactly the accepted three",
+ sorted(PREDICT_CONTROLS - NAMED), ["hooks_raise", "hooks_timeout", "lang_guess"])
+
# --------------------------------------------------------------------- report
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL: