Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
Merge pull request #542 from aashish254/feat-cli-questions
Nandakishorf888f4378611
diff --cc README.mdindex 4666369e,f37cb797..4cd6755e--- a/README.md+++ b/README.md@@@ -222,15 -198,22 +222,25 @@@ Installing the package also installs alaya "I was charged twice, please refund" # routing decision only; works offline, no downloadlaya "Refactor this service" --predict # full answers (downloads the checkpoint on first use)laya "Mein Konto wurde zweimal belastet" --lang de # force a language instead of detecting it+laya "My payment failed twice" --model ml # pin a checkpoint: names, aliases and casing all resolve as the SDK resolves themlaya "My payment failed twice" --preset triage # answer a ready-made preset (triage, email, guard, moderation, router)+laya --batch tickets.txt --predict # score a file of requests, one per line, in one batch+cat tickets.txt | laya --batch - --predict --json # stdin; one JSON line of answers per request+ laya "Where is my card" --questions intents.json # answer your own questions, written in a JSON filelaya # interactive mode```-Routing alone never downloads a checkpoint, so it returns in milliseconds. `--predict` loads the routed checkpoint, which needs network access to the Hugging Face hub the first time; if a checkpoint cannot be downloaded, the CLI says so instead of crashing.+Routing alone never downloads a checkpoint, so it returns in milliseconds. `--predict` loads the routed checkpoint, which needs network access to the Hugging Face hub the first time; if a checkpoint cannot be downloaded, the CLI says so instead of crashing. `--batch` (with or without `--predict`) sends the whole file through `Router.predict_batch` in one process, so the requests share checkpoint loads and forward passes — measured 2.6x on 20 tickets vs looping `predict` one by one, with `--batch-size N` to bound the forward pass and `--json` for JSONL output. Batch routing (`laya --batch FILE`, no `--predict`) likewise answers with `route_batch` in one pass, still without loading anything.+ `--questions` takes the same question dict the SDK takes, as JSON: either the mapping itself, or+ `{"state_key": "body", "questions": {...}}` when the question's instructions name a field other than+ `request`. It implies `--predict`, and a question set with many labels usually wants+ `--head-max-len` with it: on 58 MASSIVE-INTENT labels written as one choice question, the English+ checkpoint goes from 24/58 correct at its default 192-token option budget to 34/58 at+ `--head-max-len 384`, for about 1.4x the per-request time on CPU. Widening it further costs the+ accuracy back, because `max_len` then leaves fewer tokens for the request itself. [Honest+ limits](#honest-limits) describes the same budget ceiling for a 77-option question.+---## Try it locally: web GUI + JSON API@@@ -1116,7 -993,7 +1126,7 @@@ failure; it does not establish calibrat* `laya` (English) defaults to 512 context (`head_max_len = 192`, ~320 tokens for state).* `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`, ~768 tokens for state; mmBERT-base encoder supports up to 8,192 with RoPE).At default settings, a 77-option question like Banking77 allocates only `(256 - 16) // 77` ≈ 3–4 tokens per label, which causes accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question:- 1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. Both are also per-request: `predict(state, questions, head_max_len=512, max_len=1024)` widens one question without changing the agent for everyone else, and every LangChain node takes the same two arguments ([LangChain guide](docs/langchain.md)).- 1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. `laya --questions` takes the same two budgets as `--max-len` / `--head-max-len`.++ 1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. Both are also per-request: `predict(state, questions, head_max_len=512, max_len=1024)` widens one question without changing the agent for everyone else, and every LangChain node takes the same two arguments ([LangChain guide](docs/langchain.md)). `laya --questions` takes the same two budgets as `--max-len` / `--head-max-len`.2. Or shortlist with embeddings and run one forward pass on the top `k` labels (`predict_shortlist`, example below). `predict` and `system_one` still score every criterion they are given.3. Or split the label set yourself into a coarse question and a fine question.diff --cc laya/cli.pyindex 8ea3b611,c59a8e07..2a5e846f--- a/laya/cli.py+++ b/laya/cli.py@@@ -4,17 -4,16 +4,22 @@@laya "Refactor this service" --predict # full answers, loads the checkpointlaya # interactive modelaya "Mein Konto wurde zweimal belastet" --lang de # explicit language+ laya "My payment failed twice" --model ml # pin a checkpoint, by name or aliaslaya "My payment failed twice" --preset triage # a ready-made question preset+ laya --batch tickets.txt --predict # a file of requests, one per line+ cat tickets.txt | laya --batch - --predict --json # stdin, one JSON line of answers each+ laya "Where is my card" --questions intents.json # your own questions, from a JSON file+ laya "Where is my card" --questions intents.json --head-max-len 384Routing (the default) never downloads a checkpoint, so it works offline andreturns in milliseconds. --predict loads the routed checkpoint on first use,which needs network access to the Hugging Face hub. --preset answers one of the+ready-made question presets from laya.presets and implies --predict.+--batch scores a whole file (or stdin, with `-`) through Router.predict_batch in one+process, so the states share checkpoint loads and forward passes; --json prints JSONL.+ ready-made question presets from laya.presets and implies --predict. --questions+ answers the question set in a JSON file instead; --max-len and --head-max-len+ raise its token budget for the request, which is what a many-label question needs."""import argparse@@@ -80,15 -58,17 +85,24 @@@ def build_parser()parser.add_argument("--preset", choices=sorted(PRESETS), metavar="NAME",help="answer a ready-made question preset (%s) instead of the router questions; implies --predict"% ", ".join(sorted(PRESETS)))+ parser.add_argument("--questions", metavar="FILE",+ help="a JSON file of your own typed questions to answer instead of the router questions"+ " or a preset; implies --predict")+ parser.add_argument("--max-len", type=int, dest="max_len", metavar="N",+ help="token budget for the whole request (state plus options); defaults to the"+ " checkpoint's own budget")+ parser.add_argument("--head-max-len", type=int, dest="head_max_len", metavar="N",+ help="token budget the choice options share; raise it when a question has many"+ " labels, so each keeps enough tokens to stay distinct")parser.add_argument("--device", help="torch device, e.g. cpu or cuda")parser.add_argument("--json", action="store_true", help="print the raw result as JSON")+ parser.add_argument("--batch", metavar="FILE",+ help="score a file of requests, one per line (use '-' for stdin), instead "+ "of a single text; implies neither --predict nor --preset, but the "+ "same modes apply: routing by default, answers with --predict/--preset")+ parser.add_argument("--batch-size", type=int, default=None, metavar="N",+ help="states per forward pass in --batch mode; the default sends each "+ "routed group in one pass")return parser@@@ -162,57 -202,6 +236,58 @@@ def run(text, args, router=None)return 0+def read_batch_lines(source):+ """One request per non-blank line of FILE, or of stdin for '-'."""+ if source == "-":+ lines = sys.stdin.read().splitlines()+ else:+ with open(source, encoding="utf-8") as handle:+ lines = handle.read().splitlines()+ return [line.strip() for line in lines if line.strip()]+++def run_batch(lines, args, router=None):+ """Route or predict a whole file in one call, so states share checkpoint loads and+ forward passes; returns 0 on success, 2 on a handled error."""+ router = router or make_router(args)+ overrides = {key: value for key, value in (("model", args.model), ("task", args.task),+ ("lang", args.lang)) if value is not None}+ try:- if args.predict or args.preset:- questions = PRESETS[args.preset]() if args.preset else laya.router_questions()- key = PRESET_STATE_KEYS.get(args.preset, "request")++ if args.predict or args.preset or getattr(args, "questions", None):++ questions, key = resolve_questions(args)++ # Router.predict_batch reads the token budget per request (#583).++ overrides.update(budget_overrides(args))+ requests = [{"state": {key: line}, "questions": questions, **overrides}+ for line in lines]+ results = router.predict_batch(requests, batch_size=args.batch_size)+ for line, result in zip(lines, results):+ if args.json:+ print(json.dumps(result, ensure_ascii=False, default=str))+ else:+ print("# %s" % (line if len(line) <= 72 else line[:69] + "..."))+ show_answers(result)+ print()+ else:+ decisions = router.route_batch([{"state": {"text": line}, "questions": {},+ **overrides} for line in lines])+ for line, decision in zip(lines, decisions):+ if args.json:+ print(json.dumps(dict(decision), ensure_ascii=False, default=str))+ else:+ print("%-14s %s <- %s" % (decision["model"], decision["reason"],+ line if len(line) <= 48 else line[:45] + "..."))+ except ValueError as error:+ print("laya: %s" % error, file=sys.stderr)+ return 2+ except (ImportError, OSError, RuntimeError) as error:+ print("laya: could not run Laya (%s)." % error, file=sys.stderr)+ print("Check that the dependencies are installed and the checkpoints can be "+ "downloaded from the Hugging Face hub (network access is needed on first use).",+ file=sys.stderr)+ return 2+ return 0++def interactive(args):print("Laya interactive mode. Type a request and press Enter; Ctrl-D or 'quit' to exit.")router = make_router(args)diff --cc tests/test_cli.pyindex 634e4485,6e79c9bc..e6125ac7--- a/tests/test_cli.py+++ b/tests/test_cli.py@@@ -211,118 -154,142 +216,247 @@@ check("preset: absent means router quessorted(stub.questions) == sorted(cli.PRESETS["router"]()),str(sorted(stub.questions or {})))- code, out, err, stub = 0, "", "", None- try:- run_cli(["hi", "--preset", "bogus"])- except SystemExit as exit_:- code = exit_.code+ code, out, err, stub = run_cli(["hi", "--preset", "bogus"])check("preset: unknown name rejected by argparse", code == 2, "got %r" % code)+# --------------------------------------------------------------------- batch mode+import json # noqa: E402+import tempfile # noqa: E402+++class BatchRouter:+ """Records whole-batch calls; any single-state call is a bug in batch mode."""++ def __init__(self):+ self.predict_batch_calls = []+ self.route_batch_calls = []++ def predict_batch(self, requests, batch_size=None):+ self.predict_batch_calls.append((requests, batch_size))+ return [{"model": "laya-rl", "answers": {"difficulty": {"score": float(i)}}, "usage": {}}+ for i in range(len(requests))]++ def route_batch(self, requests):+ self.route_batch_calls.append(requests)+ return [StubDecision() for _ in requests]++ def route(self, state, **kwargs):+ raise AssertionError("single route() called in batch mode")++ def predict(self, state, questions, **kwargs):+ raise AssertionError("single predict() called in batch mode")+++def run_batch_cli(argv, router=None):+ stub = router or BatchRouter()+ original = cli.make_router+ cli.make_router = lambda args: stub+ out, err = io.StringIO(), io.StringIO()+ try:+ with redirect_stdout(out), redirect_stderr(err):+ code = cli.main(argv)+ finally:+ cli.make_router = original+ return code, out.getvalue(), err.getvalue(), stub+++tmp = tempfile.mkdtemp()+path = os.path.join(tmp, "requests.txt")+with open(path, "w", encoding="utf-8") as handle:+ handle.write("first ticket\n\n second ticket \n")++code, out, err, stub = run_batch_cli(["--batch", path, "--predict", "--batch-size", "8"])+check("batch predict: exit code", code == 0, "got %r %s" % (code, err))+requests, batch_size = stub.predict_batch_calls[0]+check("batch predict: exactly one predict_batch call", len(stub.predict_batch_calls) == 1)+check("batch predict: blank lines skipped, lines stripped",+ [r["state"]["request"] for r in requests] == ["first ticket", "second ticket"],+ str([r["state"] for r in requests]))+check("batch predict: --batch-size forwarded", batch_size == 8, str(batch_size))+check("batch predict: router questions on every request",+ sorted(requests[0]["questions"]) == sorted(cli.PRESETS["router"]()),+ str(sorted(requests[0]["questions"])))+check("batch predict: answers printed for both states", out.count("difficulty") == 2, out)++code, out, err, stub = run_batch_cli(["--batch", path, "--predict", "--json"])+parsed = [json.loads(line) for line in out.splitlines() if line.strip()]+check("batch --json: one JSONL record per state", len(parsed) == 2+ and all("answers" in p for p in parsed), out)++code, out, err, stub = run_batch_cli(["--batch", path])+check("batch route: one route_batch call, no checkpoint",+ len(stub.route_batch_calls) == 1 and stub.predict_batch_calls == [])+check("batch route: one line per state", out.count("multilingual") == 2, out)++code, out, err, stub = run_batch_cli(["--batch", path, "--preset", "triage"])+requests = stub.predict_batch_calls[0][0]+check("batch preset: state key follows the preset",+ all(set(r["state"]) == {"message"} for r in requests), str([r["state"] for r in requests]))++code, out, err, stub = run_batch_cli(["--batch", path, "--predict", "--model", "english",+ "--lang", "de"])+first = stub.predict_batch_calls[0][0][0]+check("batch flags: model/lang forwarded onto every request",+ first.get("model") == "english" and first.get("lang") == "de", str(first))++code, out, err, stub = run_batch_cli(["--batch", os.path.join(tmp, "missing.txt")])+check("batch missing file: exit 2 with a named failure",+ code == 2 and "could not read" in err, "code %r err %r" % (code, err))++code, out, err, stub = run_batch_cli(["--batch", path, "inline text"])+check("batch: text and --batch are mutually exclusive",+ code == 2 and "not both" in err, "code %r err %r" % (code, err))++empty = os.path.join(tmp, "empty.txt")+open(empty, "w").close()+code, out, err, stub = run_batch_cli(["--batch", empty, "--predict"])+check("batch empty file: exit 2, no calls",+ code == 2 and "no requests" in err and stub.predict_batch_calls == [],+ "code %r err %r" % (code, err))++_original_stdin = sys.stdin+sys.stdin = io.StringIO("piped one\npiped two\n")+try:+ code, out, err, stub = run_batch_cli(["--batch", "-", "--predict", "--json"])+finally:+ sys.stdin = _original_stdin+check("batch -: reads stdin", code == 0+ and len(stub.predict_batch_calls[0][0]) == 2+ and len([l for l in out.splitlines() if l.strip()]) == 2, "code %r err %r" % (code, err))++ # --------------------------------------------------------------------- --questions FILE+ # The CLI could only ever answer the five built-in presets, while the library's whole input is a+ # user-written question dict. A file is the only way to get one onto the command line, so the+ # loader is the seam under test here: what it accepts, and what the request ends up under.+ import json as _json # noqa: E402+ import tempfile as _tempfile # noqa: E402++ _TMP = _tempfile.mkdtemp(prefix="laya-cli-questions-")+++ def write_json(name, payload):+ path = os.path.join(_TMP, name)+ with open(path, "w", encoding="utf-8") as handle:+ _json.dump(payload, handle)+ return path+++ INTENTS = {+ "intent": {"type": "choice", "instructions": "Which intent?",+ "criteria": {"card_arrival": "where is my card", "fee": "a charge appeared"}},+ }+ INTENTS_PATH = write_json("intents.json", INTENTS)+ KEYED_PATH = write_json("keyed.json", {"state_key": "body",+ "questions": {"spam": {"type": "noul",+ "instructions": "Is `body` spam?"}}})++ code, out, err, stub = run_cli(["My payment failed twice", "--questions", INTENTS_PATH],+ router=QuestionRecorder())+ check("questions: exit code", code == 0, "got %r, err %r" % (code, err))+ check("questions: implies --predict", stub.questions == INTENTS, str(stub.questions))+ check("questions: state under the default key", stub.states[0] == {"request": "My payment failed twice"},+ str(stub.states[0]))+ check("questions: no standalone route call", stub.route_calls == [])+ check("questions: prints answers", "intent" in out, out)++ # A file that declares its own key has to win: the instructions name `body`, and asking about a+ # field that is not there is the #426 failure this path keeps reproducing.+ code, out, err, stub = run_cli(["hello", "--questions", KEYED_PATH], router=QuestionRecorder())+ check("questions: declared state_key used", stub.states[0] == {"body": "hello"}, str(stub.states[0]))+ check("questions: questions unwrapped",+ sorted(stub.questions) == ["spam"], str(sorted(stub.questions or {})))++ # The CLI validates the file's shape and defers question semantics to core, so a question set is+ # handed over unchanged rather than normalised twice.+ code, out, err, stub = run_cli(["hi", "--questions", write_json("odd.json", {"a": {"type": "bogus"}})],+ router=QuestionRecorder())+ check("questions: nonsense type reaches predict unchanged",+ stub.questions == {"a": {"type": "bogus"}}, str(stub.questions))++ for name, payload, fragment in (+ ("empty.json", {}, "holds no questions"),+ ("list.json", [1], "must be a JSON object"),+ ("scalar.json", {"a": 5}, "must map to an object"),+ ("badwrap.json", {"questions": [1]}, "'questions' field"),+ ("badkey.json", {"state_key": "", "questions": {"a": {"type": "noul"}}}, "'state_key'"),+ ):+ code, out, err, stub = run_cli(["hi", "--questions", write_json(name, payload)],+ router=QuestionRecorder())+ check("questions %s: exit code 2" % name, code == 2, "got %r" % code)+ check("questions %s: names the problem" % name, fragment in err, err)++ code, out, err, stub = run_cli(["hi", "--questions", os.path.join(_TMP, "missing.json")],+ router=QuestionRecorder())+ check("questions: missing file is a handled error",+ code == 2 and "no such --questions file" in err, "code %r, err %r" % (code, err))++ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH, "--preset", "triage"],+ router=QuestionRecorder())+ check("questions: two sources refuse each other",+ code == 2 and "pass one" in err, "code %r, err %r" % (code, err))++ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH, "--json"],+ router=QuestionRecorder())+ check("questions: --json prints the raw answers", code == 0 and '"intent"' in out,+ "code %r, out %r" % (code, out))++ # The same file answers every preset's questions too, so --questions replaces none of them.+ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH, "--model", "english"],+ router=QuestionRecorder())+ check("questions: --model still forwarded", stub.kwargs.get("model") == "english", str(stub.kwargs))++ # --------------------------------------------------------------------- the token budget+ # `--questions` is what makes a many-label question reachable from the shell, and a many-label+ # question is exactly what overflows the shared option budget. Both halves have to be here: on+ # `laya` (head budget 192) the 58-label Massive-Intent question scores 24/58 by default and+ # 34/58 at --head-max-len 384, so the CLI that can ask the question must also be able to fit it.+ # Nothing set has to mean nothing sent: the checkpoint's own budget stays in charge, and a+ # `max_len=None` in the call would be a different promise from the one `predict` documents.+ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH], router=QuestionRecorder())+ check("budget: default sends no override", "max_len" not in stub.kwargs and+ "head_max_len" not in stub.kwargs, str(stub.kwargs))++ for argv, expected in (+ (["--max-len", "1024"], {"max_len": 1024}),+ (["--head-max-len", "384"], {"head_max_len": 384}),+ (["--max-len", "1024", "--head-max-len", "384"], {"max_len": 1024, "head_max_len": 384}),+ # A zero budget is a real value, not an absent one.+ (["--head-max-len", "0"], {"head_max_len": 0}),+ ):+ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH] + argv, router=QuestionRecorder())+ sent = {k: v for k, v in stub.kwargs.items() if k in ("max_len", "head_max_len")}+ check("budget %s: forwarded" % " ".join(argv), sent == expected, "%r != %r" % (sent, expected))++ # The budget belongs to the request, not to how the questions arrived.+ for argv in (["--predict"], ["--preset", "triage"], ["--questions", INTENTS_PATH]):+ code, out, err, stub = run_cli(["hi"] + argv + ["--head-max-len", "512"], router=QuestionRecorder())+ check("budget on %s: forwarded" % " ".join(argv), stub.kwargs.get("head_max_len") == 512,+ str(stub.kwargs))+ check("budget on %s: questions unaffected" % " ".join(argv), stub.questions is not None)++ # A head budget cannot exceed the request budget it lives inside; that is core's rule, so the CLI+ # passes the pair through and reports core's words rather than inventing its own.+ class BudgetRejectingRouter(QuestionRecorder):+ def predict(self, state, questions, **kwargs):+ self.kwargs = kwargs+ if kwargs.get("head_max_len", 0) > kwargs.get("max_len", 1 << 30):+ raise ValueError("head_max_len must not exceed max_len")+ return {"answers": {}}+++ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH, "--max-len", "128",+ "--head-max-len", "384"], router=BudgetRejectingRouter())+ check("budget: a rejected pair exits 2", code == 2, "got %r" % code)+ check("budget: core's message reaches the shell", "head_max_len must not exceed max_len" in err, err)+ check("budget: both flags reached predict together",+ (stub.kwargs.get("max_len"), stub.kwargs.get("head_max_len")) == (128, 384), str(stub.kwargs))++ for value in ("abc", ""):+ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH, "--head-max-len=%s" % value],+ router=QuestionRecorder())+ check("budget %r: rejected by argparse" % value, code == 2 and "invalid int value" in err,+ "code %r, err %r" % (code, err))+# --------------------------------------------------------------------- reportprint("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL: