Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

Merge pull request #542 from aashish254/feat-cli-questions

Nandakishorf888f4378611
diff --cc README.md
index 4666369e,f37cb797..4cd6755e
--- a/README.md
+++ b/README.md
@@@ -222,15 -198,22 +222,25 @@@ Installing the package also installs a
laya "I was charged twice, please refund" # routing decision only; works offline, no download
laya "Refactor this service" --predict # full answers (downloads the checkpoint on first use)
laya "Mein Konto wurde zweimal belastet" --lang de # force a language instead of detecting it
+laya "My payment failed twice" --model ml # pin a checkpoint: names, aliases and casing all resolve as the SDK resolves them
laya "My payment failed twice" --preset triage # answer a ready-made preset (triage, email, guard, moderation, router)
+laya --batch tickets.txt --predict # score a file of requests, one per line, in one batch
+cat tickets.txt | laya --batch - --predict --json # stdin; one JSON line of answers per request
+ laya "Where is my card" --questions intents.json # answer your own questions, written in a JSON file
laya # interactive mode
```
-Routing alone never downloads a checkpoint, so it returns in milliseconds. `--predict` loads the routed checkpoint, which needs network access to the Hugging Face hub the first time; if a checkpoint cannot be downloaded, the CLI says so instead of crashing.
+Routing alone never downloads a checkpoint, so it returns in milliseconds. `--predict` loads the routed checkpoint, which needs network access to the Hugging Face hub the first time; if a checkpoint cannot be downloaded, the CLI says so instead of crashing. `--batch` (with or without `--predict`) sends the whole file through `Router.predict_batch` in one process, so the requests share checkpoint loads and forward passes — measured 2.6x on 20 tickets vs looping `predict` one by one, with `--batch-size N` to bound the forward pass and `--json` for JSONL output. Batch routing (`laya --batch FILE`, no `--predict`) likewise answers with `route_batch` in one pass, still without loading anything.
+ `--questions` takes the same question dict the SDK takes, as JSON: either the mapping itself, or
+ `{"state_key": "body", "questions": {...}}` when the question's instructions name a field other than
+ `request`. It implies `--predict`, and a question set with many labels usually wants
+ `--head-max-len` with it: on 58 MASSIVE-INTENT labels written as one choice question, the English
+ checkpoint goes from 24/58 correct at its default 192-token option budget to 34/58 at
+ `--head-max-len 384`, for about 1.4x the per-request time on CPU. Widening it further costs the
+ accuracy back, because `max_len` then leaves fewer tokens for the request itself. [Honest
+ limits](#honest-limits) describes the same budget ceiling for a 77-option question.
+
---
## Try it locally: web GUI + JSON API
@@@ -1116,7 -993,7 +1126,7 @@@ failure; it does not establish calibrat
* `laya` (English) defaults to 512 context (`head_max_len = 192`, ~320 tokens for state).
* `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`, ~768 tokens for state; mmBERT-base encoder supports up to 8,192 with RoPE).
At default settings, a 77-option question like Banking77 allocates only `(256 - 16) // 77` ≈ 3–4 tokens per label, which causes accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question:
- 1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. Both are also per-request: `predict(state, questions, head_max_len=512, max_len=1024)` widens one question without changing the agent for everyone else, and every LangChain node takes the same two arguments ([LangChain guide](docs/langchain.md)).
- 1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. `laya --questions` takes the same two budgets as `--max-len` / `--head-max-len`.
++ 1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. Both are also per-request: `predict(state, questions, head_max_len=512, max_len=1024)` widens one question without changing the agent for everyone else, and every LangChain node takes the same two arguments ([LangChain guide](docs/langchain.md)). `laya --questions` takes the same two budgets as `--max-len` / `--head-max-len`.
2. Or shortlist with embeddings and run one forward pass on the top `k` labels (`predict_shortlist`, example below). `predict` and `system_one` still score every criterion they are given.
3. Or split the label set yourself into a coarse question and a fine question.
diff --cc laya/cli.py
index 8ea3b611,c59a8e07..2a5e846f
--- a/laya/cli.py
+++ b/laya/cli.py
@@@ -4,17 -4,16 +4,22 @@@
laya "Refactor this service" --predict # full answers, loads the checkpoint
laya # interactive mode
laya "Mein Konto wurde zweimal belastet" --lang de # explicit language
+ laya "My payment failed twice" --model ml # pin a checkpoint, by name or alias
laya "My payment failed twice" --preset triage # a ready-made question preset
+ laya --batch tickets.txt --predict # a file of requests, one per line
+ cat tickets.txt | laya --batch - --predict --json # stdin, one JSON line of answers each
+ laya "Where is my card" --questions intents.json # your own questions, from a JSON file
+ laya "Where is my card" --questions intents.json --head-max-len 384
Routing (the default) never downloads a checkpoint, so it works offline and
returns in milliseconds. --predict loads the routed checkpoint on first use,
which needs network access to the Hugging Face hub. --preset answers one of the
+ready-made question presets from laya.presets and implies --predict.
+--batch scores a whole file (or stdin, with `-`) through Router.predict_batch in one
+process, so the states share checkpoint loads and forward passes; --json prints JSONL.
+ ready-made question presets from laya.presets and implies --predict. --questions
+ answers the question set in a JSON file instead; --max-len and --head-max-len
+ raise its token budget for the request, which is what a many-label question needs.
"""
import argparse
@@@ -80,15 -58,17 +85,24 @@@ def build_parser()
parser.add_argument("--preset", choices=sorted(PRESETS), metavar="NAME",
help="answer a ready-made question preset (%s) instead of the router questions; implies --predict"
% ", ".join(sorted(PRESETS)))
+ parser.add_argument("--questions", metavar="FILE",
+ help="a JSON file of your own typed questions to answer instead of the router questions"
+ " or a preset; implies --predict")
+ parser.add_argument("--max-len", type=int, dest="max_len", metavar="N",
+ help="token budget for the whole request (state plus options); defaults to the"
+ " checkpoint's own budget")
+ parser.add_argument("--head-max-len", type=int, dest="head_max_len", metavar="N",
+ help="token budget the choice options share; raise it when a question has many"
+ " labels, so each keeps enough tokens to stay distinct")
parser.add_argument("--device", help="torch device, e.g. cpu or cuda")
parser.add_argument("--json", action="store_true", help="print the raw result as JSON")
+ parser.add_argument("--batch", metavar="FILE",
+ help="score a file of requests, one per line (use '-' for stdin), instead "
+ "of a single text; implies neither --predict nor --preset, but the "
+ "same modes apply: routing by default, answers with --predict/--preset")
+ parser.add_argument("--batch-size", type=int, default=None, metavar="N",
+ help="states per forward pass in --batch mode; the default sends each "
+ "routed group in one pass")
return parser
@@@ -162,57 -202,6 +236,58 @@@ def run(text, args, router=None)
return 0
+def read_batch_lines(source):
+ """One request per non-blank line of FILE, or of stdin for '-'."""
+ if source == "-":
+ lines = sys.stdin.read().splitlines()
+ else:
+ with open(source, encoding="utf-8") as handle:
+ lines = handle.read().splitlines()
+ return [line.strip() for line in lines if line.strip()]
+
+
+def run_batch(lines, args, router=None):
+ """Route or predict a whole file in one call, so states share checkpoint loads and
+ forward passes; returns 0 on success, 2 on a handled error."""
+ router = router or make_router(args)
+ overrides = {key: value for key, value in (("model", args.model), ("task", args.task),
+ ("lang", args.lang)) if value is not None}
+ try:
- if args.predict or args.preset:
- questions = PRESETS[args.preset]() if args.preset else laya.router_questions()
- key = PRESET_STATE_KEYS.get(args.preset, "request")
++ if args.predict or args.preset or getattr(args, "questions", None):
++ questions, key = resolve_questions(args)
++ # Router.predict_batch reads the token budget per request (#583).
++ overrides.update(budget_overrides(args))
+ requests = [{"state": {key: line}, "questions": questions, **overrides}
+ for line in lines]
+ results = router.predict_batch(requests, batch_size=args.batch_size)
+ for line, result in zip(lines, results):
+ if args.json:
+ print(json.dumps(result, ensure_ascii=False, default=str))
+ else:
+ print("# %s" % (line if len(line) <= 72 else line[:69] + "..."))
+ show_answers(result)
+ print()
+ else:
+ decisions = router.route_batch([{"state": {"text": line}, "questions": {},
+ **overrides} for line in lines])
+ for line, decision in zip(lines, decisions):
+ if args.json:
+ print(json.dumps(dict(decision), ensure_ascii=False, default=str))
+ else:
+ print("%-14s %s <- %s" % (decision["model"], decision["reason"],
+ line if len(line) <= 48 else line[:45] + "..."))
+ except ValueError as error:
+ print("laya: %s" % error, file=sys.stderr)
+ return 2
+ except (ImportError, OSError, RuntimeError) as error:
+ print("laya: could not run Laya (%s)." % error, file=sys.stderr)
+ print("Check that the dependencies are installed and the checkpoints can be "
+ "downloaded from the Hugging Face hub (network access is needed on first use).",
+ file=sys.stderr)
+ return 2
+ return 0
+
+
def interactive(args):
print("Laya interactive mode. Type a request and press Enter; Ctrl-D or 'quit' to exit.")
router = make_router(args)
diff --cc tests/test_cli.py
index 634e4485,6e79c9bc..e6125ac7
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@@ -211,118 -154,142 +216,247 @@@ check("preset: absent means router ques
sorted(stub.questions) == sorted(cli.PRESETS["router"]()),
str(sorted(stub.questions or {})))
- code, out, err, stub = 0, "", "", None
- try:
- run_cli(["hi", "--preset", "bogus"])
- except SystemExit as exit_:
- code = exit_.code
+ code, out, err, stub = run_cli(["hi", "--preset", "bogus"])
check("preset: unknown name rejected by argparse", code == 2, "got %r" % code)
+# --------------------------------------------------------------------- batch mode
+import json # noqa: E402
+import tempfile # noqa: E402
+
+
+class BatchRouter:
+ """Records whole-batch calls; any single-state call is a bug in batch mode."""
+
+ def __init__(self):
+ self.predict_batch_calls = []
+ self.route_batch_calls = []
+
+ def predict_batch(self, requests, batch_size=None):
+ self.predict_batch_calls.append((requests, batch_size))
+ return [{"model": "laya-rl", "answers": {"difficulty": {"score": float(i)}}, "usage": {}}
+ for i in range(len(requests))]
+
+ def route_batch(self, requests):
+ self.route_batch_calls.append(requests)
+ return [StubDecision() for _ in requests]
+
+ def route(self, state, **kwargs):
+ raise AssertionError("single route() called in batch mode")
+
+ def predict(self, state, questions, **kwargs):
+ raise AssertionError("single predict() called in batch mode")
+
+
+def run_batch_cli(argv, router=None):
+ stub = router or BatchRouter()
+ original = cli.make_router
+ cli.make_router = lambda args: stub
+ out, err = io.StringIO(), io.StringIO()
+ try:
+ with redirect_stdout(out), redirect_stderr(err):
+ code = cli.main(argv)
+ finally:
+ cli.make_router = original
+ return code, out.getvalue(), err.getvalue(), stub
+
+
+tmp = tempfile.mkdtemp()
+path = os.path.join(tmp, "requests.txt")
+with open(path, "w", encoding="utf-8") as handle:
+ handle.write("first ticket\n\n second ticket \n")
+
+code, out, err, stub = run_batch_cli(["--batch", path, "--predict", "--batch-size", "8"])
+check("batch predict: exit code", code == 0, "got %r %s" % (code, err))
+requests, batch_size = stub.predict_batch_calls[0]
+check("batch predict: exactly one predict_batch call", len(stub.predict_batch_calls) == 1)
+check("batch predict: blank lines skipped, lines stripped",
+ [r["state"]["request"] for r in requests] == ["first ticket", "second ticket"],
+ str([r["state"] for r in requests]))
+check("batch predict: --batch-size forwarded", batch_size == 8, str(batch_size))
+check("batch predict: router questions on every request",
+ sorted(requests[0]["questions"]) == sorted(cli.PRESETS["router"]()),
+ str(sorted(requests[0]["questions"])))
+check("batch predict: answers printed for both states", out.count("difficulty") == 2, out)
+
+code, out, err, stub = run_batch_cli(["--batch", path, "--predict", "--json"])
+parsed = [json.loads(line) for line in out.splitlines() if line.strip()]
+check("batch --json: one JSONL record per state", len(parsed) == 2
+ and all("answers" in p for p in parsed), out)
+
+code, out, err, stub = run_batch_cli(["--batch", path])
+check("batch route: one route_batch call, no checkpoint",
+ len(stub.route_batch_calls) == 1 and stub.predict_batch_calls == [])
+check("batch route: one line per state", out.count("multilingual") == 2, out)
+
+code, out, err, stub = run_batch_cli(["--batch", path, "--preset", "triage"])
+requests = stub.predict_batch_calls[0][0]
+check("batch preset: state key follows the preset",
+ all(set(r["state"]) == {"message"} for r in requests), str([r["state"] for r in requests]))
+
+code, out, err, stub = run_batch_cli(["--batch", path, "--predict", "--model", "english",
+ "--lang", "de"])
+first = stub.predict_batch_calls[0][0][0]
+check("batch flags: model/lang forwarded onto every request",
+ first.get("model") == "english" and first.get("lang") == "de", str(first))
+
+code, out, err, stub = run_batch_cli(["--batch", os.path.join(tmp, "missing.txt")])
+check("batch missing file: exit 2 with a named failure",
+ code == 2 and "could not read" in err, "code %r err %r" % (code, err))
+
+code, out, err, stub = run_batch_cli(["--batch", path, "inline text"])
+check("batch: text and --batch are mutually exclusive",
+ code == 2 and "not both" in err, "code %r err %r" % (code, err))
+
+empty = os.path.join(tmp, "empty.txt")
+open(empty, "w").close()
+code, out, err, stub = run_batch_cli(["--batch", empty, "--predict"])
+check("batch empty file: exit 2, no calls",
+ code == 2 and "no requests" in err and stub.predict_batch_calls == [],
+ "code %r err %r" % (code, err))
+
+_original_stdin = sys.stdin
+sys.stdin = io.StringIO("piped one\npiped two\n")
+try:
+ code, out, err, stub = run_batch_cli(["--batch", "-", "--predict", "--json"])
+finally:
+ sys.stdin = _original_stdin
+check("batch -: reads stdin", code == 0
+ and len(stub.predict_batch_calls[0][0]) == 2
+ and len([l for l in out.splitlines() if l.strip()]) == 2, "code %r err %r" % (code, err))
+
+ # --------------------------------------------------------------------- --questions FILE
+ # The CLI could only ever answer the five built-in presets, while the library's whole input is a
+ # user-written question dict. A file is the only way to get one onto the command line, so the
+ # loader is the seam under test here: what it accepts, and what the request ends up under.
+ import json as _json # noqa: E402
+ import tempfile as _tempfile # noqa: E402
+
+ _TMP = _tempfile.mkdtemp(prefix="laya-cli-questions-")
+
+
+ def write_json(name, payload):
+ path = os.path.join(_TMP, name)
+ with open(path, "w", encoding="utf-8") as handle:
+ _json.dump(payload, handle)
+ return path
+
+
+ INTENTS = {
+ "intent": {"type": "choice", "instructions": "Which intent?",
+ "criteria": {"card_arrival": "where is my card", "fee": "a charge appeared"}},
+ }
+ INTENTS_PATH = write_json("intents.json", INTENTS)
+ KEYED_PATH = write_json("keyed.json", {"state_key": "body",
+ "questions": {"spam": {"type": "noul",
+ "instructions": "Is `body` spam?"}}})
+
+ code, out, err, stub = run_cli(["My payment failed twice", "--questions", INTENTS_PATH],
+ router=QuestionRecorder())
+ check("questions: exit code", code == 0, "got %r, err %r" % (code, err))
+ check("questions: implies --predict", stub.questions == INTENTS, str(stub.questions))
+ check("questions: state under the default key", stub.states[0] == {"request": "My payment failed twice"},
+ str(stub.states[0]))
+ check("questions: no standalone route call", stub.route_calls == [])
+ check("questions: prints answers", "intent" in out, out)
+
+ # A file that declares its own key has to win: the instructions name `body`, and asking about a
+ # field that is not there is the #426 failure this path keeps reproducing.
+ code, out, err, stub = run_cli(["hello", "--questions", KEYED_PATH], router=QuestionRecorder())
+ check("questions: declared state_key used", stub.states[0] == {"body": "hello"}, str(stub.states[0]))
+ check("questions: questions unwrapped",
+ sorted(stub.questions) == ["spam"], str(sorted(stub.questions or {})))
+
+ # The CLI validates the file's shape and defers question semantics to core, so a question set is
+ # handed over unchanged rather than normalised twice.
+ code, out, err, stub = run_cli(["hi", "--questions", write_json("odd.json", {"a": {"type": "bogus"}})],
+ router=QuestionRecorder())
+ check("questions: nonsense type reaches predict unchanged",
+ stub.questions == {"a": {"type": "bogus"}}, str(stub.questions))
+
+ for name, payload, fragment in (
+ ("empty.json", {}, "holds no questions"),
+ ("list.json", [1], "must be a JSON object"),
+ ("scalar.json", {"a": 5}, "must map to an object"),
+ ("badwrap.json", {"questions": [1]}, "'questions' field"),
+ ("badkey.json", {"state_key": "", "questions": {"a": {"type": "noul"}}}, "'state_key'"),
+ ):
+ code, out, err, stub = run_cli(["hi", "--questions", write_json(name, payload)],
+ router=QuestionRecorder())
+ check("questions %s: exit code 2" % name, code == 2, "got %r" % code)
+ check("questions %s: names the problem" % name, fragment in err, err)
+
+ code, out, err, stub = run_cli(["hi", "--questions", os.path.join(_TMP, "missing.json")],
+ router=QuestionRecorder())
+ check("questions: missing file is a handled error",
+ code == 2 and "no such --questions file" in err, "code %r, err %r" % (code, err))
+
+ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH, "--preset", "triage"],
+ router=QuestionRecorder())
+ check("questions: two sources refuse each other",
+ code == 2 and "pass one" in err, "code %r, err %r" % (code, err))
+
+ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH, "--json"],
+ router=QuestionRecorder())
+ check("questions: --json prints the raw answers", code == 0 and '"intent"' in out,
+ "code %r, out %r" % (code, out))
+
+ # The same file answers every preset's questions too, so --questions replaces none of them.
+ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH, "--model", "english"],
+ router=QuestionRecorder())
+ check("questions: --model still forwarded", stub.kwargs.get("model") == "english", str(stub.kwargs))
+
+ # --------------------------------------------------------------------- the token budget
+ # `--questions` is what makes a many-label question reachable from the shell, and a many-label
+ # question is exactly what overflows the shared option budget. Both halves have to be here: on
+ # `laya` (head budget 192) the 58-label Massive-Intent question scores 24/58 by default and
+ # 34/58 at --head-max-len 384, so the CLI that can ask the question must also be able to fit it.
+ # Nothing set has to mean nothing sent: the checkpoint's own budget stays in charge, and a
+ # `max_len=None` in the call would be a different promise from the one `predict` documents.
+ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH], router=QuestionRecorder())
+ check("budget: default sends no override", "max_len" not in stub.kwargs and
+ "head_max_len" not in stub.kwargs, str(stub.kwargs))
+
+ for argv, expected in (
+ (["--max-len", "1024"], {"max_len": 1024}),
+ (["--head-max-len", "384"], {"head_max_len": 384}),
+ (["--max-len", "1024", "--head-max-len", "384"], {"max_len": 1024, "head_max_len": 384}),
+ # A zero budget is a real value, not an absent one.
+ (["--head-max-len", "0"], {"head_max_len": 0}),
+ ):
+ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH] + argv, router=QuestionRecorder())
+ sent = {k: v for k, v in stub.kwargs.items() if k in ("max_len", "head_max_len")}
+ check("budget %s: forwarded" % " ".join(argv), sent == expected, "%r != %r" % (sent, expected))
+
+ # The budget belongs to the request, not to how the questions arrived.
+ for argv in (["--predict"], ["--preset", "triage"], ["--questions", INTENTS_PATH]):
+ code, out, err, stub = run_cli(["hi"] + argv + ["--head-max-len", "512"], router=QuestionRecorder())
+ check("budget on %s: forwarded" % " ".join(argv), stub.kwargs.get("head_max_len") == 512,
+ str(stub.kwargs))
+ check("budget on %s: questions unaffected" % " ".join(argv), stub.questions is not None)
+
+ # A head budget cannot exceed the request budget it lives inside; that is core's rule, so the CLI
+ # passes the pair through and reports core's words rather than inventing its own.
+ class BudgetRejectingRouter(QuestionRecorder):
+ def predict(self, state, questions, **kwargs):
+ self.kwargs = kwargs
+ if kwargs.get("head_max_len", 0) > kwargs.get("max_len", 1 << 30):
+ raise ValueError("head_max_len must not exceed max_len")
+ return {"answers": {}}
+
+
+ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH, "--max-len", "128",
+ "--head-max-len", "384"], router=BudgetRejectingRouter())
+ check("budget: a rejected pair exits 2", code == 2, "got %r" % code)
+ check("budget: core's message reaches the shell", "head_max_len must not exceed max_len" in err, err)
+ check("budget: both flags reached predict together",
+ (stub.kwargs.get("max_len"), stub.kwargs.get("head_max_len")) == (128, 384), str(stub.kwargs))
+
+ for value in ("abc", ""):
+ code, out, err, stub = run_cli(["hi", "--questions", INTENTS_PATH, "--head-max-len=%s" % value],
+ router=QuestionRecorder())
+ check("budget %r: rejected by argparse" % value, code == 2 and "invalid int value" in err,
+ "code %r, err %r" % (code, err))
+
# --------------------------------------------------------------------- report
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL: