Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

release: 0.4.25, the Router's vega lane

Nandakishord5a78081fd79
diff --git a/README.md b/README.md
index d34bf4a4..cc79bb8b 100644
--- a/README.md
+++ b/README.md
@@ -243,6 +243,18 @@ The shipped checkpoints work zero-shot, but fine-tuning on decisions from your o
**[nandhakishorm.github.io/laya](https://nandhakishorm.github.io/laya/)**: guides for [prediction hooks](https://nandhakishorm.github.io/laya/hooks/), [schema-driven decisions](https://nandhakishorm.github.io/laya/structured/), [Docker](https://nandhakishorm.github.io/laya/docker/) and [LangChain and LangGraph](https://nandhakishorm.github.io/laya/langchain/), plus a full [API reference](https://nandhakishorm.github.io/laya/reference/).
+## What's new in 0.4.25
+
+* **The `Router` has a vega lane.** When vegaml is installed, `Router()` routes what the encoders
+ cannot read to `laya.Agent(vegaml=True)`, built on first use: `image=`, `audio=` or `video=` in
+ place of a state, a state past 8,192 tokens (`vega_over_tokens`), and any question the vega agent
+ has a fitted head for (`Router(vega=agent)` after `agent.fit(...)`). Text the encoders can read
+ keeps going to the encoders with the same decisions and reasons as before, `Router(vega=False)`
+ turns the lane off, `model="vega"` selects it, and `predict_batch` stays on the encoders. The CLI
+ gained `--image`, `--audio` and `--video` for the same lane; the HTTP server refuses the three
+ keys with a message saying where media goes. Details in
+ [the routing guide](https://nandhakishorm.github.io/laya/routing/#the-vega-lane).
+
## What's new in 0.4.24
* **Video, through the same call: `agent.predict(video="call.mp4", questions=questions)`** on
@@ -435,6 +447,7 @@ laya --batch tickets.txt --predict # score a file of requests,
cat tickets.txt | laya --batch - --predict --json # stdin; one JSON line of answers per request
laya "Where is my card" --questions intents.json # answer your own questions, written in a JSON file
laya "Refund my card" --predict --min-confidence 0.9 # mark an answer the model is unsure of
+laya --image invoice.png --preset triage # a picture, a recording (--audio) or a clip (--video) on the vega lane
laya # interactive mode
```
@@ -656,6 +669,8 @@ downloaded models between runs.
Laya ships three checkpoints. The built-in **`Router`** is the recommended entry point: it evaluates any state in any language, automatically detects scripts and languages in sub-milliseconds, and dispatches to the optimal checkpoint in a single forward pass.
+> **Since 0.4.25, with vegaml installed:** the same `Router` also takes `image=`, `audio=` or `video=` in place of a state and sends them, states past 8,192 tokens, and questions with fitted heads to `laya.Agent(vegaml=True)`; everything else routes exactly as below. `Router(vega=False)` turns that off. See [the vega lane](https://nandhakishorm.github.io/laya/routing/#the-vega-lane).
+
### Minimal example (30 seconds)
```python
diff --git a/docs/cli-mcp.md b/docs/cli-mcp.md
index d1767d02..a7774f00 100644
--- a/docs/cli-mcp.md
+++ b/docs/cli-mcp.md
@@ -86,6 +86,16 @@ The main controls are:
- `--task NAME` forces the typed-decisions workflow instead of detecting it.
- `--device cpu|cuda|...` passes a device choice to the Router.
- `--json` emits machine-readable output.
+- `--image FILE`, `--audio FILE` or `--video FILE` answer the questions about a picture, a recording
+ or a clip instead of a text. These go to the Router's vega lane (`laya.Agent(vegaml=True)`, on by
+ default when vegaml is installed, which it is with laya 0.4.23 and newer), imply `--predict`, and
+ take the place of the text argument. A recording is transcribed locally; a clip is decided on its
+ speech if it has any, else on sampled frames.
+
+```bash
+laya --image invoice.png --preset triage
+laya --audio call.wav --questions questions.json --json
+```
### Use a built-in preset
diff --git a/docs/http-api.md b/docs/http-api.md
index 2fa55550..8912324e 100644
--- a/docs/http-api.md
+++ b/docs/http-api.md
@@ -229,7 +229,10 @@ hook arguments `predict` also takes -- `hooks`, `on_predict_start`, `on_predict_
that runs inside the server process, and the last two say how the hooks a deployment installed
execute, so no value a caller sends has a meaning here. The same five are refused client-side by a
LangChain node with a `base_url` (`laya.integrations.langchain`), so a chain and a raw HTTP client
-now get the same answer.
+now get the same answer. `image`, `audio` and `video`, which `predict` takes since 0.4.25 to route a
+picture, a recording or a clip to the vega lane, are refused with a `422` as well: this endpoint
+takes text states only, and media goes through `laya.Agent(vegaml=True)` or a `Router` with the vega
+lane in the process that holds the model.
`model` is accepted so a Jev client can keep sending one. The public Hugging Face ids
(`convaiinnovations/laya-multilingual`, `convaiinnovations/laya-typed-decisions`), the checkpoint
diff --git a/docs/routing.md b/docs/routing.md
index ff63e8fd..f52b96a9 100644
--- a/docs/routing.md
+++ b/docs/routing.md
@@ -70,6 +70,47 @@ print(decision.reason) # why that checkpoint was selected
`decision["model"]` and `decision["reason"]`. `Router.predict()` includes the same decision under
the result's `"routing"` key.
+## The vega lane
+
+Since 0.4.25 a `Router` has a fourth lane beside the three encoder checkpoints: `vega`, served by
+`laya.Agent(vegaml=True)`, the frozen 0.8B universal reader with vegaml's decision engine. It is on
+by default when the `vegaml` package is installed (it ships with laya 0.4.23 and newer) and built
+on first use, so a Router costs nothing more to construct than before. Text the encoders can read
+keeps going to the encoders, with the same decisions and the same reasons as before. The lane only
+takes what the encoders cannot:
+
+| input | routed to vega because |
+|---|---|
+| `image=`, `audio=` or `video=` instead of a state | the encoders have no vision or speech path |
+| a state past `vega_over_tokens` (8,192, the most any encoder reads) | the encoders would truncate it; vega reads up to 73,728 tokens whole |
+| a question the vega agent has a fitted head for | the head was fitted on your rows for that question |
+
+```python
+from laya import Router
+
+router = Router() # the lane is on if vegaml is installed
+router.predict(ticket, questions) # English text: english, as before
+router.predict(seventy_page_contract, questions) # about 40,000 tokens: vega, read whole
+router.predict(None, questions, image="invoice.png") # vega
+router.predict(None, questions, audio="call.wav") # vega, transcribed then decided
+router.route(long_text, questions).reason # "state is about 40000 tokens, past the 8192 ..."
+```
+
+`Router(vega=False)` turns the lane off and the Router behaves exactly as it did before, including
+refusing media with a message that names the lane. `Router(vega=True)` insists on it and raises at
+construction when vegaml is missing. `Router(vega="4b")` names a different reader (any name
+`laya.Agent(vegaml=...)` takes), and `Router(vega=agent)` attaches an agent you already built, which
+is how fitted heads reach the Router: fit on the agent, hand it over, and the questions it serves
+route to it. `model="vega"` selects the lane explicitly like any other checkpoint name.
+
+Two things stay on the encoders on purpose. `predict_batch` never routes to vega, because a batch
+shares one forward pass on one checkpoint; send media and long states one at a time. And the HTTP
+server refuses `image`, `audio` and `video` in a body with a message saying where media goes, since
+the JSON surface has no media encoding yet.
+
+Answers from the lane carry vega's fields on top of laya's: `prediction_set`, `abstain`, `unbound`
+and `route`; see [vega inside laya](vegaml.md).
+
## Explain a detection result
Use `explain=True` to inspect why built-in detection reached its result, without loading a
diff --git a/docs/vegaml.md b/docs/vegaml.md
index 7b4f750b..ff6196cd 100644
--- a/docs/vegaml.md
+++ b/docs/vegaml.md
@@ -122,6 +122,12 @@ which measures what the picture itself contributed.
## Model switching
+Since 0.4.25 the ordinary `laya.Router()` does this by itself when vegaml is installed: images,
+speech, video, states past 8,192 tokens and questions with fitted heads go to vega, everything else
+to the encoders, with `Router(vega=False)` to turn it off. See
+[the vega lane](routing.md#the-vega-lane). `laya.vega.Router` below is the explicit two-model form.
+
+
```python
from laya.vega import Router
diff --git a/laya/__init__.py b/laya/__init__.py
index 998e42fc..b795750d 100644
--- a/laya/__init__.py
+++ b/laya/__init__.py
@@ -35,7 +35,7 @@ from .presets import (
from .router import DEFAULT_MODELS, RouteDecision, Router
from .structured import DecisionResult, decide, decide_batch
-__version__ = "0.4.24"
+__version__ = "0.4.25"
# Routing, language detection and email cleaning are pure Python. The torch-backed names are
# resolved lazily so that `import laya` -- and therefore `from laya import Router` or
diff --git a/laya/cli.py b/laya/cli.py
index 93b3bd94..4b2f4633 100644
--- a/laya/cli.py
+++ b/laya/cli.py
@@ -128,6 +128,15 @@ def build_parser():
help="flag an answer whose calibrated confidence falls below P (0.0-1.0) as"
" low-confidence; the answer itself is kept, so this asks for a warning"
" rather than a refusal. Needs --predict, --preset or --questions.")
+ parser.add_argument("--image", metavar="FILE",
+ help="answer the questions about a picture instead of a text: routes to the vega lane "
+ "(needs vegaml, which ships with laya); implies --predict")
+ parser.add_argument("--audio", metavar="FILE",
+ help="answer the questions about a recording (wav, flac, ogg, mp3): transcribed locally, "
+ "then decided on the vega lane; implies --predict")
+ parser.add_argument("--video", metavar="FILE",
+ help="answer the questions about a clip: its speech if it has any, else sampled frames, "
+ "on the vega lane; implies --predict")
parser.add_argument("--device", help="torch device, e.g. cpu or cuda")
parser.add_argument("--json", action="store_true", help="print the raw result as JSON")
parser.add_argument("--explain", action="store_true",
@@ -283,10 +292,31 @@ def resolve_questions(args):
return laya.router_questions(), PRESET_STATE_KEYS["router"]
+def media_override(args):
+ """The three media arguments of `Router.predict`, as the flags gave them: at most one set,
+ the others None, so every control the parser names reaches the call."""
+ given = {key: (getattr(args, key, None) or None) for key in ("image", "audio", "video")}
+ if sum(1 for value in given.values() if value) > 1:
+ raise ValueError("pass one of --image, --audio, --video, not %s"
+ % " and ".join("--" + k for k, v in given.items() if v))
+ return given
+
+
def run(text, args, router=None):
"""Route or predict one request; returns 0 on success, 2 on a handled error."""
router = router or make_router(args)
try:
+ media = media_override(args)
+ if any(media.values()):
+ questions, _ = resolve_questions(args)
+ result = router.predict(None, questions, model=args.model, task=args.task, lang=args.lang,
+ lang_guess=args.lang_guess, **media,
+ **budget_overrides(args), **abstention_override(args))
+ if args.json:
+ print(json.dumps(result, ensure_ascii=False, indent=2, default=str))
+ else:
+ show_answers(result)
+ return 0
if args.predict or args.preset or args.questions:
questions, state_key = resolve_questions(args)
# Each preset's instructions name the field they read -- `` `message` ``,
@@ -300,7 +330,7 @@ def run(text, args, router=None):
state = {state_key: text}
result = router.predict(state, questions,
model=args.model, task=args.task, lang=args.lang,
- lang_guess=args.lang_guess,
+ lang_guess=args.lang_guess, **media,
**budget_overrides(args), **abstention_override(args))
if args.json:
print(json.dumps(result, ensure_ascii=False, indent=2, default=str))
@@ -518,6 +548,11 @@ def main(argv=None):
print("laya: no requests found in %s." % args.batch, file=sys.stderr)
return 2
return run_batch(lines, args)
+ if any(getattr(args, key, None) for key in ("image", "audio", "video")):
+ if text:
+ print("laya: pass either a text or one of --image, --audio, --video, not both.", file=sys.stderr)
+ return 2
+ return run("", args)
if not text:
return interactive(args)
return run(text, args)
diff --git a/laya/mcp/remote.py b/laya/mcp/remote.py
index 526a6032..3c80f7a0 100644
--- a/laya/mcp/remote.py
+++ b/laya/mcp/remote.py
@@ -191,7 +191,16 @@ class RemoteRouter(Router):
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
min_confidence: Optional[float] = None,
+ image: Any = None,
+ audio: Any = None,
+ video: Any = None,
) -> Dict[str, Any]:
+ if any(value is not None for value in (image, audio, video)):
+ # `Router.predict` takes these since 0.4.25 for the vega lane; laya-serve has no media
+ # encoding, so the remote says so here rather than sending a file name as a state
+ raise RemoteError("unsupported_remote",
+ "images, recordings and clips cannot be sent to laya-serve; use "
+ "laya.Agent(vegaml=True) or a local Router with the vega lane")
if hooks_timeout is not None:
raise RemoteError("unsupported_remote", "hooks_timeout belongs to the server's hooks")
if any(value is not None for value in (hooks, on_predict_start, on_predict_end, hooks_raise)):
diff --git a/laya/router.py b/laya/router.py
index 65d48e3c..924dc2c3 100644
--- a/laya/router.py
+++ b/laya/router.py
@@ -55,6 +55,13 @@ DEFAULT_MODELS = {
"typed-decisions": (BUNDLE_REPO, "typed-decisions"),
}
+#: The lane for what the encoders cannot read, served by vegaml's universal reader through
+#: `laya.Agent(vegaml=True)` when the Router is built with `vega` on (the default when vegaml is
+#: installed): images, speech and video, states past the encoders' 8,192 tokens, and questions
+#: with fitted heads. Text the encoders can read keeps going to the encoders exactly as before.
+VEGA_MODEL = "vega"
+VEGA_REPO = "nandakishorm/vega-U"
+
# The same checkpoints also live in their own repos, for anyone who prefers them.
STANDALONE_MODELS = {
"english": "convaiinnovations/laya",
@@ -63,6 +70,19 @@ STANDALONE_MODELS = {
}
+def _estimate_tokens(state: Any) -> int:
+ """A cheap token estimate for routing: characters over four, over the flattened state.
+ Only the decision "can an encoder read this at all" rests on it, so a rough count is enough."""
+ if state is None:
+ return 0
+ try:
+ from .lang import state_text
+ text = state_text(state, max_chars=1 << 30)
+ except Exception:
+ text = str(state)
+ return len(text) // 4
+
+
def _repo_str(spec):
"""Human-readable id for a model spec: 'repo' or 'repo/subfolder'."""
repo, sub = _split(spec)
@@ -724,6 +744,15 @@ class Router(HookRegistry):
hooks_timeout: Optional[float] = None,
agent_kwargs: Optional[Dict[str, Any]] = None,
sha256_digests: Optional[Dict[str, Optional[Dict[str, str]]]] = None,
+ # The vega lane. "auto": on when the vegaml package is installed, built on first use;
+ # True: on, and an ImportError now if vegaml is missing; False or None: off, and the Router
+ # behaves exactly as it did before 0.4.25; a string: the vegaml reader to use ("4b", a Hub
+ # id, an attached local dir); an already built `laya.Agent(vegaml=...)`: used as is, and its
+ # fitted heads route the questions they were fitted for.
+ vega: Any = "auto",
+ # A state estimated past this many tokens goes to vega, because no encoder checkpoint can
+ # read it whole; 8,192 is `laya-multilingual` at `max_len=8192`, the most any encoder takes.
+ vega_over_tokens: int = 8192,
):
self.hooks = normalise_hooks(hooks, on_predict_start, on_predict_end)
self.hooks_raise = bool(hooks_raise)
@@ -787,9 +816,49 @@ class Router(HookRegistry):
self._lock = threading.RLock()
self._build_lock = threading.Lock()
self._loading: Dict[str, _InFlightBuild] = {}
+ self.vega_over_tokens = int(vega_over_tokens)
+ self._vega_spec: Any = None
+ self._install_vega(vega)
if preload:
self.preload()
+ def _install_vega(self, vega: Any) -> None:
+ """Register the vega lane according to `vega`, without importing vegaml here: a lazy lane
+ is only looked up by `find_spec`, so building a Router stays as light as it was."""
+ if vega is None or vega is False:
+ return
+ if hasattr(vega, "predict") and not isinstance(vega, str):
+ self._add(VEGA_MODEL, None)
+ self._agents[VEGA_MODEL] = vega
+ self._order.append(VEGA_MODEL)
+ self.max_loaded = max(self.max_loaded, len(self._agents))
+ self._vega_spec = vega
+ return
+ if isinstance(vega, str) and vega != "auto":
+ self._add(VEGA_MODEL, None)
+ self._vega_spec = vega
+ return
+ import importlib.util
+ available = importlib.util.find_spec("vegaml") is not None
+ if vega is True and not available:
+ raise ImportError("Router(vega=True) needs the vegaml package: pip install -U laya "
+ "(vegaml ships with laya 0.4.23 and newer)")
+ if available:
+ self._add(VEGA_MODEL, None)
+ self._vega_spec = True
+
+ @property
+ def vega_enabled(self) -> bool:
+ """Whether this Router has a vega lane to route media, long states and fitted heads to."""
+ return VEGA_MODEL in self.models
+
+ def _vega_heads(self) -> set:
+ """The question ids a resident vega agent has fitted heads for, or nothing."""
+ agent = self._agents.get(VEGA_MODEL)
+ engine = getattr(agent, "vega", None)
+ heads = getattr(engine, "heads", None)
+ return set(heads) if isinstance(heads, dict) else set()
+
# ------------------------------------------------------------------ registry
def resolve(self, name: Any) -> str:
"""Canonical key of a checkpoint this Router knows: built-in, aliased, or registered.
@@ -1003,6 +1072,9 @@ class Router(HookRegistry):
def _build(self, key: str):
"""Construct the Agent for `key`: config is read under `_lock`, the build runs outside it."""
from .agent import Agent
+ if key == VEGA_MODEL and self._vega_spec is not None and isinstance(self._vega_spec, (bool, str)):
+ from .vega import VegaAgent
+ return VegaAgent(vegaml=self._vega_spec, device=self.device, token=self.token)
with self._lock:
if self.models.get(key) is None:
raise ValueError("checkpoint %r has no source to load from: it was attached as a built Agent "
@@ -1199,6 +1271,13 @@ class Router(HookRegistry):
hint = hint(state)
return _english_from_code(hint)
+ def _vega_repo(self) -> str:
+ spec = self._vega_spec
+ if isinstance(spec, str):
+ return spec
+ model_id = getattr(spec, "model_id", None)
+ return model_id if isinstance(model_id, str) else VEGA_REPO
+
# ------------------------------------------------------------------ routing
def route(
self,
@@ -1213,9 +1292,14 @@ class Router(HookRegistry):
hooks_timeout: Optional[float] = None,
*,
explain: bool = False,
+ media: Optional[str] = None,
+ allow_vega: bool = True,
) -> RouteDecision:
"""Decide which checkpoint to use, then let `on_route` hooks observe or replace it.
+ `media` is "image", "audio" or "video" when the caller is passing one of those instead of
+ a text state; see `predict`. `allow_vega=False` keeps the decision on the encoders.
+
`ctx.decision` is the `RouteDecision`; a hook may replace it (for example to pin a
checkpoint) and the replacement is what gets returned and used. `hooks` are per-call
hooks, appended after any installed on the Router.
@@ -1226,7 +1310,7 @@ class Router(HookRegistry):
if questions is not None and not isinstance(questions, dict):
raise TypeError("questions must be a dict of question id -> definition, got %s" % type(questions).__name__)
decision = self._route(state, questions, model=model, task=task, lang=lang, lang_guess=lang_guess,
- **({"explain": True} if explain else {}))
+ media=media, allow_vega=allow_vega, **({"explain": True} if explain else {}))
raise_errors = self.hooks_raise if hooks_raise is None else bool(hooks_raise)
active = compose_hooks(self.hooks, hooks)
timeout = self.hooks_timeout if hooks_timeout is None else validate_timeout(hooks_timeout)
@@ -1246,11 +1330,18 @@ class Router(HookRegistry):
lang_guess: Optional[Any] = None,
*,
explain: bool = False,
+ media: Optional[str] = None,
+ allow_vega: bool = True,
) -> RouteDecision:
"""Decide which checkpoint to use, without loading or running anything.
- Precedence: explicit `model` > explicit `task` > detected workflow (opt-in) >
- explicit `lang` > `lang_guess` > detected script/language > default.
+ Precedence: explicit `model` > explicit `task` > the vega lane (an image, a recording or a
+ clip; a question with a fitted head; a state past `vega_over_tokens`) > detected workflow
+ (opt-in) > explicit `lang` > `lang_guess` > detected script/language > default.
+
+ `media` names the kind of non-text input the caller passed ("image", "audio" or "video");
+ `allow_vega=False` keeps a decision on the encoders, which is what `predict_batch` does,
+ because a batch shares one forward pass on one checkpoint.
`lang_guess` is an opt-in hint -- a language code or a callable taking the state --
checked after an explicit `lang` and before the built-in detection. It only answers
@@ -1271,6 +1362,28 @@ class Router(HookRegistry):
return RouteDecision(model=key, repo=_repo_str(self.models.get(key)), reason="explicit task=%r" % task,
detection=None, workflow=None)
+ vega_on = allow_vega and self.vega_enabled
+ if media is not None:
+ if not vega_on:
+ raise ValueError("an %s needs the vega lane, which this Router does not have: install "
+ "vegaml (pip install -U laya) and build the Router with vega='auto' or "
+ "vega=True, or pass media to laya.Agent(vegaml=True) directly" % media)
+ return RouteDecision(model=VEGA_MODEL, repo=self._vega_repo(),
+ reason="%s input: the encoders have no vision or speech path" % media,
+ detection=None, workflow=None)
+ if vega_on:
+ fitted = sorted(q for q in (questions or {}) if q in self._vega_heads())
+ if fitted:
+ return RouteDecision(model=VEGA_MODEL, repo=self._vega_repo(),
+ reason="vega has fitted heads for %s" % ", ".join(fitted),
+ detection=None, workflow=None)
+ n_tokens = _estimate_tokens(state)
+ if n_tokens > self.vega_over_tokens:
+ return RouteDecision(model=VEGA_MODEL, repo=self._vega_repo(),
+ reason="state is about %d tokens, past the %d the encoders can read; "
+ "vega reads it whole" % (n_tokens, self.vega_over_tokens),
+ detection=None, workflow=None)
+
workflow = match_typed_decisions_workflow(questions or {})
if workflow and self.auto_task_detection:
return RouteDecision(model="typed-decisions", repo=_repo_str(self.models["typed-decisions"]),
@@ -1353,16 +1466,29 @@ class Router(HookRegistry):
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
min_confidence: Optional[float] = None,
+ image: Any = None,
+ audio: Any = None,
+ video: Any = None,
) -> Dict[str, Any]:
"""Route, then answer every question in one forward pass on the chosen checkpoint.
+ `image`, `audio` or `video` in place of `state` route to the vega lane, which is the only
+ checkpoint that can read them; so does a state past `vega_over_tokens` and a question the
+ vega agent has a fitted head for. Everything else routes exactly as before.
+
The result is the usual `system_one` payload plus a `routing` key recording the decision.
Router-level `on_predict_start` / `on_predict_end` hooks wrap the whole route+infer call
and see `ctx.decision`; see `laya.hooks`. `max_len` / `head_max_len` override the agent
token budget for this call (a start hook may set `ctx.max_len` / `ctx.head_max_len`).
"""
- if state is None:
- raise TypeError("state must not be None; pass a string, dict, or list")
+ media_given = [k for k, v in (("image", image), ("audio", audio), ("video", video)) if v is not None]
+ if len(media_given) > 1:
+ raise ValueError("pass one of image=, audio=, video=; got %s" % ", ".join(media_given))
+ media = media_given[0] if media_given else None
+ if state is None and media is None:
+ raise TypeError("state must not be None; pass a string, dict, or list (or image=, audio=, video=)")
+ if state is not None and media is not None:
+ raise ValueError("pass either a state or %s=, not both" % media)
if not isinstance(questions, dict):
raise TypeError("questions must be a dict of question id -> definition, got %s" % type(questions).__name__)
mc = check_min_confidence(min_confidence) if min_confidence is not None else None
@@ -1389,7 +1515,7 @@ class Router(HookRegistry):
# Per-call hooks apply to the whole call, including on_route inside route().
decision = self.route(state, questions, model=model, task=task, lang=lang,
lang_guess=lang_guess, hooks=hooks, hooks_raise=hooks_raise,
- hooks_timeout=hooks_timeout)
+ hooks_timeout=hooks_timeout, media=media)
ctx.decision = dict(decision)
ctx.model = decision["model"]
agent = self.load(decision["model"])
@@ -1421,14 +1547,20 @@ class Router(HookRegistry):
overrides["max_len"] = ctx.max_len
if ctx.head_max_len is not None:
overrides["head_max_len"] = ctx.head_max_len
- try:
- result = agent.system_one(ctx.states[0], ctx.questions,
- lang=effective_lang, **overrides)
- except TypeError as e:
- if "unexpected keyword argument 'lang'" in str(e):
- result = agent.system_one(ctx.states[0], ctx.questions, **overrides)
- else:
- raise
+ if decision["model"] == VEGA_MODEL:
+ # vega's predict takes laya's shape plus the media; `max_len` and
+ # `head_max_len` are recorded there and not applied
+ result = agent.predict(ctx.states[0], ctx.questions, lang=effective_lang,
+ image=image, audio=audio, video=video, **overrides)
+ else:
+ try:
+ result = agent.system_one(ctx.states[0], ctx.questions,
+ lang=effective_lang, **overrides)
+ except TypeError as e:
+ if "unexpected keyword argument 'lang'" in str(e):
+ result = agent.system_one(ctx.states[0], ctx.questions, **overrides)
+ else:
+ raise
finally:
_SKIP_DEFAULTS.reset(skip)
if scan is not None:
@@ -1633,6 +1765,7 @@ class Router(HookRegistry):
hooks=hooks,
hooks_raise=hooks_raise,
hooks_timeout=hooks_timeout,
+ allow_vega=False,
**({"explain": True} if explain else {}),
)
)
@@ -1645,6 +1778,7 @@ class Router(HookRegistry):
task=request.get("task"),
lang=request.get("lang"),
lang_guess=request.get("lang_guess"),
+ allow_vega=False,
**({"explain": True} if explain else {}),
)
)
@@ -1656,6 +1790,7 @@ class Router(HookRegistry):
task=request.get("task"),
lang=request.get("lang"),
lang_guess=request.get("lang_guess"),
+ allow_vega=False,
**({"explain": True} if explain else {}),
)
ctx = PredictContext(
diff --git a/laya/serve.py b/laya/serve.py
index b9a3d52d..07fef247 100644
--- a/laya/serve.py
+++ b/laya/serve.py
@@ -176,6 +176,11 @@ DEFAULT_MAX_TOKEN_BUDGET = 8192
# language it reads, how many tokens it gets and where it abstains are all plain data.
BODY_CONTROLS = ("model", "max_len", "head_max_len", "task", "lang", "lang_guess", "min_confidence")
BODY_REFUSALS = ("hooks", "on_predict_start", "on_predict_end", "hooks_raise", "hooks_timeout")
+# `image`, `audio` and `video` are `Router.predict` arguments since 0.4.25 and route to the vega
+# lane; the HTTP body has no media encoding yet, so they are refused here rather than accepted as
+# strings and passed on, and the response says where media goes (laya.Agent(vegaml=True) locally).
+# A tuple of their own, because ``BODY_REFUSALS`` is pinned as the hook set LangChain refuses too.
+MEDIA_REFUSALS = ("image", "audio", "video")
# ``Router.predict_batch`` reads two different kinds of control: call-level keyword arguments that
# apply to the whole batch (chunking, padding, the abstention threshold) and per-request keys that
@@ -202,9 +207,12 @@ BATCH_BODY_OUT_PARAMETERS = ("diagnostics",)
# Route-only requests cannot apply prediction budgets, abstention or batch controls.
# Hook policy belongs to the operator, including the on_route callback itself.
ROUTE_BODY_CONTROLS = ("model", "task", "lang", "lang_guess", "explain")
-ROUTE_BODY_REFUSALS = BODY_REFUSALS + (
+ROUTE_BODY_REFUSALS = BODY_REFUSALS + MEDIA_REFUSALS + (
"on_route", "max_len", "head_max_len", "min_confidence", "batch_size",
"sort_by_length", "diagnostics", "states",
+ # `Router.route` takes these two since 0.4.25: `media` names a non-text input and
+ # `allow_vega` is how `predict_batch` keeps a batch on the encoders; neither is a body control
+ "media", "allow_vega",
)
@@ -672,6 +680,13 @@ def _refuse_body_refusals(body: Dict[str, Any]) -> None:
from fastapi import HTTPException
given = sorted(key for key in BODY_REFUSALS if key in body and body[key] is not None)
+ media = sorted(key for key in MEDIA_REFUSALS if key in body and body[key] is not None)
+ if media:
+ raise HTTPException(
+ status_code=422,
+ detail="%s cannot be sent to this endpoint, which takes text states only; images, "
+ "recordings and clips go through laya.Agent(vegaml=True) or a Router with the "
+ "vega lane, in the process that holds the model" % ", ".join(media))
if given:
raise HTTPException(
status_code=422,
diff --git a/nix/package.nix b/nix/package.nix
index 0a7dca73..a94767f8 100644
--- a/nix/package.nix
+++ b/nix/package.nix
@@ -12,7 +12,7 @@ let
# side, where tests/test_packaging.py compares all three declarations (this one,
# pyproject.toml and laya.__version__) and fails CI when they diverge. Nothing
# did, which is how this sat at 0.3.4 while the package reached 0.3.20.
- version = "0.4.24";
+ version = "0.4.25";
src = ../.;
format = "setuptools";
propagatedBuildInputs = with python3Packages; [
diff --git a/pyproject.toml b/pyproject.toml
index ccfa994d..2b6ce84a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "laya"
-version = "0.4.24"
+version = "0.4.25"
description = "Fast, non-autoregressive System 1 decision engine with calibrated probabilities"
readme = "README.md"
requires-python = ">=3.10"
diff --git a/scripts/test_suites.py b/scripts/test_suites.py
index 64b6308f..57117d3c 100644
--- a/scripts/test_suites.py
+++ b/scripts/test_suites.py
@@ -128,6 +128,7 @@ PYTEST_SUITES = [
"tests/test_compile.py",
"tests/test_finetune_entrypoints.py",
"tests/test_vega.py",
+ "tests/test_router_vega.py",
]
diff --git a/tests/test_router_vega.py b/tests/test_router_vega.py
new file mode 100644
index 00000000..15b2da56
--- /dev/null
+++ b/tests/test_router_vega.py
@@ -0,0 +1,150 @@
+"""The vega lane on `laya.Router`: media, long states and fitted heads go to vega; everything the
+encoders can read keeps going to the encoders exactly as before 0.4.25.
+
+Weight-free: a stub stands in for `laya.Agent(vegaml=True)`, so the decisions are tested, not
+the engine. The lane is off when vegaml is absent and `vega` is left at "auto", which is what
+the first test pins, since that is every pre-0.4.25 deployment's situation.
+"""
+import importlib.util
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+from laya.router import VEGA_MODEL, Router, _estimate_tokens # noqa: E402
+
+HAS_VEGAML = importlib.util.find_spec("vegaml") is not None
+Q = {"dept": {"type": "choice", "instructions": "Which team?",
+ "criteria": {"billing": "money", "technical": "bugs"}},
+ "urgent": {"type": "noul", "instructions": "Urgent?"}}
+ENGLISH = "Please refund the duplicate charge on invoice 4411 today."
+LONG = ("The contract between the parties states the following. " * 3000) # ~40k tokens
+
+
+class _StubEngine:
+ def __init__(self, heads):
+ self.heads = heads
+
+
+class _StubVega:
+ """What the Router needs from a vega agent: `.vega.heads`, `predict`, `predict_long`."""
+ model_id = "stub/vega"
+
+ def __init__(self, heads=()):
+ self.vega = _StubEngine({h: object() for h in heads})
+ self.calls = []
+
+ def predict(self, state=None, questions=None, lang=None, image=None, audio=None, video=None, **kw):
+ self.calls.append({"state": state, "image": image, "audio": audio, "video": video, "lang": lang, **kw})
+ answers = {q: {"choice": "billing", "probabilities": {"billing": 0.9, "technical": 0.1},
+ "answer_confidence": 0.9, "confidence": 0.9, "route": "auto"} if d["type"] == "choice"
+ else {"noul": 0.2, "answer_confidence": 0.8, "confidence": 0.8, "route": "auto"}
+ for q, d in questions.items()}
+ return {"answers": answers, "usage": {"input_tokens": 10, "output_tokens": 0}, "latency_ms": 1.0,
+ "modality": "image" if image is not None else "audio" if audio is not None else
+ "video" if video is not None else "text"}
+
+ system_one = predict
+
+ def predict_long(self, state, questions, **kw):
+ out = self.predict(state, questions, **kw)
+ out["usage"]["windows"] = 1
+ return out
+
+
+# ---------------------------------------------------------------------- off by default without vegaml
[email protected](HAS_VEGAML, reason="vegaml is installed here, so auto turns the lane on")
+def test_without_vegaml_the_lane_is_off_and_nothing_changes():
+ r = Router()
+ assert not r.vega_enabled and VEGA_MODEL not in r.models
+ assert r.route(ENGLISH).model == "english"
+ assert r.route(LONG).model != VEGA_MODEL, "a long state stays on the encoders when there is no lane"
+ with pytest.raises(ValueError, match="needs the vega lane"):
+ r.route(None, Q, media="image")
+ with pytest.raises(ImportError, match="pip install -U laya"):
+ Router(vega=True)
+
+
+def test_vega_false_keeps_the_old_behaviour_even_with_an_agent_available():
+ r = Router(vega=False)
+ assert not r.vega_enabled
+ assert r.route(LONG).model != VEGA_MODEL
+ with pytest.raises(ValueError, match="needs the vega lane"):
+ r.predict(None, Q, image=object())
+
+
+# ------------------------------------------------------------------------------- the lane itself
+def test_an_attached_vega_agent_turns_the_lane_on_and_routes_what_the_encoders_cannot_read():
+ stub = _StubVega(heads=("urgent",))
+ r = Router(vega=stub)
+ assert r.vega_enabled and r.load(VEGA_MODEL) is stub
+ # text the encoders can read is untouched
+ d = r.route(ENGLISH, {"dept": Q["dept"]})
+ assert d.model == "english" and d.reason == "English Latin text"
+ # media
+ d = r.route(None, Q, media="image")
+ assert d.model == VEGA_MODEL and "image input" in d.reason and d["repo"] == "stub/vega"
+ # a state past the encoders' budget
+ d = r.route(LONG, {"dept": Q["dept"]})
+ assert d.model == VEGA_MODEL and "past the 8192" in d.reason
+ # a question with a fitted head, however short the state
+ d = r.route(ENGLISH, Q)
+ assert d.model == VEGA_MODEL and "fitted heads for urgent" in d.reason
+ # explicit model= still wins, both ways
+ assert r.route(LONG, Q, model="english").model == "english"
+ assert r.route(ENGLISH, {"dept": Q["dept"]}, model="vega").model == VEGA_MODEL
+
+
+def test_predict_hands_media_and_long_states_to_vega_and_stamps_the_routing():
+ stub = _StubVega()
+ r = Router(vega=stub)
+ out = r.predict(None, Q, image="invoice.png")
+ assert out["routing"]["model"] == VEGA_MODEL and out["modality"] == "image"
+ assert stub.calls[-1]["image"] == "invoice.png" and stub.calls[-1]["state"] is None
+ out = r.predict(LONG, Q)
+ assert out["routing"]["model"] == VEGA_MODEL and stub.calls[-1]["state"] == LONG
+ out = r.predict_long(LONG, Q)
+ assert out["routing"]["model"] == VEGA_MODEL and out["usage"]["windows"] == 1
+ with pytest.raises(ValueError, match="pass one of"):
+ r.predict(None, Q, image="a", audio="b")
+ with pytest.raises(ValueError, match="not both"):
+ r.predict(ENGLISH, Q, video="c")
+
+
+def test_vega_over_tokens_is_the_line_and_the_estimate_is_characters_over_four():
+ stub = _StubVega()
+ assert _estimate_tokens("a" * 400) == 100 and _estimate_tokens({"k": "b" * 40}) == 10
+ r = Router(vega=stub, vega_over_tokens=100)
+ assert r.route("x" * 404, {"dept": Q["dept"]}).model == VEGA_MODEL
+ assert r.route("x" * 396, {"dept": Q["dept"]}).model != VEGA_MODEL
+
+
+def test_predict_batch_never_routes_to_vega():
+ stub = _StubVega(heads=("urgent",))
+ r = Router(vega=stub)
+ decisions = [r._route(LONG, Q, allow_vega=False), r._route(ENGLISH, Q, allow_vega=False)]
+ assert all(d.model != VEGA_MODEL for d in decisions)
+ with pytest.raises(ValueError, match="needs the vega lane"):
+ r._route(None, Q, media="image", allow_vega=False)
+
+
+def test_a_reader_name_registers_a_lazy_lane():
+ r = Router(vega="4b")
+ assert r.vega_enabled and r.route(None, Q, media="audio")["repo"] == "4b"
+ assert r.models[VEGA_MODEL] is None, "no repo source: the lane is built by VegaAgent, not Agent"
+ assert VEGA_MODEL in r.registered
+
+
[email protected](not HAS_VEGAML, reason="needs vegaml to test the auto lane")
+def test_auto_turns_the_lane_on_when_vegaml_is_present_without_importing_it():
+ before = "vegaml" in sys.modules
+ r = Router()
+ assert r.vega_enabled
+ assert ("vegaml" in sys.modules) == before, "building a Router must not import vegaml"
+ assert r.route(ENGLISH, {"dept": Q["dept"]}).model == "english"
+ assert r.route(LONG, {"dept": Q["dept"]}).model == VEGA_MODEL
+
+
+if __name__ == "__main__":
+ sys.exit(pytest.main([__file__, "-q"]))
diff --git a/tests/test_serve.py b/tests/test_serve.py
index 7e568038..70dda790 100644
--- a/tests/test_serve.py
+++ b/tests/test_serve.py
@@ -27,6 +27,7 @@ from laya.serve import ( # noqa: E402
BATCH_BODY_ITEM_CONTROLS,
BODY_CONTROLS,
BODY_REFUSALS,
+ MEDIA_REFUSALS,
DEFAULT_BODY_READ_TIMEOUT,
DEFAULT_LOG_LEVEL,
DEFAULT_MAX_TOKEN_BUDGET,
@@ -2920,8 +2921,8 @@ def test_every_predict_control_is_forwarded_or_refused():
from laya.router import Router
taken = set(inspect.signature(Router.predict).parameters) - {"self", "state", "questions"}
- declared = set(BODY_CONTROLS) | set(BODY_REFUSALS)
- assert not set(BODY_CONTROLS) & set(BODY_REFUSALS)
+ declared = set(BODY_CONTROLS) | set(BODY_REFUSALS) | set(MEDIA_REFUSALS)
+ assert not set(BODY_CONTROLS) & set(BODY_REFUSALS) and not set(BODY_CONTROLS) & set(MEDIA_REFUSALS)
assert taken == declared, "predict() takes %s; serve declares %s" % (
sorted(taken), sorted(declared))
@@ -3052,7 +3053,8 @@ def test_a_hook_control_is_refused_not_dropped(monkeypatch, key):
says that itself instead of answering as though the control had been honoured.
"""
value = {"hooks": [], "on_predict_start": "cache", "on_predict_end": "audit",
- "hooks_raise": False, "hooks_timeout": 5.0}[key]
+ "hooks_raise": False, "hooks_timeout": 5.0,
+ "image": "invoice.png", "audio": "call.wav", "video": "clip.mp4"}[key]
client, fake = _budget_client(monkeypatch)
r = client.post("/v1/systemone", json={**REQ, key: value})
assert r.status_code == 422, r.text
@@ -3331,6 +3333,23 @@ def test_bad_control_on_batch_is_refused_before_inference(monkeypatch, key, valu
assert not fake.batch_calls, "a refused body must not reach predict_batch"
[email protected]("key", MEDIA_REFUSALS)
+def test_media_is_refused_on_both_endpoints_with_a_pointer(monkeypatch, key):
+ """`image`, `audio` and `video` are `Router.predict` arguments since 0.4.25, so the single
+ endpoint has to say it cannot take them rather than forward a file name as a state; the batch
+ endpoint refuses them for the same reason. The answer names where media goes."""
+ client, fake = _budget_client(monkeypatch)
+ r = client.post("/v1/systemone", json={**REQ, key: "%s.bin" % key})
+ assert r.status_code == 422, r.text
+ assert key in r.json()["detail"] and "vegaml=True" in r.json()["detail"]
+ assert not fake.calls
+ client, fake = _batch_client(monkeypatch)
+ r = client.post("/v1/systemone/batch", json={**BATCH_REQ, key: "%s.bin" % key})
+ assert r.status_code == 422, r.text
+ assert key in r.json()["detail"] and "cannot be sent to this endpoint" in r.json()["detail"]
+ assert not fake.batch_calls
+
+
@pytest.mark.parametrize("key", BODY_REFUSALS)
def test_batch_hook_control_is_refused_not_dropped(monkeypatch, key):
"""The batch endpoint must honour the same refusal policy the single endpoint already states.
@@ -3341,7 +3360,8 @@ def test_batch_hook_control_is_refused_not_dropped(monkeypatch, key):
reason they are on ``/v1/systemone``: they run inside the server process.
"""
value = {"hooks": [], "on_predict_start": "cache", "on_predict_end": "audit",
- "hooks_raise": False, "hooks_timeout": 5.0}[key]
+ "hooks_raise": False, "hooks_timeout": 5.0,
+ "image": "invoice.png", "audio": "call.wav", "video": "clip.mp4"}[key]
client, fake = _batch_client(monkeypatch)
r = client.post("/v1/systemone/batch", json={**BATCH_REQ, key: value})
assert r.status_code == 422, r.text