Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

Agent(vegaml=True): vegaml's engine behind laya's surface

Nandakishor9ae2930f81dd
diff --git a/README.md b/README.md
index 085f3c10..ac9c4df4 100644
--- a/README.md
+++ b/README.md
@@ -1018,6 +1018,58 @@ for measurements and limitations.
---
+## vega inside laya: `Agent(vegaml=True)`
+
+One argument swaps the engine under laya's surface for [vegaml](https://github.com/NandhaKishorM/vegaml)'s:
+a frozen 0.8B universal reader with a small physics engine on top. Every method keeps its
+laya name; the answers gain what the engine knows that an encoder does not.
+
+```python
+import laya
+
+agent = laya.Agent(vegaml=True) # pip install 'laya[vegaml]' (Python 3.10+)
+out = agent.predict(state, questions)
+
+ans = out["answers"]["department"]
+ans["choice"] # as always
+ans["prediction_set"] # the conformal set at alpha=0.1
+ans["abstain"] # the engine's own refusal
+ans["unbound"] # far from every well: likely off-distribution
+ans["route"], ans["route_reason"] # "auto" or "review", decided for you
+```
+
+| | laya checkpoints | `Agent(vegaml=True)` |
+|---|---|---|
+| context | 512 or 1,024 tokens, windowed past that | 73k tokens, read whole |
+| learning from your labels | gradient fine-tune | `agent.fit(rows)`: a head per question over frozen features, seconds |
+| calibration | shipped temperature map | refit on your rows: temperature, conformal threshold, escalation band |
+| refusal | `min_confidence` you choose | `abstain`, `unbound`, the conformal set, the fitted band |
+| images, speech | no | `agent.predict(image=img, ...)`, `agent.predict(audio=wav, ...)` |
+| speed on text (T4) | ~16 ms a decision batched | ~72 ms a decision |
+
+```python
+agent.fit(rows) # rows: {"state", "questions", "gold" or "expected"}
+agent.predict(state, questions) # heads serve where they beat the engine; route from the band
+agent.save_calibration("fit/"); agent.load_calibration("fit/")
+agent.predict_long(document, questions) # no windowing up to 73k tokens
+agent.predict(image=img, questions=questions) # the reader's vision tower
+agent.predict(audio=wav, questions=questions, sampling_rate=16000) # transcribed, then decided
+
+from laya.vega import Router
+router = Router(fast=laya.Agent(), vega=agent) # laya for short English text, vega otherwise
+router.predict(state, questions)["routing"] # {"model": "fast" | "vega", "reason": ...}
+```
+
+**Nothing here needs a threshold from you.** `route` comes from the fitted band when you have
+run `fit` or `calibrate`, and otherwise from the engine's own conformal set, abstain flag and
+`unbound` flag. `alpha` (miscoverage, 0.1) and `target_precision` (0.95) are targets with
+defaults, not confidences to guess at. Gate on `route`, not on `confidence`: a fitted head can
+report 1.000 on an answer it gets wrong, and `route` on the same answer is still right, because
+it comes from the set and the band rather than the top probability.
+
+`laya.vega` is imported only when the argument is given; without `vegaml` installed, core laya
+is unchanged and `Agent(vegaml=True)` says what to install.
+
## Automated Confidence Gating
Because Laya's probabilities are trained with strictly proper scoring rules (RLCD), confidence scores are statistically meaningful:
diff --git a/docs/vegaml.md b/docs/vegaml.md
new file mode 100644
index 00000000..0261912a
--- /dev/null
+++ b/docs/vegaml.md
@@ -0,0 +1,121 @@
+# vegaml inside laya: `Agent(vegaml=True)`
+
+One argument swaps the engine under laya's surface for [vegaml](https://github.com/NandhaKishorM/vegaml)'s:
+a frozen 0.8B universal reader with a small physics engine on top. Every method keeps its laya
+name, and the answers gain what that engine knows that an encoder does not. The class behind it
+is `laya.VegaAgent`, in `laya/vega.py`.
+
+```bash
+pip install 'laya[vegaml]' # Python 3.10 or newer; core laya stays on 3.9
+```
+
+```python
+import laya
+
+agent = laya.Agent(vegaml=True) # the 0.8B universal reader
+agent = laya.load(vegaml="4b") # a larger reader; or a Hub id, or a local dir
+
+out = agent.predict(state, questions)
+ans = out["answers"]["department"]
+ans["choice"] # as always
+ans["probabilities"], ans["answer_confidence"] # as always: the calibrated one is max p
+ans["prediction_set"] # the conformal set at alpha (0.1)
+ans["abstain"] # the engine's own refusal
+ans["unbound"] # far from every well: likely off-distribution
+ans["route"], ans["route_reason"] # "auto" or "review", decided for you
+```
+
+## What changes, feature by feature
+
+| | laya checkpoints | `Agent(vegaml=True)` |
+|---|---|---|
+| reader | ModernBERT encoder, ~150M | frozen decoder reader, 0.8B (or the 4B) |
+| context | 512 or 1,024 tokens, `predict_long` windows past that | 73k tokens, read whole |
+| learning from your labels | gradient fine-tune | `agent.fit(rows)`: a head per question over the frozen features, seconds |
+| calibration | a shipped temperature map | refit on your rows: temperature, conformal threshold, escalation band |
+| refusal | `min_confidence` you choose | `abstain`, `unbound`, the conformal set, the fitted band |
+| images and speech | no | `predict(image=...)`, `predict(audio=...)` |
+| speed on text (T4) | ~16 ms a decision batched | ~72 ms a decision; each served head adds a read |
+| memory | under 1 GB | 2 to 4 GB |
+
+## Fit and calibrate
+
+```python
+rows = [{"state": "...", "questions": questions, "gold": {"department": "billing"}}, ...]
+report = agent.fit(rows) # heads, then temperature + conformal + band, one call
+report["serving"] # the questions whose head actually beat the engine
+agent.predict(state, questions) # heads serve where they earned it; route from the band
+agent.save_calibration("fit/") # heads.npz + calibration.json
+agent.load_calibration("fit/") # on a fresh VegaAgent, identical answers and routes
+```
+
+Rows are the same `{"state", "questions", "gold" or "expected"}` shape laya's fine-tune reads,
+so a corpus prepared for one fits the other unchanged. A head is kept only if it beats chance
+under cross-validation and served only if it also beats the engine on the same rows:
+`report["heads"][qid]["used_by_auto"]` is the fact to read, not `fitted`.
+
+`agent.calibrate(rows)` (also reachable as `fit_temperatures`) does the calibration half alone.
+The argmax is invariant under a temperature, so it moves confidence, sets, `abstain` and
+`route`, never which option wins.
+
+## Automatic thresholding
+
+Nothing here needs a confidence number from you. `route` is decided per answer, in this order:
+
+1. A fitted band, if `fit` or `calibrate` has run: its line for that question was placed where
+ precision holds at `target_precision` (0.95) on your rows, with a Wilson bound to spare.
+2. Otherwise the engine's own machinery, with no labels at all: `review` when it abstains,
+ when the answer is `unbound`, or when the conformal set at `alpha` (0.1) holds more than
+ one option; `auto` only when all three agree.
+
+`alpha` and `target_precision` are targets with defaults, not thresholds to guess at.
+`agent.thresholds()` prints what is deciding right now. laya's `min_confidence` is still
+accepted and applied on top when given.
+
+Gate on `route`, not on `confidence`. A fitted head can report 1.000 on an answer it gets wrong;
+`route` on the same answer is still right, because it comes from the set and the band rather
+than the top probability.
+
+## Images, speech, long documents
+
+```python
+agent.predict(image=img, questions=questions) # PIL image, path, or array
+agent.predict(audio=wave, questions=questions, sampling_rate=16000) # transcribed, then decided
+agent.predict_long(document, questions) # one pass up to 73k tokens
+```
+
+The modality is chosen by which argument you passed; the questions and the answer shape are the
+same. An audio answer carries `transcript` and `asr`, because a decision made on words you never
+saw cannot be audited. `predict_image_text` answers the same questions with the image withheld,
+which measures what the picture itself contributed.
+
+## Model switching
+
+```python
+from laya.vega import Router
+
+router = Router(fast=laya.Agent(), vega=agent)
+out = router.predict(state, questions)
+out["routing"] # {"model": "fast" | "vega", "reason": "..."}
+```
+
+The rule, in order: an image or audio goes to vega; a state longer than the fast model's context
+goes to vega; non-English or non-Latin text goes to vega; a question with a fitted head on vega
+goes to vega; everything else goes to the fast encoder. `model=` forces one. The result carries
+`routing` in the shape `laya.Router` already returns.
+
+## Your own reader
+
+```python
+laya.VegaAgent.attach("Qwen/Qwen3.5-4B", texts, "my-reader/") # no labels needed
+agent = laya.Agent(vegaml="my-reader/")
+```
+
+## What stays the same, and what is not here
+
+`predict`, `system_one`, `predict_batch`, `predict_long`, `fit_temperatures`,
+`save_calibration`, `load_calibration` and the hooks all keep their names. `predict_batch` is a
+loop: vegaml packs a state's questions into one pass, not many states. `fit_binning` raises,
+because histogram binning is laya's own calibration and on vegaml the calibrated probabilities
+come from `calibrate()`. Without the `vegaml` package installed, `laya.vega` is never imported
+and `Agent(vegaml=True)` says what to install.
diff --git a/laya/__init__.py b/laya/__init__.py
index 1f1388ec..c9abb582 100644
--- a/laya/__init__.py
+++ b/laya/__init__.py
@@ -44,6 +44,7 @@ _LAZY_ATTRS = {
# stdlib-only module, but keep it lazy so `import laya` does not pull in hashlib
"PINNED_REVISIONS": (".revisions", "PINNED_REVISIONS"),
"Agent": (".agent", "Agent"),
+ "VegaAgent": (".vega", "VegaAgent"), # vegaml's engine behind laya's surface
"RLAgent": (".agent", "RLAgent"),
"load": (".agent", "load"),
"fit_temperatures": (".calibrate", "fit_temperatures"),
@@ -100,6 +101,7 @@ def __dir__():
__all__ = [
"Agent",
"RLAgent",
+ "VegaAgent",
"load",
"fit_temperatures",
"fit_one_temperature",
diff --git a/laya/agent.py b/laya/agent.py
index dec08ac0..6cab09cd 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -533,6 +533,17 @@ class Agent(HookRegistry):
_reduce_overhead = False
_gate = _InferenceGate()
+ def __new__(cls, *args, **kwargs):
+ # `laya.Agent(vegaml=True)` builds a VegaAgent: vegaml's engine behind laya's surface.
+ # The dispatch is here rather than in a factory so that the one argument is the whole
+ # change for a caller, and `isinstance(agent, laya.Agent)` stays true either way.
+ # VegaAgent overrides __new__ itself, so there is no recursion through this branch.
+ vegaml = kwargs.get("vegaml")
+ if vegaml is not None and vegaml is not False and cls is Agent:
+ from .vega import VegaAgent
+ return VegaAgent.__new__(VegaAgent)
+ return object.__new__(cls)
+
def __init__(
self,
model_id_or_path: str = "convaiinnovations/laya",
@@ -555,9 +566,14 @@ class Agent(HookRegistry):
compile_warmup: bool = True,
compile_cache: bool = False,
compile_mode: str = "default",
+ vegaml: Union[bool, str, None] = None,
):
"""Load a Laya checkpoint.
+ `vegaml=True` (or a size, Hub id or directory) loads vegaml's engine instead and returns
+ a :class:`laya.vega.VegaAgent`; see that module. On this path the argument is accepted
+ so that `vegaml=False`/`None` are legal and mean the ordinary checkpoint.
+
`backend` selects "eager", "auto", "compile" or "tilelang"; see `laya.backends`.
It takes precedence over `fast` and `compile`. Omit it to keep those legacy flags.
For ONNX use `load(backend="onnx")` instead.
@@ -2174,9 +2190,13 @@ def load(model_id_or_path: str = "convaiinnovations/laya", device: Optional[str]
hooks_timeout: Optional[float] = None,
calibration: Optional[str] = None, backend: Optional[str] = None,
onnx_path: Optional[str] = None, compile_warmup: bool = True,
- compile_cache: bool = False, compile_mode: str = "default") -> Union[Agent, "ONNXAgent"]:
+ compile_cache: bool = False, compile_mode: str = "default",
+ vegaml: Union[bool, str, None] = None) -> Union[Agent, "ONNXAgent", "VegaAgent"]:
"""Load a Laya agent.
+ laya.load(vegaml=True) # vegaml's engine, laya's API
+ laya.load(vegaml="4b") # a larger reader
+
`subfolder` picks one checkpoint out of a repo that bundles several:
laya.load("convaiinnovations/laya") # English (repo root)
@@ -2200,6 +2220,13 @@ def load(model_id_or_path: str = "convaiinnovations/laya", device: Optional[str]
`hooks` / `on_predict_start` / `on_predict_end` observe or shape every prediction; see
`laya.hooks`. `calibration` is the same optional JSON path accepted by `Agent`.
"""
+ if vegaml is not None and vegaml is not False:
+ # vegaml's engine behind laya's surface; everything laya-specific below (registry names,
+ # ONNX, TileLang, torch.compile) is about ModernBERT checkpoints and does not apply.
+ from .vega import VegaAgent
+ return VegaAgent(model_id_or_path if model_id_or_path != "convaiinnovations/laya" else None,
+ device=device, token=token, vegaml=vegaml, revision=revision, hooks=hooks,
+ on_predict_start=on_predict_start, on_predict_end=on_predict_end)
# A registry name or alias resolves to the same (repo, subfolder) the Router would
# pick, instead of being handed to the Hub as a repo id. A caller who spelled out a
# subfolder is loading that subfolder of whatever repo they named, so the name is only
diff --git a/laya/vega.py b/laya/vega.py
new file mode 100644
index 00000000..2f97eaa5
--- /dev/null
+++ b/laya/vega.py
@@ -0,0 +1,678 @@
+"""vega inside laya: every vega feature, under laya's own names.
+
+ import laya
+
+ agent = laya.Agent(vegaml=True) # the 0.8B universal reader + the engine
+ agent = laya.load(vegaml=True) # the same
+ agent = laya.Agent(vegaml="4b") # the 4B reader; or a Hub id, or a local dir
+
+ out = agent.predict(state, questions) # laya's shape, plus vega's fields
+ out["answers"]["dept"]["choice"] # as always
+ out["answers"]["dept"]["prediction_set"] # the conformal set at alpha
+ out["answers"]["dept"]["abstain"] # the engine's own refusal
+ out["answers"]["dept"]["unbound"] # far from every well: likely off-distribution
+ out["answers"]["dept"]["route"] # "auto" or "review", decided for you
+
+ agent.fit(rows, questions) # heads + temperature + conformal + band, one call
+ agent.predict(image=img, questions=questions) # pictures, through the reader's vision tower
+ agent.predict(audio=wav, questions=questions) # speech, transcribed then decided
+ agent.predict_long(very_long_state, questions) # 73k tokens, no windowing
+ agent.save_calibration("fit/") # heads and calibration, reloadable
+ laya.vega.Router(fast=laya.Agent(), ...) # model switching: laya for short English text,
+ # vega for everything else
+
+**What the argument buys.** laya's own checkpoints are ModernBERT encoders: fast, small, text
+only, with a temperature-scaled confidence and no notion of "I have not seen anything like
+this". ``vegaml=True`` swaps the engine underneath for vegaml's: a frozen decoder reader with a
+small physics engine on top, which adds a conformal prediction set, an abstain flag, an
+out-of-distribution flag (``unbound``), heads fitted on your labels over the frozen features,
+a calibration that is refit on your rows rather than shipped, 73k tokens of context, and the
+same typed interface over images and speech. Everything is reachable through the methods laya
+already has, so a caller switches by changing one argument.
+
+**What it costs.** A 0.8B reader against a ~150M encoder: roughly five times the latency on
+plain text (measured 72 ms a decision against 16 ms batched on a T4), two to four gigabytes
+of memory, and Python 3.10 or newer, which vegaml needs. ``pip install laya[vegaml]``. Core laya
+is untouched when the argument is not given, which is why this lives in its own module and is
+imported only on demand.
+
+**Automatic thresholding, which is the question this module exists to answer.** A caller never
+sets a confidence number. ``route`` is decided per answer, in this order:
+
+1. If a band has been fitted (``fit`` or ``calibrate`` ran on labelled rows), the band's own
+ line for that question decides: it was placed where precision holds at
+ ``target_precision`` (default 0.95) on your rows, with a 95% Wilson bound to spare.
+2. Otherwise the engine's own machinery decides, no labels required: ``review`` when it
+ abstains at the configured risk, when the answer is ``unbound``, or when the conformal set
+ at ``alpha`` (default 0.10) holds more than one option; ``auto`` only when all three agree.
+
+Both are targets, not thresholds: ``alpha`` is the miscoverage the sets are allowed, and
+``target_precision`` is the precision the band is held to. Neither is a confidence a human had
+to guess at, and both have defaults. The house rule behind this is measured, not stylistic: a
+fitted head once returned confidence 1.000 on every one of sixteen held-out records including
+both of its mistakes, while ``route`` on the same records was still right, because route comes
+from the conformal set and the band rather than from the top-1 probability. **Gate on
+``route``, never on ``confidence``.** laya's ``min_confidence`` is still accepted for
+compatibility and applied on top when given, but nothing here needs it.
+
+Two vegaml facts this module is built around, both of which pass every test until they
+matter: a head-served answer carries no ``abstain`` and no ``prediction_set`` of its own, so
+the fitted path gets its route from the band; and ``vegaml.calibrate.calibrate`` returns a
+Calibration that nothing applies unless the caller does, so :meth:`predict` applies it on
+every call itself.
+"""
+from __future__ import annotations
+
+import json
+import math
+import os
+import threading
+import time
+from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple, Union
+
+from .agent import Agent
+
+__all__ = ["VegaAgent", "Router", "is_available", "HEAD_ARRAYS"]
+
+#: what a fitted TTT head is, in full: standardisation, projection and the linear readout.
+#: vegaml gives the heads no save of their own, so this module writes these arrays itself.
+HEAD_ARRAYS = ("mu", "sd", "c", "P", "s", "W", "b")
+
+_INSTALL = ("vegaml=True needs the vegaml package, which needs Python 3.10 or newer: "
+ "pip install 'laya[vegaml]' (or: pip install vegaml)")
+
+
+def is_available() -> bool:
+ """Whether the vega engine can be imported in this interpreter."""
+ try:
+ import vegaml # noqa: F401
+ except ImportError:
+ return False
+ return True
+
+
+# ----------------------------------------------------------------------------- translation
+def _to_vega_questions(questions: Mapping[str, Mapping]) -> Dict[str, Dict]:
+ """laya's question map in vegaml's vocabulary.
+
+ The two schemas are the same typed-decision design and differ in one word: laya's yes/no
+ is ``noul``, vegaml's is ``boolean``. laya's noul-only ``labels`` (the two sides named)
+ becomes vegaml's boolean ``criteria``, which is the same information under the other key.
+ """
+ out = {}
+ for qid, q in questions.items():
+ t = q["type"]
+ vq: Dict[str, Any] = {"instructions": q["instructions"]}
+ if t == "noul":
+ vq["type"] = "boolean"
+ crit = q.get("criteria")
+ lab = q.get("labels")
+ if isinstance(crit, dict):
+ vq["criteria"] = crit
+ elif isinstance(lab, dict):
+ vq["criteria"] = {"false": lab.get("false", "no"), "true": lab.get("true", "yes")}
+ elif t == "score":
+ vq["type"] = "score"
+ vq["criteria"] = list(q["criteria"])
+ elif t == "choice":
+ vq["type"] = "choice"
+ crit = q["criteria"]
+ vq["criteria"] = (dict(crit) if isinstance(crit, dict)
+ else {str(c): None for c in crit})
+ else:
+ raise ValueError("question %r has unknown type %r (choice, score or noul)" % (qid, t))
+ out[qid] = vq
+ return out
+
+
+def _labels_for(q: Mapping) -> List[str]:
+ """The option ids a question's answer spans, in laya's own spelling."""
+ t = q["type"]
+ if t == "noul":
+ return ["false", "true"]
+ if t == "score":
+ return [str(i) for i in range(len(q["criteria"]))]
+ crit = q["criteria"]
+ return [str(k) for k in (crit if isinstance(crit, dict) else crit)]
+
+
+def _entropy_confidence(probs: Mapping[str, float]) -> float:
+ """``1 - H(p)/log(k)``: what laya reports as ``confidence``. Not an error rate.
+
+ Kept so a laya caller reading ``confidence`` sees the quantity laya has always put there;
+ the calibrated one is ``answer_confidence`` (max p), on vega exactly as on laya.
+ """
+ vals = [v for v in probs.values() if v > 0]
+ k = len(probs)
+ if k < 2:
+ return 1.0
+ h = -sum(v * math.log(v) for v in vals)
+ return round(max(0.0, min(1.0, 1.0 - h / math.log(k))), 4)
+
+
+def _gold_label(q: Mapping, gold: Any) -> str:
+ """A training row's answer as the label vegaml's fit wants.
+
+ Accepts laya's three shapes for an answer: a bare value, ``{"label": value}`` (the
+ typed-decisions dataset), or ``{"probabilities": {...}}`` where the argmax is taken.
+ """
+ if isinstance(gold, dict):
+ if "label" in gold:
+ gold = gold["label"]
+ elif "probabilities" in gold and gold["probabilities"]:
+ gold = max(gold["probabilities"], key=gold["probabilities"].get)
+ elif "noul" in gold:
+ gold = bool(float(gold["noul"]) >= 0.5)
+ t = q["type"]
+ if t == "noul":
+ s = str(gold).strip().lower()
+ return "true" if s in ("true", "yes", "1", "t", "y") else "false"
+ if t == "score":
+ if isinstance(gold, str) and not gold.isdigit():
+ crit = list(q["criteria"])
+ return str(crit.index(gold))
+ return str(int(gold))
+ return str(gold)
+
+
+def _fold(answer: Mapping, q: Mapping) -> Dict[str, Any]:
+ """One vegaml answer in laya's shape, with vega's extra fields kept.
+
+ The engine keys a boolean's probabilities ``"False"``/``"True"`` and a head keys them
+ ``"false"``/``"true"``; laya spells them lower case, so the keys are folded here once and
+ scoring code downstream never sees two spellings. Not folding this is how a benchmark read
+ every yes/no task at 0.000 for a day.
+ """
+ t = q["type"]
+ probs = {str(k): float(v) for k, v in (answer.get("probs") or {}).items()}
+ if t == "noul":
+ low = {k.lower(): v for k, v in probs.items()}
+ probs = {"false": low.get("false", 0.0), "true": low.get("true", 0.0)}
+ pset = answer.get("prediction_set")
+ if pset is not None:
+ pset = [str(p).lower() if t == "noul" else str(p) for p in pset]
+ out: Dict[str, Any] = {
+ "type": t,
+ "probabilities": {k: round(v, 6) for k, v in probs.items()},
+ "answer_confidence": (round(float(answer["confidence"]), 6)
+ if answer.get("confidence") is not None
+ else (round(max(probs.values()), 6) if probs else None)),
+ "confidence": _entropy_confidence(probs) if probs else None,
+ "prediction_set": pset,
+ "abstain": answer.get("abstain"),
+ "unbound": answer.get("unbound"),
+ "readout": answer.get("readout", "engine"),
+ }
+ if t == "noul":
+ out["noul"] = round(probs.get("true", 0.0), 6)
+ elif t == "score":
+ best = max(probs, key=probs.get) if probs else None
+ level = answer.get("score")
+ out["score"] = int(level) if level is not None else (int(best) if best else None)
+ if answer.get("expected_score") is not None:
+ out["expected_score"] = round(float(answer["expected_score"]), 4)
+ out["legend"] = {str(i): c for i, c in enumerate(q["criteria"])}
+ else:
+ ch = answer.get("choice")
+ out["choice"] = str(ch) if ch is not None else (max(probs, key=probs.get) if probs else None)
+ for k in ("settle_steps", "energy", "adapter", "gate", "view_agreement", "cv_accuracy"):
+ if k in answer:
+ out[k] = answer[k]
+ return out
+
+
+# ----------------------------------------------------------------------------- the agent
+class VegaAgent(Agent):
+ """``laya.Agent`` with vegaml's engine underneath. Built by ``laya.Agent(vegaml=...)``.
+
+ Subclasses :class:`laya.Agent` so ``isinstance`` checks, hooks and callers that duck-type
+ a laya agent keep working, but it does **not** run ``Agent.__init__``: there is no
+ ModernBERT checkpoint to load. Everything a laya caller touches is reimplemented here on
+ top of ``vegaml.Vega``.
+ """
+
+ _backend = "vega"
+
+ def __new__(cls, *args, **kwargs):
+ # Agent.__new__ dispatches here on vegaml=...; coming back through it would recurse.
+ return object.__new__(cls)
+
+ def __init__(self, model_id_or_path: Optional[str] = None, device: Optional[str] = None,
+ token: Optional[str] = None, *, vegaml: Union[bool, str, None] = True,
+ backbone: Optional[str] = None, max_len: Optional[int] = None,
+ alpha: float = 0.1, risk: Union[str, float] = "0.1",
+ target_precision: float = 0.95, folds: int = 5,
+ mode: str = "auto", revision: Optional[str] = None,
+ hooks=None, on_predict_start=None, on_predict_end=None, **ignored):
+ """Load vega.
+
+ ``vegaml`` is the argument that got us here: ``True`` loads the default 0.8B reader,
+ a string is a size (``"4b"``), a Hub repository or a local directory, exactly what
+ ``vegaml.load`` takes. ``model_id_or_path`` is accepted for positional compatibility
+ with ``laya.Agent`` and used when ``vega`` is just ``True``. ``backbone`` names the
+ reader inside a universal checkpoint. ``alpha`` is the conformal miscoverage,
+ ``risk`` the abstention risk level the checkpoint was calibrated at, and
+ ``target_precision`` what :meth:`fit` holds the band to.
+ """
+ try:
+ import vegaml as _vegaml_pkg
+ except ImportError as exc:
+ raise ImportError(_INSTALL) from exc
+ from .hooks import normalise_hooks
+
+ which = vegaml if isinstance(vegaml, str) and vegaml else model_id_or_path
+ self.vega = _vegaml_pkg.load(which, device=device, backbone=backbone, max_len=max_len,
+ mode=mode, revision=revision, token=token)
+ self.model_id = self.model_id_or_path = which or "nandakishorm/vega-U"
+ self.subfolder = backbone
+ self.alpha = str(alpha)
+ self.risk = str(risk)
+ self.target_precision = float(target_precision)
+ self.folds = int(folds)
+ self.calibration = None # a vegaml Calibration, applied on every predict
+ self.fit_report: Dict[str, Any] = {}
+ self.hooks = normalise_hooks(hooks, on_predict_start, on_predict_end)
+ self._lock = threading.RLock()
+ self.device = getattr(self.vega.engine, "device", device)
+ cfg = getattr(self.vega.engine, "cfg", {}) or {}
+ self.max_len = int(cfg.get("max_len", max_len or 0) or 0)
+
+ # ------------------------------------------------------------------ introspection
+ def __repr__(self) -> str:
+ heads = sorted(self.vega.heads)
+ return "VegaAgent(%s, max_len=%s, heads=%d, calibrated=%s)" % (
+ self.model_id, self.max_len, len(heads), self.calibration is not None)
+
+ @property
+ def context_tokens(self) -> int:
+ """How many tokens one :meth:`predict` reads before truncating. 73k on the shipped
+ universal reader; laya's own checkpoints stop at 512 or 1,024 and window past that."""
+ return self.max_len
+
+ # ------------------------------------------------------------------ predicting
+ def predict(self, state: Any = None, questions: Optional[Mapping] = None,
+ lang: Optional[str] = None, *, image: Any = None, audio: Any = None,
+ mode: Optional[str] = None, views: int = 1, max_len: Optional[int] = None,
+ head_max_len: Optional[int] = None, min_confidence: Optional[float] = None,
+ asr: Optional[str] = None, language: Optional[str] = None,
+ sampling_rate: Optional[int] = None, hooks=None, on_predict_start=None,
+ on_predict_end=None, **ignored) -> Dict[str, Any]:
+ """Answer typed questions about text, an image, or a recording, in laya's shape.
+
+ One of ``state``, ``image`` or ``audio``. The modality is chosen by which you passed,
+ which is the whole of "modal switching": the same questions, the same answer shape,
+ the same route. ``lang`` is accepted for compatibility and recorded; the universal
+ reader is multilingual and needs no checkpoint switch.
+
+ ``max_len`` and ``head_max_len`` are laya's option-budget knobs; vega truncates
+ options at 32 tokens and instructions at 256 by itself, so they are recorded and not
+ applied. ``min_confidence`` is applied on top of the automatic route when given, the
+ way laya's gate applies it, and nothing here needs it.
+ """
+ if questions is None:
+ raise ValueError("predict needs `questions`")
+ given = sum(x is not None for x in (state, image, audio))
+ if given != 1:
+ raise ValueError("predict takes exactly one of state=, image=, audio=; got %d" % given)
+ if not questions:
+ return {"answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0},
+ "latency_ms": 0.0, "modality": "text"}
+ vq = _to_vega_questions(questions)
+ started = time.perf_counter()
+ with self._lock:
+ if image is not None:
+ raw = self.vega.decide_image(image, vq, mode=mode, alpha=self.alpha, risk=self.risk)
+ modality = "image"
+ elif audio is not None:
+ raw = self.vega.decide_audio(audio, vq, asr=asr, language=language,
+ sampling_rate=sampling_rate, mode=mode,
+ alpha=self.alpha, views=views)
+ modality = "audio"
+ else:
+ raw = self.vega.decide(state, vq, mode=mode, alpha=self.alpha, views=views)
+ modality = "text"
+ if self.calibration is not None:
+ raw = self.calibration.apply(_pad_for_calibration(raw, self.calibration))
+ answers = {}
+ for qid, q in questions.items():
+ a = _fold(raw["answers"][qid], q)
+ a.update(self._route(raw["answers"][qid], a, min_confidence))
+ answers[qid] = a
+ out = {
+ "answers": answers,
+ "usage": {"input_tokens": int(raw.get("n_tokens") or 0), "output_tokens": 0},
+ "latency_ms": round(1000 * (time.perf_counter() - started), 1),
+ "modality": modality,
+ "mode": raw.get("mode", mode or self.vega.mode),
+ "views": raw.get("views", views),
+ "calibrated": self.calibration is not None,
+ }
+ if lang is not None:
+ out["lang"] = lang
+ for k in ("transcript", "asr"):
+ if k in raw:
+ out[k] = raw[k]
+ return out
+
+ system_one = predict
+
+ def predict_image(self, image: Any, questions: Mapping, **kw) -> Dict[str, Any]:
+ return self.predict(image=image, questions=questions, **kw)
+
+ def predict_image_text(self, image: Any, questions: Mapping, **kw) -> Dict[str, Any]:
+ """Parity check: the vision path with the image withheld, to see what the picture
+ itself contributed."""
+ raw = self.vega.decide_image_text(image, _to_vega_questions(questions), **kw)
+ return {"answers": {qid: _fold(raw["answers"][qid], q) for qid, q in questions.items()},
+ "modality": "image_withheld"}
+
+ def predict_audio(self, audio: Any, questions: Mapping, **kw) -> Dict[str, Any]:
+ return self.predict(audio=audio, questions=questions, **kw)
+
+ def predict_batch(self, states: Sequence[Any], questions: Mapping, batch_size: Optional[int] = None,
+ lang: Optional[str] = None, **kw) -> List[Dict[str, Any]]:
+ """Every state, one decide call each. vega packs a state's questions into one pass,
+ not many states, so this is a loop with laya's signature rather than a faster path.
+ ``batch_size`` is accepted and ignored."""
+ return [self.predict(s, questions, lang=lang, **kw) for s in states]
+
+ def predict_long(self, state: Any, questions: Mapping, window: Optional[int] = None,
+ stride: Optional[int] = None, aggregate: str = "auto", **kw) -> Dict[str, Any]:
+ """A long state, read whole.
+
+ laya's own checkpoints stop at 512 or 1,024 tokens and this method windows past that.
+ The universal reader holds 73k tokens, so the state is read in one pass up to that and
+ the per-window aggregation never happens; ``window``, ``stride`` and ``aggregate`` are
+ accepted for compatibility and recorded. Past the reader's own limit the engine
+ truncates, and ``usage.truncated`` says so, which is the one case where the caller may
+ still want to window and vote.
+ """
+ out = self.predict(state, questions, **kw)
+ n = int(out["usage"].get("input_tokens") or 0)
+ out["usage"]["windows"] = 1
+ out["usage"]["context_tokens"] = self.max_len
+ out["usage"]["truncated"] = bool(self.max_len and n >= self.max_len)
+ out["long"] = {"window": window, "stride": stride, "aggregate": aggregate,
+ "applied": False, "why": "read whole, within %d tokens" % self.max_len}
+ return out
+
+ # ------------------------------------------------------------------ the route
+ def _route(self, raw_answer: Mapping, folded: Mapping, min_confidence: Optional[float]) -> Dict:
+ """``route`` and why, decided without a user-set number. See the module docstring."""
+ if "route" in raw_answer and raw_answer["route"] in ("auto", "review"):
+ route, why = raw_answer["route"], "band: " + str(raw_answer.get("route_reason", ""))
+ else:
+ reasons = []
+ if folded.get("abstain"):
+ reasons.append("the engine abstains at risk %s" % self.risk)
+ if folded.get("unbound"):
+ reasons.append("unbound: far from every well, likely off-distribution")
+ pset = folded.get("prediction_set")
+ if pset is not None and len(pset) > 1:
+ reasons.append("conformal set at alpha %s holds %d options" % (self.alpha, len(pset)))
+ if pset is None and folded.get("abstain") is None:
+ # a head-served answer with no band fitted: nothing here can vouch for it
+ reasons.append("head answer with no fitted band; fit() or calibrate() first")
+ route = "review" if reasons else "auto"
+ why = "; ".join(reasons) if reasons else (
+ "conformal singleton, not abstaining, in distribution")
+ if min_confidence is not None and route == "auto":
+ c = folded.get("answer_confidence")
+ if c is None or c < float(min_confidence):
+ route, why = "review", "min_confidence %.3f not met (%s)" % (min_confidence, c)
+ return {"route": route, "route_reason": why}
+
+ def thresholds(self) -> Dict[str, Any]:
+ """What decides ``route`` right now, per question, so nothing about it is hidden."""
+ out: Dict[str, Any] = {"alpha": self.alpha, "risk": self.risk,
+ "target_precision": self.target_precision,
+ "policy": ("band" if self.calibration is not None
+ else "conformal set + abstain + unbound")}
+ if self.calibration is not None:
+ band = self.calibration.band
+ out["band"] = ({"default": band.default, "questions": band.questions}
+ if band is not None else None)
+ out["calibration"] = {q: {k: v for k, v in c.items() if k != "labels"}
+ for q, c in self.calibration.questions.items()}
+ out["heads"] = {q: {"cv_accuracy": cv, "serving": self.vega._head_worth_using(q, {
+ "type": "choice", "criteria": {l: None for l in h.labels}})}
+ for q, (h, cv) in self.vega.heads.items()}
+ return out
+
+ # ------------------------------------------------------------------ fitting
+ def _examples(self, rows: Sequence[Mapping], questions: Optional[Mapping]) -> Tuple[list, Dict]:
+ """laya training rows as vegaml fit examples: ``(state, {qid: label}, {qid: question})``.
+
+ Rows are ``{"state", "questions"?, "gold"|"expected"}``: the shape ``items_from_rows``
+ reads, so a corpus prepared for laya's fine-tune fits vega's heads unchanged. A row
+ without its own ``questions`` uses ``questions``.
+ """
+ examples, sheet = [], {}
+ for r in rows:
+ qs = r.get("questions") or questions
+ if not qs:
+ raise ValueError("a row has no questions and none were given")
+ vq = _to_vega_questions(qs)
+ sheet.update(vq)
+ gold = r.get("gold") if r.get("gold") is not None else r.get("expected")
+ if not isinstance(gold, Mapping):
+ raise ValueError("each row needs `gold` or `expected` as {question_id: answer}")
+ labels = {qid: _gold_label(qs[qid], g) for qid, g in gold.items() if qid in qs}
+ if labels:
+ examples.append((r["state"], labels, vq))
+ return examples, sheet
+
+ def fit(self, rows: Sequence[Mapping], questions: Optional[Mapping] = None,
+ folds: Optional[int] = None, calibrate: bool = True,
+ target_precision: Optional[float] = None) -> Dict[str, Any]:
+ """Heads on your labels, then the calibration and the band, in one call.
+
+ What laya's fine-tune does with gradient steps, this does with a small head per
+ question over the frozen features: seconds on a GPU for a few hundred rows, and the
+ reader is never touched. A head is kept only if it beats chance under cross-validation
+ and SERVED only if it also beats the engine on the same rows, so ``report[qid]
+ ["used_by_auto"]`` is the fact to read, not ``fitted``. With ``calibrate`` (the
+ default) the same rows then refit a temperature and a conformal threshold per
+ question and place the band, which is where the automatic ``route`` comes from.
+ Nothing is scored on a fitted row by this method; held-out evaluation is yours.
+ """
+ examples, sheet = self._examples(rows, questions)
+ if not examples:
+ raise ValueError("no trainable rows")
+ with self._lock:
+ report = self.vega.fit(examples, sheet, folds=folds or self.folds)
+ self.fit_report = {"heads": report, "n_rows": len(examples),
+ "serving": sorted(q for q, r in report.items() if r.get("used_by_auto")),
+ "kept": sorted(q for q, r in report.items() if r.get("fitted"))}
+ if calibrate:
+ self.fit_report["calibration"] = self.calibrate(
+ rows, questions, target_precision=target_precision, _examples=(examples, sheet))
+ return self.fit_report
+
+ def calibrate(self, rows: Sequence[Mapping], questions: Optional[Mapping] = None,
+ target_precision: Optional[float] = None, alpha: Optional[float] = None,
+ _examples=None) -> Dict[str, Any]:
+ """Refit the probabilities and place the band on your labelled rows.
+
+ A temperature per question, a conformal threshold on the corrected probabilities so
+ the sets cover at ``1 - alpha`` on rows like yours, and the escalation band at
+ ``target_precision``. The argmax is invariant under temperature, so this moves
+ confidence, sets, ``abstain`` and ``route`` and never which option wins. Sharper
+ answers need :meth:`fit`. Applied on every :meth:`predict` from now on.
+ """
+ from vegaml.calibrate import calibrate as _calibrate
+
+ examples, sheet = _examples or self._examples(rows, questions)
+ tp = self.target_precision if target_precision is None else float(target_precision)
+ a = float(self.alpha) if alpha is None else float(alpha)
+ with self._lock:
+ self.calibration = _calibrate(self.vega, examples, sheet, alpha=a,
+ target_precision=tp, readout="auto")
+ rep = {q: {k: r.get(k) for k in ("temperature", "qhat", "ece_before", "ece_after",
+ "examples") if k in r}
+ for q, r in self.calibration.report.items()}
+ band = self.calibration.band
+ return {"questions": rep, "target_precision": tp, "alpha": a,
+ "band": None if band is None else
+ {q: round(float(v["threshold"]), 4) for q, v in band.questions.items()}}
+
+ fit_temperatures = calibrate
+
+ # ------------------------------------------------------------------ persistence
+ def save_calibration(self, path: str) -> Dict[str, Any]:
+ """Heads and calibration to ``path`` (a directory), reloadable with
+ :meth:`load_calibration`. vegaml's heads have no save of their own, so the arrays are
+ written here; the Calibration is vegaml's own JSON."""
+ import numpy as np
+
+ os.makedirs(path, exist_ok=True)
+ arrays, meta = {}, {"heads": {}, "model": self.model_id, "alpha": self.alpha,
+ "risk": self.risk, "target_precision": self.target_precision}
+ for qid, (head, cv) in self.vega.heads.items():
+ if not getattr(head, "ready", False):
+ continue
+ for name in HEAD_ARRAYS:
+ arrays["%s::%s" % (qid, name)] = np.asarray(getattr(head, name))
+ meta["heads"][qid] = {"labels": [str(x) for x in head.labels],
+ "k": int(head.k), "l2": float(head.l2),
+ "cv": None if cv is None else float(cv),
+ "engine_accuracy": head.engine_accuracy}
+ arrays["meta"] = np.array(json.dumps(meta))
+ np.savez(os.path.join(path, "heads.npz"), **arrays)
+ if self.calibration is not None:
+ self.calibration.save(os.path.join(path, "calibration.json"))
+ return {"path": path, "heads": sorted(meta["heads"]),
+ "calibrated": self.calibration is not None}
+
+ def load_calibration(self, path: str) -> Dict[str, Any]:
+ import numpy as np
+ from vegaml.calibrate import Calibration
+ from vegaml.ttt import TTTHead
+
+ z = np.load(os.path.join(path, "heads.npz"), allow_pickle=False)
+ meta = json.loads(str(z["meta"]))
+ loaded = []
+ for qid, m in meta["heads"].items():
+ head = TTTHead(m["labels"], k=m["k"], l2=m["l2"])
+ for name in HEAD_ARRAYS:
+ setattr(head, name, z["%s::%s" % (qid, name)])
+ head.ready = True
+ head.engine_accuracy = m.get("engine_accuracy")
+ self.vega.heads[qid] = (head, m.get("cv"))
+ loaded.append(qid)
+ cal_path = os.path.join(path, "calibration.json")
+ if os.path.exists(cal_path):
+ self.calibration = Calibration.load(cal_path)
+ return {"heads": loaded, "calibrated": self.calibration is not None}
+
+ # ------------------------------------------------------------------ readers
+ @staticmethod
+ def attach(backbone: str, texts: Sequence[str], out_dir: str, **kw) -> Dict[str, Any]:
+ """Attach your own frozen reader to the universal engine, from your own text, with no
+ labels: writes a model directory that ``laya.Agent(vegaml=out_dir)`` then loads."""
+ import vegaml
+ return vegaml.attach(backbone, texts, out_dir, **kw)
+
+ # laya's own calibration file is a different artefact; say so rather than mis-save one
+ def fit_binning(self, *a, **kw):
+ raise NotImplementedError("histogram binning is laya's own calibration; on vega the "
+ "calibrated probabilities come from calibrate()")
+
+
+def _pad_for_calibration(result: Mapping, cal) -> Dict:
+ """Zero-fill any fitted option an answer never mentioned.
+
+ ``Calibration.apply`` indexes every label it was fitted on and KeyErrors on an answer over
+ a strict subset, after the fit and the calibration have both succeeded, which is the worst
+ place for it. Zero is the correct value: the row did not offer that option.
+ """
+ out = dict(result)
+ out["answers"] = {}
+ for qid, ans in result["answers"].items():
+ fitted = cal.questions.get(qid)
+ if fitted is None or not ans.get("probs"):
+ out["answers"][qid] = ans
+ continue
+ have = {str(k).lower() for k in ans["probs"]}
+ missing = [o for o in fitted["labels"] if str(o).lower() not in have]
+ if missing:
+ ans = dict(ans)
+ ans["probs"] = dict(ans["probs"])
+ for o in missing:
+ ans["probs"][o] = 0.0
+ out["answers"][qid] = ans
+ return out
+
+
+# ----------------------------------------------------------------------------- switching
+class Router:
+ """Model switching: laya's fast encoder where it is enough, vega where it is not.
+
+ router = from laya.vega import Router
+ Router(fast=laya.Agent(), vega=laya.Agent(vegaml=True))
+ router.predict(state, questions) # -> picks, and says which under "routing"
+
+ The rule, in order: an ``image`` or ``audio`` goes to vega (the encoder has no eyes or
+ ears); a state longer than the fast model's context goes to vega (73k against 512 or
+ 1,024); non-English or non-Latin text goes to vega (the universal reader is multilingual,
+ and laya's own English checkpoint collapses on non-Latin scripts); a question with a
+ fitted head on vega goes to vega (that is what the labels bought); everything else goes to
+ the fast model. ``model=`` forces one. The result carries ``routing = {"model", "reason"}``
+ in the shape ``laya.Router`` already returns, so a caller switching from that class reads
+ nothing new.
+ """
+
+ def __init__(self, fast: Optional[Agent] = None, vega: Optional[VegaAgent] = None,
+ fast_context_tokens: int = 512, prefer_vega_when_fitted: bool = True):
+ self.fast = fast
+ self._vega = vega
+ self.fast_context_tokens = int(fast_context_tokens)
+ self.prefer_vega_when_fitted = prefer_vega_when_fitted
+
+ @property
+ def vega(self) -> VegaAgent:
+ if self._vega is None:
+ self._vega = VegaAgent()
+ return self._vega
+
+ def choose(self, state: Any = None, questions: Optional[Mapping] = None, *, image=None,
+ audio=None, lang: Optional[str] = None) -> Tuple[str, str]:
+ from .lang import is_english
+
+ if image is not None or audio is not None:
+ return "vega", "image or audio: the encoder has no vision or speech path"
+ if self.fast is None:
+ return "vega", "no fast model attached"
+ text = state if isinstance(state, str) else json.dumps(state, ensure_ascii=False)
+ if len(text) > self.fast_context_tokens * 4: # ~4 characters a token
+ return "vega", "state is longer than the fast model's %d-token context" % \
+ self.fast_context_tokens
+ if lang is not None:
+ english = str(lang).lower() in ("en", "eng", "english")
+ else:
+ english = bool(is_english(text))
+ if not english:
+ return "vega", "non-English or non-Latin text; the universal reader is multilingual"
+ if self.prefer_vega_when_fitted and questions and self._vega is not None:
+ fitted = [q for q in questions if q in self._vega.vega.heads]
+ if fitted:
+ return "vega", "fitted heads exist for %s" % ", ".join(fitted)
+ return "fast", "short English text, no fitted head: the fast encoder is enough"
+
+ def predict(self, state: Any = None, questions: Optional[Mapping] = None, *,
+ model: Optional[str] = None, image=None, audio=None, lang: Optional[str] = None,
+ **kw) -> Dict[str, Any]:
+ if model is None:
+ model, reason = self.choose(state, questions, image=image, audio=audio, lang=lang)
+ else:
+ reason = "explicit model=%s" % model
+ if model == "vega":
+ out = self.vega.predict(state, questions, lang=lang, image=image, audio=audio, **kw)
+ elif model == "fast":
+ if image is not None or audio is not None:
+ raise ValueError("the fast laya model takes text only; drop model= or use vega")
+ out = self.fast.predict(state, questions, lang=lang, **kw)
+ else:
+ raise ValueError("model must be 'fast' or 'vega', got %r" % model)
+ out["routing"] = {"model": model, "reason": reason}
+ return out
diff --git a/pyproject.toml b/pyproject.toml
index 5dd235c6..3e3620d2 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -49,6 +49,12 @@ fast = ["tilelang>=0.1.14"]
# `pip install laya[mcp]` adds the MCP stdio server (laya.mcp / the laya-mcp-server CLI).
mcp = ["mcp>=2.2.0"]
+# `pip install laya[vegaml]` puts vega's engine behind laya's surface: `laya.Agent(vegaml=True)`.
+# Conformal sets, abstain and unbound flags, heads fitted on your labels, a refit calibration,
+# 73k tokens of context, images and speech, all through laya's own method names (laya/vega.py).
+# vegaml needs Python 3.10 or newer; core laya does not, which is why this is an extra.
+vegaml = ["vegaml>=0.15.0; python_version >= '3.10'"]
+
# `pip install laya[structured]` adds pydantic for the typed-model helpers (laya/structured.py).
structured = ["pydantic>=2.0.0"]
diff --git a/scripts/test_suites.py b/scripts/test_suites.py
index 0870e0ec..64b6308f 100644
--- a/scripts/test_suites.py
+++ b/scripts/test_suites.py
@@ -127,6 +127,7 @@ PYTEST_SUITES = [
"tests/test_truncation_direction.py",
"tests/test_compile.py",
"tests/test_finetune_entrypoints.py",
+ "tests/test_vega.py",
]
diff --git a/tests/test_audit_scope.py b/tests/test_audit_scope.py
index 53f79d19..fe773e86 100644
--- a/tests/test_audit_scope.py
+++ b/tests/test_audit_scope.py
@@ -87,14 +87,15 @@ if len(od) > 1:
check_true("pyproject/core parses", len(core) == 5, core)
check("pyproject/extra tables found", sorted(extras),
["agno", "crewai", "fast", "langchain", "langgraph", "llamaindex", "mcp", "onnx", "serve",
- "structured"])
+ "structured", "vegaml"])
declared = list(core) + [s for names in extras.values() for s in names]
expected = sorted(set(declared))
check("pyproject/extras add names the core does not have",
sorted({re.split(r"[<>=!;\[ ]", s)[0] for s in declared}
- {re.split(r"[<>=!;\[ ]", s)[0] for s in core}),
["agno", "crewai", "fastapi", "langchain-core", "langgraph", "llama-index-core", "mcp", "onnx",
- "onnxruntime", "onnxscript", "pydantic", "python-multipart", "tilelang", "uvicorn"])
+ "onnxruntime", "onnxscript", "pydantic", "python-multipart", "tilelang", "uvicorn",
+ "vegaml"])
# The two scopes: shipped (blocking) is the core plus [serve]; everything else is advisory.
shipped = sorted(set(core) | set(extras.get("serve", [])))
diff --git a/tests/test_confidence.py b/tests/test_confidence.py
index 33f55c93..57b603ef 100644
--- a/tests/test_confidence.py
+++ b/tests/test_confidence.py
@@ -2240,7 +2240,7 @@ for _fn in (test_tl_kernels_drops_the_wrong_row_claim,
# --------------------------------------- DecisionResult's docstring must split confidence by question type
# `laya/structured.py`'s `DecisionResult` closed with "confidence keeps the **normalized-entropy
# value** it has always had, because that is a different quantity on a scale that depends on the
-# label count". `Agent._decode_answers` (`laya/agent.py:1394-1423`) uses two different formulas:
+# label count". `Agent._decode_answers` (`laya/agent.py:1408-1437`) uses two different formulas:
# * `choice` and `score` -> `confidence_from_probs(p, k)` = `1 - H(p) / log(k)`
# * `noul` -> `max(p_true, 1 - p_true)` (its own comment: "identical here:
# over two options `max(p_true, 1 - p_true)` is `max(p)`")
diff --git a/tests/test_download.py b/tests/test_download.py
index 74d83fb0..5c907c24 100644
--- a/tests/test_download.py
+++ b/tests/test_download.py
@@ -353,7 +353,8 @@ class DownloadTests(unittest.TestCase):
# reports and takes no override; a different asymmetry, with its
# own fix.
not_for_onnxruntime = {"fast", "compile", "device", "backend",
- "compile_warmup", "compile_cache", "compile_mode"}
+ "compile_warmup", "compile_cache", "compile_mode",
+ "vegaml"} # selects a different engine, like backend
agent_side = (set(inspect.signature(Agent.__init__).parameters)
- {"self", "model_id_or_path"} - not_for_onnxruntime)
onnx_side = set(inspect.signature(ONNXAgent.__init__).parameters) - {"self"}
diff --git a/tests/test_vega.py b/tests/test_vega.py
new file mode 100644
index 00000000..69177dec
--- /dev/null
+++ b/tests/test_vega.py
@@ -0,0 +1,337 @@
+"""`laya.Agent(vegaml=True)`: vega's engine behind laya's surface (laya/vega.py).
+
+Two layers, so the suite is useful on every machine that runs it:
+
+* **Weight-free**, always on: the question translation, the gold-label mapping, the answer
+ fold (the "False"/"True" key folding that once scored every yes/no task at 0.000), the
+ calibration padding, the automatic route policy, the Router's switching rule, and the
+ dispatch in `Agent.__new__` / `laya.load`, including the error a 3.9 interpreter gets.
+ These run without vegaml, torch weights, or the network.
+* **Model-backed**, only when `LAYA_VEGA_TESTS=1` and vegaml is importable: the real 0.8B
+ universal reader answering text, the fitted path end to end (fit, calibrate, save, reload,
+ identical answers and routes), and the band taking over the route. `LAYA_VEGA_MEDIA_TESTS=1`
+ adds the image path, which pulls the vision tower.
+
+Run: python -m pytest tests/test_vega.py -q
+"""
+import json
+import math
+import os
+import sys
+import tempfile
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import laya # noqa: E402
+from laya import vega as lv # noqa: E402
+
+VEGA = lv.is_available()
+MODEL_TESTS = VEGA and os.environ.get("LAYA_VEGA_TESTS") == "1"
+MEDIA_TESTS = MODEL_TESTS and os.environ.get("LAYA_VEGA_MEDIA_TESTS") == "1"
+
+Q = {
+ "dept": {"type": "choice", "instructions": "Which department should handle this?",
+ "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages",
+ "other": "everything else"}},
+ "urgency": {"type": "score", "instructions": "How urgent is this?",
+ "criteria": ["not urgent", "soon", "blocking"]},
+ "churn": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?",
+ "labels": {"false": "stays", "true": "threatens to leave"}},
+}
+STATE = "We were billed twice for March. Refund the duplicate today or we cancel our plan."
+
+
+# ----------------------------------------------------------------------------- translation
+def test_questions_translate_into_vegaml_vocabulary():
+ vq = lv._to_vega_questions(Q)
+ assert vq["churn"]["type"] == "boolean"
+ assert vq["churn"]["criteria"] == {"false": "stays", "true": "threatens to leave"}
+ assert vq["urgency"] == {"instructions": Q["urgency"]["instructions"], "type": "score",
+ "criteria": ["not urgent", "soon", "blocking"]}
+ assert vq["dept"]["type"] == "choice" and vq["dept"]["criteria"] == Q["dept"]["criteria"]
+
+
+def test_list_form_choice_and_unknown_type():
+ vq = lv._to_vega_questions({"k": {"type": "choice", "instructions": "x", "criteria": ["a", "b"]}})
+ assert vq["k"]["criteria"] == {"a": None, "b": None}
+ with pytest.raises(ValueError):
+ lv._to_vega_questions({"k": {"type": "ranking", "instructions": "x"}})
+
+
+def test_labels_for_each_type():
+ assert lv._labels_for(Q["churn"]) == ["false", "true"]
+ assert lv._labels_for(Q["urgency"]) == ["0", "1", "2"]
+ assert lv._labels_for(Q["dept"]) == ["billing", "technical", "other"]
+
+
+def test_gold_label_accepts_every_laya_answer_shape():
+ assert lv._gold_label(Q["churn"], True) == "true"
+ assert lv._gold_label(Q["churn"], {"label": "yes"}) == "true"
+ assert lv._gold_label(Q["churn"], {"noul": 0.2}) == "false"
+ assert lv._gold_label(Q["urgency"], 2) == "2"
+ assert lv._gold_label(Q["urgency"], "blocking") == "2"
+ assert lv._gold_label(Q["dept"], {"probabilities": {"billing": 0.1, "technical": 0.9}}) == "technical"
+ assert lv._gold_label(Q["dept"], {"label": "other"}) == "other"
+
+
+# ----------------------------------------------------------------------------- the fold
+def _engine_answer(**kw):
+ base = {"confidence": 0.9, "prediction_set": None, "abstain": False, "unbound": False,
+ "readout": "engine"}
+ base.update(kw)
+ return base
+
+
+def test_fold_lowercases_the_engines_boolean_keys():
+ # the engine keys a boolean's probabilities "False"/"True"; laya spells them lower case
+ a = lv._fold(_engine_answer(probs={"False": 0.94, "True": 0.06}, boolean=0.06,
+ prediction_set=[False]), Q["churn"])
+ assert set(a["probabilities"]) == {"false", "true"}
+ assert a["noul"] == pytest.approx(0.06)
+ assert a["prediction_set"] == ["false"]
+ assert a["answer_confidence"] == pytest.approx(0.9)
+ assert a["type"] == "noul"
+
+
+def test_fold_choice_and_score_carry_laya_fields():
+ c = lv._fold(_engine_answer(probs={"billing": 0.7, "technical": 0.2, "other": 0.1},
+ choice="billing", prediction_set=["billing"]), Q["dept"])
+ assert c["choice"] == "billing" and c["prediction_set"] == ["billing"]
+ s = lv._fold(_engine_answer(probs={"0": 0.1, "1": 0.8, "2": 0.1}, score=1,
+ expected_score=1.0), Q["urgency"])
+ assert s["score"] == 1 and s["legend"] == {"0": "not urgent", "1": "soon", "2": "blocking"}
+ assert s["expected_score"] == 1.0
+
+
+def test_fold_confidence_is_entropy_and_answer_confidence_is_max_p():
+ a = lv._fold(_engine_answer(probs={"a": 0.5, "b": 0.5}, choice="a"),
+ {"type": "choice", "instructions": "x", "criteria": {"a": None, "b": None}})
+ assert a["confidence"] == pytest.approx(0.0, abs=1e-6) # an even split has no information
+ assert a["answer_confidence"] == pytest.approx(0.9) # what the engine reported
+ assert lv._entropy_confidence({"a": 1.0, "b": 0.0}) == 1.0
+
+
+# ----------------------------------------------------------------------------- padding
+class _FakeCalibration:
+ def __init__(self, labels):
+ self.questions = {"q": {"labels": labels, "temperature": 1.0, "qhat": 0.4}}
+
+
+def test_padding_fills_fitted_options_the_answer_never_mentioned():
+ cal = _FakeCalibration(list("ABCDEFGHI"))
+ narrow = {"answers": {"q": {"probs": {"A": 0.7, "B": 0.2, "C": 0.1}}}}
+ out = lv._pad_for_calibration(narrow, cal)
+ assert set(out["answers"]["q"]["probs"]) == set("ABCDEFGHI")
+ assert out["answers"]["q"]["probs"]["G"] == 0.0
+ assert out["answers"]["q"]["probs"]["A"] == 0.7
+ assert narrow["answers"]["q"]["probs"].keys() == {"A", "B", "C"} # input untouched
+ wide = {"answers": {"q": {"probs": {k: 1 / 9 for k in "ABCDEFGHI"}}}}
+ assert lv._pad_for_calibration(wide, cal)["answers"]["q"] is wide["answers"]["q"]
+
+
+# ----------------------------------------------------------------------------- the route
+def _bare_agent():
+ """A VegaAgent with no engine, enough to exercise `_route`."""
+ a = object.__new__(lv.VegaAgent)
+ a.alpha, a.risk = "0.1", "0.1"
+ return a
+
+
+def test_route_is_auto_only_when_everything_agrees():
+ a = _bare_agent()
+ folded = {"abstain": False, "unbound": False, "prediction_set": ["x"], "answer_confidence": 0.8}
+ r = a._route({}, folded, None)
+ assert r["route"] == "auto" and "singleton" in r["route_reason"]
+
+
[email protected]("field, value, word", [
+ ("abstain", True, "abstains"),
+ ("unbound", True, "unbound"),
+ ("prediction_set", ["x", "y"], "holds 2 options"),
+])
+def test_route_reviews_on_each_engine_signal(field, value, word):
+ a = _bare_agent()
+ folded = {"abstain": False, "unbound": False, "prediction_set": ["x"], "answer_confidence": 0.8}
+ folded[field] = value
+ r = a._route({}, folded, None)
+ assert r["route"] == "review" and word in r["route_reason"]
+
+
+def test_route_prefers_a_fitted_band_and_reviews_unvouched_heads():
+ a = _bare_agent()
+ banded = a._route({"route": "review", "route_reason": "below the line"},
+ {"abstain": False, "unbound": False, "prediction_set": ["x"]}, None)
+ assert banded["route"] == "review" and banded["route_reason"].startswith("band:")
+ # a head answer carries no abstain and no set; with no band nothing can vouch for it
+ head = a._route({}, {"abstain": None, "unbound": None, "prediction_set": None,
+ "answer_confidence": 0.99}, None)
+ assert head["route"] == "review" and "no fitted band" in head["route_reason"]
+
+
+def test_min_confidence_is_applied_on_top_only_when_given():
+ a = _bare_agent()
+ folded = {"abstain": False, "unbound": False, "prediction_set": ["x"], "answer_confidence": 0.8}
+ assert a._route({}, folded, 0.7)["route"] == "auto"
+ r = a._route({}, folded, 0.9)
+ assert r["route"] == "review" and "min_confidence" in r["route_reason"]
+
+
+# ----------------------------------------------------------------------------- switching
+class _StubAgent:
+ def __init__(self, name):
+ self.name = name
+ self.vega = type("V", (), {"heads": {}})()
+
+ def predict(self, state=None, questions=None, **kw):
+ return {"answers": {q: {"choice": self.name} for q in (questions or {})}}
+
+
+def test_router_rule_in_order():
+ fast, slow = _StubAgent("fast"), _StubAgent("vega")
+ r = lv.Router(fast=fast, vega=slow, fast_context_tokens=8)
+ q = {"dept": Q["dept"]}
+ assert r.choose("I was charged twice", q)[0] == "fast"
+ assert r.choose("I was charged twice", q, image=object())[0] == "vega"
+ assert r.choose("x " * 100, q)[0] == "vega" # longer than 8 tokens
+ assert r.choose("मुझे दो बार चार्ज किया गया", q)[0] == "vega" # non-Latin script
+ assert r.choose("I was charged twice", q, lang="de")[0] == "vega"
+ slow.vega.heads["dept"] = (None, 0.9)
+ assert r.choose("I was charged twice", q)[0] == "vega" # a fitted head wins
+ out = r.predict("I was charged twice", q, model="fast")
+ assert out["routing"] == {"model": "fast", "reason": "explicit model=fast"}
+ assert out["answers"]["dept"]["choice"] == "fast"
+ with pytest.raises(ValueError):
+ r.predict(questions=q, image=object(), model="fast")
+
+
+def test_router_without_a_fast_model_always_picks_vega():
+ r = lv.Router(fast=None, vega=_StubAgent("vega"))
+ assert r.choose("hello", {"dept": Q["dept"]})[0] == "vega"
+
+
+# ----------------------------------------------------------------------------- dispatch
+def test_agent_new_does_not_divert_without_the_argument():
+ for value in (None, False):
+ obj = laya.Agent.__new__(laya.Agent, vegaml=value)
+ assert type(obj) is laya.Agent
+
+
[email protected](VEGA, reason="vegaml is installed here; the ImportError path needs it absent")
+def test_without_vegaml_the_argument_fails_with_the_install_hint():
+ with pytest.raises(ImportError, match=r"laya\[vegaml\]"):
+ laya.Agent(vegaml=True)
+ with pytest.raises(ImportError, match=r"laya\[vegaml\]"):
+ laya.load(vegaml=True)
+
+
+def test_vega_agent_is_exported_and_is_an_agent_subclass():
+ assert laya.VegaAgent is lv.VegaAgent
+ assert issubclass(lv.VegaAgent, laya.Agent)
+ assert "VegaAgent" in laya.__all__
+
+
+# ----------------------------------------------------------------------------- with weights
+needs_model = pytest.mark.skipif(
+ not MODEL_TESTS, reason="set LAYA_VEGA_TESTS=1 with vegaml installed (downloads the 0.8B reader)")
+
+
[email protected](scope="module")
+def agent():
+ if not MODEL_TESTS:
+ pytest.skip("model tests off")
+ return laya.Agent(vegaml=True)
+
+
+@needs_model
+def test_dispatch_builds_a_vega_agent(agent):
+ assert type(agent) is lv.VegaAgent and isinstance(agent, laya.Agent)
+ assert type(laya.load(vegaml=True)) is lv.VegaAgent
+ assert agent.context_tokens >= 8192
+
+
+@needs_model
+def test_text_answers_carry_laya_and_vega_fields(agent):
+ out = agent.predict(STATE, Q)
+ for qid, q in Q.items():
+ a = out["answers"][qid]
+ assert set(lv._labels_for(q)) == set(a["probabilities"])
+ assert a["route"] in ("auto", "review") and a["route_reason"]
+ assert a["abstain"] in (True, False) and a["unbound"] in (True, False)
+ assert a["prediction_set"] is None or isinstance(a["prediction_set"], list)
+ assert 0.0 <= a["answer_confidence"] <= 1.0
+ assert "choice" in out["answers"]["dept"] and "score" in out["answers"]["urgency"]
+ assert "noul" in out["answers"]["churn"]
+ assert out["usage"]["input_tokens"] > 0 and out["modality"] == "text"
+ assert agent.predict(STATE, {}) == {"answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0},
+ "latency_ms": 0.0, "modality": "text"}
+ with pytest.raises(ValueError):
+ agent.predict(STATE, Q, image="x.png")
+
+
+@needs_model
+def test_batch_long_and_thresholds(agent):
+ assert len(agent.predict_batch([STATE, "The app crashes on login."], Q)) == 2
+ lg = agent.predict_long(STATE * 20, Q)
+ assert lg["usage"]["windows"] == 1 and lg["long"]["applied"] is False
+ assert lg["usage"]["input_tokens"] > 500
+ assert agent.thresholds()["policy"] == "conformal set + abstain + unbound"
+
+
+def _synthetic_rows(n=24):
+ """Labelled rows a head can learn: two departments, two kinds of complaint."""
+ rows = []
+ bills = ["Charged twice for %s, refund the duplicate." % m for m in
+ ("March", "April", "May", "June", "July", "August", "September", "October",
+ "November", "December", "January", "February")]
+ bugs = ["The app crashes when I open %s." % s for s in
+ ("settings", "the inbox", "a report", "the dashboard", "search", "billing history",
+ "my profile", "notifications", "the export page", "the login screen", "the map",
+ "attachments")]
+ for i in range(n // 2):
+ rows.append({"state": bills[i], "questions": {"dept": Q["dept"]}, "gold": {"dept": "billing"}})
+ rows.append({"state": bugs[i], "questions": {"dept": Q["dept"]}, "expected": {"dept": "technical"}})
+ return rows
+
+
+@needs_model
+def test_fit_calibrate_save_reload_roundtrip():
+ a = laya.Agent(vegaml=True)
+ rows = _synthetic_rows()
+ rep = a.fit(rows)
+ assert rep["n_rows"] == len(rows) and "dept" in rep["heads"]
+ assert "calibration" in rep and rep["calibration"]["target_precision"] == 0.95
+ assert a.calibration is not None and a.thresholds()["policy"] == "band"
+ out = a.predict(rows[0]["state"], {"dept": Q["dept"]})
+ assert out["calibrated"] is True
+ assert out["answers"]["dept"]["route_reason"].startswith("band:")
+ with tempfile.TemporaryDirectory() as d:
+ saved = a.save_calibration(d)
+ assert saved["calibrated"] and os.path.exists(os.path.join(d, "heads.npz"))
+ b = laya.Agent(vegaml=True)
+ loaded = b.load_calibration(d)
+ assert loaded["calibrated"] and set(loaded["heads"]) == set(saved["heads"])
+ again = b.predict(rows[0]["state"], {"dept": Q["dept"]})
+ assert again["answers"]["dept"]["probabilities"] == out["answers"]["dept"]["probabilities"]
+ assert again["answers"]["dept"]["route"] == out["answers"]["dept"]["route"]
+
+
[email protected](not MEDIA_TESTS, reason="set LAYA_VEGA_MEDIA_TESTS=1 (pulls the vision tower)")
+def test_image_path(agent):
+ from PIL import Image, ImageDraw
+ img = Image.new("RGB", (320, 200), "white")
+ d = ImageDraw.Draw(img)
+ d.rectangle([20, 20, 300, 180], outline="black", width=4)
+ d.text((40, 80), "INVOICE TOTAL DUE $1,240.00", fill="black")
+ q = {"kind": {"type": "choice", "instructions": "What kind of document is this?",
+ "criteria": {"invoice": "an invoice or bill", "photo": "a photograph",
+ "form": "a blank form"}}}
+ out = agent.predict(image=img, questions=q)
+ assert out["modality"] == "image" and out["answers"]["kind"]["readout"] == "engine"
+ assert out["answers"]["kind"]["choice"] in q["kind"]["criteria"]
+ assert agent.predict_image_text(img, q)["modality"] == "image_withheld"
+
+
+if __name__ == "__main__":
+ sys.exit(pytest.main([__file__, "-q"]))