Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
Agent(vegaml=True): vegaml's engine behind laya's surface
Nandakishor9ae2930f81dd
diff --git a/README.md b/README.mdindex 085f3c10..ac9c4df4 100644--- a/README.md+++ b/README.md@@ -1018,6 +1018,58 @@ for measurements and limitations.---+## vega inside laya: `Agent(vegaml=True)`++One argument swaps the engine under laya's surface for [vegaml](https://github.com/NandhaKishorM/vegaml)'s:+a frozen 0.8B universal reader with a small physics engine on top. Every method keeps its+laya name; the answers gain what the engine knows that an encoder does not.++```python+import laya++agent = laya.Agent(vegaml=True) # pip install 'laya[vegaml]' (Python 3.10+)+out = agent.predict(state, questions)++ans = out["answers"]["department"]+ans["choice"] # as always+ans["prediction_set"] # the conformal set at alpha=0.1+ans["abstain"] # the engine's own refusal+ans["unbound"] # far from every well: likely off-distribution+ans["route"], ans["route_reason"] # "auto" or "review", decided for you+```++| | laya checkpoints | `Agent(vegaml=True)` |+|---|---|---|+| context | 512 or 1,024 tokens, windowed past that | 73k tokens, read whole |+| learning from your labels | gradient fine-tune | `agent.fit(rows)`: a head per question over frozen features, seconds |+| calibration | shipped temperature map | refit on your rows: temperature, conformal threshold, escalation band |+| refusal | `min_confidence` you choose | `abstain`, `unbound`, the conformal set, the fitted band |+| images, speech | no | `agent.predict(image=img, ...)`, `agent.predict(audio=wav, ...)` |+| speed on text (T4) | ~16 ms a decision batched | ~72 ms a decision |++```python+agent.fit(rows) # rows: {"state", "questions", "gold" or "expected"}+agent.predict(state, questions) # heads serve where they beat the engine; route from the band+agent.save_calibration("fit/"); agent.load_calibration("fit/")+agent.predict_long(document, questions) # no windowing up to 73k tokens+agent.predict(image=img, questions=questions) # the reader's vision tower+agent.predict(audio=wav, questions=questions, sampling_rate=16000) # transcribed, then decided++from laya.vega import Router+router = Router(fast=laya.Agent(), vega=agent) # laya for short English text, vega otherwise+router.predict(state, questions)["routing"] # {"model": "fast" | "vega", "reason": ...}+```++**Nothing here needs a threshold from you.** `route` comes from the fitted band when you have+run `fit` or `calibrate`, and otherwise from the engine's own conformal set, abstain flag and+`unbound` flag. `alpha` (miscoverage, 0.1) and `target_precision` (0.95) are targets with+defaults, not confidences to guess at. Gate on `route`, not on `confidence`: a fitted head can+report 1.000 on an answer it gets wrong, and `route` on the same answer is still right, because+it comes from the set and the band rather than the top probability.++`laya.vega` is imported only when the argument is given; without `vegaml` installed, core laya+is unchanged and `Agent(vegaml=True)` says what to install.+## Automated Confidence GatingBecause Laya's probabilities are trained with strictly proper scoring rules (RLCD), confidence scores are statistically meaningful:diff --git a/docs/vegaml.md b/docs/vegaml.mdnew file mode 100644index 00000000..0261912a--- /dev/null+++ b/docs/vegaml.md@@ -0,0 +1,121 @@+# vegaml inside laya: `Agent(vegaml=True)`++One argument swaps the engine under laya's surface for [vegaml](https://github.com/NandhaKishorM/vegaml)'s:+a frozen 0.8B universal reader with a small physics engine on top. Every method keeps its laya+name, and the answers gain what that engine knows that an encoder does not. The class behind it+is `laya.VegaAgent`, in `laya/vega.py`.++```bash+pip install 'laya[vegaml]' # Python 3.10 or newer; core laya stays on 3.9+```++```python+import laya++agent = laya.Agent(vegaml=True) # the 0.8B universal reader+agent = laya.load(vegaml="4b") # a larger reader; or a Hub id, or a local dir++out = agent.predict(state, questions)+ans = out["answers"]["department"]+ans["choice"] # as always+ans["probabilities"], ans["answer_confidence"] # as always: the calibrated one is max p+ans["prediction_set"] # the conformal set at alpha (0.1)+ans["abstain"] # the engine's own refusal+ans["unbound"] # far from every well: likely off-distribution+ans["route"], ans["route_reason"] # "auto" or "review", decided for you+```++## What changes, feature by feature++| | laya checkpoints | `Agent(vegaml=True)` |+|---|---|---|+| reader | ModernBERT encoder, ~150M | frozen decoder reader, 0.8B (or the 4B) |+| context | 512 or 1,024 tokens, `predict_long` windows past that | 73k tokens, read whole |+| learning from your labels | gradient fine-tune | `agent.fit(rows)`: a head per question over the frozen features, seconds |+| calibration | a shipped temperature map | refit on your rows: temperature, conformal threshold, escalation band |+| refusal | `min_confidence` you choose | `abstain`, `unbound`, the conformal set, the fitted band |+| images and speech | no | `predict(image=...)`, `predict(audio=...)` |+| speed on text (T4) | ~16 ms a decision batched | ~72 ms a decision; each served head adds a read |+| memory | under 1 GB | 2 to 4 GB |++## Fit and calibrate++```python+rows = [{"state": "...", "questions": questions, "gold": {"department": "billing"}}, ...]+report = agent.fit(rows) # heads, then temperature + conformal + band, one call+report["serving"] # the questions whose head actually beat the engine+agent.predict(state, questions) # heads serve where they earned it; route from the band+agent.save_calibration("fit/") # heads.npz + calibration.json+agent.load_calibration("fit/") # on a fresh VegaAgent, identical answers and routes+```++Rows are the same `{"state", "questions", "gold" or "expected"}` shape laya's fine-tune reads,+so a corpus prepared for one fits the other unchanged. A head is kept only if it beats chance+under cross-validation and served only if it also beats the engine on the same rows:+`report["heads"][qid]["used_by_auto"]` is the fact to read, not `fitted`.++`agent.calibrate(rows)` (also reachable as `fit_temperatures`) does the calibration half alone.+The argmax is invariant under a temperature, so it moves confidence, sets, `abstain` and+`route`, never which option wins.++## Automatic thresholding++Nothing here needs a confidence number from you. `route` is decided per answer, in this order:++1. A fitted band, if `fit` or `calibrate` has run: its line for that question was placed where+ precision holds at `target_precision` (0.95) on your rows, with a Wilson bound to spare.+2. Otherwise the engine's own machinery, with no labels at all: `review` when it abstains,+ when the answer is `unbound`, or when the conformal set at `alpha` (0.1) holds more than+ one option; `auto` only when all three agree.++`alpha` and `target_precision` are targets with defaults, not thresholds to guess at.+`agent.thresholds()` prints what is deciding right now. laya's `min_confidence` is still+accepted and applied on top when given.++Gate on `route`, not on `confidence`. A fitted head can report 1.000 on an answer it gets wrong;+`route` on the same answer is still right, because it comes from the set and the band rather+than the top probability.++## Images, speech, long documents++```python+agent.predict(image=img, questions=questions) # PIL image, path, or array+agent.predict(audio=wave, questions=questions, sampling_rate=16000) # transcribed, then decided+agent.predict_long(document, questions) # one pass up to 73k tokens+```++The modality is chosen by which argument you passed; the questions and the answer shape are the+same. An audio answer carries `transcript` and `asr`, because a decision made on words you never+saw cannot be audited. `predict_image_text` answers the same questions with the image withheld,+which measures what the picture itself contributed.++## Model switching++```python+from laya.vega import Router++router = Router(fast=laya.Agent(), vega=agent)+out = router.predict(state, questions)+out["routing"] # {"model": "fast" | "vega", "reason": "..."}+```++The rule, in order: an image or audio goes to vega; a state longer than the fast model's context+goes to vega; non-English or non-Latin text goes to vega; a question with a fitted head on vega+goes to vega; everything else goes to the fast encoder. `model=` forces one. The result carries+`routing` in the shape `laya.Router` already returns.++## Your own reader++```python+laya.VegaAgent.attach("Qwen/Qwen3.5-4B", texts, "my-reader/") # no labels needed+agent = laya.Agent(vegaml="my-reader/")+```++## What stays the same, and what is not here++`predict`, `system_one`, `predict_batch`, `predict_long`, `fit_temperatures`,+`save_calibration`, `load_calibration` and the hooks all keep their names. `predict_batch` is a+loop: vegaml packs a state's questions into one pass, not many states. `fit_binning` raises,+because histogram binning is laya's own calibration and on vegaml the calibrated probabilities+come from `calibrate()`. Without the `vegaml` package installed, `laya.vega` is never imported+and `Agent(vegaml=True)` says what to install.diff --git a/laya/__init__.py b/laya/__init__.pyindex 1f1388ec..c9abb582 100644--- a/laya/__init__.py+++ b/laya/__init__.py@@ -44,6 +44,7 @@ _LAZY_ATTRS = {# stdlib-only module, but keep it lazy so `import laya` does not pull in hashlib"PINNED_REVISIONS": (".revisions", "PINNED_REVISIONS"),"Agent": (".agent", "Agent"),+ "VegaAgent": (".vega", "VegaAgent"), # vegaml's engine behind laya's surface"RLAgent": (".agent", "RLAgent"),"load": (".agent", "load"),"fit_temperatures": (".calibrate", "fit_temperatures"),@@ -100,6 +101,7 @@ def __dir__():__all__ = ["Agent","RLAgent",+ "VegaAgent","load","fit_temperatures","fit_one_temperature",diff --git a/laya/agent.py b/laya/agent.pyindex dec08ac0..6cab09cd 100644--- a/laya/agent.py+++ b/laya/agent.py@@ -533,6 +533,17 @@ class Agent(HookRegistry):_reduce_overhead = False_gate = _InferenceGate()+ def __new__(cls, *args, **kwargs):+ # `laya.Agent(vegaml=True)` builds a VegaAgent: vegaml's engine behind laya's surface.+ # The dispatch is here rather than in a factory so that the one argument is the whole+ # change for a caller, and `isinstance(agent, laya.Agent)` stays true either way.+ # VegaAgent overrides __new__ itself, so there is no recursion through this branch.+ vegaml = kwargs.get("vegaml")+ if vegaml is not None and vegaml is not False and cls is Agent:+ from .vega import VegaAgent+ return VegaAgent.__new__(VegaAgent)+ return object.__new__(cls)+def __init__(self,model_id_or_path: str = "convaiinnovations/laya",@@ -555,9 +566,14 @@ class Agent(HookRegistry):compile_warmup: bool = True,compile_cache: bool = False,compile_mode: str = "default",+ vegaml: Union[bool, str, None] = None,):"""Load a Laya checkpoint.+ `vegaml=True` (or a size, Hub id or directory) loads vegaml's engine instead and returns+ a :class:`laya.vega.VegaAgent`; see that module. On this path the argument is accepted+ so that `vegaml=False`/`None` are legal and mean the ordinary checkpoint.+`backend` selects "eager", "auto", "compile" or "tilelang"; see `laya.backends`.It takes precedence over `fast` and `compile`. Omit it to keep those legacy flags.For ONNX use `load(backend="onnx")` instead.@@ -2174,9 +2190,13 @@ def load(model_id_or_path: str = "convaiinnovations/laya", device: Optional[str]hooks_timeout: Optional[float] = None,calibration: Optional[str] = None, backend: Optional[str] = None,onnx_path: Optional[str] = None, compile_warmup: bool = True,- compile_cache: bool = False, compile_mode: str = "default") -> Union[Agent, "ONNXAgent"]:+ compile_cache: bool = False, compile_mode: str = "default",+ vegaml: Union[bool, str, None] = None) -> Union[Agent, "ONNXAgent", "VegaAgent"]:"""Load a Laya agent.+ laya.load(vegaml=True) # vegaml's engine, laya's API+ laya.load(vegaml="4b") # a larger reader+`subfolder` picks one checkpoint out of a repo that bundles several:laya.load("convaiinnovations/laya") # English (repo root)@@ -2200,6 +2220,13 @@ def load(model_id_or_path: str = "convaiinnovations/laya", device: Optional[str]`hooks` / `on_predict_start` / `on_predict_end` observe or shape every prediction; see`laya.hooks`. `calibration` is the same optional JSON path accepted by `Agent`."""+ if vegaml is not None and vegaml is not False:+ # vegaml's engine behind laya's surface; everything laya-specific below (registry names,+ # ONNX, TileLang, torch.compile) is about ModernBERT checkpoints and does not apply.+ from .vega import VegaAgent+ return VegaAgent(model_id_or_path if model_id_or_path != "convaiinnovations/laya" else None,+ device=device, token=token, vegaml=vegaml, revision=revision, hooks=hooks,+ on_predict_start=on_predict_start, on_predict_end=on_predict_end)# A registry name or alias resolves to the same (repo, subfolder) the Router would# pick, instead of being handed to the Hub as a repo id. A caller who spelled out a# subfolder is loading that subfolder of whatever repo they named, so the name is onlydiff --git a/laya/vega.py b/laya/vega.pynew file mode 100644index 00000000..2f97eaa5--- /dev/null+++ b/laya/vega.py@@ -0,0 +1,678 @@+"""vega inside laya: every vega feature, under laya's own names.++ import laya++ agent = laya.Agent(vegaml=True) # the 0.8B universal reader + the engine+ agent = laya.load(vegaml=True) # the same+ agent = laya.Agent(vegaml="4b") # the 4B reader; or a Hub id, or a local dir++ out = agent.predict(state, questions) # laya's shape, plus vega's fields+ out["answers"]["dept"]["choice"] # as always+ out["answers"]["dept"]["prediction_set"] # the conformal set at alpha+ out["answers"]["dept"]["abstain"] # the engine's own refusal+ out["answers"]["dept"]["unbound"] # far from every well: likely off-distribution+ out["answers"]["dept"]["route"] # "auto" or "review", decided for you++ agent.fit(rows, questions) # heads + temperature + conformal + band, one call+ agent.predict(image=img, questions=questions) # pictures, through the reader's vision tower+ agent.predict(audio=wav, questions=questions) # speech, transcribed then decided+ agent.predict_long(very_long_state, questions) # 73k tokens, no windowing+ agent.save_calibration("fit/") # heads and calibration, reloadable+ laya.vega.Router(fast=laya.Agent(), ...) # model switching: laya for short English text,+ # vega for everything else++**What the argument buys.** laya's own checkpoints are ModernBERT encoders: fast, small, text+only, with a temperature-scaled confidence and no notion of "I have not seen anything like+this". ``vegaml=True`` swaps the engine underneath for vegaml's: a frozen decoder reader with a+small physics engine on top, which adds a conformal prediction set, an abstain flag, an+out-of-distribution flag (``unbound``), heads fitted on your labels over the frozen features,+a calibration that is refit on your rows rather than shipped, 73k tokens of context, and the+same typed interface over images and speech. Everything is reachable through the methods laya+already has, so a caller switches by changing one argument.++**What it costs.** A 0.8B reader against a ~150M encoder: roughly five times the latency on+plain text (measured 72 ms a decision against 16 ms batched on a T4), two to four gigabytes+of memory, and Python 3.10 or newer, which vegaml needs. ``pip install laya[vegaml]``. Core laya+is untouched when the argument is not given, which is why this lives in its own module and is+imported only on demand.++**Automatic thresholding, which is the question this module exists to answer.** A caller never+sets a confidence number. ``route`` is decided per answer, in this order:++1. If a band has been fitted (``fit`` or ``calibrate`` ran on labelled rows), the band's own+ line for that question decides: it was placed where precision holds at+ ``target_precision`` (default 0.95) on your rows, with a 95% Wilson bound to spare.+2. Otherwise the engine's own machinery decides, no labels required: ``review`` when it+ abstains at the configured risk, when the answer is ``unbound``, or when the conformal set+ at ``alpha`` (default 0.10) holds more than one option; ``auto`` only when all three agree.++Both are targets, not thresholds: ``alpha`` is the miscoverage the sets are allowed, and+``target_precision`` is the precision the band is held to. Neither is a confidence a human had+to guess at, and both have defaults. The house rule behind this is measured, not stylistic: a+fitted head once returned confidence 1.000 on every one of sixteen held-out records including+both of its mistakes, while ``route`` on the same records was still right, because route comes+from the conformal set and the band rather than from the top-1 probability. **Gate on+``route``, never on ``confidence``.** laya's ``min_confidence`` is still accepted for+compatibility and applied on top when given, but nothing here needs it.++Two vegaml facts this module is built around, both of which pass every test until they+matter: a head-served answer carries no ``abstain`` and no ``prediction_set`` of its own, so+the fitted path gets its route from the band; and ``vegaml.calibrate.calibrate`` returns a+Calibration that nothing applies unless the caller does, so :meth:`predict` applies it on+every call itself.+"""+from __future__ import annotations++import json+import math+import os+import threading+import time+from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple, Union++from .agent import Agent++__all__ = ["VegaAgent", "Router", "is_available", "HEAD_ARRAYS"]++#: what a fitted TTT head is, in full: standardisation, projection and the linear readout.+#: vegaml gives the heads no save of their own, so this module writes these arrays itself.+HEAD_ARRAYS = ("mu", "sd", "c", "P", "s", "W", "b")++_INSTALL = ("vegaml=True needs the vegaml package, which needs Python 3.10 or newer: "+ "pip install 'laya[vegaml]' (or: pip install vegaml)")+++def is_available() -> bool:+ """Whether the vega engine can be imported in this interpreter."""+ try:+ import vegaml # noqa: F401+ except ImportError:+ return False+ return True+++# ----------------------------------------------------------------------------- translation+def _to_vega_questions(questions: Mapping[str, Mapping]) -> Dict[str, Dict]:+ """laya's question map in vegaml's vocabulary.++ The two schemas are the same typed-decision design and differ in one word: laya's yes/no+ is ``noul``, vegaml's is ``boolean``. laya's noul-only ``labels`` (the two sides named)+ becomes vegaml's boolean ``criteria``, which is the same information under the other key.+ """+ out = {}+ for qid, q in questions.items():+ t = q["type"]+ vq: Dict[str, Any] = {"instructions": q["instructions"]}+ if t == "noul":+ vq["type"] = "boolean"+ crit = q.get("criteria")+ lab = q.get("labels")+ if isinstance(crit, dict):+ vq["criteria"] = crit+ elif isinstance(lab, dict):+ vq["criteria"] = {"false": lab.get("false", "no"), "true": lab.get("true", "yes")}+ elif t == "score":+ vq["type"] = "score"+ vq["criteria"] = list(q["criteria"])+ elif t == "choice":+ vq["type"] = "choice"+ crit = q["criteria"]+ vq["criteria"] = (dict(crit) if isinstance(crit, dict)+ else {str(c): None for c in crit})+ else:+ raise ValueError("question %r has unknown type %r (choice, score or noul)" % (qid, t))+ out[qid] = vq+ return out+++def _labels_for(q: Mapping) -> List[str]:+ """The option ids a question's answer spans, in laya's own spelling."""+ t = q["type"]+ if t == "noul":+ return ["false", "true"]+ if t == "score":+ return [str(i) for i in range(len(q["criteria"]))]+ crit = q["criteria"]+ return [str(k) for k in (crit if isinstance(crit, dict) else crit)]+++def _entropy_confidence(probs: Mapping[str, float]) -> float:+ """``1 - H(p)/log(k)``: what laya reports as ``confidence``. Not an error rate.++ Kept so a laya caller reading ``confidence`` sees the quantity laya has always put there;+ the calibrated one is ``answer_confidence`` (max p), on vega exactly as on laya.+ """+ vals = [v for v in probs.values() if v > 0]+ k = len(probs)+ if k < 2:+ return 1.0+ h = -sum(v * math.log(v) for v in vals)+ return round(max(0.0, min(1.0, 1.0 - h / math.log(k))), 4)+++def _gold_label(q: Mapping, gold: Any) -> str:+ """A training row's answer as the label vegaml's fit wants.++ Accepts laya's three shapes for an answer: a bare value, ``{"label": value}`` (the+ typed-decisions dataset), or ``{"probabilities": {...}}`` where the argmax is taken.+ """+ if isinstance(gold, dict):+ if "label" in gold:+ gold = gold["label"]+ elif "probabilities" in gold and gold["probabilities"]:+ gold = max(gold["probabilities"], key=gold["probabilities"].get)+ elif "noul" in gold:+ gold = bool(float(gold["noul"]) >= 0.5)+ t = q["type"]+ if t == "noul":+ s = str(gold).strip().lower()+ return "true" if s in ("true", "yes", "1", "t", "y") else "false"+ if t == "score":+ if isinstance(gold, str) and not gold.isdigit():+ crit = list(q["criteria"])+ return str(crit.index(gold))+ return str(int(gold))+ return str(gold)+++def _fold(answer: Mapping, q: Mapping) -> Dict[str, Any]:+ """One vegaml answer in laya's shape, with vega's extra fields kept.++ The engine keys a boolean's probabilities ``"False"``/``"True"`` and a head keys them+ ``"false"``/``"true"``; laya spells them lower case, so the keys are folded here once and+ scoring code downstream never sees two spellings. Not folding this is how a benchmark read+ every yes/no task at 0.000 for a day.+ """+ t = q["type"]+ probs = {str(k): float(v) for k, v in (answer.get("probs") or {}).items()}+ if t == "noul":+ low = {k.lower(): v for k, v in probs.items()}+ probs = {"false": low.get("false", 0.0), "true": low.get("true", 0.0)}+ pset = answer.get("prediction_set")+ if pset is not None:+ pset = [str(p).lower() if t == "noul" else str(p) for p in pset]+ out: Dict[str, Any] = {+ "type": t,+ "probabilities": {k: round(v, 6) for k, v in probs.items()},+ "answer_confidence": (round(float(answer["confidence"]), 6)+ if answer.get("confidence") is not None+ else (round(max(probs.values()), 6) if probs else None)),+ "confidence": _entropy_confidence(probs) if probs else None,+ "prediction_set": pset,+ "abstain": answer.get("abstain"),+ "unbound": answer.get("unbound"),+ "readout": answer.get("readout", "engine"),+ }+ if t == "noul":+ out["noul"] = round(probs.get("true", 0.0), 6)+ elif t == "score":+ best = max(probs, key=probs.get) if probs else None+ level = answer.get("score")+ out["score"] = int(level) if level is not None else (int(best) if best else None)+ if answer.get("expected_score") is not None:+ out["expected_score"] = round(float(answer["expected_score"]), 4)+ out["legend"] = {str(i): c for i, c in enumerate(q["criteria"])}+ else:+ ch = answer.get("choice")+ out["choice"] = str(ch) if ch is not None else (max(probs, key=probs.get) if probs else None)+ for k in ("settle_steps", "energy", "adapter", "gate", "view_agreement", "cv_accuracy"):+ if k in answer:+ out[k] = answer[k]+ return out+++# ----------------------------------------------------------------------------- the agent+class VegaAgent(Agent):+ """``laya.Agent`` with vegaml's engine underneath. Built by ``laya.Agent(vegaml=...)``.++ Subclasses :class:`laya.Agent` so ``isinstance`` checks, hooks and callers that duck-type+ a laya agent keep working, but it does **not** run ``Agent.__init__``: there is no+ ModernBERT checkpoint to load. Everything a laya caller touches is reimplemented here on+ top of ``vegaml.Vega``.+ """++ _backend = "vega"++ def __new__(cls, *args, **kwargs):+ # Agent.__new__ dispatches here on vegaml=...; coming back through it would recurse.+ return object.__new__(cls)++ def __init__(self, model_id_or_path: Optional[str] = None, device: Optional[str] = None,+ token: Optional[str] = None, *, vegaml: Union[bool, str, None] = True,+ backbone: Optional[str] = None, max_len: Optional[int] = None,+ alpha: float = 0.1, risk: Union[str, float] = "0.1",+ target_precision: float = 0.95, folds: int = 5,+ mode: str = "auto", revision: Optional[str] = None,+ hooks=None, on_predict_start=None, on_predict_end=None, **ignored):+ """Load vega.++ ``vegaml`` is the argument that got us here: ``True`` loads the default 0.8B reader,+ a string is a size (``"4b"``), a Hub repository or a local directory, exactly what+ ``vegaml.load`` takes. ``model_id_or_path`` is accepted for positional compatibility+ with ``laya.Agent`` and used when ``vega`` is just ``True``. ``backbone`` names the+ reader inside a universal checkpoint. ``alpha`` is the conformal miscoverage,+ ``risk`` the abstention risk level the checkpoint was calibrated at, and+ ``target_precision`` what :meth:`fit` holds the band to.+ """+ try:+ import vegaml as _vegaml_pkg+ except ImportError as exc:+ raise ImportError(_INSTALL) from exc+ from .hooks import normalise_hooks++ which = vegaml if isinstance(vegaml, str) and vegaml else model_id_or_path+ self.vega = _vegaml_pkg.load(which, device=device, backbone=backbone, max_len=max_len,+ mode=mode, revision=revision, token=token)+ self.model_id = self.model_id_or_path = which or "nandakishorm/vega-U"+ self.subfolder = backbone+ self.alpha = str(alpha)+ self.risk = str(risk)+ self.target_precision = float(target_precision)+ self.folds = int(folds)+ self.calibration = None # a vegaml Calibration, applied on every predict+ self.fit_report: Dict[str, Any] = {}+ self.hooks = normalise_hooks(hooks, on_predict_start, on_predict_end)+ self._lock = threading.RLock()+ self.device = getattr(self.vega.engine, "device", device)+ cfg = getattr(self.vega.engine, "cfg", {}) or {}+ self.max_len = int(cfg.get("max_len", max_len or 0) or 0)++ # ------------------------------------------------------------------ introspection+ def __repr__(self) -> str:+ heads = sorted(self.vega.heads)+ return "VegaAgent(%s, max_len=%s, heads=%d, calibrated=%s)" % (+ self.model_id, self.max_len, len(heads), self.calibration is not None)++ @property+ def context_tokens(self) -> int:+ """How many tokens one :meth:`predict` reads before truncating. 73k on the shipped+ universal reader; laya's own checkpoints stop at 512 or 1,024 and window past that."""+ return self.max_len++ # ------------------------------------------------------------------ predicting+ def predict(self, state: Any = None, questions: Optional[Mapping] = None,+ lang: Optional[str] = None, *, image: Any = None, audio: Any = None,+ mode: Optional[str] = None, views: int = 1, max_len: Optional[int] = None,+ head_max_len: Optional[int] = None, min_confidence: Optional[float] = None,+ asr: Optional[str] = None, language: Optional[str] = None,+ sampling_rate: Optional[int] = None, hooks=None, on_predict_start=None,+ on_predict_end=None, **ignored) -> Dict[str, Any]:+ """Answer typed questions about text, an image, or a recording, in laya's shape.++ One of ``state``, ``image`` or ``audio``. The modality is chosen by which you passed,+ which is the whole of "modal switching": the same questions, the same answer shape,+ the same route. ``lang`` is accepted for compatibility and recorded; the universal+ reader is multilingual and needs no checkpoint switch.++ ``max_len`` and ``head_max_len`` are laya's option-budget knobs; vega truncates+ options at 32 tokens and instructions at 256 by itself, so they are recorded and not+ applied. ``min_confidence`` is applied on top of the automatic route when given, the+ way laya's gate applies it, and nothing here needs it.+ """+ if questions is None:+ raise ValueError("predict needs `questions`")+ given = sum(x is not None for x in (state, image, audio))+ if given != 1:+ raise ValueError("predict takes exactly one of state=, image=, audio=; got %d" % given)+ if not questions:+ return {"answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0},+ "latency_ms": 0.0, "modality": "text"}+ vq = _to_vega_questions(questions)+ started = time.perf_counter()+ with self._lock:+ if image is not None:+ raw = self.vega.decide_image(image, vq, mode=mode, alpha=self.alpha, risk=self.risk)+ modality = "image"+ elif audio is not None:+ raw = self.vega.decide_audio(audio, vq, asr=asr, language=language,+ sampling_rate=sampling_rate, mode=mode,+ alpha=self.alpha, views=views)+ modality = "audio"+ else:+ raw = self.vega.decide(state, vq, mode=mode, alpha=self.alpha, views=views)+ modality = "text"+ if self.calibration is not None:+ raw = self.calibration.apply(_pad_for_calibration(raw, self.calibration))+ answers = {}+ for qid, q in questions.items():+ a = _fold(raw["answers"][qid], q)+ a.update(self._route(raw["answers"][qid], a, min_confidence))+ answers[qid] = a+ out = {+ "answers": answers,+ "usage": {"input_tokens": int(raw.get("n_tokens") or 0), "output_tokens": 0},+ "latency_ms": round(1000 * (time.perf_counter() - started), 1),+ "modality": modality,+ "mode": raw.get("mode", mode or self.vega.mode),+ "views": raw.get("views", views),+ "calibrated": self.calibration is not None,+ }+ if lang is not None:+ out["lang"] = lang+ for k in ("transcript", "asr"):+ if k in raw:+ out[k] = raw[k]+ return out++ system_one = predict++ def predict_image(self, image: Any, questions: Mapping, **kw) -> Dict[str, Any]:+ return self.predict(image=image, questions=questions, **kw)++ def predict_image_text(self, image: Any, questions: Mapping, **kw) -> Dict[str, Any]:+ """Parity check: the vision path with the image withheld, to see what the picture+ itself contributed."""+ raw = self.vega.decide_image_text(image, _to_vega_questions(questions), **kw)+ return {"answers": {qid: _fold(raw["answers"][qid], q) for qid, q in questions.items()},+ "modality": "image_withheld"}++ def predict_audio(self, audio: Any, questions: Mapping, **kw) -> Dict[str, Any]:+ return self.predict(audio=audio, questions=questions, **kw)++ def predict_batch(self, states: Sequence[Any], questions: Mapping, batch_size: Optional[int] = None,+ lang: Optional[str] = None, **kw) -> List[Dict[str, Any]]:+ """Every state, one decide call each. vega packs a state's questions into one pass,+ not many states, so this is a loop with laya's signature rather than a faster path.+ ``batch_size`` is accepted and ignored."""+ return [self.predict(s, questions, lang=lang, **kw) for s in states]++ def predict_long(self, state: Any, questions: Mapping, window: Optional[int] = None,+ stride: Optional[int] = None, aggregate: str = "auto", **kw) -> Dict[str, Any]:+ """A long state, read whole.++ laya's own checkpoints stop at 512 or 1,024 tokens and this method windows past that.+ The universal reader holds 73k tokens, so the state is read in one pass up to that and+ the per-window aggregation never happens; ``window``, ``stride`` and ``aggregate`` are+ accepted for compatibility and recorded. Past the reader's own limit the engine+ truncates, and ``usage.truncated`` says so, which is the one case where the caller may+ still want to window and vote.+ """+ out = self.predict(state, questions, **kw)+ n = int(out["usage"].get("input_tokens") or 0)+ out["usage"]["windows"] = 1+ out["usage"]["context_tokens"] = self.max_len+ out["usage"]["truncated"] = bool(self.max_len and n >= self.max_len)+ out["long"] = {"window": window, "stride": stride, "aggregate": aggregate,+ "applied": False, "why": "read whole, within %d tokens" % self.max_len}+ return out++ # ------------------------------------------------------------------ the route+ def _route(self, raw_answer: Mapping, folded: Mapping, min_confidence: Optional[float]) -> Dict:+ """``route`` and why, decided without a user-set number. See the module docstring."""+ if "route" in raw_answer and raw_answer["route"] in ("auto", "review"):+ route, why = raw_answer["route"], "band: " + str(raw_answer.get("route_reason", ""))+ else:+ reasons = []+ if folded.get("abstain"):+ reasons.append("the engine abstains at risk %s" % self.risk)+ if folded.get("unbound"):+ reasons.append("unbound: far from every well, likely off-distribution")+ pset = folded.get("prediction_set")+ if pset is not None and len(pset) > 1:+ reasons.append("conformal set at alpha %s holds %d options" % (self.alpha, len(pset)))+ if pset is None and folded.get("abstain") is None:+ # a head-served answer with no band fitted: nothing here can vouch for it+ reasons.append("head answer with no fitted band; fit() or calibrate() first")+ route = "review" if reasons else "auto"+ why = "; ".join(reasons) if reasons else (+ "conformal singleton, not abstaining, in distribution")+ if min_confidence is not None and route == "auto":+ c = folded.get("answer_confidence")+ if c is None or c < float(min_confidence):+ route, why = "review", "min_confidence %.3f not met (%s)" % (min_confidence, c)+ return {"route": route, "route_reason": why}++ def thresholds(self) -> Dict[str, Any]:+ """What decides ``route`` right now, per question, so nothing about it is hidden."""+ out: Dict[str, Any] = {"alpha": self.alpha, "risk": self.risk,+ "target_precision": self.target_precision,+ "policy": ("band" if self.calibration is not None+ else "conformal set + abstain + unbound")}+ if self.calibration is not None:+ band = self.calibration.band+ out["band"] = ({"default": band.default, "questions": band.questions}+ if band is not None else None)+ out["calibration"] = {q: {k: v for k, v in c.items() if k != "labels"}+ for q, c in self.calibration.questions.items()}+ out["heads"] = {q: {"cv_accuracy": cv, "serving": self.vega._head_worth_using(q, {+ "type": "choice", "criteria": {l: None for l in h.labels}})}+ for q, (h, cv) in self.vega.heads.items()}+ return out++ # ------------------------------------------------------------------ fitting+ def _examples(self, rows: Sequence[Mapping], questions: Optional[Mapping]) -> Tuple[list, Dict]:+ """laya training rows as vegaml fit examples: ``(state, {qid: label}, {qid: question})``.++ Rows are ``{"state", "questions"?, "gold"|"expected"}``: the shape ``items_from_rows``+ reads, so a corpus prepared for laya's fine-tune fits vega's heads unchanged. A row+ without its own ``questions`` uses ``questions``.+ """+ examples, sheet = [], {}+ for r in rows:+ qs = r.get("questions") or questions+ if not qs:+ raise ValueError("a row has no questions and none were given")+ vq = _to_vega_questions(qs)+ sheet.update(vq)+ gold = r.get("gold") if r.get("gold") is not None else r.get("expected")+ if not isinstance(gold, Mapping):+ raise ValueError("each row needs `gold` or `expected` as {question_id: answer}")+ labels = {qid: _gold_label(qs[qid], g) for qid, g in gold.items() if qid in qs}+ if labels:+ examples.append((r["state"], labels, vq))+ return examples, sheet++ def fit(self, rows: Sequence[Mapping], questions: Optional[Mapping] = None,+ folds: Optional[int] = None, calibrate: bool = True,+ target_precision: Optional[float] = None) -> Dict[str, Any]:+ """Heads on your labels, then the calibration and the band, in one call.++ What laya's fine-tune does with gradient steps, this does with a small head per+ question over the frozen features: seconds on a GPU for a few hundred rows, and the+ reader is never touched. A head is kept only if it beats chance under cross-validation+ and SERVED only if it also beats the engine on the same rows, so ``report[qid]+ ["used_by_auto"]`` is the fact to read, not ``fitted``. With ``calibrate`` (the+ default) the same rows then refit a temperature and a conformal threshold per+ question and place the band, which is where the automatic ``route`` comes from.+ Nothing is scored on a fitted row by this method; held-out evaluation is yours.+ """+ examples, sheet = self._examples(rows, questions)+ if not examples:+ raise ValueError("no trainable rows")+ with self._lock:+ report = self.vega.fit(examples, sheet, folds=folds or self.folds)+ self.fit_report = {"heads": report, "n_rows": len(examples),+ "serving": sorted(q for q, r in report.items() if r.get("used_by_auto")),+ "kept": sorted(q for q, r in report.items() if r.get("fitted"))}+ if calibrate:+ self.fit_report["calibration"] = self.calibrate(+ rows, questions, target_precision=target_precision, _examples=(examples, sheet))+ return self.fit_report++ def calibrate(self, rows: Sequence[Mapping], questions: Optional[Mapping] = None,+ target_precision: Optional[float] = None, alpha: Optional[float] = None,+ _examples=None) -> Dict[str, Any]:+ """Refit the probabilities and place the band on your labelled rows.++ A temperature per question, a conformal threshold on the corrected probabilities so+ the sets cover at ``1 - alpha`` on rows like yours, and the escalation band at+ ``target_precision``. The argmax is invariant under temperature, so this moves+ confidence, sets, ``abstain`` and ``route`` and never which option wins. Sharper+ answers need :meth:`fit`. Applied on every :meth:`predict` from now on.+ """+ from vegaml.calibrate import calibrate as _calibrate++ examples, sheet = _examples or self._examples(rows, questions)+ tp = self.target_precision if target_precision is None else float(target_precision)+ a = float(self.alpha) if alpha is None else float(alpha)+ with self._lock:+ self.calibration = _calibrate(self.vega, examples, sheet, alpha=a,+ target_precision=tp, readout="auto")+ rep = {q: {k: r.get(k) for k in ("temperature", "qhat", "ece_before", "ece_after",+ "examples") if k in r}+ for q, r in self.calibration.report.items()}+ band = self.calibration.band+ return {"questions": rep, "target_precision": tp, "alpha": a,+ "band": None if band is None else+ {q: round(float(v["threshold"]), 4) for q, v in band.questions.items()}}++ fit_temperatures = calibrate++ # ------------------------------------------------------------------ persistence+ def save_calibration(self, path: str) -> Dict[str, Any]:+ """Heads and calibration to ``path`` (a directory), reloadable with+ :meth:`load_calibration`. vegaml's heads have no save of their own, so the arrays are+ written here; the Calibration is vegaml's own JSON."""+ import numpy as np++ os.makedirs(path, exist_ok=True)+ arrays, meta = {}, {"heads": {}, "model": self.model_id, "alpha": self.alpha,+ "risk": self.risk, "target_precision": self.target_precision}+ for qid, (head, cv) in self.vega.heads.items():+ if not getattr(head, "ready", False):+ continue+ for name in HEAD_ARRAYS:+ arrays["%s::%s" % (qid, name)] = np.asarray(getattr(head, name))+ meta["heads"][qid] = {"labels": [str(x) for x in head.labels],+ "k": int(head.k), "l2": float(head.l2),+ "cv": None if cv is None else float(cv),+ "engine_accuracy": head.engine_accuracy}+ arrays["meta"] = np.array(json.dumps(meta))+ np.savez(os.path.join(path, "heads.npz"), **arrays)+ if self.calibration is not None:+ self.calibration.save(os.path.join(path, "calibration.json"))+ return {"path": path, "heads": sorted(meta["heads"]),+ "calibrated": self.calibration is not None}++ def load_calibration(self, path: str) -> Dict[str, Any]:+ import numpy as np+ from vegaml.calibrate import Calibration+ from vegaml.ttt import TTTHead++ z = np.load(os.path.join(path, "heads.npz"), allow_pickle=False)+ meta = json.loads(str(z["meta"]))+ loaded = []+ for qid, m in meta["heads"].items():+ head = TTTHead(m["labels"], k=m["k"], l2=m["l2"])+ for name in HEAD_ARRAYS:+ setattr(head, name, z["%s::%s" % (qid, name)])+ head.ready = True+ head.engine_accuracy = m.get("engine_accuracy")+ self.vega.heads[qid] = (head, m.get("cv"))+ loaded.append(qid)+ cal_path = os.path.join(path, "calibration.json")+ if os.path.exists(cal_path):+ self.calibration = Calibration.load(cal_path)+ return {"heads": loaded, "calibrated": self.calibration is not None}++ # ------------------------------------------------------------------ readers+ @staticmethod+ def attach(backbone: str, texts: Sequence[str], out_dir: str, **kw) -> Dict[str, Any]:+ """Attach your own frozen reader to the universal engine, from your own text, with no+ labels: writes a model directory that ``laya.Agent(vegaml=out_dir)`` then loads."""+ import vegaml+ return vegaml.attach(backbone, texts, out_dir, **kw)++ # laya's own calibration file is a different artefact; say so rather than mis-save one+ def fit_binning(self, *a, **kw):+ raise NotImplementedError("histogram binning is laya's own calibration; on vega the "+ "calibrated probabilities come from calibrate()")+++def _pad_for_calibration(result: Mapping, cal) -> Dict:+ """Zero-fill any fitted option an answer never mentioned.++ ``Calibration.apply`` indexes every label it was fitted on and KeyErrors on an answer over+ a strict subset, after the fit and the calibration have both succeeded, which is the worst+ place for it. Zero is the correct value: the row did not offer that option.+ """+ out = dict(result)+ out["answers"] = {}+ for qid, ans in result["answers"].items():+ fitted = cal.questions.get(qid)+ if fitted is None or not ans.get("probs"):+ out["answers"][qid] = ans+ continue+ have = {str(k).lower() for k in ans["probs"]}+ missing = [o for o in fitted["labels"] if str(o).lower() not in have]+ if missing:+ ans = dict(ans)+ ans["probs"] = dict(ans["probs"])+ for o in missing:+ ans["probs"][o] = 0.0+ out["answers"][qid] = ans+ return out+++# ----------------------------------------------------------------------------- switching+class Router:+ """Model switching: laya's fast encoder where it is enough, vega where it is not.++ router = from laya.vega import Router+ Router(fast=laya.Agent(), vega=laya.Agent(vegaml=True))+ router.predict(state, questions) # -> picks, and says which under "routing"++ The rule, in order: an ``image`` or ``audio`` goes to vega (the encoder has no eyes or+ ears); a state longer than the fast model's context goes to vega (73k against 512 or+ 1,024); non-English or non-Latin text goes to vega (the universal reader is multilingual,+ and laya's own English checkpoint collapses on non-Latin scripts); a question with a+ fitted head on vega goes to vega (that is what the labels bought); everything else goes to+ the fast model. ``model=`` forces one. The result carries ``routing = {"model", "reason"}``+ in the shape ``laya.Router`` already returns, so a caller switching from that class reads+ nothing new.+ """++ def __init__(self, fast: Optional[Agent] = None, vega: Optional[VegaAgent] = None,+ fast_context_tokens: int = 512, prefer_vega_when_fitted: bool = True):+ self.fast = fast+ self._vega = vega+ self.fast_context_tokens = int(fast_context_tokens)+ self.prefer_vega_when_fitted = prefer_vega_when_fitted++ @property+ def vega(self) -> VegaAgent:+ if self._vega is None:+ self._vega = VegaAgent()+ return self._vega++ def choose(self, state: Any = None, questions: Optional[Mapping] = None, *, image=None,+ audio=None, lang: Optional[str] = None) -> Tuple[str, str]:+ from .lang import is_english++ if image is not None or audio is not None:+ return "vega", "image or audio: the encoder has no vision or speech path"+ if self.fast is None:+ return "vega", "no fast model attached"+ text = state if isinstance(state, str) else json.dumps(state, ensure_ascii=False)+ if len(text) > self.fast_context_tokens * 4: # ~4 characters a token+ return "vega", "state is longer than the fast model's %d-token context" % \+ self.fast_context_tokens+ if lang is not None:+ english = str(lang).lower() in ("en", "eng", "english")+ else:+ english = bool(is_english(text))+ if not english:+ return "vega", "non-English or non-Latin text; the universal reader is multilingual"+ if self.prefer_vega_when_fitted and questions and self._vega is not None:+ fitted = [q for q in questions if q in self._vega.vega.heads]+ if fitted:+ return "vega", "fitted heads exist for %s" % ", ".join(fitted)+ return "fast", "short English text, no fitted head: the fast encoder is enough"++ def predict(self, state: Any = None, questions: Optional[Mapping] = None, *,+ model: Optional[str] = None, image=None, audio=None, lang: Optional[str] = None,+ **kw) -> Dict[str, Any]:+ if model is None:+ model, reason = self.choose(state, questions, image=image, audio=audio, lang=lang)+ else:+ reason = "explicit model=%s" % model+ if model == "vega":+ out = self.vega.predict(state, questions, lang=lang, image=image, audio=audio, **kw)+ elif model == "fast":+ if image is not None or audio is not None:+ raise ValueError("the fast laya model takes text only; drop model= or use vega")+ out = self.fast.predict(state, questions, lang=lang, **kw)+ else:+ raise ValueError("model must be 'fast' or 'vega', got %r" % model)+ out["routing"] = {"model": model, "reason": reason}+ return outdiff --git a/pyproject.toml b/pyproject.tomlindex 5dd235c6..3e3620d2 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -49,6 +49,12 @@ fast = ["tilelang>=0.1.14"]# `pip install laya[mcp]` adds the MCP stdio server (laya.mcp / the laya-mcp-server CLI).mcp = ["mcp>=2.2.0"]+# `pip install laya[vegaml]` puts vega's engine behind laya's surface: `laya.Agent(vegaml=True)`.+# Conformal sets, abstain and unbound flags, heads fitted on your labels, a refit calibration,+# 73k tokens of context, images and speech, all through laya's own method names (laya/vega.py).+# vegaml needs Python 3.10 or newer; core laya does not, which is why this is an extra.+vegaml = ["vegaml>=0.15.0; python_version >= '3.10'"]+# `pip install laya[structured]` adds pydantic for the typed-model helpers (laya/structured.py).structured = ["pydantic>=2.0.0"]diff --git a/scripts/test_suites.py b/scripts/test_suites.pyindex 0870e0ec..64b6308f 100644--- a/scripts/test_suites.py+++ b/scripts/test_suites.py@@ -127,6 +127,7 @@ PYTEST_SUITES = ["tests/test_truncation_direction.py","tests/test_compile.py","tests/test_finetune_entrypoints.py",+ "tests/test_vega.py",]diff --git a/tests/test_audit_scope.py b/tests/test_audit_scope.pyindex 53f79d19..fe773e86 100644--- a/tests/test_audit_scope.py+++ b/tests/test_audit_scope.py@@ -87,14 +87,15 @@ if len(od) > 1:check_true("pyproject/core parses", len(core) == 5, core)check("pyproject/extra tables found", sorted(extras),["agno", "crewai", "fast", "langchain", "langgraph", "llamaindex", "mcp", "onnx", "serve",- "structured"])+ "structured", "vegaml"])declared = list(core) + [s for names in extras.values() for s in names]expected = sorted(set(declared))check("pyproject/extras add names the core does not have",sorted({re.split(r"[<>=!;\[ ]", s)[0] for s in declared}- {re.split(r"[<>=!;\[ ]", s)[0] for s in core}),["agno", "crewai", "fastapi", "langchain-core", "langgraph", "llama-index-core", "mcp", "onnx",- "onnxruntime", "onnxscript", "pydantic", "python-multipart", "tilelang", "uvicorn"])+ "onnxruntime", "onnxscript", "pydantic", "python-multipart", "tilelang", "uvicorn",+ "vegaml"])# The two scopes: shipped (blocking) is the core plus [serve]; everything else is advisory.shipped = sorted(set(core) | set(extras.get("serve", [])))diff --git a/tests/test_confidence.py b/tests/test_confidence.pyindex 33f55c93..57b603ef 100644--- a/tests/test_confidence.py+++ b/tests/test_confidence.py@@ -2240,7 +2240,7 @@ for _fn in (test_tl_kernels_drops_the_wrong_row_claim,# --------------------------------------- DecisionResult's docstring must split confidence by question type# `laya/structured.py`'s `DecisionResult` closed with "confidence keeps the **normalized-entropy# value** it has always had, because that is a different quantity on a scale that depends on the-# label count". `Agent._decode_answers` (`laya/agent.py:1394-1423`) uses two different formulas:+# label count". `Agent._decode_answers` (`laya/agent.py:1408-1437`) uses two different formulas:# * `choice` and `score` -> `confidence_from_probs(p, k)` = `1 - H(p) / log(k)`# * `noul` -> `max(p_true, 1 - p_true)` (its own comment: "identical here:# over two options `max(p_true, 1 - p_true)` is `max(p)`")diff --git a/tests/test_download.py b/tests/test_download.pyindex 74d83fb0..5c907c24 100644--- a/tests/test_download.py+++ b/tests/test_download.py@@ -353,7 +353,8 @@ class DownloadTests(unittest.TestCase):# reports and takes no override; a different asymmetry, with its# own fix.not_for_onnxruntime = {"fast", "compile", "device", "backend",- "compile_warmup", "compile_cache", "compile_mode"}+ "compile_warmup", "compile_cache", "compile_mode",+ "vegaml"} # selects a different engine, like backendagent_side = (set(inspect.signature(Agent.__init__).parameters)- {"self", "model_id_or_path"} - not_for_onnxruntime)onnx_side = set(inspect.signature(ONNXAgent.__init__).parameters) - {"self"}diff --git a/tests/test_vega.py b/tests/test_vega.pynew file mode 100644index 00000000..69177dec--- /dev/null+++ b/tests/test_vega.py@@ -0,0 +1,337 @@+"""`laya.Agent(vegaml=True)`: vega's engine behind laya's surface (laya/vega.py).++Two layers, so the suite is useful on every machine that runs it:++* **Weight-free**, always on: the question translation, the gold-label mapping, the answer+ fold (the "False"/"True" key folding that once scored every yes/no task at 0.000), the+ calibration padding, the automatic route policy, the Router's switching rule, and the+ dispatch in `Agent.__new__` / `laya.load`, including the error a 3.9 interpreter gets.+ These run without vegaml, torch weights, or the network.+* **Model-backed**, only when `LAYA_VEGA_TESTS=1` and vegaml is importable: the real 0.8B+ universal reader answering text, the fitted path end to end (fit, calibrate, save, reload,+ identical answers and routes), and the band taking over the route. `LAYA_VEGA_MEDIA_TESTS=1`+ adds the image path, which pulls the vision tower.++Run: python -m pytest tests/test_vega.py -q+"""+import json+import math+import os+import sys+import tempfile++import pytest++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))++import laya # noqa: E402+from laya import vega as lv # noqa: E402++VEGA = lv.is_available()+MODEL_TESTS = VEGA and os.environ.get("LAYA_VEGA_TESTS") == "1"+MEDIA_TESTS = MODEL_TESTS and os.environ.get("LAYA_VEGA_MEDIA_TESTS") == "1"++Q = {+ "dept": {"type": "choice", "instructions": "Which department should handle this?",+ "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages",+ "other": "everything else"}},+ "urgency": {"type": "score", "instructions": "How urgent is this?",+ "criteria": ["not urgent", "soon", "blocking"]},+ "churn": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?",+ "labels": {"false": "stays", "true": "threatens to leave"}},+}+STATE = "We were billed twice for March. Refund the duplicate today or we cancel our plan."+++# ----------------------------------------------------------------------------- translation+def test_questions_translate_into_vegaml_vocabulary():+ vq = lv._to_vega_questions(Q)+ assert vq["churn"]["type"] == "boolean"+ assert vq["churn"]["criteria"] == {"false": "stays", "true": "threatens to leave"}+ assert vq["urgency"] == {"instructions": Q["urgency"]["instructions"], "type": "score",+ "criteria": ["not urgent", "soon", "blocking"]}+ assert vq["dept"]["type"] == "choice" and vq["dept"]["criteria"] == Q["dept"]["criteria"]+++def test_list_form_choice_and_unknown_type():+ vq = lv._to_vega_questions({"k": {"type": "choice", "instructions": "x", "criteria": ["a", "b"]}})+ assert vq["k"]["criteria"] == {"a": None, "b": None}+ with pytest.raises(ValueError):+ lv._to_vega_questions({"k": {"type": "ranking", "instructions": "x"}})+++def test_labels_for_each_type():+ assert lv._labels_for(Q["churn"]) == ["false", "true"]+ assert lv._labels_for(Q["urgency"]) == ["0", "1", "2"]+ assert lv._labels_for(Q["dept"]) == ["billing", "technical", "other"]+++def test_gold_label_accepts_every_laya_answer_shape():+ assert lv._gold_label(Q["churn"], True) == "true"+ assert lv._gold_label(Q["churn"], {"label": "yes"}) == "true"+ assert lv._gold_label(Q["churn"], {"noul": 0.2}) == "false"+ assert lv._gold_label(Q["urgency"], 2) == "2"+ assert lv._gold_label(Q["urgency"], "blocking") == "2"+ assert lv._gold_label(Q["dept"], {"probabilities": {"billing": 0.1, "technical": 0.9}}) == "technical"+ assert lv._gold_label(Q["dept"], {"label": "other"}) == "other"+++# ----------------------------------------------------------------------------- the fold+def _engine_answer(**kw):+ base = {"confidence": 0.9, "prediction_set": None, "abstain": False, "unbound": False,+ "readout": "engine"}+ base.update(kw)+ return base+++def test_fold_lowercases_the_engines_boolean_keys():+ # the engine keys a boolean's probabilities "False"/"True"; laya spells them lower case+ a = lv._fold(_engine_answer(probs={"False": 0.94, "True": 0.06}, boolean=0.06,+ prediction_set=[False]), Q["churn"])+ assert set(a["probabilities"]) == {"false", "true"}+ assert a["noul"] == pytest.approx(0.06)+ assert a["prediction_set"] == ["false"]+ assert a["answer_confidence"] == pytest.approx(0.9)+ assert a["type"] == "noul"+++def test_fold_choice_and_score_carry_laya_fields():+ c = lv._fold(_engine_answer(probs={"billing": 0.7, "technical": 0.2, "other": 0.1},+ choice="billing", prediction_set=["billing"]), Q["dept"])+ assert c["choice"] == "billing" and c["prediction_set"] == ["billing"]+ s = lv._fold(_engine_answer(probs={"0": 0.1, "1": 0.8, "2": 0.1}, score=1,+ expected_score=1.0), Q["urgency"])+ assert s["score"] == 1 and s["legend"] == {"0": "not urgent", "1": "soon", "2": "blocking"}+ assert s["expected_score"] == 1.0+++def test_fold_confidence_is_entropy_and_answer_confidence_is_max_p():+ a = lv._fold(_engine_answer(probs={"a": 0.5, "b": 0.5}, choice="a"),+ {"type": "choice", "instructions": "x", "criteria": {"a": None, "b": None}})+ assert a["confidence"] == pytest.approx(0.0, abs=1e-6) # an even split has no information+ assert a["answer_confidence"] == pytest.approx(0.9) # what the engine reported+ assert lv._entropy_confidence({"a": 1.0, "b": 0.0}) == 1.0+++# ----------------------------------------------------------------------------- padding+class _FakeCalibration:+ def __init__(self, labels):+ self.questions = {"q": {"labels": labels, "temperature": 1.0, "qhat": 0.4}}+++def test_padding_fills_fitted_options_the_answer_never_mentioned():+ cal = _FakeCalibration(list("ABCDEFGHI"))+ narrow = {"answers": {"q": {"probs": {"A": 0.7, "B": 0.2, "C": 0.1}}}}+ out = lv._pad_for_calibration(narrow, cal)+ assert set(out["answers"]["q"]["probs"]) == set("ABCDEFGHI")+ assert out["answers"]["q"]["probs"]["G"] == 0.0+ assert out["answers"]["q"]["probs"]["A"] == 0.7+ assert narrow["answers"]["q"]["probs"].keys() == {"A", "B", "C"} # input untouched+ wide = {"answers": {"q": {"probs": {k: 1 / 9 for k in "ABCDEFGHI"}}}}+ assert lv._pad_for_calibration(wide, cal)["answers"]["q"] is wide["answers"]["q"]+++# ----------------------------------------------------------------------------- the route+def _bare_agent():+ """A VegaAgent with no engine, enough to exercise `_route`."""+ a = object.__new__(lv.VegaAgent)+ a.alpha, a.risk = "0.1", "0.1"+ return a+++def test_route_is_auto_only_when_everything_agrees():+ a = _bare_agent()+ folded = {"abstain": False, "unbound": False, "prediction_set": ["x"], "answer_confidence": 0.8}+ r = a._route({}, folded, None)+ assert r["route"] == "auto" and "singleton" in r["route_reason"]++[email protected]("field, value, word", [+ ("abstain", True, "abstains"),+ ("unbound", True, "unbound"),+ ("prediction_set", ["x", "y"], "holds 2 options"),+])+def test_route_reviews_on_each_engine_signal(field, value, word):+ a = _bare_agent()+ folded = {"abstain": False, "unbound": False, "prediction_set": ["x"], "answer_confidence": 0.8}+ folded[field] = value+ r = a._route({}, folded, None)+ assert r["route"] == "review" and word in r["route_reason"]+++def test_route_prefers_a_fitted_band_and_reviews_unvouched_heads():+ a = _bare_agent()+ banded = a._route({"route": "review", "route_reason": "below the line"},+ {"abstain": False, "unbound": False, "prediction_set": ["x"]}, None)+ assert banded["route"] == "review" and banded["route_reason"].startswith("band:")+ # a head answer carries no abstain and no set; with no band nothing can vouch for it+ head = a._route({}, {"abstain": None, "unbound": None, "prediction_set": None,+ "answer_confidence": 0.99}, None)+ assert head["route"] == "review" and "no fitted band" in head["route_reason"]+++def test_min_confidence_is_applied_on_top_only_when_given():+ a = _bare_agent()+ folded = {"abstain": False, "unbound": False, "prediction_set": ["x"], "answer_confidence": 0.8}+ assert a._route({}, folded, 0.7)["route"] == "auto"+ r = a._route({}, folded, 0.9)+ assert r["route"] == "review" and "min_confidence" in r["route_reason"]+++# ----------------------------------------------------------------------------- switching+class _StubAgent:+ def __init__(self, name):+ self.name = name+ self.vega = type("V", (), {"heads": {}})()++ def predict(self, state=None, questions=None, **kw):+ return {"answers": {q: {"choice": self.name} for q in (questions or {})}}+++def test_router_rule_in_order():+ fast, slow = _StubAgent("fast"), _StubAgent("vega")+ r = lv.Router(fast=fast, vega=slow, fast_context_tokens=8)+ q = {"dept": Q["dept"]}+ assert r.choose("I was charged twice", q)[0] == "fast"+ assert r.choose("I was charged twice", q, image=object())[0] == "vega"+ assert r.choose("x " * 100, q)[0] == "vega" # longer than 8 tokens+ assert r.choose("मुझे दो बार चार्ज किया गया", q)[0] == "vega" # non-Latin script+ assert r.choose("I was charged twice", q, lang="de")[0] == "vega"+ slow.vega.heads["dept"] = (None, 0.9)+ assert r.choose("I was charged twice", q)[0] == "vega" # a fitted head wins+ out = r.predict("I was charged twice", q, model="fast")+ assert out["routing"] == {"model": "fast", "reason": "explicit model=fast"}+ assert out["answers"]["dept"]["choice"] == "fast"+ with pytest.raises(ValueError):+ r.predict(questions=q, image=object(), model="fast")+++def test_router_without_a_fast_model_always_picks_vega():+ r = lv.Router(fast=None, vega=_StubAgent("vega"))+ assert r.choose("hello", {"dept": Q["dept"]})[0] == "vega"+++# ----------------------------------------------------------------------------- dispatch+def test_agent_new_does_not_divert_without_the_argument():+ for value in (None, False):+ obj = laya.Agent.__new__(laya.Agent, vegaml=value)+ assert type(obj) is laya.Agent++[email protected](VEGA, reason="vegaml is installed here; the ImportError path needs it absent")+def test_without_vegaml_the_argument_fails_with_the_install_hint():+ with pytest.raises(ImportError, match=r"laya\[vegaml\]"):+ laya.Agent(vegaml=True)+ with pytest.raises(ImportError, match=r"laya\[vegaml\]"):+ laya.load(vegaml=True)+++def test_vega_agent_is_exported_and_is_an_agent_subclass():+ assert laya.VegaAgent is lv.VegaAgent+ assert issubclass(lv.VegaAgent, laya.Agent)+ assert "VegaAgent" in laya.__all__+++# ----------------------------------------------------------------------------- with weights+needs_model = pytest.mark.skipif(+ not MODEL_TESTS, reason="set LAYA_VEGA_TESTS=1 with vegaml installed (downloads the 0.8B reader)")++[email protected](scope="module")+def agent():+ if not MODEL_TESTS:+ pytest.skip("model tests off")+ return laya.Agent(vegaml=True)+++@needs_model+def test_dispatch_builds_a_vega_agent(agent):+ assert type(agent) is lv.VegaAgent and isinstance(agent, laya.Agent)+ assert type(laya.load(vegaml=True)) is lv.VegaAgent+ assert agent.context_tokens >= 8192+++@needs_model+def test_text_answers_carry_laya_and_vega_fields(agent):+ out = agent.predict(STATE, Q)+ for qid, q in Q.items():+ a = out["answers"][qid]+ assert set(lv._labels_for(q)) == set(a["probabilities"])+ assert a["route"] in ("auto", "review") and a["route_reason"]+ assert a["abstain"] in (True, False) and a["unbound"] in (True, False)+ assert a["prediction_set"] is None or isinstance(a["prediction_set"], list)+ assert 0.0 <= a["answer_confidence"] <= 1.0+ assert "choice" in out["answers"]["dept"] and "score" in out["answers"]["urgency"]+ assert "noul" in out["answers"]["churn"]+ assert out["usage"]["input_tokens"] > 0 and out["modality"] == "text"+ assert agent.predict(STATE, {}) == {"answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0},+ "latency_ms": 0.0, "modality": "text"}+ with pytest.raises(ValueError):+ agent.predict(STATE, Q, image="x.png")+++@needs_model+def test_batch_long_and_thresholds(agent):+ assert len(agent.predict_batch([STATE, "The app crashes on login."], Q)) == 2+ lg = agent.predict_long(STATE * 20, Q)+ assert lg["usage"]["windows"] == 1 and lg["long"]["applied"] is False+ assert lg["usage"]["input_tokens"] > 500+ assert agent.thresholds()["policy"] == "conformal set + abstain + unbound"+++def _synthetic_rows(n=24):+ """Labelled rows a head can learn: two departments, two kinds of complaint."""+ rows = []+ bills = ["Charged twice for %s, refund the duplicate." % m for m in+ ("March", "April", "May", "June", "July", "August", "September", "October",+ "November", "December", "January", "February")]+ bugs = ["The app crashes when I open %s." % s for s in+ ("settings", "the inbox", "a report", "the dashboard", "search", "billing history",+ "my profile", "notifications", "the export page", "the login screen", "the map",+ "attachments")]+ for i in range(n // 2):+ rows.append({"state": bills[i], "questions": {"dept": Q["dept"]}, "gold": {"dept": "billing"}})+ rows.append({"state": bugs[i], "questions": {"dept": Q["dept"]}, "expected": {"dept": "technical"}})+ return rows+++@needs_model+def test_fit_calibrate_save_reload_roundtrip():+ a = laya.Agent(vegaml=True)+ rows = _synthetic_rows()+ rep = a.fit(rows)+ assert rep["n_rows"] == len(rows) and "dept" in rep["heads"]+ assert "calibration" in rep and rep["calibration"]["target_precision"] == 0.95+ assert a.calibration is not None and a.thresholds()["policy"] == "band"+ out = a.predict(rows[0]["state"], {"dept": Q["dept"]})+ assert out["calibrated"] is True+ assert out["answers"]["dept"]["route_reason"].startswith("band:")+ with tempfile.TemporaryDirectory() as d:+ saved = a.save_calibration(d)+ assert saved["calibrated"] and os.path.exists(os.path.join(d, "heads.npz"))+ b = laya.Agent(vegaml=True)+ loaded = b.load_calibration(d)+ assert loaded["calibrated"] and set(loaded["heads"]) == set(saved["heads"])+ again = b.predict(rows[0]["state"], {"dept": Q["dept"]})+ assert again["answers"]["dept"]["probabilities"] == out["answers"]["dept"]["probabilities"]+ assert again["answers"]["dept"]["route"] == out["answers"]["dept"]["route"]++[email protected](not MEDIA_TESTS, reason="set LAYA_VEGA_MEDIA_TESTS=1 (pulls the vision tower)")+def test_image_path(agent):+ from PIL import Image, ImageDraw+ img = Image.new("RGB", (320, 200), "white")+ d = ImageDraw.Draw(img)+ d.rectangle([20, 20, 300, 180], outline="black", width=4)+ d.text((40, 80), "INVOICE TOTAL DUE $1,240.00", fill="black")+ q = {"kind": {"type": "choice", "instructions": "What kind of document is this?",+ "criteria": {"invoice": "an invoice or bill", "photo": "a photograph",+ "form": "a blank form"}}}+ out = agent.predict(image=img, questions=q)+ assert out["modality"] == "image" and out["answers"]["kind"]["readout"] == "engine"+ assert out["answers"]["kind"]["choice"] in q["kind"]["criteria"]+ assert agent.predict_image_text(img, q)["modality"] == "image_withheld"+++if __name__ == "__main__":+ sys.exit(pytest.main([__file__, "-q"]))