Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(integrations): forward lang and min_confidence through the framework wrappers

Aashish7f214109fe9a
diff --git a/docs/crewai.md b/docs/crewai.md
index 0c196b22..da66f11f 100644
--- a/docs/crewai.md
+++ b/docs/crewai.md
@@ -5,7 +5,7 @@ Laya provides sub-35ms, non-autoregressive decision components for **CrewAI** mu
* **`LayaCrewRouter`**: Sub-35ms task delegation router replacing LLM managers in hierarchical crews.
* **`LayaTaskGuard`**: Pre-execution task guardrail screening prompts and instructions for jailbreaks, injections, and policy violations.
-Both take core's per-call decision controls -- the two token budgets (`max_len`, `head_max_len`) and the five prediction-hook arguments (`hooks`, `on_predict_start`, `on_predict_end`, `hooks_raise`, `hooks_timeout`) -- see [Per-call decision controls](#5-per-call-decision-controls).
+Both take core's per-call decision controls -- the two token budgets (`max_len`, `head_max_len`), the language and abstention controls (`lang`, `min_confidence`) and the five prediction-hook arguments (`hooks`, `on_predict_start`, `on_predict_end`, `hooks_raise`, `hooks_timeout`) -- see [Per-call decision controls](#5-per-call-decision-controls).
Supports both **local in-process inference** (`Agent` or `Router`) and **remote HTTP inference** against your own `laya-serve` instance without requiring PyTorch on edge clients.
@@ -129,7 +129,8 @@ The remote client uses Python's standard library `urllib` with zero heavy depend
## 5. Per-call decision controls
`LayaCrewRouter` and `LayaTaskGuard` take the same per-call arguments the core API does: the two
-token budgets (`max_len`, `head_max_len`) and the five prediction-hook arguments (`hooks`,
+token budgets (`max_len`, `head_max_len`), the language and abstention controls (`lang`,
+`min_confidence`), and the five prediction-hook arguments (`hooks`,
`on_predict_start`, `on_predict_end`, `hooks_raise`, `hooks_timeout`). They are per instance, so a
crew with a large roster can be given room while the rest of the pipeline keeps the checkpoint's
defaults.
@@ -175,3 +176,21 @@ integration](langchain.md#7-widening-the-token-budget-for-many-options) for that
executes inside `predict`, and no wire format carries it. Install hooks in the process that runs
inference. The two budgets do travel to a remote node, in the request body, up to its
`LAYA_MAX_TOKEN_BUDGET` ceiling; a larger value comes back as a 422.
+
+### Language and abstention
+
+`lang` pins the language the task is routed and answered in -- selecting the answering
+checkpoint's per-language calibration instead of relying on built-in detection -- and
+`min_confidence` is core's abstention gate: a decision under it comes back as an abstention rather
+than a forced delegation. Both are read by `Agent.predict` and `Router.predict` alike and accepted by
+`laya-serve` in the request body, so a router or guard forwards them on the local and the remote
+path. An unset one is omitted, not sent as `None`, so it cannot shadow the deployment's own default;
+`min_confidence=0.0` and `lang=""` are real values and are forwarded as given.
+
+```python
+router = LayaCrewRouter(
+ confidence_threshold=0.80,
+ lang="fr", # route a French-language crew in French
+ min_confidence=0.3, # abstain on a delegation the model is not sure about
+)
+```
diff --git a/docs/langchain.md b/docs/langchain.md
index da3b9655..3ca5d902 100644
--- a/docs/langchain.md
+++ b/docs/langchain.md
@@ -8,7 +8,7 @@ Laya provides fast, non-autoregressive decision components for **LangChain** and
* **`LayaEvaluator`**: Rubric-based output grading and hallucination evaluation.
* **`LayaDecision`**: Schema-driven decisions -- a JSON schema or pydantic model in, schema-shaped values out.
-Every node also takes core's five per-call prediction-hook arguments (`hooks`, `on_predict_start`, `on_predict_end`, `hooks_raise`, `hooks_timeout`).
+Every node also takes core's per-call decision controls -- the two token budgets (`max_len`, `head_max_len`), the language and abstention controls (`lang`, `min_confidence`), and the five prediction-hook arguments (`hooks`, `on_predict_start`, `on_predict_end`, `hooks_raise`, `hooks_timeout`).
Supports both **local in-process inference** (`Agent` or `Router`) and **remote HTTP inference** against your own `laya-serve` without requiring PyTorch on edge clients.
@@ -351,6 +351,28 @@ checkpoint's defaults, which is the point of not raising `agent.cfg["head_max_le
request body, and `laya-serve` applies them up to its `LAYA_MAX_TOKEN_BUDGET` ceiling (8192 by
default); a larger value comes back as a 422.
+### Language and abstention
+
+Every runnable also takes `lang` and `min_confidence`, the two per-request controls `Agent.predict`
+and `Router.predict` both read and `laya-serve` both accepts in the body. `lang` pins the language
+the state is routed and answered in -- select the answering checkpoint's per-language calibration
+rather than relying on built-in detection -- and `min_confidence` is core's abstention gate: a
+decision under it comes back as an abstention instead of a forced choice. Both are forwarded on the
+local and the remote path, and an unset one is omitted rather than sent as `None` so it cannot
+shadow a checkpoint's own default. `min_confidence=0.0` and `lang=""` are real values, not absences,
+and are forwarded as given.
+
+```python
+router = LayaRouter(
+ criteria={"billing": "invoices", "tech": "bugs"},
+ lang="es", # route and answer in Spanish
+ min_confidence=0.3, # abstain below a 0.3 calibrated confidence
+)
+```
+
+Unlike `task` and `lang_guess` -- Router-only routing keywords a direct `Agent.predict` rejects --
+these two are safe on every runner and every deployment.
+
---
## 8. Prediction Hooks on a Single Node
diff --git a/docs/llamaindex.md b/docs/llamaindex.md
index f75eea4a..d543dd12 100644
--- a/docs/llamaindex.md
+++ b/docs/llamaindex.md
@@ -145,7 +145,8 @@ The remote client uses Python's standard library `urllib` with zero heavy depend
## 6. Per-call decision controls
`LayaSingleSelector`, `LayaMultiSelector` and `LayaQueryRouter` take the same per-call arguments the
-core API does: the two token budgets (`max_len`, `head_max_len`) and the five prediction-hook
+core API does: the two token budgets (`max_len`, `head_max_len`), the language and abstention
+controls (`lang`, `min_confidence`), and the five prediction-hook
arguments (`hooks`, `on_predict_start`, `on_predict_end`, `hooks_raise`, `hooks_timeout`). They are
per selector, so a wide routing step can be given room while the rest of the pipeline keeps the
checkpoint's defaults.
@@ -185,3 +186,21 @@ integration](langchain.md#7-widening-the-token-budget-for-many-options) for that
that runs inside `predict`, and no wire format carries it. Install hooks in the process that runs
inference. The two budgets do travel to a remote node, in the request body, up to its
`LAYA_MAX_TOKEN_BUDGET` ceiling; a larger value comes back as a 422.
+
+### Language and abstention
+
+`lang` pins the language the query is routed and answered in -- selecting the answering
+checkpoint's per-language calibration instead of relying on built-in detection -- and
+`min_confidence` is core's abstention gate: a decision under it comes back as an abstention rather
+than a forced selection. Both are read by `Agent.predict` and `Router.predict` alike and accepted by
+`laya-serve` in the request body, so a selector forwards them on the local and the remote path. An
+unset one is omitted, not sent as `None`, so it cannot shadow the deployment's own default;
+`min_confidence=0.0` and `lang=""` are real values and are forwarded as given.
+
+```python
+selector = LayaSingleSelector(
+ instructions="Which tool or query engine is best suited to answer this query?",
+ lang="de", # answer German queries in German
+ min_confidence=0.3, # abstain when no tool clears a 0.3 confidence
+)
+```
diff --git a/laya/integrations/_controls.py b/laya/integrations/_controls.py
index a8b1caaa..999f5ce4 100644
--- a/laya/integrations/_controls.py
+++ b/laya/integrations/_controls.py
@@ -18,6 +18,14 @@ from typing import Any, Dict, Optional
# laya-serve accepts in the request body up to its `LAYA_MAX_TOKEN_BUDGET` ceiling.
PREDICT_CONTROLS = ("max_len", "head_max_len")
+# The per-call language and abstention controls. Unlike `task` / `lang_guess` -- Router-only
+# routing keywords a direct `Agent.predict` does not accept -- both `lang` and `min_confidence`
+# are read by `Agent.predict`/`system_one` AND `Router.predict`, and laya-serve takes each in the
+# request body, so the same kwargs are safe on the local and the remote path alike. `lang` routes
+# non-English text and selects the answering checkpoint's per-language calibration;
+# `min_confidence` is core's abstention gate (#361).
+DECISION_CONTROLS = ("lang", "min_confidence")
+
# The per-call hook family. These are Python callables and flags that run inside `predict`, so
# they exist only on the local path -- see `reject_remote_hooks`.
HOOK_CONTROLS = ("hooks", "on_predict_start", "on_predict_end", "hooks_raise", "hooks_timeout")
@@ -35,13 +43,29 @@ def budget_kwargs(max_len: Optional[int] = None,
if value is not None}
+def decision_kwargs(lang: Optional[str] = None,
+ min_confidence: Optional[float] = None) -> Dict[str, Any]:
+ """The language and abstention overrides, with the unset ones omitted.
+
+ Absent rather than `None`: a `None` `lang` would shadow the checkpoint's own detection and a
+ `None` `min_confidence` would override the runner's abstention default with nothing. Both use
+ the `is not None` test rather than truthiness -- an empty `lang` is core's documented "fall
+ through to detection", and a `0.0` abstention gate is a real decision (abstain over nothing),
+ not an absence.
+ """
+ return {key: value for key, value in (("lang", lang), ("min_confidence", min_confidence))
+ if value is not None}
+
+
def predict_kwargs(model: Optional[str] = None, max_len: Optional[int] = None,
- head_max_len: Optional[int] = None) -> Dict[str, Any]:
+ head_max_len: Optional[int] = None, lang: Optional[str] = None,
+ min_confidence: Optional[float] = None) -> Dict[str, Any]:
"""The per-request overrides a local runner accepts, with the unset ones omitted."""
kwargs: Dict[str, Any] = {}
if model:
kwargs["model"] = model
kwargs.update(budget_kwargs(max_len, head_max_len))
+ kwargs.update(decision_kwargs(lang, min_confidence))
return kwargs
diff --git a/laya/integrations/crewai.py b/laya/integrations/crewai.py
index 435d96c1..8f712b75 100644
--- a/laya/integrations/crewai.py
+++ b/laya/integrations/crewai.py
@@ -55,6 +55,7 @@ class LayaTaskGuardError(ValueError):
# `predict` call and the same laya-serve request body, and three copies of the rule is how two of
# them came to forward only `model`.
from ._controls import budget_kwargs as _budget_kwargs, hook_kwargs as _hook_kwargs # noqa: E402
+from ._controls import decision_kwargs as _decision_kwargs # noqa: E402
from ._controls import predict_kwargs as _predict_kwargs, reject_remote_hooks as _reject_remote_hooks # noqa: E402
# One class for every integration, so `except LayaLowConfidenceError` catches all of them.
@@ -146,11 +147,15 @@ def _call_remote(
timeout: float = 10.0,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
) -> Dict[str, Any]:
"""Send decision request to a remote laya-serve HTTP instance using standard library urllib.
`max_len` / `head_max_len` travel in the body; laya-serve applies them up to its
- `LAYA_MAX_TOKEN_BUDGET` ceiling and answers a larger value with 422.
+ `LAYA_MAX_TOKEN_BUDGET` ceiling and answers a larger value with 422. `lang` / `min_confidence`
+ ride in the same body (they are laya-serve `BODY_CONTROLS` too): the language codes the state
+ is routed and answered in, and core's abstention gate.
"""
url = base_url.rstrip("/")
if not url.endswith("/v1/systemone"):
@@ -163,6 +168,10 @@ def _call_remote(
payload["max_len"] = max_len
if head_max_len is not None:
payload["head_max_len"] = head_max_len
+ if lang is not None:
+ payload["lang"] = lang
+ if min_confidence is not None:
+ payload["min_confidence"] = min_confidence
data = json.dumps(payload).encode("utf-8")
headers = {"Content-Type": "application/json"}
@@ -204,6 +213,8 @@ def _execute_decision(
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -219,9 +230,11 @@ def _execute_decision(
if base_url:
_reject_remote_hooks(hook_kwargs, base_url)
budget = _budget_kwargs(max_len, head_max_len)
- return _call_remote(base_url, state, questions, api_key=api_key, model=model, **budget)
+ decision = _decision_kwargs(lang, min_confidence)
+ return _call_remote(base_url, state, questions, api_key=api_key, model=model,
+ **budget, **decision)
runner = agent if agent is not None else _get_default_router()
- kwargs = _predict_kwargs(model, max_len, head_max_len)
+ kwargs = _predict_kwargs(model, max_len, head_max_len, lang, min_confidence)
kwargs.update(hook_kwargs)
return runner.predict(state, questions, **kwargs)
@@ -245,6 +258,8 @@ class LayaCrewRouter:
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -261,6 +276,8 @@ class LayaCrewRouter:
self.model = model
self.max_len = max_len
self.head_max_len = head_max_len
+ self.lang = lang
+ self.min_confidence = min_confidence
self.hooks = hooks
self.on_predict_start = on_predict_start
self.on_predict_end = on_predict_end
@@ -297,6 +314,8 @@ class LayaCrewRouter:
model=self.model,
max_len=self.max_len,
head_max_len=self.head_max_len,
+ lang=self.lang,
+ min_confidence=self.min_confidence,
hooks=self.hooks,
on_predict_start=self.on_predict_start,
on_predict_end=self.on_predict_end,
@@ -414,6 +433,8 @@ class LayaTaskGuard:
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -430,6 +451,8 @@ class LayaTaskGuard:
self.model = model
self.max_len = max_len
self.head_max_len = head_max_len
+ self.lang = lang
+ self.min_confidence = min_confidence
self.hooks = hooks
self.on_predict_start = on_predict_start
self.on_predict_end = on_predict_end
@@ -455,6 +478,8 @@ class LayaTaskGuard:
model=self.model,
max_len=self.max_len,
head_max_len=self.head_max_len,
+ lang=self.lang,
+ min_confidence=self.min_confidence,
hooks=self.hooks,
on_predict_start=self.on_predict_start,
on_predict_end=self.on_predict_end,
diff --git a/laya/integrations/langchain.py b/laya/integrations/langchain.py
index d6282877..868b19ca 100644
--- a/laya/integrations/langchain.py
+++ b/laya/integrations/langchain.py
@@ -30,6 +30,7 @@ except ImportError:
# at the same `predict` call and the same laya-serve body; keeping the rule in three places is how
# two of them ended up forwarding only `model`.
from ._controls import budget_kwargs as _budget_kwargs, hook_kwargs as _hook_kwargs
+from ._controls import decision_kwargs as _decision_kwargs
from ._controls import predict_kwargs as _predict_kwargs
from ._controls import reject_remote_hooks as _reject_remote_hooks
@@ -157,11 +158,15 @@ def _call_remote(
timeout: float = 10.0,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
) -> Dict[str, Any]:
"""Send decision request to a remote laya-serve HTTP instance using standard library urllib.
`max_len` / `head_max_len` travel in the body; laya-serve applies them up to its
- `LAYA_MAX_TOKEN_BUDGET` ceiling and answers a larger value with 422.
+ `LAYA_MAX_TOKEN_BUDGET` ceiling and answers a larger value with 422. `lang` / `min_confidence`
+ ride in the same body (they are laya-serve `BODY_CONTROLS` too): the language codes the state
+ for routing and calibration, and the abstention gate flags a low-confidence answer.
"""
url = base_url.rstrip("/")
if not url.endswith("/v1/systemone"):
@@ -174,6 +179,10 @@ def _call_remote(
payload["max_len"] = max_len
if head_max_len is not None:
payload["head_max_len"] = head_max_len
+ if lang is not None:
+ payload["lang"] = lang
+ if min_confidence is not None:
+ payload["min_confidence"] = min_confidence
data = json.dumps(payload).encode("utf-8")
headers = {"Content-Type": "application/json"}
@@ -215,6 +224,8 @@ def _execute_decision(
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -225,9 +236,11 @@ def _execute_decision(
if base_url:
_reject_remote_hooks(hook_kwargs, base_url)
budget = _budget_kwargs(max_len, head_max_len)
- return _call_remote(base_url, state, questions, api_key=api_key, model=model, **budget)
+ decision = _decision_kwargs(lang, min_confidence)
+ return _call_remote(base_url, state, questions, api_key=api_key, model=model,
+ **budget, **decision)
runner = agent if agent is not None else _get_default_router()
- kwargs = _predict_kwargs(model, max_len, head_max_len)
+ kwargs = _predict_kwargs(model, max_len, head_max_len, lang, min_confidence)
kwargs.update(hook_kwargs)
return runner.predict(state, questions, **kwargs)
@@ -381,6 +394,8 @@ class LayaRouter(_BatchedRunnable, RunnableSerializable):
model: Optional[str] = None
max_len: Optional[int] = None
head_max_len: Optional[int] = None
+ lang: Optional[str] = None
+ min_confidence: Optional[float] = None
hooks: Optional[Any] = None
on_predict_start: Optional[Any] = None
on_predict_end: Optional[Any] = None
@@ -406,6 +421,8 @@ class LayaRouter(_BatchedRunnable, RunnableSerializable):
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -426,6 +443,8 @@ class LayaRouter(_BatchedRunnable, RunnableSerializable):
model=model,
max_len=max_len,
head_max_len=head_max_len,
+ lang=lang,
+ min_confidence=min_confidence,
hooks=hooks,
on_predict_start=on_predict_start,
on_predict_end=on_predict_end,
@@ -445,6 +464,8 @@ class LayaRouter(_BatchedRunnable, RunnableSerializable):
self.model = model
self.max_len = max_len
self.head_max_len = head_max_len
+ self.lang = lang
+ self.min_confidence = min_confidence
self.hooks = hooks
self.on_predict_start = on_predict_start
self.on_predict_end = on_predict_end
@@ -488,6 +509,8 @@ class LayaRouter(_BatchedRunnable, RunnableSerializable):
model=self.model,
max_len=self.max_len,
head_max_len=self.head_max_len,
+ lang=self.lang,
+ min_confidence=self.min_confidence,
hooks=self.hooks,
on_predict_start=self.on_predict_start,
on_predict_end=self.on_predict_end,
@@ -527,6 +550,8 @@ class LayaGuardrail(_BatchedRunnable, RunnableSerializable):
model: Optional[str] = None
max_len: Optional[int] = None
head_max_len: Optional[int] = None
+ lang: Optional[str] = None
+ min_confidence: Optional[float] = None
hooks: Optional[Any] = None
on_predict_start: Optional[Any] = None
on_predict_end: Optional[Any] = None
@@ -550,6 +575,8 @@ class LayaGuardrail(_BatchedRunnable, RunnableSerializable):
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -572,6 +599,8 @@ class LayaGuardrail(_BatchedRunnable, RunnableSerializable):
model=model,
max_len=max_len,
head_max_len=head_max_len,
+ lang=lang,
+ min_confidence=min_confidence,
hooks=hooks,
on_predict_start=on_predict_start,
on_predict_end=on_predict_end,
@@ -591,6 +620,8 @@ class LayaGuardrail(_BatchedRunnable, RunnableSerializable):
self.model = model
self.max_len = max_len
self.head_max_len = head_max_len
+ self.lang = lang
+ self.min_confidence = min_confidence
self.hooks = hooks
self.on_predict_start = on_predict_start
self.on_predict_end = on_predict_end
@@ -684,6 +715,8 @@ class LayaGuardrail(_BatchedRunnable, RunnableSerializable):
model=self.model,
max_len=self.max_len,
head_max_len=self.head_max_len,
+ lang=self.lang,
+ min_confidence=self.min_confidence,
hooks=self.hooks,
on_predict_start=self.on_predict_start,
on_predict_end=self.on_predict_end,
@@ -711,6 +744,8 @@ class LayaTriage(_BatchedRunnable, RunnableSerializable):
model: Optional[str] = None
max_len: Optional[int] = None
head_max_len: Optional[int] = None
+ lang: Optional[str] = None
+ min_confidence: Optional[float] = None
hooks: Optional[Any] = None
on_predict_start: Optional[Any] = None
on_predict_end: Optional[Any] = None
@@ -730,6 +765,8 @@ class LayaTriage(_BatchedRunnable, RunnableSerializable):
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -746,6 +783,8 @@ class LayaTriage(_BatchedRunnable, RunnableSerializable):
model=model,
max_len=max_len,
head_max_len=head_max_len,
+ lang=lang,
+ min_confidence=min_confidence,
hooks=hooks,
on_predict_start=on_predict_start,
on_predict_end=on_predict_end,
@@ -761,6 +800,8 @@ class LayaTriage(_BatchedRunnable, RunnableSerializable):
self.model = model
self.max_len = max_len
self.head_max_len = head_max_len
+ self.lang = lang
+ self.min_confidence = min_confidence
self.hooks = hooks
self.on_predict_start = on_predict_start
self.on_predict_end = on_predict_end
@@ -804,6 +845,8 @@ class LayaTriage(_BatchedRunnable, RunnableSerializable):
model=self.model,
max_len=self.max_len,
head_max_len=self.head_max_len,
+ lang=self.lang,
+ min_confidence=self.min_confidence,
hooks=self.hooks,
on_predict_start=self.on_predict_start,
on_predict_end=self.on_predict_end,
@@ -831,6 +874,8 @@ class LayaEvaluator(_BatchedRunnable, RunnableSerializable):
model: Optional[str] = None
max_len: Optional[int] = None
head_max_len: Optional[int] = None
+ lang: Optional[str] = None
+ min_confidence: Optional[float] = None
hooks: Optional[Any] = None
on_predict_start: Optional[Any] = None
on_predict_end: Optional[Any] = None
@@ -851,6 +896,8 @@ class LayaEvaluator(_BatchedRunnable, RunnableSerializable):
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -868,6 +915,8 @@ class LayaEvaluator(_BatchedRunnable, RunnableSerializable):
model=model,
max_len=max_len,
head_max_len=head_max_len,
+ lang=lang,
+ min_confidence=min_confidence,
hooks=hooks,
on_predict_start=on_predict_start,
on_predict_end=on_predict_end,
@@ -884,6 +933,8 @@ class LayaEvaluator(_BatchedRunnable, RunnableSerializable):
self.model = model
self.max_len = max_len
self.head_max_len = head_max_len
+ self.lang = lang
+ self.min_confidence = min_confidence
self.hooks = hooks
self.on_predict_start = on_predict_start
self.on_predict_end = on_predict_end
@@ -902,6 +953,8 @@ class LayaEvaluator(_BatchedRunnable, RunnableSerializable):
model=self.model,
max_len=self.max_len,
head_max_len=self.head_max_len,
+ lang=self.lang,
+ min_confidence=self.min_confidence,
hooks=self.hooks,
on_predict_start=self.on_predict_start,
on_predict_end=self.on_predict_end,
@@ -927,6 +980,8 @@ class LayaEvaluator(_BatchedRunnable, RunnableSerializable):
model=self.model,
max_len=self.max_len,
head_max_len=self.head_max_len,
+ lang=self.lang,
+ min_confidence=self.min_confidence,
hooks=self.hooks,
on_predict_start=self.on_predict_start,
on_predict_end=self.on_predict_end,
diff --git a/laya/integrations/llamaindex.py b/laya/integrations/llamaindex.py
index 260df8f3..057e1361 100644
--- a/laya/integrations/llamaindex.py
+++ b/laya/integrations/llamaindex.py
@@ -64,6 +64,7 @@ except ImportError:
# `predict` call and the same laya-serve request body, and three copies of the rule is how two of
# them came to forward only `model`.
from ._controls import budget_kwargs as _budget_kwargs, hook_kwargs as _hook_kwargs # noqa: E402
+from ._controls import decision_kwargs as _decision_kwargs # noqa: E402
from ._controls import predict_kwargs as _predict_kwargs, reject_remote_hooks as _reject_remote_hooks # noqa: E402
# One class for every integration, so `except LayaLowConfidenceError` catches all of them.
@@ -130,11 +131,15 @@ def _call_remote(
timeout: float = 10.0,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
) -> Dict[str, Any]:
"""Send decision request to a remote laya-serve HTTP instance using standard library urllib.
`max_len` / `head_max_len` travel in the body; laya-serve applies them up to its
- `LAYA_MAX_TOKEN_BUDGET` ceiling and answers a larger value with 422.
+ `LAYA_MAX_TOKEN_BUDGET` ceiling and answers a larger value with 422. `lang` / `min_confidence`
+ ride in the same body (they are laya-serve `BODY_CONTROLS` too): the language codes the state
+ is routed and answered in, and core's abstention gate.
"""
url = base_url.rstrip("/")
if not url.endswith("/v1/systemone"):
@@ -147,6 +152,10 @@ def _call_remote(
payload["max_len"] = max_len
if head_max_len is not None:
payload["head_max_len"] = head_max_len
+ if lang is not None:
+ payload["lang"] = lang
+ if min_confidence is not None:
+ payload["min_confidence"] = min_confidence
data = json.dumps(payload).encode("utf-8")
headers = {"Content-Type": "application/json"}
@@ -188,6 +197,8 @@ def _execute_decision(
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -203,9 +214,11 @@ def _execute_decision(
if base_url:
_reject_remote_hooks(hook_kwargs, base_url)
budget = _budget_kwargs(max_len, head_max_len)
- return _call_remote(base_url, state, questions, api_key=api_key, model=model, **budget)
+ decision = _decision_kwargs(lang, min_confidence)
+ return _call_remote(base_url, state, questions, api_key=api_key, model=model,
+ **budget, **decision)
runner = agent if agent is not None else _get_default_router()
- kwargs = _predict_kwargs(model, max_len, head_max_len)
+ kwargs = _predict_kwargs(model, max_len, head_max_len, lang, min_confidence)
kwargs.update(hook_kwargs)
return runner.predict(state, questions, **kwargs)
@@ -230,6 +243,8 @@ class LayaSingleSelector(BaseSelector):
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -246,6 +261,8 @@ class LayaSingleSelector(BaseSelector):
self.model = model
self.max_len = max_len
self.head_max_len = head_max_len
+ self.lang = lang
+ self.min_confidence = min_confidence
self.hooks = hooks
self.on_predict_start = on_predict_start
self.on_predict_end = on_predict_end
@@ -282,6 +299,8 @@ class LayaSingleSelector(BaseSelector):
model=self.model,
max_len=self.max_len,
head_max_len=self.head_max_len,
+ lang=self.lang,
+ min_confidence=self.min_confidence,
hooks=self.hooks,
on_predict_start=self.on_predict_start,
on_predict_end=self.on_predict_end,
@@ -370,6 +389,8 @@ class LayaMultiSelector(BaseSelector):
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -386,6 +407,8 @@ class LayaMultiSelector(BaseSelector):
self.model = model
self.max_len = max_len
self.head_max_len = head_max_len
+ self.lang = lang
+ self.min_confidence = min_confidence
self.hooks = hooks
self.on_predict_start = on_predict_start
self.on_predict_end = on_predict_end
@@ -421,6 +444,8 @@ class LayaMultiSelector(BaseSelector):
model=self.model,
max_len=self.max_len,
head_max_len=self.head_max_len,
+ lang=self.lang,
+ min_confidence=self.min_confidence,
hooks=self.hooks,
on_predict_start=self.on_predict_start,
on_predict_end=self.on_predict_end,
@@ -519,6 +544,8 @@ class LayaQueryRouter:
model: Optional[str] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None,
+ lang: Optional[str] = None,
+ min_confidence: Optional[float] = None,
hooks: Optional[Any] = None,
on_predict_start: Optional[Any] = None,
on_predict_end: Optional[Any] = None,
@@ -536,6 +563,8 @@ class LayaQueryRouter:
self.model = model
self.max_len = max_len
self.head_max_len = head_max_len
+ self.lang = lang
+ self.min_confidence = min_confidence
self.hooks = hooks
self.on_predict_start = on_predict_start
self.on_predict_end = on_predict_end
@@ -568,6 +597,8 @@ class LayaQueryRouter:
model=self.model,
max_len=self.max_len,
head_max_len=self.head_max_len,
+ lang=self.lang,
+ min_confidence=self.min_confidence,
hooks=self.hooks,
on_predict_start=self.on_predict_start,
on_predict_end=self.on_predict_end,
diff --git a/tests/test_crewai.py b/tests/test_crewai.py
index bebbaa32..fb7a3e60 100644
--- a/tests/test_crewai.py
+++ b/tests/test_crewai.py
@@ -298,7 +298,8 @@ from laya.integrations import langchain as langchain_module
from laya.integrations import llamaindex as llamaindex_module
from laya.router import Router
-CONTROLS = tuple(_controls.PREDICT_CONTROLS) + tuple(_controls.HOOK_CONTROLS)
+CONTROLS = (tuple(_controls.PREDICT_CONTROLS) + tuple(_controls.DECISION_CONTROLS)
+ + tuple(_controls.HOOK_CONTROLS))
def _params(fn):
@@ -308,6 +309,8 @@ def _params(fn):
# The shared tuples name the arguments the shared builders accept, in both directions.
check("controls/budget tuple names budget_kwargs",
set(_params(_controls.budget_kwargs)), set(_controls.PREDICT_CONTROLS))
+check("controls/decision tuple names decision_kwargs",
+ set(_params(_controls.decision_kwargs)), set(_controls.DECISION_CONTROLS))
check("controls/hook tuple names hook_kwargs",
set(_params(_controls.hook_kwargs)), set(_controls.HOOK_CONTROLS))
@@ -354,7 +357,8 @@ class RecordingAgent:
}
-ALL_CONTROLS = {"max_len": 1024, "head_max_len": 512, "hooks": ["H"], "on_predict_start": "S",
+ALL_CONTROLS = {"max_len": 1024, "head_max_len": 512, "lang": "fr", "min_confidence": 0.4,
+ "hooks": ["H"], "on_predict_start": "S",
"on_predict_end": "E", "hooks_raise": True, "hooks_timeout": 0.5}
@@ -384,13 +388,16 @@ def guard_call(**controls):
for label, call in (("router", crew_call), ("guard", guard_call)):
check("controls/%s with nothing set sends nothing" % label, call(), {})
- check("controls/%s forwards all seven" % label, call(**ALL_CONTROLS), ALL_CONTROLS)
+ check("controls/%s forwards every control" % label, call(**ALL_CONTROLS), ALL_CONTROLS)
check("controls/%s forwards one budget alone" % label, call(head_max_len=256),
{"head_max_len": 256})
+ check("controls/%s forwards lang alone" % label, call(lang="fr"), {"lang": "fr"})
+ check("controls/%s forwards min_confidence alone" % label, call(min_confidence=0.4),
+ {"min_confidence": 0.4})
# 0 and [] are decisions, not absences: truthiness tests here would drop them.
check("controls/%s keeps falsy values" % label,
- call(head_max_len=0, hooks=[], hooks_raise=False),
- {"head_max_len": 0, "hooks": [], "hooks_raise": False})
+ call(head_max_len=0, hooks=[], hooks_raise=False, min_confidence=0.0),
+ {"head_max_len": 0, "hooks": [], "hooks_raise": False, "min_confidence": 0.0})
check("controls/%s alongside model" % label,
call(model="laya-multilingual", max_len=1024),
{"model": "laya-multilingual", "max_len": 1024})
@@ -426,6 +433,16 @@ check("controls/remote body omits unset budgets",
[k for k in remote_body({}) if k in _controls.PREDICT_CONTROLS], [])
check("controls/remote body keeps a zero",
remote_body({"head_max_len": 0}).get("head_max_len"), 0)
+# `lang` / `min_confidence` are laya-serve `BODY_CONTROLS` too, so the same override reaches the
+# remote node -- and an unset one stays out of the body rather than shadowing the deployment.
+check("controls/remote body carries the decision controls",
+ {k: v for k, v in remote_body({"lang": "es", "min_confidence": 0.3}).items()
+ if k in _controls.DECISION_CONTROLS},
+ {"lang": "es", "min_confidence": 0.3})
+check("controls/remote body omits unset decision controls",
+ [k for k in remote_body({}) if k in _controls.DECISION_CONTROLS], [])
+check("controls/remote body keeps min_confidence=0.0",
+ remote_body({"min_confidence": 0.0}).get("min_confidence"), 0.0)
# A hook is a Python callable that runs inside `predict`; a serve node cannot receive one. Saying
# so beats reporting success after never calling it.
diff --git a/tests/test_langchain.py b/tests/test_langchain.py
index e56c09ad..1a3fd79d 100644
--- a/tests/test_langchain.py
+++ b/tests/test_langchain.py
@@ -992,6 +992,120 @@ finally:
langchain_module.urllib.request.build_opener = _real_build_opener
+# --------------------------------------------------------------- 7b. Per-request lang / abstention
+#
+# `lang` and `min_confidence` are core's language routing and abstention gate (#361). Both are
+# read by `Agent.predict`/`system_one` AND `Router.predict`, and both are laya-serve
+# `BODY_CONTROLS`, so they are safe to forward on the local and the remote path alike -- unlike
+# `task` / `lang_guess`, which are Router-only. A wrapper that drops them cannot route a
+# non-English state or measure the abstention gate it is supposed to honor.
+from laya.integrations import _controls # noqa: E402
+import inspect # noqa: E402
+from laya.agent import Agent # noqa: E402
+from laya.router import Router # noqa: E402
+
+DECISION_CRITERIA = {"billing": "invoices", "tech": "bugs"}
+DECISION_NODES = (
+ ("router", lambda a, **kw: LayaRouter(DECISION_CRITERIA, agent=a, **kw)),
+ ("guardrail", lambda a, **kw: LayaGuardrail(
+ questions={"jailbreak": {"type": "noul", "instructions": "jailbreak?"}}, agent=a, **kw)),
+ ("triage", lambda a, **kw: LayaTriage(agent=a, **kw)),
+ ("evaluator", lambda a, **kw: LayaEvaluator(
+ questions={"faithful": {"type": "noul", "instructions": "faithful?"}}, agent=a, **kw)),
+)
+
+check("decision/_controls names the tuple", _controls.DECISION_CONTROLS, ("lang", "min_confidence"))
+for _c in _controls.DECISION_CONTROLS:
+ check_true("decision/%s accepted by Agent.system_one" % _c,
+ _c in set(inspect.signature(Agent.system_one).parameters))
+ check_true("decision/%s accepted by Router.predict" % _c,
+ _c in set(inspect.signature(Router.predict).parameters))
+
+for name, build in DECISION_NODES:
+ plain = BudgetAgent()
+ build(plain).invoke("some state")
+ check("decision/%s default sends nothing" % name, plain.kwargs[0], {})
+
+ both = BudgetAgent()
+ build(both, lang="fr", min_confidence=0.4).invoke("some state")
+ check("decision/%s forwards both" % name, both.kwargs[0],
+ {"lang": "fr", "min_confidence": 0.4})
+
+ # Each knob works alone; an override built from one must not carry the other.
+ one = BudgetAgent()
+ build(one, lang="es").invoke("some state")
+ check("decision/%s forwards lang alone" % name, one.kwargs[0], {"lang": "es"})
+
+ # A 0.0 abstention gate is a real decision ("abstain over nothing"), not an absence.
+ zero = BudgetAgent()
+ build(zero, min_confidence=0.0).invoke("some state")
+ check("decision/%s keeps min_confidence=0.0" % name, zero.kwargs[0], {"min_confidence": 0.0})
+
+ mixed = BudgetAgent()
+ build(mixed, model="laya-multilingual", lang="de").invoke("some state")
+ check("decision/%s with model" % name, mixed.kwargs[0],
+ {"model": "laya-multilingual", "lang": "de"})
+
+
+# A remote node forwards them in the request body instead of dropping them.
+decision_remote_calls = []
+_decision_real_call_remote = langchain_module._call_remote
+
+
+def decision_spy_call_remote(base_url, state, questions, api_key=None, model=None, **extras):
+ decision_remote_calls.append(dict({"model": model}, **extras))
+ return {"answers": {"route": {"type": "choice", "choice": "billing", "confidence": 0.9}}}
+
+
+langchain_module._call_remote = decision_spy_call_remote
+try:
+ for field, value in (("lang", "it"), ("min_confidence", 0.25), ("min_confidence", 0.0)):
+ LayaRouter(DECISION_CRITERIA, base_url="http://laya:8000", **{field: value}).invoke("x")
+ check("decision/remote forwards %s=%r" % (field, value), decision_remote_calls[-1],
+ {"model": None, field: value})
+ # Nothing set sends nothing.
+ LayaRouter(DECISION_CRITERIA, base_url="http://laya:8000").invoke("x")
+ check_true("decision/remote omits unset decision controls",
+ not any(k in decision_remote_calls[-1] for k in _controls.DECISION_CONTROLS),
+ repr(decision_remote_calls[-1]))
+finally:
+ langchain_module._call_remote = _decision_real_call_remote
+
+# ...and the real `_call_remote` writes them into the JSON body.
+_decision_sent = {}
+
+
+class _DecisionFakeResponse:
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *exc):
+ return False
+
+ def read(self):
+ return b'{"answers": {}}'
+
+
+class _DecisionFakeOpener:
+ def open(self, req, timeout=None):
+ _decision_sent["body"] = json.loads(req.data.decode("utf-8"))
+ return _DecisionFakeResponse()
+
+
+_decision_real_build_opener = langchain_module.urllib.request.build_opener
+langchain_module.urllib.request.build_opener = lambda *a, **k: _DecisionFakeOpener()
+try:
+ langchain_module._call_remote("http://laya:8000", "x", {}, lang="fr", min_confidence=0.0)
+ check("decision/_call_remote sends lang", _decision_sent["body"].get("lang"), "fr")
+ check("decision/_call_remote keeps min_confidence=0.0",
+ _decision_sent["body"].get("min_confidence"), 0.0)
+ langchain_module._call_remote("http://laya:8000", "x", {})
+ check_true("decision/_call_remote omits unset decision controls",
+ "lang" not in _decision_sent["body"] and "min_confidence" not in _decision_sent["body"])
+finally:
+ langchain_module.urllib.request.build_opener = _decision_real_build_opener
+
+
# --------------------------------------------------------------- 7. Prediction hooks
from laya.integrations import langchain as langchain_module # noqa: E402
from laya.hooks import PredictContext # noqa: E402
diff --git a/tests/test_llamaindex.py b/tests/test_llamaindex.py
index dcfbb4ce..09718f76 100644
--- a/tests/test_llamaindex.py
+++ b/tests/test_llamaindex.py
@@ -409,7 +409,8 @@ from laya.integrations import langchain as langchain_module
from laya.integrations import llamaindex as llamaindex_module
from laya.router import Router
-CONTROLS = tuple(_controls.PREDICT_CONTROLS) + tuple(_controls.HOOK_CONTROLS)
+CONTROLS = (tuple(_controls.PREDICT_CONTROLS) + tuple(_controls.DECISION_CONTROLS)
+ + tuple(_controls.HOOK_CONTROLS))
def _params(fn):
@@ -418,6 +419,8 @@ def _params(fn):
check("controls/budget tuple names budget_kwargs",
set(_params(_controls.budget_kwargs)), set(_controls.PREDICT_CONTROLS))
+check("controls/decision tuple names decision_kwargs",
+ set(_params(_controls.decision_kwargs)), set(_controls.DECISION_CONTROLS))
check("controls/hook tuple names hook_kwargs",
set(_params(_controls.hook_kwargs)), set(_controls.HOOK_CONTROLS))
@@ -464,7 +467,8 @@ class RecordingAgent:
}
-ALL_CONTROLS = {"max_len": 1024, "head_max_len": 512, "hooks": ["H"], "on_predict_start": "S",
+ALL_CONTROLS = {"max_len": 1024, "head_max_len": 512, "lang": "fr", "min_confidence": 0.4,
+ "hooks": ["H"], "on_predict_start": "S",
"on_predict_end": "E", "hooks_raise": True, "hooks_timeout": 0.5}
# One entry per class that can take a decision here: (label, class, constructor extras, one call).
@@ -508,14 +512,19 @@ def remote_body(cls, extra, run, **controls):
for _label, _cls, _extra, _run in SURFACES:
check("controls/%s with nothing set sends nothing" % _label,
local_kwargs(_cls, _extra, _run), {})
- check("controls/%s forwards all seven" % _label,
+ check("controls/%s forwards every control" % _label,
local_kwargs(_cls, _extra, _run, **ALL_CONTROLS), ALL_CONTROLS)
check("controls/%s forwards one budget alone" % _label,
local_kwargs(_cls, _extra, _run, head_max_len=256), {"head_max_len": 256})
+ check("controls/%s forwards lang alone" % _label,
+ local_kwargs(_cls, _extra, _run, lang="fr"), {"lang": "fr"})
+ check("controls/%s forwards min_confidence alone" % _label,
+ local_kwargs(_cls, _extra, _run, min_confidence=0.4), {"min_confidence": 0.4})
# 0 and [] are decisions, not absences: truthiness tests here would drop them.
check("controls/%s keeps falsy values" % _label,
- local_kwargs(_cls, _extra, _run, head_max_len=0, hooks=[], hooks_raise=False),
- {"head_max_len": 0, "hooks": [], "hooks_raise": False})
+ local_kwargs(_cls, _extra, _run, head_max_len=0, hooks=[], hooks_raise=False,
+ min_confidence=0.0),
+ {"head_max_len": 0, "hooks": [], "hooks_raise": False, "min_confidence": 0.0})
check("controls/%s alongside model" % _label,
local_kwargs(_cls, _extra, _run, model="laya-multilingual", max_len=1024),
{"model": "laya-multilingual", "max_len": 1024})
@@ -528,6 +537,16 @@ for _label, _cls, _extra, _run in SURFACES:
[k for k in remote_body(_cls, _extra, _run) if k in _controls.PREDICT_CONTROLS], [])
check("controls/%s remote body keeps a zero" % _label,
remote_body(_cls, _extra, _run, head_max_len=0).get("head_max_len"), 0)
+ # `lang` / `min_confidence` are laya-serve `BODY_CONTROLS`, so the same override reaches the
+ # remote node -- and an unset one stays out of the body rather than shadowing the deployment.
+ check("controls/%s remote body carries the decision controls" % _label,
+ {k: v for k, v in remote_body(_cls, _extra, _run, lang="es", min_confidence=0.3).items()
+ if k in _controls.DECISION_CONTROLS},
+ {"lang": "es", "min_confidence": 0.3})
+ check("controls/%s remote body omits unset decision controls" % _label,
+ [k for k in remote_body(_cls, _extra, _run) if k in _controls.DECISION_CONTROLS], [])
+ check("controls/%s remote body keeps min_confidence=0.0" % _label,
+ remote_body(_cls, _extra, _run, min_confidence=0.0).get("min_confidence"), 0.0)
# A hook is a Python callable that runs inside `predict`; a serve node cannot receive one.
for _c, _sample in HOOK_SAMPLES:
try: