Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(mcp): share a local model server through LAYA_BASE_URL

Rishit Sharmac8ddefa4836a
diff --git a/README.md b/README.md
index 061732d3..c55ff1c3 100644
--- a/README.md
+++ b/README.md
@@ -1187,6 +1187,12 @@ pip install "laya[mcp]"
laya-mcp-server # or: python -m laya.mcp.server
```
+To share a `laya-serve` on your own hardware across editor sessions, set
+`LAYA_BASE_URL=http://127.0.0.1:8000` in each MCP client's environment. Its prediction tools use
+HTTP and import no torch; routing stays local. Set `LAYA_API_KEY` to the server's bearer key and
+`LAYA_REMOTE_TIMEOUT` to the allowed HTTP wait (300 seconds by default). `laya_shortlist` is
+unavailable in this mode. See [shared-server MCP setup](docs/cli-mcp.md#share-one-model-server-across-mcp-sessions).
+
Example MCP client configuration (stdio transport):
```json
diff --git a/docs/cli-mcp.md b/docs/cli-mcp.md
index b09d5588..97edf633 100644
--- a/docs/cli-mcp.md
+++ b/docs/cli-mcp.md
@@ -229,10 +229,48 @@ the server is ready.
| `LAYA_THREADS` | PyTorch default | Caps Torch intra-op threads for CPU inference; keep it at or below the physical core count. |
| `LAYA_AUTO_TASK` | `0` | Set to `1` to let a request auto-route to the `typed-decisions` checkpoint. Same meaning as in `laya.serve`; it does not preload that checkpoint, so `LAYA_MODELS` still decides what is built at startup. |
| `LAYA_DEFAULT_MODEL` | `english` | The checkpoint a state with no language evidence falls back to, same meaning as in `laya.serve`. Unlike `laya.serve`, an unresolvable name does not stop the server: it comes back as a `router construction failed` tool error on the next call, because a stdio server has no startup to refuse. |
+| `LAYA_BASE_URL` | unset | Send predictions to a `laya-serve` on your own hardware instead of loading checkpoints in each MCP process. A bare `host:port` is read as HTTP. |
+| `LAYA_REMOTE_TIMEOUT` | `300` | HTTP timeout in seconds when `LAYA_BASE_URL` is set, including the server's cold load. Invalid or non-positive values use the default. |
-The stock `laya-mcp-server` launcher creates its Router without installing hooks. If you need
-prediction hooks, use a custom launcher that installs them, for example with
-`laya.hooks.set_default_hooks`, before the server builds its Router. The environment variables
+### Share one model server across MCP sessions
+
+Run one local HTTP server and point each MCP client's environment at it:
+
+```bash
+LAYA_HOST=127.0.0.1 LAYA_PRELOAD=0 LAYA_IDLE_UNLOAD_SECONDS=300 laya-serve
+```
+
+```json
+{
+ "mcpServers": {
+ "laya": {
+ "command": "laya-mcp-server",
+ "env": {"LAYA_BASE_URL": "http://127.0.0.1:8000"}
+ }
+ }
+}
+```
+
+Install `laya[serve]` where the HTTP server runs. MCP still uses stdio with the editor; its
+prediction tools use HTTP to reach your server. `laya_predict`, `laya_predict_batch`, `laya_decide`
+and `laya_preset` use the server's original state, instructions and option descriptions.
+Heterogeneous batches send one `/v1/systemone` request per item, preserving input order;
+`batch_size` and `sort_by_length` do not change the server's execution. `laya_status` reports the
+server's `/health`; `laya_route` and `laya_route_batch` stay local and need no model or HTTP request.
+The MCP process imports no torch and loads no checkpoint, including when `LAYA_THREADS` or
+`LAYA_PRELOAD` is set.
+
+Set the same `LAYA_API_KEY` in both processes when the server requires a bearer token. Keep
+`LAYA_DEFAULT_MODEL` and `LAYA_AUTO_TASK` aligned so local routing previews match the server's
+actual routing. Device and preload settings belong to the HTTP server. The first call after an
+idle unload waits for a cold load; raise `LAYA_REMOTE_TIMEOUT` if that takes longer than 300 seconds.
+`laya_shortlist` and prediction hook overrides return `unsupported_remote`, since their code needs
+the model process. HTTP errors retain the server's detail text as MCP tool errors. With
+`LAYA_BASE_URL` unset, the MCP server keeps loading checkpoints in its own process.
+
+The stock `laya-mcp-server` launcher creates its Router without installing hooks. Install prediction
+hooks in the process that runs inference: the MCP process in local mode, or the HTTP server in
+shared-server mode. A custom launcher can use `laya.hooks.set_default_hooks` before building its Router. The environment variables
above configure model lifecycle, not hook registration. The client still decides when to call a
tool and what to do with the returned decision.
diff --git a/laya/mcp/remote.py b/laya/mcp/remote.py
new file mode 100644
index 00000000..0631e622
--- /dev/null
+++ b/laya/mcp/remote.py
@@ -0,0 +1,239 @@
+"""A Router that answers over HTTP from a running ``laya-serve`` instead of loading weights.
+
+``laya-mcp-server`` builds a ``Router`` and loads checkpoints into its own process. That is the
+right default for one agent on one machine, but it means every MCP client that opens a session
+holds its own copy of the model: two Claude sessions plus a ``laya-serve`` for scripts is three
+times 1.3 GB for the same checkpoint. Set ``LAYA_BASE_URL`` and the MCP server uses this class
+instead: ``predict`` and ``predict_batch`` become ``POST /v1/systemone`` calls to that server,
+and the MCP process itself stays at a few tens of megabytes, with no torch import at all.
+
+Routing stays local and free. ``RemoteRouter`` is a ``Router``, so ``route`` / ``route_batch``
+run the same language and script detection the server runs, without a model. The server makes
+its own routing decision for the actual answer and reports it in ``routing``; the two agree when
+the MCP process and the server see the same ``LAYA_DEFAULT_MODEL`` / ``LAYA_AUTO_TASK``.
+
+Not available remotely: ``load`` (and therefore ``laya_shortlist``, which needs the answering
+checkpoint's embeddings in-process), ``preload`` and ``unload`` (no-ops: the server owns its
+checkpoints' lifetime, see ``LAYA_IDLE_UNLOAD_SECONDS``), and per-call hooks (a hook is a callable
+that has to live where inference runs, exactly as ``laya.serve`` already refuses them).
+
+Standard library only: ``urllib`` carries the requests, so the extra pulls in nothing new.
+"""
+from __future__ import annotations
+
+import json
+import math
+import os
+import urllib.error
+import urllib.request
+import urllib.parse
+from typing import Any, Dict, List, Optional, Sequence, Union
+
+from ..router import Router
+from ..serve import BODY_CONTROLS, BODY_REFUSALS
+
+DEFAULT_TIMEOUT_S = 300.0
+
+# Request fields `laya.serve` reads off a /v1/systemone body, in the order the single-request
+# handler validates them. Anything not in this list has no wire form and is refused or dropped.
+_WIRE_CONTROLS = BODY_CONTROLS
+
+
+class RemoteError(RuntimeError):
+ """An HTTP-level failure talking to laya-serve. ``code`` is a tool-error style slug."""
+
+ def __init__(self, code: str, message: str):
+ super().__init__(message)
+ self.code = code
+ self.message = message
+
+
+def _timeout_from_env() -> float:
+ raw = os.environ.get("LAYA_REMOTE_TIMEOUT", "").strip()
+ if not raw:
+ return DEFAULT_TIMEOUT_S
+ try:
+ seconds = float(raw)
+ except ValueError:
+ return DEFAULT_TIMEOUT_S
+ return seconds if math.isfinite(seconds) and seconds > 0 else DEFAULT_TIMEOUT_S
+
+
+class RemoteRouter(Router):
+ """Route locally, answer from a ``laya-serve`` at ``base_url``.
+
+ ``base_url`` is the server origin (``http://127.0.0.1:8000``); the ``/v1/systemone`` paths are
+ appended here. ``api_key``, when given, is sent as ``Authorization: Bearer`` and must match the
+ server's ``LAYA_API_KEY``.
+ """
+
+ def __init__(self, base_url: str, api_key: Optional[str] = None,
+ timeout: Optional[float] = None, **router_kwargs: Any):
+ # device/preload make no sense for a router that never builds an Agent.
+ router_kwargs.pop("device", None)
+ router_kwargs.pop("preload", None)
+ super().__init__(**router_kwargs)
+ self.base_url = base_url.rstrip("/")
+ parsed = urllib.parse.urlsplit(self.base_url)
+ if parsed.scheme not in ("http", "https") or not parsed.hostname or parsed.query or parsed.fragment:
+ raise ValueError("base_url must be an HTTP(S) server URL without a query or fragment")
+ self.api_key = api_key or None
+ self.timeout = _timeout_from_env() if timeout is None else float(timeout)
+ if not math.isfinite(self.timeout) or self.timeout <= 0:
+ raise ValueError("timeout must be a finite positive number")
+
+ # ------------------------------------------------------------------ transport
+ def _request(self, path: str, body: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
+ data = None if body is None else json.dumps(body).encode("utf-8")
+ headers = {"Accept": "application/json"}
+ if data is not None:
+ headers["Content-Type"] = "application/json"
+ if self.api_key:
+ headers["Authorization"] = "Bearer " + self.api_key
+ req = urllib.request.Request(self.base_url + path, data=data, headers=headers,
+ method="POST" if data is not None else "GET")
+ try:
+ with urllib.request.urlopen(req, timeout=self.timeout) as resp:
+ raw = resp.read()
+ except urllib.error.HTTPError as exc:
+ detail = _error_detail(exc)
+ if exc.code == 401:
+ raise RemoteError("unauthorized", "laya-serve refused the bearer token: " + detail) from exc
+ if exc.code in (400, 413, 422):
+ raise RemoteError("invalid_request", detail) from exc
+ if exc.code == 503:
+ raise RemoteError("busy", "laya-serve is busy: " + detail) from exc
+ raise RemoteError("upstream_error", "laya-serve answered HTTP %d: %s" % (exc.code, detail)) from exc
+ except urllib.error.URLError as exc:
+ if isinstance(exc.reason, TimeoutError):
+ raise RemoteError("timeout", "laya-serve did not answer within %.0fs" % self.timeout) from exc
+ raise RemoteError("unreachable", "cannot reach laya-serve at %s: %s" % (self.base_url, exc.reason)) from exc
+ except TimeoutError as exc:
+ raise RemoteError("timeout", "laya-serve did not answer within %.0fs" % self.timeout) from exc
+ try:
+ payload = json.loads(raw.decode("utf-8"))
+ except (UnicodeDecodeError, json.JSONDecodeError) as exc:
+ raise RemoteError("upstream_error", "laya-serve returned a non-JSON body") from exc
+ if not isinstance(payload, dict):
+ raise RemoteError("upstream_error", "laya-serve returned a non-object body")
+ return payload
+
+ # ------------------------------------------------------------------ server state
+ def health(self) -> Dict[str, Any]:
+ """The server's ``GET /health`` payload (liveness only when the bearer is wrong)."""
+ return self._request("/health")
+
+ @property
+ def loaded(self) -> List[str]:
+ """Checkpoints resident on the server right now, or [] when it cannot be asked."""
+ try:
+ value = self.health().get("loaded")
+ except RemoteError:
+ return []
+ return [v for v in value if isinstance(v, str)] if isinstance(value, list) else []
+
+ @property
+ def loaded_revisions(self) -> Dict[str, Optional[str]]:
+ try:
+ value = self.health().get("revisions")
+ except RemoteError:
+ return {}
+ return dict(value) if isinstance(value, dict) else {}
+
+ # ------------------------------------------------------------------ lifecycle (server-owned)
+ def load(self, name: str):
+ raise RemoteError(
+ "unsupported_remote",
+ "checkpoint %r lives in laya-serve at %s; a remote router cannot hand out an Agent"
+ % (name, self.base_url),
+ )
+
+ def preload(self, names: Optional[List[str]] = None):
+ return self # the server decides what is resident
+
+ def unload(self, name: Optional[str] = None):
+ return None # the server decides when to free memory (LAYA_IDLE_UNLOAD_SECONDS)
+
+ def attach(self, name: str, agent: Any):
+ raise RemoteError("unsupported_remote", "a remote router holds no agents to attach to")
+
+ # ------------------------------------------------------------------ answering
+ def predict(
+ self,
+ state: Union[str, dict, list],
+ questions: Dict[str, Any],
+ model: Optional[str] = None,
+ task: Optional[str] = None,
+ lang: Optional[str] = None,
+ lang_guess: Optional[Any] = None,
+ hooks=None,
+ on_predict_start=None,
+ on_predict_end=None,
+ hooks_raise: Optional[bool] = None,
+ hooks_timeout: Optional[float] = None,
+ max_len: Optional[int] = None,
+ head_max_len: Optional[int] = None,
+ min_confidence: Optional[float] = None,
+ ) -> Dict[str, Any]:
+ if any(value is not None for value in (hooks, on_predict_start, on_predict_end, hooks_raise, hooks_timeout)):
+ raise RemoteError("unsupported_remote",
+ "hooks are callables and cannot be sent to laya-serve; install them on the server")
+ body: Dict[str, Any] = {"state": state, "questions": questions}
+ controls = {"model": model, "task": task, "lang": lang, "lang_guess": lang_guess,
+ "max_len": max_len, "head_max_len": head_max_len, "min_confidence": min_confidence}
+ for key in _WIRE_CONTROLS:
+ value = controls[key]
+ if value is not None and not (key == "model" and value == "auto"):
+ if key == "lang_guess" and not isinstance(value, str):
+ raise RemoteError("unsupported_remote", "a callable lang_guess cannot be sent over HTTP")
+ body[key] = value
+ return self._request("/v1/systemone", body)
+
+ def predict_batch(
+ self,
+ requests: Sequence[Dict[str, Any]],
+ batch_size: Optional[int] = None,
+ hooks_timeout: Optional[float] = None,
+ min_confidence: Optional[float] = None,
+ sort_by_length: bool = False,
+ ) -> List[Dict[str, Any]]:
+ """One ``/v1/systemone`` call per request, answers in input order.
+
+ ``laya-serve``'s ``/v1/systemone/batch`` shares one ``questions`` object across every state,
+ while a Router batch lets each request carry its own questions and controls, so the general
+ form is the per-request call. ``batch_size`` and ``sort_by_length`` shape forward passes and
+ have no meaning here; they are accepted and ignored so callers written against ``Router``
+ keep working. The server applies its own batching inside each call.
+ """
+ results: List[Dict[str, Any]] = []
+ if hooks_timeout is not None:
+ raise RemoteError("unsupported_remote", "hooks_timeout belongs to the server's hooks")
+ for item in requests:
+ if not isinstance(item, dict):
+ raise TypeError("predict_batch requests must be dicts, got %s" % type(item).__name__)
+ if any(item.get(key) is not None for key in BODY_REFUSALS):
+ raise RemoteError("unsupported_remote", "per-request hooks cannot be sent to laya-serve")
+ kwargs = {k: item[k] for k in ("model", "task", "lang", "lang_guess", "max_len", "head_max_len")
+ if k in item}
+ if min_confidence is not None:
+ kwargs["min_confidence"] = min_confidence
+ results.append(self.predict(item["state"], item["questions"], **kwargs))
+ return results
+
+ def __repr__(self) -> str:
+ return "RemoteRouter(base_url=%r, timeout=%g)" % (self.base_url, self.timeout)
+
+
+def _error_detail(exc: urllib.error.HTTPError) -> str:
+ """The server's ``detail`` string when the error body is FastAPI-shaped, else the status text."""
+ try:
+ body = json.loads(exc.read().decode("utf-8"))
+ except Exception: # noqa: BLE001 -- any unreadable body falls back to the HTTP reason
+ return str(exc.reason)
+ if isinstance(body, dict):
+ detail = body.get("detail")
+ if isinstance(detail, str):
+ return detail
+ if detail is not None:
+ return json.dumps(detail, ensure_ascii=False)
+ return str(exc.reason)
diff --git a/laya/mcp/server.py b/laya/mcp/server.py
index 3eb05a65..39035cb4 100644
--- a/laya/mcp/server.py
+++ b/laya/mcp/server.py
@@ -18,6 +18,8 @@ Environment (same meaning as laya.serve where it exists):
LAYA_DEFAULT_MODEL the checkpoint a state with no language evidence falls back to
(same as laya.serve, including its aliases); an unset value leaves
it to Router, an unresolvable one is a tool error.
+ LAYA_BASE_URL answer from a running laya-serve over HTTP; unset keeps models local
+ LAYA_REMOTE_TIMEOUT HTTP timeout in seconds for remote mode (default 300)
"""
from __future__ import annotations
@@ -126,6 +128,20 @@ def _models_from_env() -> list[str]:
return names or list(_DEFAULT_MODELS)
+def _remote_base_url() -> str | None:
+ """``LAYA_BASE_URL``: a running laya-serve to answer from instead of loading checkpoints here.
+
+ Set it to the server origin (``http://127.0.0.1:8000``). Blank or unset means the default,
+ in-process router. Scheme-less values are accepted as http so a bare ``host:port`` works.
+ """
+ raw = os.environ.get("LAYA_BASE_URL", "").strip()
+ if not raw:
+ return None
+ if "://" not in raw:
+ raw = "http://" + raw
+ return raw.rstrip("/")
+
+
def _ensure_router() -> Any:
"""Build the Router from the environment, following the laya.serve contract.
@@ -145,6 +161,19 @@ def _ensure_router() -> Any:
with _ROUTER_LOCK:
if _ROUTER is not None:
return _ROUTER
+ base_url = _remote_base_url()
+ if base_url:
+ # Remote mode: no checkpoint is built here, no torch is imported. The RemoteRouter
+ # routes locally (pure Python) and answers over HTTP from a running laya-serve.
+ try:
+ from .remote import RemoteRouter
+ router = RemoteRouter(base_url, api_key=os.environ.get("LAYA_API_KEY") or None,
+ auto_task_detection=_env_bool("LAYA_AUTO_TASK", False),
+ **_default_model_option())
+ except Exception as exc:
+ raise ToolError("internal_error", f"remote router construction failed: {exc}") from exc
+ _ROUTER = router
+ return router
try:
from laya import Router
except Exception as exc:
@@ -205,7 +234,11 @@ def _wrap(fn, **kwargs) -> str:
@server.tool(name="laya_status")
def laya_status_tool() -> str:
"""Report the device actually in use per loaded checkpoint (or the configured preference, flagged as such, when nothing is loaded), torch CUDA availability, loaded checkpoints, and package versions."""
- return _wrap(laya_status, router=_ROUTER, preload=_env_bool("LAYA_PRELOAD", True))
+ # In remote mode the router is cheap (no checkpoint, no torch), so build it here if a status
+ # call comes first: the remote report then describes the server instead of a torch probe of
+ # this process, and `device_report()`'s torch import never happens in a remote MCP process.
+ router = _ROUTER if (_ROUTER is not None or not _remote_base_url()) else _router_or_error()
+ return _wrap(laya_status, router=router, preload=_env_bool("LAYA_PRELOAD", True))
@server.tool(
@@ -478,7 +511,11 @@ def main() -> None:
if os.name == "nt":
os.environ.setdefault("HF_HUB_DISABLE_SYMLINKS", "1")
os.environ.setdefault("HF_HUB_DISABLE_SYMLINKS_WARNING", "1")
- if _env_bool("LAYA_PRELOAD", True):
+ if _remote_base_url():
+ # Nothing to preload: the server at LAYA_BASE_URL owns the checkpoints. Say so once, so a
+ # reader of the client's stderr knows which process is answering.
+ print(f"[laya-mcp] remote mode: answering from laya-serve at {_remote_base_url()}", file=sys.stderr)
+ elif _env_bool("LAYA_PRELOAD", True):
try:
_ensure_router()
except Exception as exc:
diff --git a/laya/mcp/tools.py b/laya/mcp/tools.py
index 2ac1c22f..80539d47 100644
--- a/laya/mcp/tools.py
+++ b/laya/mcp/tools.py
@@ -342,6 +342,14 @@ def _overrides(task: Any, lang: Any, max_len: Any, head_max_len: Any) -> dict:
return {name: value for name, value in values.items() if value is not None}
+def _raise_if_remote_error(exc: BaseException) -> None:
+ """Turn transport failures into the ToolError the client can read."""
+ from .remote import RemoteError
+
+ if isinstance(exc, RemoteError):
+ raise ToolError(exc.code, exc.message) from exc
+
+
def _normalize_answers(raw: Any) -> dict:
if not isinstance(raw, dict):
raise ToolError("internal_error", "predict returned non-object answers")
@@ -457,7 +465,11 @@ def laya_predict(
return router.predict(state_d, questions_d, model=model_name, **budget)
started = time.perf_counter()
- result = _run()
+ try:
+ result = _run()
+ except Exception as exc:
+ _raise_if_remote_error(exc)
+ raise
latency_ms = (time.perf_counter() - started) * 1000.0
norm = _normalize_result(result)
@@ -589,6 +601,13 @@ def _resident_or_load(router: Any, name: str) -> Any:
resident = router_agent(router, name)
if resident is not None:
return resident
+ if hasattr(router, "base_url") and hasattr(router, "health"):
+ raise ToolError(
+ "unsupported_remote",
+ "laya_shortlist embeds options with the answering checkpoint in-process; it is not "
+ "available when LAYA_BASE_URL points the MCP server at a remote laya-serve. Use "
+ "laya_predict with head_max_len raised, or run the MCP server without LAYA_BASE_URL.",
+ )
load = getattr(router, "load", None)
if load is None:
raise ToolError("models_not_ready", f"checkpoint {name!r} is not loaded")
@@ -657,6 +676,15 @@ def laya_shortlist(
(``lang_guess`` only routes, so like ``task`` it is stripped before the answering pass);
``min_confidence`` flags a kept-label answer the checkpoint is unsure of.
"""
+ # Before the heavy imports below: a remote router has no in-process encoder to embed with,
+ # and the refusal should not cost the MCP process a torch import on the way to saying so.
+ if router is not None and agent is None and hasattr(router, "base_url") and hasattr(router, "health"):
+ raise ToolError(
+ "unsupported_remote",
+ "laya_shortlist embeds options with the answering checkpoint in-process; it is not "
+ "available when LAYA_BASE_URL points the MCP server at a remote laya-serve. Use "
+ "laya_predict with head_max_len raised, or run the MCP server without LAYA_BASE_URL.",
+ )
# Lazy: keeps numpy/shortlist out of module import for laya.mcp.tools.
from laya.shortlist import (
DEFAULT_SHORTLIST_K,
@@ -822,7 +850,39 @@ def laya_preset(
)
+def _remote_status(router: Any, preload: bool) -> dict:
+ """``laya_status`` for a remote router: the server's own /health, no torch import here."""
+ versions: dict[str, str | None] = {"laya": None}
+ try:
+ versions["laya"] = getattr(__import__("laya"), "__version__", "unknown")
+ except Exception:
+ pass
+ out: dict[str, Any] = {
+ "mode": "remote",
+ "base_url": router.base_url,
+ "router_preload": False,
+ "router_ready": True,
+ "package_versions": versions,
+ }
+ try:
+ health = router.health()
+ except Exception as exc: # noqa: BLE001 -- the status tool reports, it never raises
+ out["server"] = None
+ out["server_error"] = f"{type(exc).__name__}: {exc}"
+ out["loaded"] = []
+ return out
+ out["server"] = health
+ loaded = health.get("loaded")
+ out["loaded"] = [v for v in loaded if isinstance(v, str)] if isinstance(loaded, list) else []
+ for key in ("device", "device_is_preference", "checkpoint_devices"):
+ if key in health:
+ out[key] = health[key]
+ return out
+
+
def laya_status(*, router: Any = None, loaded: list[str] | None = None, preload: bool = True) -> dict:
+ if router is not None and hasattr(router, "base_url") and hasattr(router, "health"):
+ return _remote_status(router, preload)
report = device_report()
versions: dict[str, str | None] = {
"laya": None,
@@ -1038,7 +1098,11 @@ def laya_predict_batch(
if sort_by_length:
kwargs["sort_by_length"] = True
results = router.predict_batch(items, **kwargs)
- except TypeError as exc:
+ except Exception as exc:
+ _raise_if_remote_error(exc)
+ if not isinstance(exc, TypeError):
+ raise
+
# A Router without batch support raises at the call itself; anything
# else is a real bug and must surface unchanged.
raise ToolError(
@@ -1161,6 +1225,9 @@ def laya_decide(
model=model_name, min_confidence=min_conf)
except SchemaError as exc:
raise ToolError("invalid_schema", str(exc)) from exc
+ except Exception as exc:
+ _raise_if_remote_error(exc)
+ raise
latency_ms = (time.perf_counter() - started) * 1000.0
answers = _normalize_answers(details.answers)
diff --git a/scripts/test_suites.py b/scripts/test_suites.py
index ffe1ab20..fbab819a 100644
--- a/scripts/test_suites.py
+++ b/scripts/test_suites.py
@@ -99,6 +99,7 @@ SCRIPT_SUITES = [
PYTEST_SUITES = [
"tests/test_serve.py",
+ "tests/test_mcp_remote.py",
"tests/test_router_batch.py",
"tests/test_predict_batch.py",
"tests/test_system_one_lang.py",
diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.py
index a8aa4957..e9e2b753 100644
--- a/tests/test_hooks_api.py
+++ b/tests/test_hooks_api.py
@@ -105,6 +105,20 @@ check_param("load", load, "compile", False)
check_param("ONNXAgent.__init__", ONNXAgent.__init__, "token", None)
# --------------------------------------------------------------- predict surfaces
+from laya.mcp.remote import RemoteRouter # noqa: E402
+
+check_true("RemoteRouter/is a Router", issubclass(RemoteRouter, Router))
+check_param("RemoteRouter.__init__", RemoteRouter.__init__, "base_url", inspect.Parameter.empty)
+check_param("RemoteRouter.__init__", RemoteRouter.__init__, "api_key", None)
+check_param("RemoteRouter.__init__", RemoteRouter.__init__, "timeout", None)
+for method in ("predict", "predict_batch"):
+ check("RemoteRouter/%s signature" % method,
+ [(p.name, p.kind, p.default) for p in sig(getattr(RemoteRouter, method)).values()],
+ [(p.name, p.kind, p.default) for p in sig(getattr(Router, method)).values()])
+for method in ("route", "route_batch"):
+ check_true("RemoteRouter/%s stays local" % method,
+ getattr(RemoteRouter, method) is getattr(Router, method))
+
for label, fn in (("Agent.predict_batch", Agent.predict_batch),
("Agent.system_one", Agent.system_one),
("Agent.predict_long", Agent.predict_long),
diff --git a/tests/test_mcp_remote.py b/tests/test_mcp_remote.py
new file mode 100644
index 00000000..ef412075
--- /dev/null
+++ b/tests/test_mcp_remote.py
@@ -0,0 +1,375 @@
+"""Remote mode of the MCP server (LAYA_BASE_URL): answers come from a laya-serve over HTTP.
+
+A stdlib HTTP server stands in for laya-serve, so these tests need no checkpoint, no torch
+and no external network. Run: python -m pytest tests/test_mcp_remote.py
+"""
+import json
+import threading
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+import pytest
+
+pytest.importorskip("mcp")
+
+from laya.mcp import server as mcp_server # noqa: E402
+from laya.mcp.tools import ( # noqa: E402
+ ToolError, laya_decide, laya_predict, laya_predict_batch, laya_preset, laya_route, laya_route_batch,
+ laya_shortlist, laya_status,
+)
+from laya.mcp.remote import DEFAULT_TIMEOUT_S, RemoteError, RemoteRouter # noqa: E402
+
+ANSWER = {
+ "model": "laya-rl-agent",
+ "answers": {"dept": {"type": "choice", "choice": "billing",
+ "probabilities": {"billing": 0.9, "tech": 0.1}, "confidence": 0.9}},
+ "usage": {"input_tokens": 40, "output_tokens": 0},
+ "routing": {"model": "english", "repo": "convaiinnovations/laya", "reason": "English Latin text"},
+}
+HEALTH = {"status": "ok", "loaded": ["english"], "revisions": {"english": "abc"},
+ "device": "mps", "device_is_preference": False, "checkpoint_devices": {"english": "mps"}}
+
+
+class FakeServe(BaseHTTPRequestHandler):
+ """Records every request; answers /health and /v1/systemone like laya-serve."""
+
+ requests: list = []
+ status_override: int | None = None
+ detail_override: str = "bad request"
+ require_bearer: str | None = None
+
+ def log_message(self, *args): # quiet
+ pass
+
+ def _send(self, status, payload):
+ body = json.dumps(payload).encode()
+ self.send_response(status)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ def do_GET(self):
+ FakeServe.requests.append(("GET", self.path, None, dict(self.headers)))
+ if self.path == "/health":
+ if FakeServe.require_bearer and self.headers.get("Authorization") != "Bearer " + FakeServe.require_bearer:
+ return self._send(200, {"status": "ok"})
+ return self._send(200, HEALTH)
+ self._send(404, {"detail": "Not Found"})
+
+ def do_POST(self):
+ length = int(self.headers.get("Content-Length") or 0)
+ body = json.loads(self.rfile.read(length)) if length else None
+ FakeServe.requests.append(("POST", self.path, body, dict(self.headers)))
+ if FakeServe.require_bearer and self.headers.get("Authorization") != "Bearer " + FakeServe.require_bearer:
+ return self._send(401, {"detail": "invalid or missing bearer token"})
+ if FakeServe.status_override:
+ return self._send(FakeServe.status_override, {"detail": FakeServe.detail_override})
+ if self.path == "/v1/systemone":
+ return self._send(200, ANSWER)
+ self._send(404, {"detail": "Not Found"})
+
+
+def serve():
+ FakeServe.requests = []
+ FakeServe.status_override = None
+ FakeServe.require_bearer = None
+ httpd = HTTPServer(("127.0.0.1", 0), FakeServe)
+ thread = threading.Thread(target=lambda: httpd.serve_forever(poll_interval=0.01), daemon=True)
+ thread.start()
+ try:
+ yield "http://127.0.0.1:%d" % httpd.server_address[1]
+ finally:
+ httpd.shutdown()
+ httpd.server_close()
+
+
+def _questions():
+ return {"dept": {"type": "choice", "instructions": "which team?", "criteria": {"billing": None, "tech": None}}}
+
+
+def test_predict_posts_the_serve_body_and_returns_it_unchanged(serve):
+ router = RemoteRouter(serve)
+ result = router.predict({"text": "billed twice"}, _questions(), lang="en", max_len=512, min_confidence=0.3)
+ assert result == ANSWER
+ method, path, body, headers = FakeServe.requests[-1]
+ assert (method, path) == ("POST", "/v1/systemone")
+ assert body == {"state": {"text": "billed twice"}, "questions": _questions(),
+ "lang": "en", "max_len": 512, "min_confidence": 0.3}
+ assert "Authorization" not in headers
+
+
+def test_auto_model_is_not_sent_but_a_pinned_one_is(serve):
+ router = RemoteRouter(serve)
+ router.predict("x", _questions(), model="auto")
+ assert "model" not in FakeServe.requests[-1][2]
+ router.predict("x", _questions(), model="multilingual")
+ assert FakeServe.requests[-1][2]["model"] == "multilingual"
+
+
+def test_bearer_token_is_sent_when_configured(serve):
+ FakeServe.require_bearer = "s3cret"
+ router = RemoteRouter(serve, api_key="s3cret")
+ assert router.predict("x", _questions()) == ANSWER
+ assert FakeServe.requests[-1][3]["Authorization"] == "Bearer s3cret"
+ with pytest.raises(RemoteError) as info:
+ RemoteRouter(serve, api_key="wrong").predict("x", _questions())
+ assert info.value.code == "unauthorized"
+
+
[email protected]("status, code", [(400, "invalid_request"), (422, "invalid_request"),
+ (503, "busy"), (500, "upstream_error")])
+def test_http_errors_map_to_tool_error_codes(serve, status, code):
+ FakeServe.status_override = status
+ FakeServe.detail_override = "question 'dept': criteria must be an object"
+ with pytest.raises(RemoteError) as info:
+ RemoteRouter(serve).predict("x", _questions())
+ assert info.value.code == code
+ assert "criteria must be an object" in info.value.message
+
+
+def test_unreachable_server_is_a_clean_error():
+ with pytest.raises(RemoteError) as info:
+ RemoteRouter("http://127.0.0.1:9", timeout=2).predict("x", _questions())
+ assert info.value.code == "unreachable"
+
+
+def test_loaded_and_revisions_come_from_health(serve):
+ router = RemoteRouter(serve)
+ assert router.loaded == ["english"]
+ assert router.loaded_revisions == {"english": "abc"}
+ assert RemoteRouter("http://127.0.0.1:9", timeout=2).loaded == []
+
+
+def test_routing_stays_local_and_needs_no_server(monkeypatch):
+ router = RemoteRouter("http://127.0.0.1:9", timeout=2)
+ def no_http(*args, **kwargs):
+ raise AssertionError("routing attempted an HTTP request")
+ monkeypatch.setattr(router, "_request", no_http)
+ decision = router.route({"text": "Mein Konto wurde zweimal belastet"}, _questions())
+ assert decision.model == "multilingual"
+ assert router.route_batch([{"state": "Mein Konto wurde zweimal belastet", "questions": _questions()}])[0].model == "multilingual"
+
+
+def test_predict_batch_is_one_call_per_request_in_order(serve):
+ router = RemoteRouter(serve)
+ items = [{"state": "a", "questions": _questions()},
+ {"state": "b", "questions": _questions(), "model": "english", "max_len": 256}]
+ results = router.predict_batch(items, batch_size=8, sort_by_length=True, min_confidence=0.5)
+ assert results == [ANSWER, ANSWER]
+ posts = [r for r in FakeServe.requests if r[0] == "POST"]
+ assert [p[2]["state"] for p in posts] == ["a", "b"]
+ assert posts[1][2]["model"] == "english" and posts[1][2]["max_len"] == 256
+ assert all(p[2]["min_confidence"] == 0.5 for p in posts)
+
+
+def test_lifecycle_is_server_owned(serve):
+ router = RemoteRouter(serve)
+ assert router.preload(["english"]) is router
+ assert router.unload() is None
+ with pytest.raises(RemoteError) as info:
+ router.load("english")
+ assert info.value.code == "unsupported_remote"
+
+
+def test_tools_translate_remote_errors(serve):
+ FakeServe.status_override = 422
+ FakeServe.detail_override = "question 'dept': unknown type"
+ with pytest.raises(ToolError) as info:
+ laya_predict({"text": "x"}, _questions(), router=RemoteRouter(serve))
+ assert info.value.code == "invalid_request" and "unknown type" in info.value.message
+ with pytest.raises(ToolError) as info:
+ laya_predict_batch([{"state": {"text": "x"}, "questions": _questions()}], router=RemoteRouter(serve))
+ assert info.value.code == "invalid_request"
+
+
+def test_predict_tool_shape_through_a_remote_router(serve):
+ out = laya_predict({"text": "billed twice"}, _questions(), router=RemoteRouter(serve))
+ assert out["answers"] == ANSWER["answers"]
+ assert out["routing"]["model"] == "english"
+ assert "device" not in out # no in-process agent to read a device from
+ assert out["latency_ms"] >= 0
+
+
+def test_status_reports_the_server_not_local_torch(serve):
+ out = laya_status(router=RemoteRouter(serve), preload=False)
+ assert out["mode"] == "remote" and out["base_url"] == serve
+ assert out["loaded"] == ["english"] and out["device"] == "mps"
+ assert out["server"]["status"] == "ok"
+ assert "torch_version" not in out
+ down = laya_status(router=RemoteRouter("http://127.0.0.1:9", timeout=2))
+ assert down["mode"] == "remote" and down["loaded"] == [] and "server_error" in down
+
+
+def test_shortlist_is_refused_in_remote_mode(serve):
+ many = {"pick": {"type": "choice", "instructions": "pick", "criteria": {"opt%d" % i: None for i in range(60)}}}
+ with pytest.raises(ToolError) as info:
+ laya_shortlist({"text": "x"}, many, k=5, router=RemoteRouter(serve))
+ assert info.value.code == "unsupported_remote"
+
+
+def test_server_builds_a_remote_router_from_the_environment(monkeypatch, serve):
+ monkeypatch.setenv("LAYA_BASE_URL", serve + "/")
+ monkeypatch.setenv("LAYA_API_KEY", "k")
+ monkeypatch.setattr(mcp_server, "_ROUTER", None)
+ router = mcp_server._ensure_router()
+ assert isinstance(router, RemoteRouter)
+ assert router.base_url == serve and router.api_key == "k"
+ monkeypatch.setattr(mcp_server, "_ROUTER", None)
+
+
[email protected]("raw, expected", [("", None), (" ", None), ("127.0.0.1:8123", "http://127.0.0.1:8123"),
+ ("http://h:1/", "http://h:1"), ("https://laya.example", "https://laya.example")])
+def test_base_url_normalisation(monkeypatch, raw, expected):
+ monkeypatch.setenv("LAYA_BASE_URL", raw)
+ assert mcp_server._remote_base_url() == expected
+
+
[email protected]("raw", ["bad", "0", "-1", "nan", "inf", "1e999"])
+def test_remote_timeout_rejects_invalid_environment(monkeypatch, raw):
+ monkeypatch.setenv("LAYA_REMOTE_TIMEOUT", raw)
+ assert RemoteRouter("http://localhost:8000").timeout == DEFAULT_TIMEOUT_S
+
+
[email protected]("timeout", [0, -1, float("nan"), float("inf")])
+def test_remote_timeout_rejects_invalid_explicit_values(timeout):
+ with pytest.raises(ValueError, match="finite positive"):
+ RemoteRouter("http://localhost:8000", timeout=timeout)
+
+
[email protected]("key, value", [
+ ("hooks", []), ("on_predict_start", lambda ctx: None), ("on_predict_end", lambda ctx: None),
+ ("hooks_raise", False), ("hooks_timeout", 1), ("lang_guess", lambda state: "en"),
+])
+def test_remote_predict_refuses_controls_without_a_wire_form(serve, key, value):
+ with pytest.raises(RemoteError) as info:
+ RemoteRouter(serve).predict("x", _questions(), **{key: value})
+ assert info.value.code == "unsupported_remote"
+ assert not FakeServe.requests
+
+
+def test_remote_batch_refuses_hook_timeouts_and_item_hooks(serve):
+ router = RemoteRouter(serve)
+ items = [{"state": "x", "questions": _questions()}]
+ with pytest.raises(RemoteError, match="hooks_timeout"):
+ router.predict_batch(items, hooks_timeout=1)
+ items[0]["hooks"] = [lambda ctx: None]
+ with pytest.raises(RemoteError, match="per-request hooks"):
+ router.predict_batch(items)
+ assert not FakeServe.requests
+
+
+def test_remote_decide_and_preset_use_the_server(serve):
+ router = RemoteRouter(serve)
+ schema = {"type": "object", "properties": {"dept": {"type": "string", "enum": ["billing", "tech"]}}}
+ result = laya_decide({"text": "billed twice"}, schema, router=router)
+ assert result["values"] == {"dept": "billing"}
+ assert FakeServe.requests[-1][2]["questions"]["dept"]["criteria"] == {"billing": None, "tech": None}
+ laya_preset("triage", {"text": "billed twice"}, router=router, preset_builder=mcp_server._preset_builder)
+ assert FakeServe.requests[-1][1] == "/v1/systemone"
+ FakeServe.status_override = 503
+ for fn in (lambda: laya_decide({"text": "x"}, schema, router=router),
+ lambda: laya_preset("triage", {"text": "x"}, router=router, preset_builder=mcp_server._preset_builder)):
+ with pytest.raises(ToolError) as info:
+ fn()
+ assert info.value.code == "busy"
+
+
+def test_remote_route_tools_do_not_contact_the_server(serve):
+ router = RemoteRouter(serve)
+ state = {"text": "Mein Konto wurde zweimal belastet"}
+ assert laya_route(state, _questions(), router=router)["model"] == "multilingual"
+ assert laya_route_batch([{"state": state, "questions": _questions()}], router=router)["decisions"][0]["model"] == "multilingual"
+ assert not FakeServe.requests
+
+
[email protected]("url", ["file:///tmp/model", "localhost:8000", "http://", "http://host?q=x", "http://host#x"])
+def test_remote_router_requires_an_http_server_url(url):
+ with pytest.raises(ValueError, match="HTTP"):
+ RemoteRouter(url)
+
+
+def test_remote_mcp_startup_and_every_tool_avoid_torch(serve):
+ import os
+ import subprocess
+ import sys
+ import textwrap
+
+ code = textwrap.dedent('''
+ import builtins, json, sys
+ original_import = builtins.__import__
+ def no_torch(name, *args, **kwargs):
+ if name == "torch" or name.startswith("torch."):
+ raise AssertionError("remote MCP imported torch")
+ return original_import(name, *args, **kwargs)
+ builtins.__import__ = no_torch
+ from laya.mcp import server
+ server.server.run = lambda: None
+ server.main()
+ state = {"text": "billed twice"}
+ questions = {"dept": {"type": "choice", "instructions": "which team?",
+ "criteria": {"billing": "payments", "tech": "bugs"}}}
+ schema = {"type": "object", "properties": {"dept": {"type": "string", "enum": ["billing", "tech"]}}}
+ items = [{"state": state, "questions": questions}]
+ calls = [lambda: server.laya_status_tool(),
+ lambda: server.laya_predict_tool(state, questions),
+ lambda: server.laya_predict_batch_tool(items),
+ lambda: server.laya_decide_tool(state, schema),
+ lambda: server.laya_preset_tool("triage", state),
+ lambda: server.laya_route_tool(state, questions),
+ lambda: server.laya_route_batch_tool(items)]
+ for call in calls:
+ out = json.loads(call())
+ assert "error" not in out, out
+ try:
+ server.laya_shortlist_tool(state, questions)
+ except Exception as exc:
+ assert "unsupported_remote" in str(exc), exc
+ else:
+ raise AssertionError("remote shortlist did not refuse the call")
+ assert "torch" not in sys.modules
+ assert not server._ROUTER._agents
+ ''')
+ env = dict(os.environ, LAYA_BASE_URL=serve, LAYA_THREADS="2", LAYA_PRELOAD="1")
+ result = subprocess.run([sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=15)
+ assert result.returncode == 0, result.stderr
+
+
+def test_remote_timeout_is_forwarded_and_http_failures_are_validated(monkeypatch, serve):
+ monkeypatch.setenv("LAYA_REMOTE_TIMEOUT", "0.25")
+ router = RemoteRouter(serve)
+ assert router.timeout == 0.25
+ assert RemoteRouter(serve, timeout=3).timeout == 3
+ from urllib.error import URLError
+ import urllib.request
+ def timed_out(*args, **kwargs):
+ raise URLError(TimeoutError("test timeout"))
+ monkeypatch.setattr(urllib.request, "urlopen", timed_out)
+ with pytest.raises(RemoteError) as info:
+ router.predict("x", _questions())
+ assert info.value.code == "timeout"
+
+
+def test_remote_transport_preserves_options_and_all_wire_controls(serve):
+ questions = {"dept": {"type": "choice", "instructions": "Which team handles `message`?",
+ "criteria": {"billing": "refunds", "tech": "bugs"}}}
+ controls = {"task": "typed_decisions", "lang": "de", "lang_guess": "de", "max_len": 512,
+ "head_max_len": 128, "min_confidence": 0.7}
+ state = {"message": "Rückerstattung bitte"}
+ RemoteRouter(serve).predict(state, questions, **controls)
+ assert FakeServe.requests[-1][2] == dict(state=state, questions=questions, **controls)
+
+
[email protected]("raw", [b"not JSON", b"\xff", b"[]", b"null"])
+def test_remote_transport_reports_malformed_responses(monkeypatch, raw):
+ from contextlib import contextmanager
+ from types import SimpleNamespace
+ import urllib.request
+
+ @contextmanager
+ def response(*args, **kwargs):
+ yield SimpleNamespace(read=lambda: raw)
+ monkeypatch.setattr(urllib.request, "urlopen", response)
+ with pytest.raises(RemoteError) as info:
+ RemoteRouter("http://localhost:8000").predict("x", _questions())
+ assert info.value.code == "upstream_error"