Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(tools): condition latency claims with hardware profiles and scope exports to integrations
LEVELING21083777bbd7cf80
diff --git a/docs/tools.md b/docs/tools.mdindex 3ccc7fd2..bcc1b4b3 100644--- a/docs/tools.md+++ b/docs/tools.md@@ -1,11 +1,11 @@# Tool & Function Selection-Laya provides sub-35ms, non-autoregressive tool routing and function selection for AI agent workflows across **LangChain**, **CrewAI**, **LlamaIndex**, and vanilla agent architectures (single-question latency measured at **32.8 ms** with `laya-multilingual` and **39.5 ms** with `laya` on a Tesla T4 GPU; 193–464 ms on CPU):+Laya provides non-autoregressive tool routing and function selection for AI agent workflows across **LangChain**, **CrewAI**, **LlamaIndex**, and vanilla agent architectures (single-question latency measured at **32.8 ms** with `laya-multilingual` and **39.5 ms** with `laya` on a Tesla T4 GPU; **p50 125.6 ms** on Apple Silicon; **193–464 ms** on CPU):-* **`LayaToolSelector`**: Sub-35ms tool router evaluating user queries and agent states against candidate tools in a single forward pass.+* **`LayaToolSelector`**: Tool routing component evaluating user queries and agent states against candidate tools in a single forward pass (~33–40 ms on Tesla T4 GPU, ~88–464 ms on CPU/Apple Silicon).* **`ToolRouteDecision`**: Structured routing result carrying the selected tool, tool index, calibrated confidence, probability distribution across tools, and fallback status.-Replaces slow, token-generating LLM tool-calling (1,000–2,500 ms) with deterministic, typed decisions executed in <35 ms with **zero token generation cost**.+Replaces slow, token-generating LLM tool-calling (1,000–2,500 ms) with deterministic, typed decisions executed in a single forward pass with **zero token generation cost**.Supports both **local in-process inference** (`Agent` or `Router`) and **remote HTTP inference** against your own `laya-serve` instance without requiring PyTorch on edge clients.@@ -19,9 +19,9 @@ pip install laya----## 1. Sub-35ms Tool Selection+## 1. Single-Pass Tool Selection-Autoregressive LLMs (such as GPT-4o or Claude 3.5 Sonnet) take 1–2 seconds generating reasoning tokens just to choose which tool to call. `LayaToolSelector` evaluates candidate tools against the query in **~33 ms**:+Autoregressive LLMs (such as GPT-4o or Claude 3.5 Sonnet) take 1–2 seconds generating reasoning tokens just to choose which tool to call. `LayaToolSelector` evaluates candidate tools against the query in a single forward pass (~33–40 ms on Tesla T4 GPU, ~88–126 ms on Apple Silicon, and 193–464 ms on CPU):```pythonfrom laya.integrations.tools import LayaToolSelector@@ -40,10 +40,10 @@ def query_database(sql: str):tools = [search_web, calculate_mortgage, query_database]-# Initialize sub-35ms tool selector+# Initialize tool selectorselector = LayaToolSelector(tools=tools)-# Select the matching tool in <35ms:+# Select the matching tool:decision = selector.select("What is the current mortgage rate on a 30-year fixed loan?")print(f"Selected Tool: {decision.tool_name}")print(f"Confidence: {decision.confidence:.3f}")diff --git a/examples/tool_selection_quickstart.py b/examples/tool_selection_quickstart.pyindex 12a2201a..7c4b03aa 100644--- a/examples/tool_selection_quickstart.py+++ b/examples/tool_selection_quickstart.py@@ -1,7 +1,7 @@"""Laya System 1 decision engine: Tool and Function Selection Quickstart.Demonstrates:-1. Sub-35ms tool selection for AI agent workflows (replaces 1-2s LLM tool calling).+1. Non-autoregressive tool selection for AI agent workflows (replaces 1-2s LLM tool calling).2. Direct execution via .call() with function arguments.3. Direct answer detection when no tool is needed (allow_direct_answer=True).4. Calibrated confidence threshold fallback gating for ambiguous requests.@@ -78,14 +78,14 @@ def general_reasoning_fallback(prompt: str) -> str:def main():print("=" * 70)- print(" Laya System 1: Sub-35ms Tool and Function Selection")+ print(" Laya System 1: Tool and Function Selection")print("=" * 70)tools = [search_web, calculate_tax, query_database]agent = _DemoAgent()# 1. Standard Tool Selection- print("\n[1] Sub-35ms Tool Selection:")+ print("\n[1] Single-Pass Tool Selection:")selector = LayaToolSelector(tools=tools, agent=agent)queries = [diff --git a/laya/__init__.py b/laya/__init__.pyindex bb85bbd5..16e4ef2e 100644--- a/laya/__init__.py+++ b/laya/__init__.py@@ -77,8 +77,6 @@ _LAZY_ATTRS = {"LayaTriage": (".integrations", "LayaTriage"),"LayaEvaluator": (".integrations", "LayaEvaluator"),"LayaDecision": (".integrations", "LayaDecision"),- "LayaToolSelector": (".integrations", "LayaToolSelector"),- "ToolRouteDecision": (".integrations", "ToolRouteDecision"),}@@ -146,8 +144,6 @@ __all__ = ["LayaTriage","LayaEvaluator","LayaDecision",- "LayaToolSelector",- "ToolRouteDecision","PredictContext","PredictHook","Hook",diff --git a/laya/integrations/tools.py b/laya/integrations/tools.pyindex b8d8d017..bc30df78 100644--- a/laya/integrations/tools.py+++ b/laya/integrations/tools.py@@ -1,6 +1,6 @@"""Tool and function selection for agent frameworks using Laya System 1 decision engine.-Provides sub-35ms, non-autoregressive tool routing and function selection for AI agent+Provides non-autoregressive tool routing and function selection for AI agentworkflows (LangChain, CrewAI, LlamaIndex, AutoGen, and custom agent loops).Replaces slow, token-generating LLM tool calling (1,000-2,500ms) with calibrated, typed@@ -39,7 +39,7 @@ def _gated_confidence(answer: Dict[str, Any]) -> Optional[float]:@dataclassclass ToolRouteDecision:- """Result of a Laya sub-35ms tool routing decision."""+ """Result of a Laya tool routing decision."""tool: Anytool_name: str@@ -52,6 +52,13 @@ class ToolRouteDecision:raw_decision: Dict[str, Any]+__all__ = [+ "LayaToolSelector",+ "ToolRouteDecision",+ "LayaLowConfidenceError",+]++def _extract_tool_metadata(tool: Any, default_index: int) -> Tuple[str, str]:"""Extract (name, description) from any tool representation."""if isinstance(tool, str):@@ -226,10 +233,11 @@ def _execute_decision(class LayaToolSelector:- """Sub-35ms tool and function router for AI agent workflows.+ """Non-autoregressive tool and function router for AI agent workflows.- Evaluates user queries and agent states against candidate tools in a single- non-autoregressive forward pass, eliminating 1-2 second LLM tool-calling latency.+ Evaluates candidate tools against a user prompt or agent state in a single+ forward pass (measured at 32.8 ms with `laya-multilingual` and 39.5 ms with+ `laya` on a Tesla T4 GPU; ~88–126 ms on Apple Silicon; 193–464 ms on CPU).Supports Python functions, LangChain/CrewAI/LlamaIndex tools, and JSON Schema specs.Works seamlessly with local in-process models (`Agent` / `Router`) and remote@@ -299,7 +307,7 @@ class LayaToolSelector:tools: Optional[Sequence[Any]] = None,instructions: Optional[str] = None,) -> ToolRouteDecision:- """Select the best candidate tool for the given query/state in sub-35ms."""+ """Select the best candidate tool for the given query/state in a single forward pass."""candidate_tools = tools if tools is not None else self.toolsif not candidate_tools and not self.allow_direct_answer:raise ValueError("No tools provided to LayaToolSelector.")diff --git a/tests/test_tools.py b/tests/test_tools.pyindex 0371ca25..dbdb1cf1 100644--- a/tests/test_tools.py+++ b/tests/test_tools.py@@ -367,8 +367,10 @@ for ctrl in HOOK_CONTROLS:integrations_mod = __import__("laya.integrations", fromlist=["__all__"])check_true("export/integrations_selector", "LayaToolSelector" in integrations_mod.__all__)check_true("export/integrations_decision", "ToolRouteDecision" in integrations_mod.__all__)-check_true("export/laya_selector", "LayaToolSelector" in laya.__all__)-check_true("export/laya_decision", "ToolRouteDecision" in laya.__all__)+tools_mod = __import__("laya.integrations.tools", fromlist=["__all__"])+check_true("export/tools_selector", "LayaToolSelector" in tools_mod.__all__)+check_true("export/tools_decision", "ToolRouteDecision" in tools_mod.__all__)+check_true("export/top_level_clean", "LayaToolSelector" not in laya.__all__ and "ToolRouteDecision" not in laya.__all__)# --------------------------------------------------------------- Report