Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(tools): condition latency claims with hardware profiles and scope exports to integrations

LEVELING21083777bbd7cf80
diff --git a/docs/tools.md b/docs/tools.md
index 3ccc7fd2..bcc1b4b3 100644
--- a/docs/tools.md
+++ b/docs/tools.md
@@ -1,11 +1,11 @@
# Tool & Function Selection
-Laya provides sub-35ms, non-autoregressive tool routing and function selection for AI agent workflows across **LangChain**, **CrewAI**, **LlamaIndex**, and vanilla agent architectures (single-question latency measured at **32.8 ms** with `laya-multilingual` and **39.5 ms** with `laya` on a Tesla T4 GPU; 193–464 ms on CPU):
+Laya provides non-autoregressive tool routing and function selection for AI agent workflows across **LangChain**, **CrewAI**, **LlamaIndex**, and vanilla agent architectures (single-question latency measured at **32.8 ms** with `laya-multilingual` and **39.5 ms** with `laya` on a Tesla T4 GPU; **p50 125.6 ms** on Apple Silicon; **193–464 ms** on CPU):
-* **`LayaToolSelector`**: Sub-35ms tool router evaluating user queries and agent states against candidate tools in a single forward pass.
+* **`LayaToolSelector`**: Tool routing component evaluating user queries and agent states against candidate tools in a single forward pass (~33–40 ms on Tesla T4 GPU, ~88–464 ms on CPU/Apple Silicon).
* **`ToolRouteDecision`**: Structured routing result carrying the selected tool, tool index, calibrated confidence, probability distribution across tools, and fallback status.
-Replaces slow, token-generating LLM tool-calling (1,000–2,500 ms) with deterministic, typed decisions executed in <35 ms with **zero token generation cost**.
+Replaces slow, token-generating LLM tool-calling (1,000–2,500 ms) with deterministic, typed decisions executed in a single forward pass with **zero token generation cost**.
Supports both **local in-process inference** (`Agent` or `Router`) and **remote HTTP inference** against your own `laya-serve` instance without requiring PyTorch on edge clients.
@@ -19,9 +19,9 @@ pip install laya
---
-## 1. Sub-35ms Tool Selection
+## 1. Single-Pass Tool Selection
-Autoregressive LLMs (such as GPT-4o or Claude 3.5 Sonnet) take 1–2 seconds generating reasoning tokens just to choose which tool to call. `LayaToolSelector` evaluates candidate tools against the query in **~33 ms**:
+Autoregressive LLMs (such as GPT-4o or Claude 3.5 Sonnet) take 1–2 seconds generating reasoning tokens just to choose which tool to call. `LayaToolSelector` evaluates candidate tools against the query in a single forward pass (~33–40 ms on Tesla T4 GPU, ~88–126 ms on Apple Silicon, and 193–464 ms on CPU):
```python
from laya.integrations.tools import LayaToolSelector
@@ -40,10 +40,10 @@ def query_database(sql: str):
tools = [search_web, calculate_mortgage, query_database]
-# Initialize sub-35ms tool selector
+# Initialize tool selector
selector = LayaToolSelector(tools=tools)
-# Select the matching tool in <35ms:
+# Select the matching tool:
decision = selector.select("What is the current mortgage rate on a 30-year fixed loan?")
print(f"Selected Tool: {decision.tool_name}")
print(f"Confidence: {decision.confidence:.3f}")
diff --git a/examples/tool_selection_quickstart.py b/examples/tool_selection_quickstart.py
index 12a2201a..7c4b03aa 100644
--- a/examples/tool_selection_quickstart.py
+++ b/examples/tool_selection_quickstart.py
@@ -1,7 +1,7 @@
"""Laya System 1 decision engine: Tool and Function Selection Quickstart.
Demonstrates:
-1. Sub-35ms tool selection for AI agent workflows (replaces 1-2s LLM tool calling).
+1. Non-autoregressive tool selection for AI agent workflows (replaces 1-2s LLM tool calling).
2. Direct execution via .call() with function arguments.
3. Direct answer detection when no tool is needed (allow_direct_answer=True).
4. Calibrated confidence threshold fallback gating for ambiguous requests.
@@ -78,14 +78,14 @@ def general_reasoning_fallback(prompt: str) -> str:
def main():
print("=" * 70)
- print(" Laya System 1: Sub-35ms Tool and Function Selection")
+ print(" Laya System 1: Tool and Function Selection")
print("=" * 70)
tools = [search_web, calculate_tax, query_database]
agent = _DemoAgent()
# 1. Standard Tool Selection
- print("\n[1] Sub-35ms Tool Selection:")
+ print("\n[1] Single-Pass Tool Selection:")
selector = LayaToolSelector(tools=tools, agent=agent)
queries = [
diff --git a/laya/__init__.py b/laya/__init__.py
index bb85bbd5..16e4ef2e 100644
--- a/laya/__init__.py
+++ b/laya/__init__.py
@@ -77,8 +77,6 @@ _LAZY_ATTRS = {
"LayaTriage": (".integrations", "LayaTriage"),
"LayaEvaluator": (".integrations", "LayaEvaluator"),
"LayaDecision": (".integrations", "LayaDecision"),
- "LayaToolSelector": (".integrations", "LayaToolSelector"),
- "ToolRouteDecision": (".integrations", "ToolRouteDecision"),
}
@@ -146,8 +144,6 @@ __all__ = [
"LayaTriage",
"LayaEvaluator",
"LayaDecision",
- "LayaToolSelector",
- "ToolRouteDecision",
"PredictContext",
"PredictHook",
"Hook",
diff --git a/laya/integrations/tools.py b/laya/integrations/tools.py
index b8d8d017..bc30df78 100644
--- a/laya/integrations/tools.py
+++ b/laya/integrations/tools.py
@@ -1,6 +1,6 @@
"""Tool and function selection for agent frameworks using Laya System 1 decision engine.
-Provides sub-35ms, non-autoregressive tool routing and function selection for AI agent
+Provides non-autoregressive tool routing and function selection for AI agent
workflows (LangChain, CrewAI, LlamaIndex, AutoGen, and custom agent loops).
Replaces slow, token-generating LLM tool calling (1,000-2,500ms) with calibrated, typed
@@ -39,7 +39,7 @@ def _gated_confidence(answer: Dict[str, Any]) -> Optional[float]:
@dataclass
class ToolRouteDecision:
- """Result of a Laya sub-35ms tool routing decision."""
+ """Result of a Laya tool routing decision."""
tool: Any
tool_name: str
@@ -52,6 +52,13 @@ class ToolRouteDecision:
raw_decision: Dict[str, Any]
+__all__ = [
+ "LayaToolSelector",
+ "ToolRouteDecision",
+ "LayaLowConfidenceError",
+]
+
+
def _extract_tool_metadata(tool: Any, default_index: int) -> Tuple[str, str]:
"""Extract (name, description) from any tool representation."""
if isinstance(tool, str):
@@ -226,10 +233,11 @@ def _execute_decision(
class LayaToolSelector:
- """Sub-35ms tool and function router for AI agent workflows.
+ """Non-autoregressive tool and function router for AI agent workflows.
- Evaluates user queries and agent states against candidate tools in a single
- non-autoregressive forward pass, eliminating 1-2 second LLM tool-calling latency.
+ Evaluates candidate tools against a user prompt or agent state in a single
+ forward pass (measured at 32.8 ms with `laya-multilingual` and 39.5 ms with
+ `laya` on a Tesla T4 GPU; ~88–126 ms on Apple Silicon; 193–464 ms on CPU).
Supports Python functions, LangChain/CrewAI/LlamaIndex tools, and JSON Schema specs.
Works seamlessly with local in-process models (`Agent` / `Router`) and remote
@@ -299,7 +307,7 @@ class LayaToolSelector:
tools: Optional[Sequence[Any]] = None,
instructions: Optional[str] = None,
) -> ToolRouteDecision:
- """Select the best candidate tool for the given query/state in sub-35ms."""
+ """Select the best candidate tool for the given query/state in a single forward pass."""
candidate_tools = tools if tools is not None else self.tools
if not candidate_tools and not self.allow_direct_answer:
raise ValueError("No tools provided to LayaToolSelector.")
diff --git a/tests/test_tools.py b/tests/test_tools.py
index 0371ca25..dbdb1cf1 100644
--- a/tests/test_tools.py
+++ b/tests/test_tools.py
@@ -367,8 +367,10 @@ for ctrl in HOOK_CONTROLS:
integrations_mod = __import__("laya.integrations", fromlist=["__all__"])
check_true("export/integrations_selector", "LayaToolSelector" in integrations_mod.__all__)
check_true("export/integrations_decision", "ToolRouteDecision" in integrations_mod.__all__)
-check_true("export/laya_selector", "LayaToolSelector" in laya.__all__)
-check_true("export/laya_decision", "ToolRouteDecision" in laya.__all__)
+tools_mod = __import__("laya.integrations.tools", fromlist=["__all__"])
+check_true("export/tools_selector", "LayaToolSelector" in tools_mod.__all__)
+check_true("export/tools_decision", "ToolRouteDecision" in tools_mod.__all__)
+check_true("export/top_level_clean", "LayaToolSelector" not in laya.__all__ and "ToolRouteDecision" not in laya.__all__)
# --------------------------------------------------------------- Report