Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(integrations): add LlamaIndex query routing and selectors

LEVELING210819f997fb8882
diff --git a/docs/.nav.yml b/docs/.nav.yml
index 2ffca4b8..d1a8a1eb 100644
--- a/docs/.nav.yml
+++ b/docs/.nav.yml
@@ -6,6 +6,7 @@ nav:
- structured.md
- hooks
- langchain.md
+ - llamaindex.md
- glob: "*"
ignore_no_matches: true
- Deployment:
diff --git a/docs/llamaindex.md b/docs/llamaindex.md
new file mode 100644
index 00000000..17ebdf59
--- /dev/null
+++ b/docs/llamaindex.md
@@ -0,0 +1,141 @@
+# LlamaIndex Integration
+
+Laya provides sub-35ms, non-autoregressive decision components for **LlamaIndex** RAG pipelines, `RouterQueryEngine`, and tool selection (single-question latency measured at **32.8 ms** with `laya-multilingual` and **39.5 ms** with `laya` on a Tesla T4 GPU; 193–464 ms on CPU):
+
+* **`LayaSingleSelector`**: Sub-35ms single-choice selector replacing `LLMSingleSelector` for `RouterQueryEngine`.
+* **`LayaMultiSelector`**: Multi-choice selector replacing `LLMMultiSelector` for composite queries spanning multiple data sources.
+* **`LayaQueryRouter`**: Standalone query dispatcher routing incoming requests directly to target query engines or callables.
+
+Supports both **local in-process inference** (`Agent` or `Router`) and **remote HTTP inference** against your own `laya-serve` instance without requiring PyTorch on edge clients.
+
+---
+
+## Installation
+
+```bash
+pip install "laya[llamaindex]"
+```
+
+---
+
+## 1. Single-Choice Routing with `RouterQueryEngine`
+
+In LlamaIndex, `RouterQueryEngine` uses a selector to decide which underlying query engine or tool should answer a question. Autoregressive LLM selectors (`LLMSingleSelector`) take 1,000–2,000 ms generating text. `LayaSingleSelector` evaluates candidate tools in **~33 ms** without token generation:
+
+```python
+from llama_index.core.query_engine import RouterQueryEngine
+from llama_index.core.tools import QueryEngineTool, ToolMetadata
+from laya.integrations.llamaindex import LayaSingleSelector
+
+# Define query engine tools
+docs_tool = QueryEngineTool(
+ query_engine=vector_index.as_query_engine(),
+ metadata=ToolMetadata(
+ name="vector_documentation",
+ description="Semantic search over technical user documentation and API guides.",
+ ),
+)
+sql_tool = QueryEngineTool(
+ query_engine=sql_index.as_query_engine(),
+ metadata=ToolMetadata(
+ name="sql_database",
+ description="Structured SQL database containing customer accounts, billing, and orders.",
+ ),
+)
+
+# Initialize Laya sub-35ms selector with confidence fallback
+selector = LayaSingleSelector(
+ confidence_threshold=0.80, # If confidence < 0.80, fall back to index 0
+ fallback_index=0,
+)
+
+router_engine = RouterQueryEngine(
+ selector=selector,
+ query_engine_tools=[docs_tool, sql_tool],
+)
+
+response = router_engine.query("What is the shipping address for order #4912?")
+print(response)
+```
+
+---
+
+## 2. Multi-Choice Selection for Composite Queries
+
+For queries that require synthesis across multiple indexes (e.g. comparing documentation specs with transactional database records), `LayaMultiSelector` evaluates candidate relevance and returns multiple selected tools:
+
+```python
+from laya.integrations.llamaindex import LayaMultiSelector
+
+multi_selector = LayaMultiSelector(
+ probability_threshold=0.25, # Select all tools with probability >= 0.25
+ max_outputs=2,
+)
+
+tools = [docs_tool.metadata, sql_tool.metadata, summary_tool.metadata]
+result = multi_selector.select(
+ tools,
+ "How does the database security policy compare with our published compliance guide?"
+)
+
+for sel in result.selections:
+ print(f"Tool: {tools[sel.index].name} | {sel.reason}")
+```
+
+---
+
+## 3. Direct Query Dispatch with `LayaQueryRouter`
+
+For direct routing without the overhead of `RouterQueryEngine`, `LayaQueryRouter` routes queries directly to dictionary-registered engines:
+
+```python
+from laya.integrations.llamaindex import LayaQueryRouter
+
+router = LayaQueryRouter(
+ query_engines={
+ "vector": vector_query_engine,
+ "sql": sql_query_engine,
+ "summary": summary_query_engine,
+ },
+ descriptions={
+ "vector": "Semantic search over product documentation and guides",
+ "sql": "Structured SQL queries for user accounts and transactions",
+ "summary": "Quarterly reports and high-level business summaries",
+ },
+ confidence_threshold=0.75,
+ fallback_key="vector",
+)
+
+# Route and execute in one call:
+response = router.query("How many active subscriptions were renewed in Q3?")
+print(response)
+```
+
+Both synchronous `query()` and asynchronous `aquery()` are supported.
+
+---
+
+## 4. Confidence Threshold Gating
+
+Like Laya's LangChain integration, `LayaSingleSelector` and `LayaQueryRouter` read calibrated `answer_confidence` (`max(p)`):
+
+- **Automatic Fallback:** Specify `fallback_index` (or `fallback_key`) to seamlessly divert uncertain queries to a safe default engine.
+- **Strict Guarding:** Set `raise_on_low_confidence=True` on `LayaSingleSelector` to raise `LayaLowConfidenceError` when input is ambiguous, allowing caller escalation.
+
+---
+
+## 5. Remote HTTP Deployments
+
+For serverless RAG, edge environments, or environments without local GPUs:
+
+```python
+from laya.integrations.llamaindex import LayaSingleSelector
+
+selector = LayaSingleSelector(
+ base_url="http://laya-serve.internal:8080",
+ confidence_threshold=0.85,
+ fallback_index=0,
+)
+```
+
+The remote client uses Python's standard library `urllib` with zero heavy dependencies, preventing cross-origin credential forwarding and matching the `/v1/systemone` specification.
diff --git a/examples/llamaindex_quickstart.py b/examples/llamaindex_quickstart.py
new file mode 100644
index 00000000..4c568962
--- /dev/null
+++ b/examples/llamaindex_quickstart.py
@@ -0,0 +1,173 @@
+"""Laya System 1 decision engine: LlamaIndex Integration Quickstart.
+
+Demonstrates:
+1. Sub-35ms single-choice tool routing in LlamaIndex with confidence fallback gating.
+2. Multi-choice routing across multiple RAG query engines for composite questions.
+3. Direct high-performance query dispatch with LayaQueryRouter.
+4. Edge/Serverless deployment against remote laya-serve HTTP instances.
+"""
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from laya.integrations.llamaindex import (
+ LayaMultiSelector,
+ LayaQueryRouter,
+ LayaSingleSelector,
+ QueryBundle,
+ ToolMetadata,
+)
+
+# Optional mock agent for environments without local GPU / PyTorch weights:
+class _DemoAgent:
+ def predict(self, state, questions, **kwargs):
+ text = str(state).lower()
+ q_id = next(iter(questions.keys()))
+ criteria = questions[q_id].get("criteria", {})
+
+ # Route database/orders queries to sql_database or database
+ if any(w in text for w in ("order", "orders", "account", "database", "balance", "user")):
+ chosen = "choice_1" if "choice_1" in criteria else "database"
+ probs = {"choice_0": 0.08, "choice_1": 0.88, "choice_2": 0.04}
+ elif "both" in text or "compare" in text or "security" in text:
+ chosen = "choice_0" if "choice_0" in criteria else "docs"
+ probs = {"choice_0": 0.52, "choice_1": 0.44, "choice_2": 0.04}
+ else:
+ chosen = "choice_0" if "choice_0" in criteria else "docs"
+ probs = {"choice_0": 0.91, "choice_1": 0.06, "choice_2": 0.03}
+ return {
+ "model": "laya-demo",
+ "answers": {
+ q_id: {
+ "choice": chosen,
+ "answer_confidence": probs.get(chosen, 0.90),
+ "confidence": probs.get(chosen, 0.90),
+ "probabilities": probs,
+ }
+ },
+ }
+
+_agent = None
+try:
+ from laya import Router
+ r = Router()
+ # Test if models/dependencies are present
+ r.predict("test", {"test": {"type": "choice", "instructions": "test", "criteria": {"a": "a"}}})
+ _agent = r
+except Exception:
+ _agent = _DemoAgent()
+
+# =====================================================================
+# 1. Sub-35ms Single-Choice RAG Selector (Replaces LLMSingleSelector)
+# =====================================================================
+# Evaluates query against candidate query engine tools in ~33 ms without
+# burning tokens or waiting 1-2 seconds for autoregressive LLM completion.
+
+tools = [
+ ToolMetadata(
+ name="vector_documentation",
+ description="Semantic search over technical user documentation and API guides.",
+ ),
+ ToolMetadata(
+ name="sql_database",
+ description="Structured SQL database containing customer accounts, billing, and orders.",
+ ),
+ ToolMetadata(
+ name="summary_index",
+ description="High-level summaries and quarterly reports of company financial performance.",
+ ),
+]
+
+# Configure selector with confidence threshold fallback
+selector = LayaSingleSelector(
+ confidence_threshold=0.80,
+ fallback_index=0, # Fallback to vector documentation if uncertain
+ agent=_agent,
+)
+
+query = "Can you show me the total number of orders placed by customer #1042 last month?"
+print(f"--- 1. Single-Choice Selector ---")
+print(f"Query: {query}")
+
+result = selector.select(tools, query)
+selected_tool = tools[result.selections[0].index]
+print(f"Selected Tool: {selected_tool.name} (Index: {result.selections[0].index})")
+print(f"Reason: {result.selections[0].reason}")
+
+
+# =====================================================================
+# 2. Multi-Choice Selector for Composite Questions (Replaces LLMMultiSelector)
+# =====================================================================
+# Identifies which subset of candidate tools are relevant for complex queries.
+
+multi_selector = LayaMultiSelector(
+ probability_threshold=0.25,
+ max_outputs=2,
+ agent=_agent,
+)
+
+composite_query = (
+ "How does the new authentication architecture documented in the guides impact "
+ "our customer account database security settings?"
+)
+print(f"\n--- 2. Multi-Choice Selector ---")
+print(f"Composite Query: {composite_query}")
+
+multi_result = multi_selector.select(tools, composite_query)
+print(f"Selected {len(multi_result.selections)} relevant tools:")
+for sel in multi_result.selections:
+ print(f" -> Tool: {tools[sel.index].name} | {sel.reason}")
+
+
+# =====================================================================
+# 3. Direct High-Performance Query Dispatch with LayaQueryRouter
+# =====================================================================
+# Dispatches incoming queries directly to backend query engines or functions.
+
+class DummyQueryEngine:
+ def __init__(self, name: str):
+ self.name = name
+
+ def query(self, query_str: str) -> str:
+ return f"[{self.name}] Synthesized response for: {query_str}"
+
+
+engines = {
+ "docs": DummyQueryEngine("DocsEngine"),
+ "database": DummyQueryEngine("SQLEngine"),
+ "finance": DummyQueryEngine("FinanceEngine"),
+}
+
+router = LayaQueryRouter(
+ query_engines=engines,
+ descriptions={
+ "docs": "Documentation and setup tutorials",
+ "database": "User transactions and database queries",
+ "finance": "Quarterly earnings and financial statements",
+ },
+ confidence_threshold=0.75,
+ fallback_key="docs",
+ agent=_agent,
+)
+
+db_query = "What is the account balance for user Alice?"
+routed_key = router.route(db_query)
+response = router.query(db_query)
+print(f"\n--- 3. LayaQueryRouter Direct Dispatch ---")
+print(f"Query: {db_query}")
+print(f"Routed Engine Key: -> {routed_key}")
+print(f"Engine Response: {response}")
+
+
+# =====================================================================
+# 4. Remote HTTP Server Execution (Zero Heavy Dependencies)
+# =====================================================================
+# Connects to your self-hosted `laya-serve` instance without requiring PyTorch on edge clients.
+remote_selector = LayaSingleSelector(
+ base_url="http://localhost:8080",
+ confidence_threshold=0.85,
+ fallback_index=0,
+)
+print(f"\n--- 4. Remote HTTP Server Deployment ---")
+print("Remote selector configured with base_url='http://localhost:8080' via standard urllib.")
diff --git a/laya/integrations/__init__.py b/laya/integrations/__init__.py
index 7ce62e40..0c79937d 100644
--- a/laya/integrations/__init__.py
+++ b/laya/integrations/__init__.py
@@ -6,6 +6,12 @@ from .langchain import (
LayaRouter,
LayaTriage,
)
+from .llamaindex import (
+ LayaLowConfidenceError,
+ LayaMultiSelector,
+ LayaQueryRouter,
+ LayaSingleSelector,
+)
__all__ = [
"LayaRouter",
@@ -13,4 +19,8 @@ __all__ = [
"LayaGuardrailError",
"LayaTriage",
"LayaEvaluator",
+ "LayaSingleSelector",
+ "LayaMultiSelector",
+ "LayaQueryRouter",
+ "LayaLowConfidenceError",
]
diff --git a/laya/integrations/llamaindex.py b/laya/integrations/llamaindex.py
new file mode 100644
index 00000000..4bf8f960
--- /dev/null
+++ b/laya/integrations/llamaindex.py
@@ -0,0 +1,525 @@
+"""LlamaIndex integration for Laya System 1 decision engine.
+
+Provides sub-35ms, non-autoregressive query routing, single/multi-tool selection,
+and real-time guardrails for LlamaIndex RAG pipelines and RouterQueryEngine.
+
+Replaces slow, token-generating LLM selectors (`LLMSingleSelector`, `LLMMultiSelector`)
+with calibrated, typed decisions executed in a single forward pass without token cost.
+
+Supports both local in-process models (`Agent` / `Router`) and remote HTTP
+deployments (your own `laya-serve`) without requiring PyTorch on edge clients.
+"""
+from __future__ import annotations
+
+import asyncio
+import json
+import threading
+import urllib.error
+import urllib.parse
+import urllib.request
+from dataclasses import dataclass
+from typing import Any, Dict, List, Optional, Sequence, Union
+
+# Optional LlamaIndex base classes
+try:
+ from llama_index.core.schema import QueryBundle
+ from llama_index.core.selectors import BaseSelector, SelectorResult, ToolSelection
+ from llama_index.core.tools import ToolMetadata
+ _LLAMA_INDEX_AVAILABLE = True
+except ImportError:
+ _LLAMA_INDEX_AVAILABLE = False
+
+ class BaseSelector: # type: ignore
+ """Lightweight shim when llama-index-core is not installed."""
+
+ def select(self, choices: Sequence[Any], query: Any) -> Any:
+ raise NotImplementedError
+
+ async def aselect(self, choices: Sequence[Any], query: Any) -> Any:
+ raise NotImplementedError
+
+ @dataclass
+ class ToolMetadata: # type: ignore
+ description: str
+ name: Optional[str] = None
+
+ @dataclass
+ class ToolSelection: # type: ignore
+ index: int
+ reason: Optional[str] = None
+
+ @dataclass
+ class SelectorResult: # type: ignore
+ selections: List[ToolSelection]
+
+ @dataclass
+ class QueryBundle: # type: ignore
+ query_str: str
+
+ def __str__(self) -> str:
+ return self.query_str
+
+
+class LayaLowConfidenceError(ValueError):
+ """Raised when a routing decision falls below the confidence threshold and no fallback is set."""
+
+ def __init__(self, message: str, confidence: float, threshold: float, raw_decision: Dict[str, Any]):
+ super().__init__(message)
+ self.confidence = confidence
+ self.threshold = threshold
+ self.raw_decision = raw_decision
+
+
+def _extract_query_str(query: Union[str, QueryBundle, Any]) -> str:
+ """Extract raw query string from query input."""
+ if isinstance(query, str):
+ return query
+ if hasattr(query, "query_str"):
+ return str(query.query_str)
+ return str(query)
+
+
+def _format_choices_criteria(choices: Sequence[Union[ToolMetadata, Dict[str, Any], Any]]) -> Dict[str, str]:
+ """Format candidate tool choices into a Laya choice criteria dictionary."""
+ criteria: Dict[str, str] = {}
+ for i, choice in enumerate(choices):
+ key = f"choice_{i}"
+ name = getattr(choice, "name", None)
+ desc = getattr(choice, "description", None)
+
+ if isinstance(choice, dict):
+ name = choice.get("name", name)
+ desc = choice.get("description", desc)
+ elif isinstance(choice, str):
+ desc = choice
+
+ if not desc:
+ desc = name or f"Option {i}"
+
+ if name:
+ criteria[key] = f"{name}: {desc}"
+ else:
+ criteria[key] = str(desc)
+
+ return criteria
+
+
+class _SameOriginRedirectHandler(urllib.request.HTTPRedirectHandler):
+ """Do not forward bearer credentials across an origin or HTTPS downgrade."""
+
+ def redirect_request(self, req, fp, code, msg, headers, newurl):
+ old = urllib.parse.urlsplit(req.full_url)
+ new = urllib.parse.urlsplit(newurl)
+ old_port = old.port or (443 if old.scheme.lower() == "https" else 80)
+ new_port = new.port or (443 if new.scheme.lower() == "https" else 80)
+ if (
+ old.scheme.lower() != new.scheme.lower()
+ or (old.hostname or "").lower() != (new.hostname or "").lower()
+ or old_port != new_port
+ ):
+ raise urllib.error.URLError("refusing cross-origin or HTTPS-downgrade redirect")
+ return super().redirect_request(req, fp, code, msg, headers, newurl)
+
+
+def _call_remote(
+ base_url: str,
+ state: Any,
+ questions: Dict[str, Any],
+ api_key: Optional[str] = None,
+ model: Optional[str] = None,
+ timeout: float = 10.0,
+) -> Dict[str, Any]:
+ """Send decision request to a remote laya-serve HTTP instance using standard library urllib."""
+ url = base_url.rstrip("/")
+ if not url.endswith("/v1/systemone"):
+ url = f"{url}/v1/systemone"
+
+ payload: Dict[str, Any] = {"state": state, "questions": questions}
+ if model:
+ payload["model"] = model
+
+ data = json.dumps(payload).encode("utf-8")
+ headers = {"Content-Type": "application/json"}
+ if api_key:
+ headers["Authorization"] = f"Bearer {api_key}"
+
+ req = urllib.request.Request(url, data=data, headers=headers, method="POST")
+ opener = urllib.request.build_opener(_SameOriginRedirectHandler())
+ try:
+ with opener.open(req, timeout=timeout) as resp:
+ return json.loads(resp.read().decode("utf-8"))
+ except urllib.error.HTTPError as e:
+ body = e.read().decode("utf-8", errors="replace")
+ raise RuntimeError(f"Laya server error {e.code}: {body}") from e
+ except Exception as e:
+ raise RuntimeError(f"Failed to connect to Laya server at {url}: {e}") from e
+
+
+_DEFAULT_ROUTER = None
+_DEFAULT_ROUTER_LOCK = threading.Lock()
+
+
+def _get_default_router():
+ global _DEFAULT_ROUTER
+ if _DEFAULT_ROUTER is None:
+ with _DEFAULT_ROUTER_LOCK:
+ if _DEFAULT_ROUTER is None:
+ from ..router import Router
+ _DEFAULT_ROUTER = Router()
+ return _DEFAULT_ROUTER
+
+
+def _execute_decision(
+ state: Any,
+ questions: Dict[str, Any],
+ agent: Optional[Any] = None,
+ base_url: Optional[str] = None,
+ api_key: Optional[str] = None,
+ model: Optional[str] = None,
+) -> Dict[str, Any]:
+ if base_url:
+ return _call_remote(base_url, state, questions, api_key=api_key, model=model)
+ runner = agent if agent is not None else _get_default_router()
+ kwargs = {"model": model} if model else {}
+ return runner.predict(state, questions, **kwargs)
+
+
+class LayaSingleSelector(BaseSelector):
+ """Sub-35ms single-choice selector for LlamaIndex RouterQueryEngine.
+
+ Replaces LLMSingleSelector to route queries to candidate query engines or tools
+ in a single non-autoregressive forward pass with zero token generation cost.
+ Supports confidence threshold gating and automatic fallback selection.
+ """
+
+ def __init__(
+ self,
+ instructions: str = "Which tool or query engine is best suited to answer this query?",
+ confidence_threshold: float = 0.0,
+ fallback_index: Optional[int] = None,
+ raise_on_low_confidence: bool = False,
+ agent: Optional[Any] = None,
+ base_url: Optional[str] = None,
+ api_key: Optional[str] = None,
+ model: Optional[str] = None,
+ ):
+ self.instructions = instructions
+ self.confidence_threshold = confidence_threshold
+ self.fallback_index = fallback_index
+ self.raise_on_low_confidence = raise_on_low_confidence
+ self.agent = agent
+ self.base_url = base_url
+ self.api_key = api_key
+ self.model = model
+ self.last_decision: Optional[Dict[str, Any]] = None
+
+ def _select(
+ self,
+ choices: Sequence[Union[ToolMetadata, Any]],
+ query: Union[QueryBundle, str],
+ ) -> SelectorResult:
+ """Select a single choice synchronously."""
+ if not choices:
+ raise ValueError("No choices provided to select from.")
+
+ query_str = _extract_query_str(query)
+ criteria = _format_choices_criteria(choices)
+
+ questions = {
+ "selector": {
+ "type": "choice",
+ "instructions": self.instructions,
+ "criteria": criteria,
+ }
+ }
+
+ res = _execute_decision(
+ query_str,
+ questions,
+ agent=self.agent,
+ base_url=self.base_url,
+ api_key=self.api_key,
+ model=self.model,
+ )
+ self.last_decision = res
+
+ ans = res.get("answers", {}).get("selector", {})
+ chosen_key = ans.get("choice")
+ conf = ans.get("answer_confidence", ans.get("confidence", 1.0))
+
+ # Map "choice_i" back to index i
+ chosen_idx: int = 0
+ if chosen_key and chosen_key.startswith("choice_"):
+ try:
+ chosen_idx = int(chosen_key.split("_")[1])
+ except (ValueError, IndexError):
+ chosen_idx = 0
+ elif chosen_key in criteria:
+ chosen_idx = list(criteria.keys()).index(chosen_key)
+
+ # Confidence gating
+ if self.confidence_threshold > 0.0 and conf < self.confidence_threshold:
+ if self.fallback_index is not None:
+ reason = (
+ f"Selected fallback index {self.fallback_index} because confidence "
+ f"({conf:.3f}) was below threshold ({self.confidence_threshold:.3f})."
+ )
+ return SelectorResult(selections=[ToolSelection(index=self.fallback_index, reason=reason)])
+ if self.raise_on_low_confidence:
+ raise LayaLowConfidenceError(
+ f"Confidence {conf:.3f} below threshold {self.confidence_threshold:.3f} "
+ f"for query: {query_str!r}",
+ confidence=conf,
+ threshold=self.confidence_threshold,
+ raw_decision=res,
+ )
+
+ choice_obj = choices[chosen_idx]
+ choice_name = getattr(choice_obj, "name", None) or f"choice_{chosen_idx}"
+ reason = f"Selected '{choice_name}' via Laya System 1 decision (confidence: {conf:.3f})."
+ return SelectorResult(selections=[ToolSelection(index=chosen_idx, reason=reason)])
+
+ def select(
+ self,
+ choices: Sequence[Union[ToolMetadata, Any]],
+ query: Union[QueryBundle, str],
+ ) -> SelectorResult:
+ """Public synchronous selection method."""
+ return self._select(choices, query)
+
+ async def _aselect(
+ self,
+ choices: Sequence[Union[ToolMetadata, Any]],
+ query: Union[QueryBundle, str],
+ ) -> SelectorResult:
+ """Select a single choice asynchronously without blocking the event loop."""
+ return await asyncio.to_thread(self._select, choices, query)
+
+ async def aselect(
+ self,
+ choices: Sequence[Union[ToolMetadata, Any]],
+ query: Union[QueryBundle, str],
+ ) -> SelectorResult:
+ """Public asynchronous selection method."""
+ return await self._aselect(choices, query)
+
+
+class LayaMultiSelector(BaseSelector):
+ """Sub-35ms multi-choice selector for LlamaIndex RouterQueryEngine.
+
+ Replaces LLMMultiSelector to evaluate and select multiple query engines or tools
+ for composite questions requiring information from multiple sources.
+ """
+
+ def __init__(
+ self,
+ instructions: str = "Which tools or query engines are relevant to answer this query?",
+ max_outputs: Optional[int] = None,
+ probability_threshold: float = 0.25,
+ default_index: int = 0,
+ agent: Optional[Any] = None,
+ base_url: Optional[str] = None,
+ api_key: Optional[str] = None,
+ model: Optional[str] = None,
+ ):
+ self.instructions = instructions
+ self.max_outputs = max_outputs
+ self.probability_threshold = probability_threshold
+ self.default_index = default_index
+ self.agent = agent
+ self.base_url = base_url
+ self.api_key = api_key
+ self.model = model
+ self.last_decision: Optional[Dict[str, Any]] = None
+
+ def _select(
+ self,
+ choices: Sequence[Union[ToolMetadata, Any]],
+ query: Union[QueryBundle, str],
+ ) -> SelectorResult:
+ if not choices:
+ raise ValueError("No choices provided to select from.")
+
+ query_str = _extract_query_str(query)
+ criteria = _format_choices_criteria(choices)
+
+ questions = {
+ "selector": {
+ "type": "choice",
+ "instructions": self.instructions,
+ "criteria": criteria,
+ }
+ }
+
+ res = _execute_decision(
+ query_str,
+ questions,
+ agent=self.agent,
+ base_url=self.base_url,
+ api_key=self.api_key,
+ model=self.model,
+ )
+ self.last_decision = res
+
+ ans = res.get("answers", {}).get("selector", {})
+ probs = ans.get("probabilities", {})
+
+ selected: List[ToolSelection] = []
+ key_list = list(criteria.keys())
+
+ # Sort candidate choices by descending probability
+ ranked = sorted(
+ key_list,
+ key=lambda k: probs.get(k, 0.0),
+ reverse=True,
+ )
+
+ for key in ranked:
+ prob = probs.get(key, 0.0)
+ if prob >= self.probability_threshold:
+ try:
+ idx = int(key.split("_")[1]) if key.startswith("choice_") else key_list.index(key)
+ except (ValueError, IndexError):
+ idx = key_list.index(key)
+
+ choice_obj = choices[idx]
+ choice_name = getattr(choice_obj, "name", None) or f"choice_{idx}"
+ selected.append(
+ ToolSelection(
+ index=idx,
+ reason=f"Selected '{choice_name}' (probability: {prob:.3f} >= {self.probability_threshold:.3f}).",
+ )
+ )
+ if self.max_outputs is not None and len(selected) >= self.max_outputs:
+ break
+
+ # Fallback to top option if none met threshold
+ if not selected and ranked:
+ top_key = ranked[0]
+ top_idx = int(top_key.split("_")[1]) if top_key.startswith("choice_") else key_list.index(top_key)
+ choice_obj = choices[top_idx]
+ choice_name = getattr(choice_obj, "name", None) or f"choice_{top_idx}"
+ prob = probs.get(top_key, 0.0)
+ selected.append(
+ ToolSelection(
+ index=top_idx,
+ reason=f"Selected top choice '{choice_name}' as fallback (probability: {prob:.3f}).",
+ )
+ )
+
+ return SelectorResult(selections=selected)
+
+ def select(
+ self,
+ choices: Sequence[Union[ToolMetadata, Any]],
+ query: Union[QueryBundle, str],
+ ) -> SelectorResult:
+ return self._select(choices, query)
+
+ async def _aselect(
+ self,
+ choices: Sequence[Union[ToolMetadata, Any]],
+ query: Union[QueryBundle, str],
+ ) -> SelectorResult:
+ return await asyncio.to_thread(self._select, choices, query)
+
+ async def aselect(
+ self,
+ choices: Sequence[Union[ToolMetadata, Any]],
+ query: Union[QueryBundle, str],
+ ) -> SelectorResult:
+ return await self._aselect(choices, query)
+
+
+class LayaQueryRouter:
+ """Direct, high-performance query dispatcher for LlamaIndex query engines.
+
+ Allows routing queries directly to target engines or callable tools without
+ the overhead of standard LLM-based query engine routers.
+ """
+
+ def __init__(
+ self,
+ query_engines: Dict[str, Any],
+ descriptions: Optional[Dict[str, str]] = None,
+ instructions: str = "Which query engine should handle this request?",
+ confidence_threshold: float = 0.0,
+ fallback_key: Optional[str] = None,
+ agent: Optional[Any] = None,
+ base_url: Optional[str] = None,
+ api_key: Optional[str] = None,
+ model: Optional[str] = None,
+ ):
+ self.query_engines = query_engines
+ self.descriptions = descriptions or {}
+ self.instructions = instructions
+ self.confidence_threshold = confidence_threshold
+ self.fallback_key = fallback_key or next(iter(query_engines.keys()), None)
+ self.agent = agent
+ self.base_url = base_url
+ self.api_key = api_key
+ self.model = model
+ self.last_decision: Optional[Dict[str, Any]] = None
+
+ def route(self, query: Union[QueryBundle, str]) -> str:
+ """Determine which query engine key should handle the query in ~33ms."""
+ query_str = _extract_query_str(query)
+ criteria = {
+ key: self.descriptions.get(key, f"Engine for {key}")
+ for key in self.query_engines.keys()
+ }
+
+ questions = {
+ "route": {
+ "type": "choice",
+ "instructions": self.instructions,
+ "criteria": criteria,
+ }
+ }
+
+ res = _execute_decision(
+ query_str,
+ questions,
+ agent=self.agent,
+ base_url=self.base_url,
+ api_key=self.api_key,
+ model=self.model,
+ )
+ self.last_decision = res
+
+ ans = res.get("answers", {}).get("route", {})
+ chosen = ans.get("choice")
+ conf = ans.get("answer_confidence", ans.get("confidence", 1.0))
+
+ if self.confidence_threshold > 0.0 and conf < self.confidence_threshold:
+ if self.fallback_key:
+ return self.fallback_key
+
+ if chosen in self.query_engines:
+ return chosen
+
+ return self.fallback_key or next(iter(self.query_engines.keys()))
+
+ def query(self, query: Union[QueryBundle, str], **kwargs: Any) -> Any:
+ """Route the query to the winning engine and execute engine.query()."""
+ key = self.route(query)
+ engine = self.query_engines[key]
+ if hasattr(engine, "query"):
+ return engine.query(query, **kwargs)
+ if callable(engine):
+ return engine(query, **kwargs)
+ raise TypeError(f"Engine registered for key {key!r} is not callable and has no .query() method.")
+
+ async def aquery(self, query: Union[QueryBundle, str], **kwargs: Any) -> Any:
+ """Asynchronously route the query and execute engine.aquery() or engine.query()."""
+ key = await asyncio.to_thread(self.route, query)
+ engine = self.query_engines[key]
+ if hasattr(engine, "aquery"):
+ return await engine.aquery(query, **kwargs)
+ if hasattr(engine, "query"):
+ return await asyncio.to_thread(engine.query, query, **kwargs)
+ if callable(engine):
+ if asyncio.iscoroutinefunction(engine):
+ return await engine(query, **kwargs)
+ return await asyncio.to_thread(engine, query, **kwargs)
+ raise TypeError(f"Engine registered for key {key!r} is not callable and has no .query() method.")
diff --git a/pyproject.toml b/pyproject.toml
index 27b66880..92512810 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -64,6 +64,11 @@ langgraph = [
"langgraph>=0.1.0",
]
+# `pip install laya[llamaindex]` adds the LlamaIndex query router & selector integrations (laya.integrations).
+llamaindex = [
+ "llama-index-core>=0.10.0",
+]
+
[project.scripts]
laya = "laya.cli:main"
laya-serve = "laya.serve:main"
diff --git a/tests/test_llamaindex.py b/tests/test_llamaindex.py
new file mode 100644
index 00000000..e056599a
--- /dev/null
+++ b/tests/test_llamaindex.py
@@ -0,0 +1,403 @@
+"""Unit tests for Laya LlamaIndex integration.
+
+Tests verify single/multi-selector logic, confidence threshold fallback gating,
+query router dispatch, and LlamaIndex schema conventions without requiring
+model downloads, GPU, or external services.
+"""
+import asyncio
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from laya.integrations.llamaindex import (
+ LayaLowConfidenceError,
+ LayaMultiSelector,
+ LayaQueryRouter,
+ LayaSingleSelector,
+ QueryBundle,
+ ToolMetadata,
+ _extract_query_str,
+ _format_choices_criteria,
+)
+
+PASS, FAIL = [], []
+
+
+def check(name, got, want):
+ if got == want:
+ PASS.append(name)
+ else:
+ FAIL.append(f"{name}:\n got {got!r}\n want {want!r}")
+
+
+def check_true(name, cond, detail=""):
+ if cond:
+ PASS.append(name)
+ else:
+ FAIL.append(f"{name} {detail}")
+
+
+# --------------------------------------------------------------- Mock Agent
+class MockLayaAgent:
+ """Mock agent returning deterministic responses for testing."""
+
+ def __init__(self, response_fn):
+ self.response_fn = response_fn
+
+ def predict(self, state, questions, **kwargs):
+ return self.response_fn(state, questions)
+
+
+# --------------------------------------------------------------- 1. Query & Criteria Formatting
+check("extract/str", _extract_query_str("What is the revenue?"), "What is the revenue?")
+check(
+ "extract/query_bundle",
+ _extract_query_str(QueryBundle(query_str="How do I reset password?")),
+ "How do I reset password?",
+)
+
+
+class CustomQueryObj:
+ def __init__(self, text):
+ self.query_str = text
+
+
+check(
+ "extract/custom_query_obj",
+ _extract_query_str(CustomQueryObj("custom text")),
+ "custom text",
+)
+
+# Test criteria formatting from ToolMetadata
+tools = [
+ ToolMetadata(name="vector_store", description="Search financial documents"),
+ ToolMetadata(name="sql_db", description="Query relational tables"),
+ {"name": "summary_tool", "description": "High level summarization"},
+ "Simple string description tool",
+]
+criteria = _format_choices_criteria(tools)
+check(
+ "criteria/vector_store",
+ criteria["choice_0"],
+ "vector_store: Search financial documents",
+)
+check(
+ "criteria/sql_db",
+ criteria["choice_1"],
+ "sql_db: Query relational tables",
+)
+check(
+ "criteria/summary_tool",
+ criteria["choice_2"],
+ "summary_tool: High level summarization",
+)
+check(
+ "criteria/string_tool",
+ criteria["choice_3"],
+ "Simple string description tool",
+)
+
+
+# --------------------------------------------------------------- 2. LayaSingleSelector
+def mock_single_selector_response(state, questions):
+ text = str(state).lower()
+ if "sql" in text or "table" in text or "database" in text:
+ chosen = "choice_1"
+ conf = 0.94
+ elif "lowconf" in text:
+ chosen = "choice_0"
+ conf = 0.42
+ else:
+ chosen = "choice_0"
+ conf = 0.91
+
+ return {
+ "model": "mock-single",
+ "answers": {
+ "selector": {
+ "choice": chosen,
+ "answer_confidence": conf,
+ "confidence": conf,
+ }
+ },
+ }
+
+
+mock_agent = MockLayaAgent(mock_single_selector_response)
+single_selector = LayaSingleSelector(agent=mock_agent)
+
+# Standard selection for choice_0
+res0 = single_selector.select(tools, "Search for quarterly 10-K report")
+check("single/index_0", res0.selections[0].index, 0)
+check_true(
+ "single/reason_0",
+ "Selected 'vector_store' via Laya System 1 decision" in res0.selections[0].reason,
+)
+
+# Standard selection for choice_1 (sql)
+res1 = single_selector.select(tools, QueryBundle(query_str="Query the SQL database for customers"))
+check("single/index_1", res1.selections[0].index, 1)
+check_true(
+ "single/reason_1",
+ "Selected 'sql_db' via Laya System 1 decision" in res1.selections[0].reason,
+)
+
+# Confidence threshold with fallback
+gated_selector = LayaSingleSelector(
+ agent=mock_agent,
+ confidence_threshold=0.80,
+ fallback_index=2,
+)
+res_fallback = gated_selector.select(tools, "lowconf query here")
+check("single/fallback_index", res_fallback.selections[0].index, 2)
+check_true(
+ "single/fallback_reason",
+ "Selected fallback index 2" in res_fallback.selections[0].reason,
+)
+
+# Confidence threshold with raise_on_low_confidence
+raising_selector = LayaSingleSelector(
+ agent=mock_agent,
+ confidence_threshold=0.80,
+ raise_on_low_confidence=True,
+)
+try:
+ raising_selector.select(tools, "lowconf query here")
+ check("single/raise_error", False, True)
+except LayaLowConfidenceError as e:
+ check("single/raise_error", True, True)
+ check("single/error_conf", e.confidence, 0.42)
+ check("single/error_thresh", e.threshold, 0.80)
+
+# Empty choices validation
+try:
+ single_selector.select([], "any query")
+ check("single/empty_choices", False, True)
+except ValueError:
+ check("single/empty_choices", True, True)
+
+
+# Async single selector
+async def run_async_single():
+ res_async = await single_selector.aselect(tools, "Search the SQL database")
+ check("single/async_index", res_async.selections[0].index, 1)
+
+
+asyncio.run(run_async_single())
+
+
+# --------------------------------------------------------------- 3. LayaMultiSelector
+def mock_multi_selector_response(state, questions):
+ text = str(state).lower()
+ if "compare" in text or "both" in text:
+ # High probability for both vector (0) and sql (1)
+ probs = {"choice_0": 0.55, "choice_1": 0.40, "choice_2": 0.05}
+ elif "none_qualify" in text:
+ # All below threshold
+ probs = {"choice_0": 0.15, "choice_1": 0.10, "choice_2": 0.05}
+ else:
+ probs = {"choice_0": 0.85, "choice_1": 0.10, "choice_2": 0.05}
+
+ return {
+ "model": "mock-multi",
+ "answers": {
+ "selector": {
+ "choice": max(probs, key=probs.get),
+ "probabilities": probs,
+ "answer_confidence": 0.85,
+ }
+ },
+ }
+
+
+mock_multi_agent = MockLayaAgent(mock_multi_selector_response)
+multi_selector = LayaMultiSelector(
+ agent=mock_multi_agent,
+ probability_threshold=0.30,
+)
+
+# Multi-select returns multiple tools
+res_multi = multi_selector.select(tools[:3], "Compare both vector docs and SQL records")
+check("multi/count", len(res_multi.selections), 2)
+check("multi/idx_0", res_multi.selections[0].index, 0)
+check("multi/idx_1", res_multi.selections[1].index, 1)
+
+# Multi-select with max_outputs limit
+limited_selector = LayaMultiSelector(
+ agent=mock_multi_agent,
+ probability_threshold=0.30,
+ max_outputs=1,
+)
+res_limited = limited_selector.select(tools[:3], "Compare both vector docs and SQL records")
+check("multi/limited_count", len(res_limited.selections), 1)
+check("multi/limited_idx", res_limited.selections[0].index, 0)
+
+# Multi-select fallback to top option when none meet threshold
+res_none = multi_selector.select(tools[:3], "none_qualify search")
+check("multi/fallback_top_count", len(res_none.selections), 1)
+check("multi/fallback_top_idx", res_none.selections[0].index, 0)
+
+
+# Async multi-selector
+async def run_async_multi():
+ res_async = await multi_selector.aselect(tools[:3], "Compare both sources")
+ check("multi/async_count", len(res_async.selections), 2)
+
+
+asyncio.run(run_async_multi())
+
+
+# --------------------------------------------------------------- 4. LayaQueryRouter
+class MockQueryEngine:
+ def __init__(self, name):
+ self.name = name
+
+ def query(self, q):
+ return f"Result from {self.name} for: {q}"
+
+ async def aquery(self, q):
+ return f"Async result from {self.name} for: {q}"
+
+
+engines = {
+ "vector": MockQueryEngine("VectorEngine"),
+ "sql": MockQueryEngine("SqlEngine"),
+ "summary": MockQueryEngine("SummaryEngine"),
+}
+
+
+def mock_router_dispatcher(state, questions):
+ text = str(state).lower()
+ if "sql" in text or "table" in text:
+ chosen = "sql"
+ conf = 0.96
+ elif "summary" in text:
+ chosen = "summary"
+ conf = 0.91
+ elif "lowconf" in text:
+ chosen = "summary"
+ conf = 0.35
+ else:
+ chosen = "vector"
+ conf = 0.89
+
+ return {
+ "model": "mock-router",
+ "answers": {
+ "route": {
+ "choice": chosen,
+ "answer_confidence": conf,
+ "confidence": conf,
+ }
+ },
+ }
+
+
+router_agent = MockLayaAgent(mock_router_dispatcher)
+laya_qr = LayaQueryRouter(
+ query_engines=engines,
+ descriptions={
+ "vector": "Semantic search over documentation",
+ "sql": "Relational SQL queries",
+ "summary": "Document summarization",
+ },
+ agent=router_agent,
+ confidence_threshold=0.70,
+ fallback_key="vector",
+)
+
+# Route direct
+check("router/route_sql", laya_qr.route("Run a query on the SQL table"), "sql")
+check("router/route_summary", laya_qr.route("Provide an executive summary"), "summary")
+check("router/route_vector", laya_qr.route("How does authentication work?"), "vector")
+
+# Confidence fallback
+check("router/route_fallback", laya_qr.route("lowconf request"), "vector")
+
+# Execute query()
+res_q = laya_qr.query("Run a query on the SQL table")
+check("router/query_exec", res_q, "Result from SqlEngine for: Run a query on the SQL table")
+
+
+# Async query
+async def run_async_router():
+ res_aq = await laya_qr.aquery("Provide an executive summary")
+ check("router/aquery_exec", res_aq, "Async result from SummaryEngine for: Provide an executive summary")
+
+
+asyncio.run(run_async_router())
+
+
+# Callable engine support
+callable_engines = {
+ "calc": lambda q: f"Calculated: {q}",
+ "search": lambda q: f"Searched: {q}",
+}
+callable_qr = LayaQueryRouter(
+ query_engines=callable_engines,
+ agent=MockLayaAgent(
+ lambda s, q: {"answers": {"route": {"choice": "calc", "answer_confidence": 0.99}}}
+ ),
+)
+check("router/callable", callable_qr.query("2 + 2"), "Calculated: 2 + 2")
+
+
+# --------------------------------------------------------------- 5. Remote HTTP Execution (Mocked)
+from unittest.mock import MagicMock, patch
+
+
+class DummyHTTPResponse:
+ def __init__(self, data_dict):
+ self.data = json.dumps(data_dict).encode("utf-8")
+
+ def read(self):
+ return self.data
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *args):
+ pass
+
+
+import json
+
+remote_response = {
+ "model": "laya-multilingual",
+ "answers": {
+ "selector": {
+ "choice": "choice_1",
+ "answer_confidence": 0.98,
+ "confidence": 0.98,
+ }
+ },
+}
+
+with patch("urllib.request.build_opener") as mock_build_opener:
+ mock_opener = MagicMock()
+ mock_opener.open.return_value = DummyHTTPResponse(remote_response)
+ mock_build_opener.return_value = mock_opener
+
+ remote_selector = LayaSingleSelector(
+ base_url="https://api.laya.ai",
+ api_key="sk-test-token",
+ )
+ res_remote = remote_selector.select(tools, "Query via remote server")
+ check("remote/index", res_remote.selections[0].index, 1)
+
+ # Check request headers and URL
+ call_args = mock_opener.open.call_args
+ req = call_args[0][0]
+ check("remote/url", req.full_url, "https://api.laya.ai/v1/systemone")
+ check("remote/auth", req.headers.get("Authorization"), "Bearer sk-test-token")
+
+
+# --------------------------------------------------------------- Results Summary
+print(f"PASS: {len(PASS)}")
+print(f"FAIL: {len(FAIL)}")
+for f in FAIL:
+ print(f" - {f}")
+
+if FAIL:
+ sys.exit(1)