Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(integrations): add LlamaIndex query routing and selectors
LEVELING210819f997fb8882
diff --git a/docs/.nav.yml b/docs/.nav.ymlindex 2ffca4b8..d1a8a1eb 100644--- a/docs/.nav.yml+++ b/docs/.nav.yml@@ -6,6 +6,7 @@ nav:- structured.md- hooks- langchain.md+ - llamaindex.md- glob: "*"ignore_no_matches: true- Deployment:diff --git a/docs/llamaindex.md b/docs/llamaindex.mdnew file mode 100644index 00000000..17ebdf59--- /dev/null+++ b/docs/llamaindex.md@@ -0,0 +1,141 @@+# LlamaIndex Integration++Laya provides sub-35ms, non-autoregressive decision components for **LlamaIndex** RAG pipelines, `RouterQueryEngine`, and tool selection (single-question latency measured at **32.8 ms** with `laya-multilingual` and **39.5 ms** with `laya` on a Tesla T4 GPU; 193–464 ms on CPU):++* **`LayaSingleSelector`**: Sub-35ms single-choice selector replacing `LLMSingleSelector` for `RouterQueryEngine`.+* **`LayaMultiSelector`**: Multi-choice selector replacing `LLMMultiSelector` for composite queries spanning multiple data sources.+* **`LayaQueryRouter`**: Standalone query dispatcher routing incoming requests directly to target query engines or callables.++Supports both **local in-process inference** (`Agent` or `Router`) and **remote HTTP inference** against your own `laya-serve` instance without requiring PyTorch on edge clients.++---++## Installation++```bash+pip install "laya[llamaindex]"+```++---++## 1. Single-Choice Routing with `RouterQueryEngine`++In LlamaIndex, `RouterQueryEngine` uses a selector to decide which underlying query engine or tool should answer a question. Autoregressive LLM selectors (`LLMSingleSelector`) take 1,000–2,000 ms generating text. `LayaSingleSelector` evaluates candidate tools in **~33 ms** without token generation:++```python+from llama_index.core.query_engine import RouterQueryEngine+from llama_index.core.tools import QueryEngineTool, ToolMetadata+from laya.integrations.llamaindex import LayaSingleSelector++# Define query engine tools+docs_tool = QueryEngineTool(+ query_engine=vector_index.as_query_engine(),+ metadata=ToolMetadata(+ name="vector_documentation",+ description="Semantic search over technical user documentation and API guides.",+ ),+)+sql_tool = QueryEngineTool(+ query_engine=sql_index.as_query_engine(),+ metadata=ToolMetadata(+ name="sql_database",+ description="Structured SQL database containing customer accounts, billing, and orders.",+ ),+)++# Initialize Laya sub-35ms selector with confidence fallback+selector = LayaSingleSelector(+ confidence_threshold=0.80, # If confidence < 0.80, fall back to index 0+ fallback_index=0,+)++router_engine = RouterQueryEngine(+ selector=selector,+ query_engine_tools=[docs_tool, sql_tool],+)++response = router_engine.query("What is the shipping address for order #4912?")+print(response)+```++---++## 2. Multi-Choice Selection for Composite Queries++For queries that require synthesis across multiple indexes (e.g. comparing documentation specs with transactional database records), `LayaMultiSelector` evaluates candidate relevance and returns multiple selected tools:++```python+from laya.integrations.llamaindex import LayaMultiSelector++multi_selector = LayaMultiSelector(+ probability_threshold=0.25, # Select all tools with probability >= 0.25+ max_outputs=2,+)++tools = [docs_tool.metadata, sql_tool.metadata, summary_tool.metadata]+result = multi_selector.select(+ tools,+ "How does the database security policy compare with our published compliance guide?"+)++for sel in result.selections:+ print(f"Tool: {tools[sel.index].name} | {sel.reason}")+```++---++## 3. Direct Query Dispatch with `LayaQueryRouter`++For direct routing without the overhead of `RouterQueryEngine`, `LayaQueryRouter` routes queries directly to dictionary-registered engines:++```python+from laya.integrations.llamaindex import LayaQueryRouter++router = LayaQueryRouter(+ query_engines={+ "vector": vector_query_engine,+ "sql": sql_query_engine,+ "summary": summary_query_engine,+ },+ descriptions={+ "vector": "Semantic search over product documentation and guides",+ "sql": "Structured SQL queries for user accounts and transactions",+ "summary": "Quarterly reports and high-level business summaries",+ },+ confidence_threshold=0.75,+ fallback_key="vector",+)++# Route and execute in one call:+response = router.query("How many active subscriptions were renewed in Q3?")+print(response)+```++Both synchronous `query()` and asynchronous `aquery()` are supported.++---++## 4. Confidence Threshold Gating++Like Laya's LangChain integration, `LayaSingleSelector` and `LayaQueryRouter` read calibrated `answer_confidence` (`max(p)`):++- **Automatic Fallback:** Specify `fallback_index` (or `fallback_key`) to seamlessly divert uncertain queries to a safe default engine.+- **Strict Guarding:** Set `raise_on_low_confidence=True` on `LayaSingleSelector` to raise `LayaLowConfidenceError` when input is ambiguous, allowing caller escalation.++---++## 5. Remote HTTP Deployments++For serverless RAG, edge environments, or environments without local GPUs:++```python+from laya.integrations.llamaindex import LayaSingleSelector++selector = LayaSingleSelector(+ base_url="http://laya-serve.internal:8080",+ confidence_threshold=0.85,+ fallback_index=0,+)+```++The remote client uses Python's standard library `urllib` with zero heavy dependencies, preventing cross-origin credential forwarding and matching the `/v1/systemone` specification.diff --git a/examples/llamaindex_quickstart.py b/examples/llamaindex_quickstart.pynew file mode 100644index 00000000..4c568962--- /dev/null+++ b/examples/llamaindex_quickstart.py@@ -0,0 +1,173 @@+"""Laya System 1 decision engine: LlamaIndex Integration Quickstart.++Demonstrates:+1. Sub-35ms single-choice tool routing in LlamaIndex with confidence fallback gating.+2. Multi-choice routing across multiple RAG query engines for composite questions.+3. Direct high-performance query dispatch with LayaQueryRouter.+4. Edge/Serverless deployment against remote laya-serve HTTP instances.+"""+import os+import sys++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))++from laya.integrations.llamaindex import (+ LayaMultiSelector,+ LayaQueryRouter,+ LayaSingleSelector,+ QueryBundle,+ ToolMetadata,+)++# Optional mock agent for environments without local GPU / PyTorch weights:+class _DemoAgent:+ def predict(self, state, questions, **kwargs):+ text = str(state).lower()+ q_id = next(iter(questions.keys()))+ criteria = questions[q_id].get("criteria", {})++ # Route database/orders queries to sql_database or database+ if any(w in text for w in ("order", "orders", "account", "database", "balance", "user")):+ chosen = "choice_1" if "choice_1" in criteria else "database"+ probs = {"choice_0": 0.08, "choice_1": 0.88, "choice_2": 0.04}+ elif "both" in text or "compare" in text or "security" in text:+ chosen = "choice_0" if "choice_0" in criteria else "docs"+ probs = {"choice_0": 0.52, "choice_1": 0.44, "choice_2": 0.04}+ else:+ chosen = "choice_0" if "choice_0" in criteria else "docs"+ probs = {"choice_0": 0.91, "choice_1": 0.06, "choice_2": 0.03}+ return {+ "model": "laya-demo",+ "answers": {+ q_id: {+ "choice": chosen,+ "answer_confidence": probs.get(chosen, 0.90),+ "confidence": probs.get(chosen, 0.90),+ "probabilities": probs,+ }+ },+ }++_agent = None+try:+ from laya import Router+ r = Router()+ # Test if models/dependencies are present+ r.predict("test", {"test": {"type": "choice", "instructions": "test", "criteria": {"a": "a"}}})+ _agent = r+except Exception:+ _agent = _DemoAgent()++# =====================================================================+# 1. Sub-35ms Single-Choice RAG Selector (Replaces LLMSingleSelector)+# =====================================================================+# Evaluates query against candidate query engine tools in ~33 ms without+# burning tokens or waiting 1-2 seconds for autoregressive LLM completion.++tools = [+ ToolMetadata(+ name="vector_documentation",+ description="Semantic search over technical user documentation and API guides.",+ ),+ ToolMetadata(+ name="sql_database",+ description="Structured SQL database containing customer accounts, billing, and orders.",+ ),+ ToolMetadata(+ name="summary_index",+ description="High-level summaries and quarterly reports of company financial performance.",+ ),+]++# Configure selector with confidence threshold fallback+selector = LayaSingleSelector(+ confidence_threshold=0.80,+ fallback_index=0, # Fallback to vector documentation if uncertain+ agent=_agent,+)++query = "Can you show me the total number of orders placed by customer #1042 last month?"+print(f"--- 1. Single-Choice Selector ---")+print(f"Query: {query}")++result = selector.select(tools, query)+selected_tool = tools[result.selections[0].index]+print(f"Selected Tool: {selected_tool.name} (Index: {result.selections[0].index})")+print(f"Reason: {result.selections[0].reason}")+++# =====================================================================+# 2. Multi-Choice Selector for Composite Questions (Replaces LLMMultiSelector)+# =====================================================================+# Identifies which subset of candidate tools are relevant for complex queries.++multi_selector = LayaMultiSelector(+ probability_threshold=0.25,+ max_outputs=2,+ agent=_agent,+)++composite_query = (+ "How does the new authentication architecture documented in the guides impact "+ "our customer account database security settings?"+)+print(f"\n--- 2. Multi-Choice Selector ---")+print(f"Composite Query: {composite_query}")++multi_result = multi_selector.select(tools, composite_query)+print(f"Selected {len(multi_result.selections)} relevant tools:")+for sel in multi_result.selections:+ print(f" -> Tool: {tools[sel.index].name} | {sel.reason}")+++# =====================================================================+# 3. Direct High-Performance Query Dispatch with LayaQueryRouter+# =====================================================================+# Dispatches incoming queries directly to backend query engines or functions.++class DummyQueryEngine:+ def __init__(self, name: str):+ self.name = name++ def query(self, query_str: str) -> str:+ return f"[{self.name}] Synthesized response for: {query_str}"+++engines = {+ "docs": DummyQueryEngine("DocsEngine"),+ "database": DummyQueryEngine("SQLEngine"),+ "finance": DummyQueryEngine("FinanceEngine"),+}++router = LayaQueryRouter(+ query_engines=engines,+ descriptions={+ "docs": "Documentation and setup tutorials",+ "database": "User transactions and database queries",+ "finance": "Quarterly earnings and financial statements",+ },+ confidence_threshold=0.75,+ fallback_key="docs",+ agent=_agent,+)++db_query = "What is the account balance for user Alice?"+routed_key = router.route(db_query)+response = router.query(db_query)+print(f"\n--- 3. LayaQueryRouter Direct Dispatch ---")+print(f"Query: {db_query}")+print(f"Routed Engine Key: -> {routed_key}")+print(f"Engine Response: {response}")+++# =====================================================================+# 4. Remote HTTP Server Execution (Zero Heavy Dependencies)+# =====================================================================+# Connects to your self-hosted `laya-serve` instance without requiring PyTorch on edge clients.+remote_selector = LayaSingleSelector(+ base_url="http://localhost:8080",+ confidence_threshold=0.85,+ fallback_index=0,+)+print(f"\n--- 4. Remote HTTP Server Deployment ---")+print("Remote selector configured with base_url='http://localhost:8080' via standard urllib.")diff --git a/laya/integrations/__init__.py b/laya/integrations/__init__.pyindex 7ce62e40..0c79937d 100644--- a/laya/integrations/__init__.py+++ b/laya/integrations/__init__.py@@ -6,6 +6,12 @@ from .langchain import (LayaRouter,LayaTriage,)+from .llamaindex import (+ LayaLowConfidenceError,+ LayaMultiSelector,+ LayaQueryRouter,+ LayaSingleSelector,+)__all__ = ["LayaRouter",@@ -13,4 +19,8 @@ __all__ = ["LayaGuardrailError","LayaTriage","LayaEvaluator",+ "LayaSingleSelector",+ "LayaMultiSelector",+ "LayaQueryRouter",+ "LayaLowConfidenceError",]diff --git a/laya/integrations/llamaindex.py b/laya/integrations/llamaindex.pynew file mode 100644index 00000000..4bf8f960--- /dev/null+++ b/laya/integrations/llamaindex.py@@ -0,0 +1,525 @@+"""LlamaIndex integration for Laya System 1 decision engine.++Provides sub-35ms, non-autoregressive query routing, single/multi-tool selection,+and real-time guardrails for LlamaIndex RAG pipelines and RouterQueryEngine.++Replaces slow, token-generating LLM selectors (`LLMSingleSelector`, `LLMMultiSelector`)+with calibrated, typed decisions executed in a single forward pass without token cost.++Supports both local in-process models (`Agent` / `Router`) and remote HTTP+deployments (your own `laya-serve`) without requiring PyTorch on edge clients.+"""+from __future__ import annotations++import asyncio+import json+import threading+import urllib.error+import urllib.parse+import urllib.request+from dataclasses import dataclass+from typing import Any, Dict, List, Optional, Sequence, Union++# Optional LlamaIndex base classes+try:+ from llama_index.core.schema import QueryBundle+ from llama_index.core.selectors import BaseSelector, SelectorResult, ToolSelection+ from llama_index.core.tools import ToolMetadata+ _LLAMA_INDEX_AVAILABLE = True+except ImportError:+ _LLAMA_INDEX_AVAILABLE = False++ class BaseSelector: # type: ignore+ """Lightweight shim when llama-index-core is not installed."""++ def select(self, choices: Sequence[Any], query: Any) -> Any:+ raise NotImplementedError++ async def aselect(self, choices: Sequence[Any], query: Any) -> Any:+ raise NotImplementedError++ @dataclass+ class ToolMetadata: # type: ignore+ description: str+ name: Optional[str] = None++ @dataclass+ class ToolSelection: # type: ignore+ index: int+ reason: Optional[str] = None++ @dataclass+ class SelectorResult: # type: ignore+ selections: List[ToolSelection]++ @dataclass+ class QueryBundle: # type: ignore+ query_str: str++ def __str__(self) -> str:+ return self.query_str+++class LayaLowConfidenceError(ValueError):+ """Raised when a routing decision falls below the confidence threshold and no fallback is set."""++ def __init__(self, message: str, confidence: float, threshold: float, raw_decision: Dict[str, Any]):+ super().__init__(message)+ self.confidence = confidence+ self.threshold = threshold+ self.raw_decision = raw_decision+++def _extract_query_str(query: Union[str, QueryBundle, Any]) -> str:+ """Extract raw query string from query input."""+ if isinstance(query, str):+ return query+ if hasattr(query, "query_str"):+ return str(query.query_str)+ return str(query)+++def _format_choices_criteria(choices: Sequence[Union[ToolMetadata, Dict[str, Any], Any]]) -> Dict[str, str]:+ """Format candidate tool choices into a Laya choice criteria dictionary."""+ criteria: Dict[str, str] = {}+ for i, choice in enumerate(choices):+ key = f"choice_{i}"+ name = getattr(choice, "name", None)+ desc = getattr(choice, "description", None)++ if isinstance(choice, dict):+ name = choice.get("name", name)+ desc = choice.get("description", desc)+ elif isinstance(choice, str):+ desc = choice++ if not desc:+ desc = name or f"Option {i}"++ if name:+ criteria[key] = f"{name}: {desc}"+ else:+ criteria[key] = str(desc)++ return criteria+++class _SameOriginRedirectHandler(urllib.request.HTTPRedirectHandler):+ """Do not forward bearer credentials across an origin or HTTPS downgrade."""++ def redirect_request(self, req, fp, code, msg, headers, newurl):+ old = urllib.parse.urlsplit(req.full_url)+ new = urllib.parse.urlsplit(newurl)+ old_port = old.port or (443 if old.scheme.lower() == "https" else 80)+ new_port = new.port or (443 if new.scheme.lower() == "https" else 80)+ if (+ old.scheme.lower() != new.scheme.lower()+ or (old.hostname or "").lower() != (new.hostname or "").lower()+ or old_port != new_port+ ):+ raise urllib.error.URLError("refusing cross-origin or HTTPS-downgrade redirect")+ return super().redirect_request(req, fp, code, msg, headers, newurl)+++def _call_remote(+ base_url: str,+ state: Any,+ questions: Dict[str, Any],+ api_key: Optional[str] = None,+ model: Optional[str] = None,+ timeout: float = 10.0,+) -> Dict[str, Any]:+ """Send decision request to a remote laya-serve HTTP instance using standard library urllib."""+ url = base_url.rstrip("/")+ if not url.endswith("/v1/systemone"):+ url = f"{url}/v1/systemone"++ payload: Dict[str, Any] = {"state": state, "questions": questions}+ if model:+ payload["model"] = model++ data = json.dumps(payload).encode("utf-8")+ headers = {"Content-Type": "application/json"}+ if api_key:+ headers["Authorization"] = f"Bearer {api_key}"++ req = urllib.request.Request(url, data=data, headers=headers, method="POST")+ opener = urllib.request.build_opener(_SameOriginRedirectHandler())+ try:+ with opener.open(req, timeout=timeout) as resp:+ return json.loads(resp.read().decode("utf-8"))+ except urllib.error.HTTPError as e:+ body = e.read().decode("utf-8", errors="replace")+ raise RuntimeError(f"Laya server error {e.code}: {body}") from e+ except Exception as e:+ raise RuntimeError(f"Failed to connect to Laya server at {url}: {e}") from e+++_DEFAULT_ROUTER = None+_DEFAULT_ROUTER_LOCK = threading.Lock()+++def _get_default_router():+ global _DEFAULT_ROUTER+ if _DEFAULT_ROUTER is None:+ with _DEFAULT_ROUTER_LOCK:+ if _DEFAULT_ROUTER is None:+ from ..router import Router+ _DEFAULT_ROUTER = Router()+ return _DEFAULT_ROUTER+++def _execute_decision(+ state: Any,+ questions: Dict[str, Any],+ agent: Optional[Any] = None,+ base_url: Optional[str] = None,+ api_key: Optional[str] = None,+ model: Optional[str] = None,+) -> Dict[str, Any]:+ if base_url:+ return _call_remote(base_url, state, questions, api_key=api_key, model=model)+ runner = agent if agent is not None else _get_default_router()+ kwargs = {"model": model} if model else {}+ return runner.predict(state, questions, **kwargs)+++class LayaSingleSelector(BaseSelector):+ """Sub-35ms single-choice selector for LlamaIndex RouterQueryEngine.++ Replaces LLMSingleSelector to route queries to candidate query engines or tools+ in a single non-autoregressive forward pass with zero token generation cost.+ Supports confidence threshold gating and automatic fallback selection.+ """++ def __init__(+ self,+ instructions: str = "Which tool or query engine is best suited to answer this query?",+ confidence_threshold: float = 0.0,+ fallback_index: Optional[int] = None,+ raise_on_low_confidence: bool = False,+ agent: Optional[Any] = None,+ base_url: Optional[str] = None,+ api_key: Optional[str] = None,+ model: Optional[str] = None,+ ):+ self.instructions = instructions+ self.confidence_threshold = confidence_threshold+ self.fallback_index = fallback_index+ self.raise_on_low_confidence = raise_on_low_confidence+ self.agent = agent+ self.base_url = base_url+ self.api_key = api_key+ self.model = model+ self.last_decision: Optional[Dict[str, Any]] = None++ def _select(+ self,+ choices: Sequence[Union[ToolMetadata, Any]],+ query: Union[QueryBundle, str],+ ) -> SelectorResult:+ """Select a single choice synchronously."""+ if not choices:+ raise ValueError("No choices provided to select from.")++ query_str = _extract_query_str(query)+ criteria = _format_choices_criteria(choices)++ questions = {+ "selector": {+ "type": "choice",+ "instructions": self.instructions,+ "criteria": criteria,+ }+ }++ res = _execute_decision(+ query_str,+ questions,+ agent=self.agent,+ base_url=self.base_url,+ api_key=self.api_key,+ model=self.model,+ )+ self.last_decision = res++ ans = res.get("answers", {}).get("selector", {})+ chosen_key = ans.get("choice")+ conf = ans.get("answer_confidence", ans.get("confidence", 1.0))++ # Map "choice_i" back to index i+ chosen_idx: int = 0+ if chosen_key and chosen_key.startswith("choice_"):+ try:+ chosen_idx = int(chosen_key.split("_")[1])+ except (ValueError, IndexError):+ chosen_idx = 0+ elif chosen_key in criteria:+ chosen_idx = list(criteria.keys()).index(chosen_key)++ # Confidence gating+ if self.confidence_threshold > 0.0 and conf < self.confidence_threshold:+ if self.fallback_index is not None:+ reason = (+ f"Selected fallback index {self.fallback_index} because confidence "+ f"({conf:.3f}) was below threshold ({self.confidence_threshold:.3f})."+ )+ return SelectorResult(selections=[ToolSelection(index=self.fallback_index, reason=reason)])+ if self.raise_on_low_confidence:+ raise LayaLowConfidenceError(+ f"Confidence {conf:.3f} below threshold {self.confidence_threshold:.3f} "+ f"for query: {query_str!r}",+ confidence=conf,+ threshold=self.confidence_threshold,+ raw_decision=res,+ )++ choice_obj = choices[chosen_idx]+ choice_name = getattr(choice_obj, "name", None) or f"choice_{chosen_idx}"+ reason = f"Selected '{choice_name}' via Laya System 1 decision (confidence: {conf:.3f})."+ return SelectorResult(selections=[ToolSelection(index=chosen_idx, reason=reason)])++ def select(+ self,+ choices: Sequence[Union[ToolMetadata, Any]],+ query: Union[QueryBundle, str],+ ) -> SelectorResult:+ """Public synchronous selection method."""+ return self._select(choices, query)++ async def _aselect(+ self,+ choices: Sequence[Union[ToolMetadata, Any]],+ query: Union[QueryBundle, str],+ ) -> SelectorResult:+ """Select a single choice asynchronously without blocking the event loop."""+ return await asyncio.to_thread(self._select, choices, query)++ async def aselect(+ self,+ choices: Sequence[Union[ToolMetadata, Any]],+ query: Union[QueryBundle, str],+ ) -> SelectorResult:+ """Public asynchronous selection method."""+ return await self._aselect(choices, query)+++class LayaMultiSelector(BaseSelector):+ """Sub-35ms multi-choice selector for LlamaIndex RouterQueryEngine.++ Replaces LLMMultiSelector to evaluate and select multiple query engines or tools+ for composite questions requiring information from multiple sources.+ """++ def __init__(+ self,+ instructions: str = "Which tools or query engines are relevant to answer this query?",+ max_outputs: Optional[int] = None,+ probability_threshold: float = 0.25,+ default_index: int = 0,+ agent: Optional[Any] = None,+ base_url: Optional[str] = None,+ api_key: Optional[str] = None,+ model: Optional[str] = None,+ ):+ self.instructions = instructions+ self.max_outputs = max_outputs+ self.probability_threshold = probability_threshold+ self.default_index = default_index+ self.agent = agent+ self.base_url = base_url+ self.api_key = api_key+ self.model = model+ self.last_decision: Optional[Dict[str, Any]] = None++ def _select(+ self,+ choices: Sequence[Union[ToolMetadata, Any]],+ query: Union[QueryBundle, str],+ ) -> SelectorResult:+ if not choices:+ raise ValueError("No choices provided to select from.")++ query_str = _extract_query_str(query)+ criteria = _format_choices_criteria(choices)++ questions = {+ "selector": {+ "type": "choice",+ "instructions": self.instructions,+ "criteria": criteria,+ }+ }++ res = _execute_decision(+ query_str,+ questions,+ agent=self.agent,+ base_url=self.base_url,+ api_key=self.api_key,+ model=self.model,+ )+ self.last_decision = res++ ans = res.get("answers", {}).get("selector", {})+ probs = ans.get("probabilities", {})++ selected: List[ToolSelection] = []+ key_list = list(criteria.keys())++ # Sort candidate choices by descending probability+ ranked = sorted(+ key_list,+ key=lambda k: probs.get(k, 0.0),+ reverse=True,+ )++ for key in ranked:+ prob = probs.get(key, 0.0)+ if prob >= self.probability_threshold:+ try:+ idx = int(key.split("_")[1]) if key.startswith("choice_") else key_list.index(key)+ except (ValueError, IndexError):+ idx = key_list.index(key)++ choice_obj = choices[idx]+ choice_name = getattr(choice_obj, "name", None) or f"choice_{idx}"+ selected.append(+ ToolSelection(+ index=idx,+ reason=f"Selected '{choice_name}' (probability: {prob:.3f} >= {self.probability_threshold:.3f}).",+ )+ )+ if self.max_outputs is not None and len(selected) >= self.max_outputs:+ break++ # Fallback to top option if none met threshold+ if not selected and ranked:+ top_key = ranked[0]+ top_idx = int(top_key.split("_")[1]) if top_key.startswith("choice_") else key_list.index(top_key)+ choice_obj = choices[top_idx]+ choice_name = getattr(choice_obj, "name", None) or f"choice_{top_idx}"+ prob = probs.get(top_key, 0.0)+ selected.append(+ ToolSelection(+ index=top_idx,+ reason=f"Selected top choice '{choice_name}' as fallback (probability: {prob:.3f}).",+ )+ )++ return SelectorResult(selections=selected)++ def select(+ self,+ choices: Sequence[Union[ToolMetadata, Any]],+ query: Union[QueryBundle, str],+ ) -> SelectorResult:+ return self._select(choices, query)++ async def _aselect(+ self,+ choices: Sequence[Union[ToolMetadata, Any]],+ query: Union[QueryBundle, str],+ ) -> SelectorResult:+ return await asyncio.to_thread(self._select, choices, query)++ async def aselect(+ self,+ choices: Sequence[Union[ToolMetadata, Any]],+ query: Union[QueryBundle, str],+ ) -> SelectorResult:+ return await self._aselect(choices, query)+++class LayaQueryRouter:+ """Direct, high-performance query dispatcher for LlamaIndex query engines.++ Allows routing queries directly to target engines or callable tools without+ the overhead of standard LLM-based query engine routers.+ """++ def __init__(+ self,+ query_engines: Dict[str, Any],+ descriptions: Optional[Dict[str, str]] = None,+ instructions: str = "Which query engine should handle this request?",+ confidence_threshold: float = 0.0,+ fallback_key: Optional[str] = None,+ agent: Optional[Any] = None,+ base_url: Optional[str] = None,+ api_key: Optional[str] = None,+ model: Optional[str] = None,+ ):+ self.query_engines = query_engines+ self.descriptions = descriptions or {}+ self.instructions = instructions+ self.confidence_threshold = confidence_threshold+ self.fallback_key = fallback_key or next(iter(query_engines.keys()), None)+ self.agent = agent+ self.base_url = base_url+ self.api_key = api_key+ self.model = model+ self.last_decision: Optional[Dict[str, Any]] = None++ def route(self, query: Union[QueryBundle, str]) -> str:+ """Determine which query engine key should handle the query in ~33ms."""+ query_str = _extract_query_str(query)+ criteria = {+ key: self.descriptions.get(key, f"Engine for {key}")+ for key in self.query_engines.keys()+ }++ questions = {+ "route": {+ "type": "choice",+ "instructions": self.instructions,+ "criteria": criteria,+ }+ }++ res = _execute_decision(+ query_str,+ questions,+ agent=self.agent,+ base_url=self.base_url,+ api_key=self.api_key,+ model=self.model,+ )+ self.last_decision = res++ ans = res.get("answers", {}).get("route", {})+ chosen = ans.get("choice")+ conf = ans.get("answer_confidence", ans.get("confidence", 1.0))++ if self.confidence_threshold > 0.0 and conf < self.confidence_threshold:+ if self.fallback_key:+ return self.fallback_key++ if chosen in self.query_engines:+ return chosen++ return self.fallback_key or next(iter(self.query_engines.keys()))++ def query(self, query: Union[QueryBundle, str], **kwargs: Any) -> Any:+ """Route the query to the winning engine and execute engine.query()."""+ key = self.route(query)+ engine = self.query_engines[key]+ if hasattr(engine, "query"):+ return engine.query(query, **kwargs)+ if callable(engine):+ return engine(query, **kwargs)+ raise TypeError(f"Engine registered for key {key!r} is not callable and has no .query() method.")++ async def aquery(self, query: Union[QueryBundle, str], **kwargs: Any) -> Any:+ """Asynchronously route the query and execute engine.aquery() or engine.query()."""+ key = await asyncio.to_thread(self.route, query)+ engine = self.query_engines[key]+ if hasattr(engine, "aquery"):+ return await engine.aquery(query, **kwargs)+ if hasattr(engine, "query"):+ return await asyncio.to_thread(engine.query, query, **kwargs)+ if callable(engine):+ if asyncio.iscoroutinefunction(engine):+ return await engine(query, **kwargs)+ return await asyncio.to_thread(engine, query, **kwargs)+ raise TypeError(f"Engine registered for key {key!r} is not callable and has no .query() method.")diff --git a/pyproject.toml b/pyproject.tomlindex 27b66880..92512810 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -64,6 +64,11 @@ langgraph = ["langgraph>=0.1.0",]+# `pip install laya[llamaindex]` adds the LlamaIndex query router & selector integrations (laya.integrations).+llamaindex = [+ "llama-index-core>=0.10.0",+]+[project.scripts]laya = "laya.cli:main"laya-serve = "laya.serve:main"diff --git a/tests/test_llamaindex.py b/tests/test_llamaindex.pynew file mode 100644index 00000000..e056599a--- /dev/null+++ b/tests/test_llamaindex.py@@ -0,0 +1,403 @@+"""Unit tests for Laya LlamaIndex integration.++Tests verify single/multi-selector logic, confidence threshold fallback gating,+query router dispatch, and LlamaIndex schema conventions without requiring+model downloads, GPU, or external services.+"""+import asyncio+import os+import sys++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))++from laya.integrations.llamaindex import (+ LayaLowConfidenceError,+ LayaMultiSelector,+ LayaQueryRouter,+ LayaSingleSelector,+ QueryBundle,+ ToolMetadata,+ _extract_query_str,+ _format_choices_criteria,+)++PASS, FAIL = [], []+++def check(name, got, want):+ if got == want:+ PASS.append(name)+ else:+ FAIL.append(f"{name}:\n got {got!r}\n want {want!r}")+++def check_true(name, cond, detail=""):+ if cond:+ PASS.append(name)+ else:+ FAIL.append(f"{name} {detail}")+++# --------------------------------------------------------------- Mock Agent+class MockLayaAgent:+ """Mock agent returning deterministic responses for testing."""++ def __init__(self, response_fn):+ self.response_fn = response_fn++ def predict(self, state, questions, **kwargs):+ return self.response_fn(state, questions)+++# --------------------------------------------------------------- 1. Query & Criteria Formatting+check("extract/str", _extract_query_str("What is the revenue?"), "What is the revenue?")+check(+ "extract/query_bundle",+ _extract_query_str(QueryBundle(query_str="How do I reset password?")),+ "How do I reset password?",+)+++class CustomQueryObj:+ def __init__(self, text):+ self.query_str = text+++check(+ "extract/custom_query_obj",+ _extract_query_str(CustomQueryObj("custom text")),+ "custom text",+)++# Test criteria formatting from ToolMetadata+tools = [+ ToolMetadata(name="vector_store", description="Search financial documents"),+ ToolMetadata(name="sql_db", description="Query relational tables"),+ {"name": "summary_tool", "description": "High level summarization"},+ "Simple string description tool",+]+criteria = _format_choices_criteria(tools)+check(+ "criteria/vector_store",+ criteria["choice_0"],+ "vector_store: Search financial documents",+)+check(+ "criteria/sql_db",+ criteria["choice_1"],+ "sql_db: Query relational tables",+)+check(+ "criteria/summary_tool",+ criteria["choice_2"],+ "summary_tool: High level summarization",+)+check(+ "criteria/string_tool",+ criteria["choice_3"],+ "Simple string description tool",+)+++# --------------------------------------------------------------- 2. LayaSingleSelector+def mock_single_selector_response(state, questions):+ text = str(state).lower()+ if "sql" in text or "table" in text or "database" in text:+ chosen = "choice_1"+ conf = 0.94+ elif "lowconf" in text:+ chosen = "choice_0"+ conf = 0.42+ else:+ chosen = "choice_0"+ conf = 0.91++ return {+ "model": "mock-single",+ "answers": {+ "selector": {+ "choice": chosen,+ "answer_confidence": conf,+ "confidence": conf,+ }+ },+ }+++mock_agent = MockLayaAgent(mock_single_selector_response)+single_selector = LayaSingleSelector(agent=mock_agent)++# Standard selection for choice_0+res0 = single_selector.select(tools, "Search for quarterly 10-K report")+check("single/index_0", res0.selections[0].index, 0)+check_true(+ "single/reason_0",+ "Selected 'vector_store' via Laya System 1 decision" in res0.selections[0].reason,+)++# Standard selection for choice_1 (sql)+res1 = single_selector.select(tools, QueryBundle(query_str="Query the SQL database for customers"))+check("single/index_1", res1.selections[0].index, 1)+check_true(+ "single/reason_1",+ "Selected 'sql_db' via Laya System 1 decision" in res1.selections[0].reason,+)++# Confidence threshold with fallback+gated_selector = LayaSingleSelector(+ agent=mock_agent,+ confidence_threshold=0.80,+ fallback_index=2,+)+res_fallback = gated_selector.select(tools, "lowconf query here")+check("single/fallback_index", res_fallback.selections[0].index, 2)+check_true(+ "single/fallback_reason",+ "Selected fallback index 2" in res_fallback.selections[0].reason,+)++# Confidence threshold with raise_on_low_confidence+raising_selector = LayaSingleSelector(+ agent=mock_agent,+ confidence_threshold=0.80,+ raise_on_low_confidence=True,+)+try:+ raising_selector.select(tools, "lowconf query here")+ check("single/raise_error", False, True)+except LayaLowConfidenceError as e:+ check("single/raise_error", True, True)+ check("single/error_conf", e.confidence, 0.42)+ check("single/error_thresh", e.threshold, 0.80)++# Empty choices validation+try:+ single_selector.select([], "any query")+ check("single/empty_choices", False, True)+except ValueError:+ check("single/empty_choices", True, True)+++# Async single selector+async def run_async_single():+ res_async = await single_selector.aselect(tools, "Search the SQL database")+ check("single/async_index", res_async.selections[0].index, 1)+++asyncio.run(run_async_single())+++# --------------------------------------------------------------- 3. LayaMultiSelector+def mock_multi_selector_response(state, questions):+ text = str(state).lower()+ if "compare" in text or "both" in text:+ # High probability for both vector (0) and sql (1)+ probs = {"choice_0": 0.55, "choice_1": 0.40, "choice_2": 0.05}+ elif "none_qualify" in text:+ # All below threshold+ probs = {"choice_0": 0.15, "choice_1": 0.10, "choice_2": 0.05}+ else:+ probs = {"choice_0": 0.85, "choice_1": 0.10, "choice_2": 0.05}++ return {+ "model": "mock-multi",+ "answers": {+ "selector": {+ "choice": max(probs, key=probs.get),+ "probabilities": probs,+ "answer_confidence": 0.85,+ }+ },+ }+++mock_multi_agent = MockLayaAgent(mock_multi_selector_response)+multi_selector = LayaMultiSelector(+ agent=mock_multi_agent,+ probability_threshold=0.30,+)++# Multi-select returns multiple tools+res_multi = multi_selector.select(tools[:3], "Compare both vector docs and SQL records")+check("multi/count", len(res_multi.selections), 2)+check("multi/idx_0", res_multi.selections[0].index, 0)+check("multi/idx_1", res_multi.selections[1].index, 1)++# Multi-select with max_outputs limit+limited_selector = LayaMultiSelector(+ agent=mock_multi_agent,+ probability_threshold=0.30,+ max_outputs=1,+)+res_limited = limited_selector.select(tools[:3], "Compare both vector docs and SQL records")+check("multi/limited_count", len(res_limited.selections), 1)+check("multi/limited_idx", res_limited.selections[0].index, 0)++# Multi-select fallback to top option when none meet threshold+res_none = multi_selector.select(tools[:3], "none_qualify search")+check("multi/fallback_top_count", len(res_none.selections), 1)+check("multi/fallback_top_idx", res_none.selections[0].index, 0)+++# Async multi-selector+async def run_async_multi():+ res_async = await multi_selector.aselect(tools[:3], "Compare both sources")+ check("multi/async_count", len(res_async.selections), 2)+++asyncio.run(run_async_multi())+++# --------------------------------------------------------------- 4. LayaQueryRouter+class MockQueryEngine:+ def __init__(self, name):+ self.name = name++ def query(self, q):+ return f"Result from {self.name} for: {q}"++ async def aquery(self, q):+ return f"Async result from {self.name} for: {q}"+++engines = {+ "vector": MockQueryEngine("VectorEngine"),+ "sql": MockQueryEngine("SqlEngine"),+ "summary": MockQueryEngine("SummaryEngine"),+}+++def mock_router_dispatcher(state, questions):+ text = str(state).lower()+ if "sql" in text or "table" in text:+ chosen = "sql"+ conf = 0.96+ elif "summary" in text:+ chosen = "summary"+ conf = 0.91+ elif "lowconf" in text:+ chosen = "summary"+ conf = 0.35+ else:+ chosen = "vector"+ conf = 0.89++ return {+ "model": "mock-router",+ "answers": {+ "route": {+ "choice": chosen,+ "answer_confidence": conf,+ "confidence": conf,+ }+ },+ }+++router_agent = MockLayaAgent(mock_router_dispatcher)+laya_qr = LayaQueryRouter(+ query_engines=engines,+ descriptions={+ "vector": "Semantic search over documentation",+ "sql": "Relational SQL queries",+ "summary": "Document summarization",+ },+ agent=router_agent,+ confidence_threshold=0.70,+ fallback_key="vector",+)++# Route direct+check("router/route_sql", laya_qr.route("Run a query on the SQL table"), "sql")+check("router/route_summary", laya_qr.route("Provide an executive summary"), "summary")+check("router/route_vector", laya_qr.route("How does authentication work?"), "vector")++# Confidence fallback+check("router/route_fallback", laya_qr.route("lowconf request"), "vector")++# Execute query()+res_q = laya_qr.query("Run a query on the SQL table")+check("router/query_exec", res_q, "Result from SqlEngine for: Run a query on the SQL table")+++# Async query+async def run_async_router():+ res_aq = await laya_qr.aquery("Provide an executive summary")+ check("router/aquery_exec", res_aq, "Async result from SummaryEngine for: Provide an executive summary")+++asyncio.run(run_async_router())+++# Callable engine support+callable_engines = {+ "calc": lambda q: f"Calculated: {q}",+ "search": lambda q: f"Searched: {q}",+}+callable_qr = LayaQueryRouter(+ query_engines=callable_engines,+ agent=MockLayaAgent(+ lambda s, q: {"answers": {"route": {"choice": "calc", "answer_confidence": 0.99}}}+ ),+)+check("router/callable", callable_qr.query("2 + 2"), "Calculated: 2 + 2")+++# --------------------------------------------------------------- 5. Remote HTTP Execution (Mocked)+from unittest.mock import MagicMock, patch+++class DummyHTTPResponse:+ def __init__(self, data_dict):+ self.data = json.dumps(data_dict).encode("utf-8")++ def read(self):+ return self.data++ def __enter__(self):+ return self++ def __exit__(self, *args):+ pass+++import json++remote_response = {+ "model": "laya-multilingual",+ "answers": {+ "selector": {+ "choice": "choice_1",+ "answer_confidence": 0.98,+ "confidence": 0.98,+ }+ },+}++with patch("urllib.request.build_opener") as mock_build_opener:+ mock_opener = MagicMock()+ mock_opener.open.return_value = DummyHTTPResponse(remote_response)+ mock_build_opener.return_value = mock_opener++ remote_selector = LayaSingleSelector(+ base_url="https://api.laya.ai",+ api_key="sk-test-token",+ )+ res_remote = remote_selector.select(tools, "Query via remote server")+ check("remote/index", res_remote.selections[0].index, 1)++ # Check request headers and URL+ call_args = mock_opener.open.call_args+ req = call_args[0][0]+ check("remote/url", req.full_url, "https://api.laya.ai/v1/systemone")+ check("remote/auth", req.headers.get("Authorization"), "Bearer sk-test-token")+++# --------------------------------------------------------------- Results Summary+print(f"PASS: {len(PASS)}")+print(f"FAIL: {len(FAIL)}")+for f in FAIL:+ print(f" - {f}")++if FAIL:+ sys.exit(1)