Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
feat(llms): standardize generate_typed() across all provider wrappers (#1553)
Sakshi Jainc47ffae8ff94
diff --git a/docs/guides/llm-integrations.md b/docs/guides/llm-integrations.mdindex 09458d58..8b6fd62d 100644--- a/docs/guides/llm-integrations.md+++ b/docs/guides/llm-integrations.md@@ -3,11 +3,11 @@ title: "LLM Integrations"description: "Connect Semantica to Groq, OpenAI, Anthropic, HuggingFace, Novita AI, and 100+ LLM providers through a unified interface."----Semantica exposes a unified provider interface — a single `.generate()` method — across Groq, OpenAI, Anthropic Claude, HuggingFace, Novita AI, and 100+ providers via LiteLLM. Use it when you need to swap providers for latency, accuracy, cost, or data-residency reasons without touching application code.+Semantica exposes a unified provider interface — the same `.generate()`, `.generate_structured()` and `.generate_typed()` methods — across Groq, OpenAI, Anthropic Claude, HuggingFace, Novita AI, and 100+ providers via LiteLLM. Use it when you need to swap providers for latency, accuracy, cost, or data-residency reasons without touching application code.## What Are LLM Integrations?-The `semantica.llms` module provides a unified interface for connecting to Large Language Model providers. Instead of learning different APIs for each provider, you use the same methods (`.generate()`, `.generate_structured()`) regardless of whether you're calling Groq, OpenAI, Anthropic, or local HuggingFace models.+The `semantica.llms` module provides a unified interface for connecting to Large Language Model providers. Instead of learning different APIs for each provider, you use the same methods (`.generate()`, `.generate_structured()`, `.generate_typed()`) regardless of whether you're calling Groq, OpenAI, Anthropic, or local HuggingFace models.**Unified interface across providers:** All LLM providers in Semantica expose identical methods, so switching from OpenAI to Anthropic requires changing only the provider constructor, not your application code.@@ -19,7 +19,7 @@ The `semantica.llms` module provides a unified interface for connecting to Large**Reduced vendor lock-in.** Avoid tying your application to a single LLM provider's API. If pricing changes or service availability issues arise, switching providers is straightforward.-**Consistent APIs.** Use the same `.generate()` and `.generate_structured()` methods across all providers instead of learning provider-specific interfaces.+**Consistent APIs.** Use the same `.generate()`, `.generate_structured()`, and `.generate_typed()` methods across all providers instead of learning provider-specific interfaces.**Multi-provider workflows.** Run fast models for initial classification and expensive frontier models for complex reasoning in the same pipeline.@@ -70,15 +70,16 @@ The unified interface means you can prototype with Groq for speed, validate accu## The Shared Interface-Every provider exposes the same two methods:+Every provider exposes the same methods:```pythonprovider.generate(prompt: str, **kwargs) -> str-provider.generate_structured(prompt: str, **kwargs) -> dict+provider.generate_structured(prompt: str, **kwargs) -> dict | list+provider.generate_typed(prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs) -> BaseModelprovider.is_available() -> bool```-`generate()` returns a plain string. `generate_structured()` instructs the model to respond in JSON and returns a parsed `dict`. `is_available()` lets you health-check the provider before committing to a call — useful in retry logic and warm-up checks.+`generate()` returns a plain string. `generate_structured()` instructs the model to respond in JSON and returns the parsed result — a `dict` for a top-level JSON object, or a `list` if the model returns a top-level JSON array. `generate_typed()` takes a Pydantic model, validates the model's output against it, and retries up to `max_retries` times with the validation error fed back into the prompt — reach for it when downstream code needs a guaranteed shape rather than best-effort JSON. `is_available()` lets you health-check the provider before committing to a call — useful in retry logic and warm-up checks.This means every place in Semantica that accepts an LLM — `query_with_reasoning()`, semantic extraction, custom reasoning loops — accepts any of these providers interchangeably.@@ -711,7 +712,7 @@ for src in best["sources"]:**Using LLMs for deterministic pattern matching that regex can handle.** If your task is extracting email addresses, phone numbers, or other pattern-based entities, regular expressions are faster, cheaper, and more reliable than LLM extraction. Use LLMs when context, ambiguity, or domain knowledge matter for correct interpretation.-**Not validating structured outputs.** The `generate_structured()` method returns parsed JSON, but LLMs can still produce malformed or incomplete structures. Always validate the returned dictionary against your expected schema before using the data downstream.+**Not validating structured outputs.** The `generate_structured()` method returns parsed JSON (a dict, or a list for a top-level array), but LLMs can still produce malformed or incomplete structures. Validate the result against your expected schema before using it downstream — or use `generate_typed()`, which validates against a Pydantic model for you.**Switching providers without testing prompt behavior.** Different models respond differently to the same prompt. A prompt optimized for GPT-4 may produce poor results with Llama or Claude. When switching providers, test your prompts and adjust temperature, instructions, or examples as needed.diff --git a/semantica/llms/__init__.py b/semantica/llms/__init__.pyindex 943a88e6..b31511ff 100644--- a/semantica/llms/__init__.py+++ b/semantica/llms/__init__.py@@ -5,6 +5,12 @@ This module provides clean, intuitive imports for LLM providers used in SemanticIt wraps the underlying provider functionality from semantica.semantic_extract.providersto provide a cleaner API.+Every provider wrapper exposes the same generation interface:+ - generate(prompt, **kwargs) -> str+ - generate_structured(prompt, **kwargs) -> dict | list+ - generate_typed(prompt, schema, max_retries=3, **kwargs) -> BaseModel+ - is_available() -> bool+Supported Providers:- Groq: Groq API for fast inference- OpenAI: OpenAI API (GPT-3.5, GPT-4, etc.)diff --git a/semantica/llms/anthropic.py b/semantica/llms/anthropic.pyindex 573d761b..54cc330b 100644--- a/semantica/llms/anthropic.py+++ b/semantica/llms/anthropic.py@@ -4,7 +4,9 @@ Anthropic LLM ProviderWrapper for Anthropic Claude API provider with clean interface"""-from typing import Any, Dict, List, Optional, Union+from typing import Any, Dict, List, Optional, Type, Union++from pydantic import BaseModelfrom ..semantic_extract.providers import AnthropicProviderfrom ..utils.exceptions import ProcessingError@@ -88,7 +90,9 @@ class Anthropic:)return self.provider.generate_structured(prompt, **kwargs)- def generate_typed(self, prompt: str, schema: Any, max_retries: int = 3, **kwargs) -> Any:+ def generate_typed(+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs+ ) -> BaseModel:"""Generate output validated against a Pydantic schema.diff --git a/semantica/llms/deepseek.py b/semantica/llms/deepseek.pyindex 545f764c..89faa18a 100644--- a/semantica/llms/deepseek.py+++ b/semantica/llms/deepseek.py@@ -4,7 +4,9 @@ DeepSeek LLM ProviderWrapper for DeepSeek API provider with clean interface."""-from typing import Any, Dict, List, Optional, Union+from typing import Any, Dict, List, Optional, Type, Union++from pydantic import BaseModelfrom ..semantic_extract.providers import DeepSeekProviderfrom ..utils.exceptions import ProcessingError@@ -88,7 +90,9 @@ class DeepSeek:)return self.provider.generate_structured(prompt, **kwargs)- def generate_typed(self, prompt: str, schema: Any, max_retries: int = 3, **kwargs) -> Any:+ def generate_typed(+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs+ ) -> BaseModel:"""Generate output validated against a Pydantic schema.diff --git a/semantica/llms/gemini.py b/semantica/llms/gemini.pyindex a8bfa163..83bf9df7 100644--- a/semantica/llms/gemini.py+++ b/semantica/llms/gemini.py@@ -4,7 +4,9 @@ Gemini LLM ProviderWrapper for Google Gemini API provider with clean interface."""-from typing import Any, Dict, List, Optional, Union+from typing import Any, Dict, List, Optional, Type, Union++from pydantic import BaseModelfrom ..semantic_extract.providers import GeminiProviderfrom ..utils.exceptions import ProcessingError@@ -88,7 +90,9 @@ class Gemini:)return self.provider.generate_structured(prompt, **kwargs)- def generate_typed(self, prompt: str, schema: Any, max_retries: int = 3, **kwargs) -> Any:+ def generate_typed(+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs+ ) -> BaseModel:"""Generate output validated against a Pydantic schema.diff --git a/semantica/llms/groq.py b/semantica/llms/groq.pyindex e3cbf43e..20fac876 100644--- a/semantica/llms/groq.py+++ b/semantica/llms/groq.py@@ -4,7 +4,9 @@ Groq LLM ProviderWrapper for Groq API provider with clean interface."""-from typing import Any, Dict, Optional+from typing import Any, Dict, List, Optional, Type, Union++from pydantic import BaseModelfrom ..semantic_extract.providers import GroqProviderfrom ..utils.exceptions import ProcessingError@@ -67,7 +69,7 @@ class Groq:)return self.provider.generate(prompt, **kwargs)- def generate_structured(self, prompt: str, **kwargs) -> Dict[str, Any]:+ def generate_structured(self, prompt: str, **kwargs) -> Union[Dict[str, Any], List[Any]]:"""Generate structured JSON output.@@ -76,7 +78,8 @@ class Groq:**kwargs: Generation optionsReturns:- Parsed JSON response as dictionary+ Parsed JSON response. A dict for a top-level JSON object, or a+ list if the model returns a top-level JSON array.Raises:ProcessingError: If provider is not available or parsing fails@@ -87,3 +90,27 @@ class Groq:)return self.provider.generate_structured(prompt, **kwargs)+ def generate_typed(+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs+ ) -> BaseModel:+ """+ Generate output validated against a Pydantic schema.++ Args:+ prompt: Input prompt text+ schema: Pydantic model class to validate the output against+ max_retries: Number of retries if validation fails (default: 3)+ **kwargs: Generation options++ Returns:+ An instance of `schema`, populated from the model's response++ Raises:+ ProcessingError: If provider is not available or generation fails+ """+ if not self.is_available():+ raise ProcessingError(+ "Groq provider not available. Set GROQ_API_KEY or pass api_key."+ )+ return self.provider.generate_typed(prompt, schema, max_retries=max_retries, **kwargs)+diff --git a/semantica/llms/huggingface.py b/semantica/llms/huggingface.pyindex bd5326ce..c7f162e0 100644--- a/semantica/llms/huggingface.py+++ b/semantica/llms/huggingface.py@@ -4,7 +4,9 @@ HuggingFace LLM ProviderWrapper for HuggingFace Transformers LLM provider with clean interface."""-from typing import Any, Dict, Optional+from typing import Any, Dict, List, Optional, Type, Union++from pydantic import BaseModelfrom ..semantic_extract.providers import HuggingFaceLLMProviderfrom ..utils.exceptions import ProcessingError@@ -78,7 +80,7 @@ class HuggingFaceLLM:)return self.provider.generate(prompt, **kwargs)- def generate_structured(self, prompt: str, **kwargs) -> Dict[str, Any]:+ def generate_structured(self, prompt: str, **kwargs) -> Union[Dict[str, Any], List[Any]]:"""Generate structured JSON output.@@ -87,7 +89,8 @@ class HuggingFaceLLM:**kwargs: Generation optionsReturns:- Parsed JSON response as dictionary+ Parsed JSON response. A dict for a top-level JSON object, or a+ list if the model returns a top-level JSON array.Raises:ProcessingError: If provider is not available or parsing fails@@ -98,3 +101,27 @@ class HuggingFaceLLM:)return self.provider.generate_structured(prompt, **kwargs)+ def generate_typed(+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs+ ) -> BaseModel:+ """+ Generate output validated against a Pydantic schema.++ Args:+ prompt: Input prompt text+ schema: Pydantic model class to validate the output against+ max_retries: Number of retries if validation fails (default: 3)+ **kwargs: Generation options++ Returns:+ An instance of `schema`, populated from the model's response++ Raises:+ ProcessingError: If provider is not available or generation fails+ """+ if not self.is_available():+ raise ProcessingError(+ "HuggingFace LLM provider not available. Install transformers library."+ )+ return self.provider.generate_typed(prompt, schema, max_retries=max_retries, **kwargs)+diff --git a/semantica/llms/litellm.py b/semantica/llms/litellm.pyindex 770db273..8c75e2e9 100644--- a/semantica/llms/litellm.py+++ b/semantica/llms/litellm.py@@ -5,7 +5,10 @@ Wrapper for LiteLLM library that provides unified access to 100+ LLM providers.Supports OpenAI, Anthropic, Groq, Azure, Bedrock, Vertex AI, and many more."""-from typing import Any, Dict, List, Optional+import time+from typing import Any, Dict, List, Optional, Type, Union++from pydantic import BaseModelfrom ..utils.exceptions import ProcessingErrorfrom ..utils.logging import get_logger@@ -123,7 +126,7 @@ class LiteLLM:logger.error(f"LiteLLM generation failed: {e}")raise ProcessingError(f"LiteLLM generation failed: {e}")- def generate_structured(self, prompt: str, **kwargs) -> Dict[str, Any]:+ def generate_structured(self, prompt: str, **kwargs) -> Union[Dict[str, Any], List[Any]]:"""Generate structured JSON output.@@ -132,7 +135,8 @@ class LiteLLM:**kwargs: Generation optionsReturns:- Parsed JSON response as dictionary+ Parsed JSON response. A dict for a top-level JSON object, or a+ list if the model returns a top-level JSON array.Raises:ProcessingError: If provider is not available or parsing fails@@ -189,3 +193,84 @@ class LiteLLM:logger.error(f"LiteLLM structured generation failed: {e}")raise ProcessingError(f"LiteLLM structured generation failed: {e}")+ def generate_typed(+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs+ ) -> BaseModel:+ """+ Generate output validated against a Pydantic schema.++ Uses the ``instructor`` library when it is installed and can be+ initialised (schema and retries handled natively). Otherwise - instructor+ missing, or its setup failing - falls back to a JSON-generation plus+ Pydantic-validation retry loop on top of ``generate_structured()``. A+ failure of the instructor completion request itself is raised, not+ retried through the fallback.++ Args:+ prompt: Input prompt text+ schema: Pydantic model class to validate the output against+ max_retries: Number of retries if validation fails (default: 3)+ **kwargs: Generation options++ Returns:+ An instance of `schema`, populated from the model's response++ Raises:+ ProcessingError: If provider is not available or generation fails+ """+ if not self.is_available():+ raise ProcessingError(+ "LiteLLM library not installed. Install with: pip install litellm"+ )++ options = {**self.config, **kwargs}++ # Preferred path: instructor drives the schema and retries natively.+ # Only an instructor setup failure falls through to the manual loop - a+ # failure of the completion request itself (auth, transport, exhausted+ # retries) is surfaced directly rather than retried a second way.+ instructor_mod, instructor_available = safe_import("instructor")+ if instructor_available:+ client = None+ try:+ client = instructor_mod.from_litellm(completion)+ except Exception as e:+ logger.warning(+ f"instructor is installed but could not be initialised for "+ f"LiteLLM ({e}); falling back to manual validation loop."+ )+ if client is not None:+ try:+ return client.chat.completions.create(+ model=self.model,+ messages=[{"role": "user", "content": prompt}],+ api_key=self.api_key,+ response_model=schema,+ max_retries=max_retries,+ **options,+ )+ except Exception as e:+ raise ProcessingError(+ f"LiteLLM typed generation failed: {e}"+ ) from e++ # Fallback: generate JSON, validate against the schema, retry with feedback.+ last_error = None+ current_prompt = prompt+ for _attempt in range(max_retries):+ try:+ data = self.generate_structured(current_prompt, **kwargs)+ return schema.model_validate(data)+ except Exception as e:+ last_error = e+ current_prompt = (+ f"{prompt}\n\nThe previous response did not match the required "+ f"schema:\n{e}\n\nReturn valid JSON that matches the schema."+ )+ if _attempt < max_retries - 1:+ time.sleep(1)++ raise ProcessingError(+ f"LiteLLM typed generation failed after {max_retries} attempts: {last_error}"+ )+diff --git a/semantica/llms/novita.py b/semantica/llms/novita.pyindex ddada85e..d99ee789 100644--- a/semantica/llms/novita.py+++ b/semantica/llms/novita.py@@ -4,7 +4,9 @@ Novita LLM ProviderWrapper for Novita AI's OpenAI-compatible API with clean interface."""-from typing import Any, Dict, List, Optional, Union+from typing import Any, Dict, List, Optional, Type, Union++from pydantic import BaseModelfrom ..semantic_extract.providers import NovitaProviderfrom ..utils.exceptions import ProcessingError@@ -88,7 +90,9 @@ class Novita:)return self.provider.generate_structured(prompt, **kwargs)- def generate_typed(self, prompt: str, schema: Any, max_retries: int = 3, **kwargs) -> Any:+ def generate_typed(+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs+ ) -> BaseModel:"""Generate output validated against a Pydantic schema.diff --git a/semantica/llms/ollama.py b/semantica/llms/ollama.pyindex bc7f4aae..c6fb5788 100644--- a/semantica/llms/ollama.py+++ b/semantica/llms/ollama.py@@ -4,7 +4,9 @@ Ollama LLM ProviderWrapper for local Ollama models with clean interface."""-from typing import Any, Dict, List, Union+from typing import Any, Dict, List, Type, Union++from pydantic import BaseModelfrom ..semantic_extract.providers import OllamaProviderfrom ..utils.exceptions import ProcessingError@@ -92,7 +94,9 @@ class Ollama:)return self.provider.generate_structured(prompt, **kwargs)- def generate_typed(self, prompt: str, schema: Any, max_retries: int = 3, **kwargs) -> Any:+ def generate_typed(+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs+ ) -> BaseModel:"""Generate output validated against a Pydantic schema.diff --git a/semantica/llms/openai.py b/semantica/llms/openai.pyindex 15a3a8df..b8615d87 100644--- a/semantica/llms/openai.py+++ b/semantica/llms/openai.py@@ -4,7 +4,9 @@ OpenAI LLM ProviderWrapper for OpenAI API provider with clean interface."""-from typing import Any, Dict, Optional+from typing import Any, Dict, List, Optional, Type, Union++from pydantic import BaseModelfrom ..semantic_extract.providers import OpenAIProviderfrom ..utils.exceptions import ProcessingError@@ -67,7 +69,7 @@ class OpenAI:)return self.provider.generate(prompt, **kwargs)- def generate_structured(self, prompt: str, **kwargs) -> Dict[str, Any]:+ def generate_structured(self, prompt: str, **kwargs) -> Union[Dict[str, Any], List[Any]]:"""Generate structured JSON output.@@ -76,7 +78,8 @@ class OpenAI:**kwargs: Generation optionsReturns:- Parsed JSON response as dictionary+ Parsed JSON response. A dict for a top-level JSON object, or a+ list if the model returns a top-level JSON array.Raises:ProcessingError: If provider is not available or parsing fails@@ -87,3 +90,27 @@ class OpenAI:)return self.provider.generate_structured(prompt, **kwargs)+ def generate_typed(+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs+ ) -> BaseModel:+ """+ Generate output validated against a Pydantic schema.++ Args:+ prompt: Input prompt text+ schema: Pydantic model class to validate the output against+ max_retries: Number of retries if validation fails (default: 3)+ **kwargs: Generation options++ Returns:+ An instance of `schema`, populated from the model's response++ Raises:+ ProcessingError: If provider is not available or generation fails+ """+ if not self.is_available():+ raise ProcessingError(+ "OpenAI provider not available. Set OPENAI_API_KEY or pass api_key."+ )+ return self.provider.generate_typed(prompt, schema, max_retries=max_retries, **kwargs)+diff --git a/tests/test_llm_groq.py b/tests/test_llm_groq.pynew file mode 100644index 00000000..5245b376--- /dev/null+++ b/tests/test_llm_groq.py@@ -0,0 +1,82 @@+"""Tests for the Groq LLM provider wrapper (semantica.llms.Groq)."""++from unittest.mock import MagicMock++import pytest++from semantica.llms import Groq+from semantica.utils.exceptions import ProcessingError+++def test_construction_stores_model_and_api_key():+ llm = Groq(model="llama-3.1-8b-instant", api_key="fake-key")+ assert llm.model == "llama-3.1-8b-instant"+ assert llm.api_key == "fake-key"+++def test_is_available_false_with_no_key(monkeypatch):+ monkeypatch.delenv("GROQ_API_KEY", raising=False)+ llm = Groq(api_key=None)+ assert llm.is_available() is False+++def test_generate_raises_clear_error_when_unavailable(monkeypatch):+ monkeypatch.delenv("GROQ_API_KEY", raising=False)+ llm = Groq(api_key=None)+ with pytest.raises(ProcessingError, match="Groq provider not available"):+ llm.generate("hello")+++def test_generate_forwards_to_the_real_provider_when_available():+ llm = Groq(api_key="fake-key")+ llm.provider = MagicMock()+ llm.provider.is_available.return_value = True+ llm.provider.generate.return_value = "a fake response"++ result = llm.generate("hello", temperature=0.5)++ assert result == "a fake response"+ llm.provider.generate.assert_called_once_with("hello", temperature=0.5)+++def test_generate_structured_forwards_to_the_real_provider():+ llm = Groq(api_key="fake-key")+ llm.provider = MagicMock()+ llm.provider.is_available.return_value = True+ llm.provider.generate_structured.return_value = {"key": "value"}++ result = llm.generate_structured("hello")++ assert result == {"key": "value"}+ llm.provider.generate_structured.assert_called_once_with("hello")+++def test_generate_typed_forwards_schema_max_retries_and_extra_kwargs():+ """Also verifies the **kwargs contract: anything beyond schema/max_retries+ (e.g. temperature) must reach the underlying provider unchanged."""+ llm = Groq(api_key="fake-key")+ llm.provider = MagicMock()+ llm.provider.is_available.return_value = True+ fake_schema = object()+ llm.provider.generate_typed.return_value = "typed result"++ result = llm.generate_typed("hello", fake_schema, max_retries=5, temperature=0.5)++ assert result == "typed result"+ llm.provider.generate_typed.assert_called_once_with(+ "hello", fake_schema, max_retries=5, temperature=0.5+ )+++def test_generate_structured_raises_clear_error_when_unavailable(monkeypatch):+ monkeypatch.delenv("GROQ_API_KEY", raising=False)+ llm = Groq(api_key=None)+ with pytest.raises(ProcessingError, match="Groq provider not available"):+ llm.generate_structured("hello")+++def test_generate_typed_raises_clear_error_when_unavailable(monkeypatch):+ monkeypatch.delenv("GROQ_API_KEY", raising=False)+ llm = Groq(api_key=None)+ with pytest.raises(ProcessingError, match="Groq provider not available"):+ llm.generate_typed("hello", object())diff --git a/tests/test_llm_huggingface.py b/tests/test_llm_huggingface.pynew file mode 100644index 00000000..24bfd562--- /dev/null+++ b/tests/test_llm_huggingface.py@@ -0,0 +1,79 @@+"""Tests for the HuggingFace LLM provider wrapper (semantica.llms.HuggingFaceLLM).++The wrapper builds a real ``HuggingFaceLLMProvider`` in ``__init__``, which would+load a model from disk/network. These tests patch that class so we only exercise+the wrapper's delegation and error handling.+"""++from unittest.mock import patch++import pytest++from semantica.utils.exceptions import ProcessingError++PROVIDER_PATH = "semantica.llms.huggingface.HuggingFaceLLMProvider"+++def _make_wrapper(available=True):+ from semantica.llms import HuggingFaceLLM++ with patch(PROVIDER_PATH) as provider_cls:+ provider_cls.return_value.is_available.return_value = available+ return HuggingFaceLLM(model_name="gpt2")+++def test_construction_stores_model_name():+ hf = _make_wrapper()+ assert hf.model_name == "gpt2"+ assert hf.model == "gpt2" # alias kept for cross-provider consistency+++def test_generate_raises_clear_error_when_unavailable():+ hf = _make_wrapper(available=False)+ with pytest.raises(ProcessingError, match="HuggingFace LLM provider not available"):+ hf.generate("hello")+++def test_generate_forwards_to_the_real_provider_when_available():+ hf = _make_wrapper()+ hf.provider.generate.return_value = "a fake response"++ result = hf.generate("hello", max_new_tokens=10)++ assert result == "a fake response"+ hf.provider.generate.assert_called_once_with("hello", max_new_tokens=10)+++def test_generate_structured_forwards_to_the_real_provider():+ hf = _make_wrapper()+ hf.provider.generate_structured.return_value = {"key": "value"}++ result = hf.generate_structured("hello")++ assert result == {"key": "value"}+ hf.provider.generate_structured.assert_called_once_with("hello")+++def test_generate_typed_forwards_schema_and_max_retries():+ hf = _make_wrapper()+ fake_schema = object()+ hf.provider.generate_typed.return_value = "typed result"++ result = hf.generate_typed("hello", fake_schema, max_retries=5)++ assert result == "typed result"+ hf.provider.generate_typed.assert_called_once_with(+ "hello", fake_schema, max_retries=5+ )+++def test_generate_structured_raises_clear_error_when_unavailable():+ hf = _make_wrapper(available=False)+ with pytest.raises(ProcessingError, match="HuggingFace LLM provider not available"):+ hf.generate_structured("hello")+++def test_generate_typed_raises_clear_error_when_unavailable():+ hf = _make_wrapper(available=False)+ with pytest.raises(ProcessingError, match="HuggingFace LLM provider not available"):+ hf.generate_typed("hello", object())diff --git a/tests/test_llm_litellm.py b/tests/test_llm_litellm.pynew file mode 100644index 00000000..5c7294c8--- /dev/null+++ b/tests/test_llm_litellm.py@@ -0,0 +1,190 @@+"""Tests for the LiteLLM provider wrapper (semantica.llms.LiteLLM).++Unlike the other wrappers, LiteLLM does not sit on a ``BaseProvider`` - it calls+``litellm.completion`` directly - so ``generate_typed()`` has its own+implementation: it uses ``instructor`` when that package is importable, and+otherwise falls back to a validate-and-retry loop on top of+``generate_structured()``. Which path runs depends only on whether ``instructor``+imports, so every test here pins that explicitly via ``safe_import``.+"""++from unittest.mock import MagicMock++import pytest+from pydantic import BaseModel++import semantica.llms.litellm as litellm_module+from semantica.llms.litellm import LiteLLM+from semantica.utils.exceptions import ProcessingError+++class _Point(BaseModel):+ x: int+ y: int+++def litellm_available(monkeypatch):+ """Pretend the litellm library is installed."""+ monkeypatch.setattr(litellm_module, "LITELLM_AVAILABLE", True)+ monkeypatch.setattr(litellm_module, "completion", MagicMock(), raising=False)+++def without_instructor(monkeypatch):+ """Force ``safe_import("instructor")`` to report unavailable.++ Without this the selected code path would depend on whether ``instructor``+ happens to be installed in the test environment (it is part of the repo's+ all-extras set), so the manual-fallback tests would not reliably exercise+ the fallback.+ """+ real = litellm_module.safe_import+ monkeypatch.setattr(+ litellm_module,+ "safe_import",+ lambda name, *a, **k: (None, False) if name == "instructor" else real(name, *a, **k),+ )+++def _with_instructor(monkeypatch, fake_client):+ """Point ``safe_import("instructor")`` at a fake instructor module."""+ fake_instructor = MagicMock()+ fake_instructor.from_litellm.return_value = fake_client+ real = litellm_module.safe_import+ monkeypatch.setattr(+ litellm_module,+ "safe_import",+ lambda name, *a, **k: (fake_instructor, True)+ if name == "instructor"+ else real(name, *a, **k),+ )+ return fake_instructor+++def _with_instructor_init_failure(monkeypatch, error=None):+ """Point ``safe_import("instructor")`` at a module whose ``from_litellm()`` raises.++ Simulates instructor being importable but unable to wrap this LiteLLM+ ``completion`` (e.g. an incompatible instructor/litellm version pairing).+ """+ fake_instructor = MagicMock()+ fake_instructor.from_litellm.side_effect = error or RuntimeError("incompatible client")+ real = litellm_module.safe_import+ monkeypatch.setattr(+ litellm_module,+ "safe_import",+ lambda name, *a, **k: (fake_instructor, True)+ if name == "instructor"+ else real(name, *a, **k),+ )+ return fake_instructor+++def test_construction_raises_without_litellm_installed(monkeypatch):+ monkeypatch.setattr(litellm_module, "LITELLM_AVAILABLE", False)+ with pytest.raises(ProcessingError, match="LiteLLM library not installed"):+ LiteLLM(model="openai/gpt-4o")+++def test_construction_stores_model_and_api_key(litellm_available):+ llm = LiteLLM(model="openai/gpt-4o", api_key="fake-key")+ assert llm.model == "openai/gpt-4o"+ assert llm.api_key == "fake-key"+++def test_generate_typed_raises_when_unavailable(litellm_available, monkeypatch):+ llm = LiteLLM(model="openai/gpt-4o")+ monkeypatch.setattr(litellm_module, "LITELLM_AVAILABLE", False)+ with pytest.raises(ProcessingError, match="LiteLLM library not installed"):+ llm.generate_typed("hello", _Point)+++# --- preferred path: instructor is available -------------------------------+++def test_generate_typed_prefers_instructor_when_available(litellm_available, monkeypatch):+ fake_client = MagicMock()+ fake_client.chat.completions.create.return_value = _Point(x=3, y=4)+ _with_instructor(monkeypatch, fake_client)++ llm = LiteLLM(model="openai/gpt-4o", api_key="k")+ llm.generate_structured = MagicMock() # must not be touched on this path++ result = llm.generate_typed("give me a point", _Point, max_retries=2)++ assert result == _Point(x=3, y=4)+ llm.generate_structured.assert_not_called()+ _, kwargs = fake_client.chat.completions.create.call_args+ assert kwargs["response_model"] is _Point+ assert kwargs["max_retries"] == 2+ assert kwargs["model"] == "openai/gpt-4o"+++def test_generate_typed_surfaces_instructor_completion_error(litellm_available, monkeypatch):+ """A failing completion request is raised, not retried through the manual loop."""+ fake_client = MagicMock()+ fake_client.chat.completions.create.side_effect = RuntimeError("401 unauthorized")+ _with_instructor(monkeypatch, fake_client)++ llm = LiteLLM(model="openai/gpt-4o", api_key="k")+ llm.generate_structured = MagicMock()++ with pytest.raises(ProcessingError, match="LiteLLM typed generation failed"):+ llm.generate_typed("give me a point", _Point)++ llm.generate_structured.assert_not_called()+++def test_generate_typed_falls_back_when_instructor_init_fails(litellm_available, monkeypatch):+ """Regression test: instructor is installed but from_litellm() raises - this+ must fall through to the manual loop and still succeed, not propagate the+ init failure or skip generation."""+ fake_instructor = _with_instructor_init_failure(monkeypatch)++ llm = LiteLLM(model="openai/gpt-4o")+ llm.generate_structured = MagicMock(return_value={"x": 5, "y": 6})++ result = llm.generate_typed("give me a point", _Point)++ assert isinstance(result, _Point)+ assert (result.x, result.y) == (5, 6)+ fake_instructor.from_litellm.assert_called_once()+ llm.generate_structured.assert_called_once()+++# --- fallback path: instructor is not available -----------------------------+++def test_generate_typed_validates_structured_output(litellm_available, without_instructor):+ llm = LiteLLM(model="openai/gpt-4o")+ llm.generate_structured = MagicMock(return_value={"x": 1, "y": 2})++ result = llm.generate_typed("give me a point", _Point)++ assert isinstance(result, _Point)+ assert (result.x, result.y) == (1, 2)+ llm.generate_structured.assert_called_once()+++def test_generate_typed_retries_then_raises_on_bad_output(+ litellm_available, without_instructor, monkeypatch+):+ sleep_calls = []+ monkeypatch.setattr(litellm_module.time, "sleep", sleep_calls.append)++ llm = LiteLLM(model="openai/gpt-4o")+ llm.generate_structured = MagicMock(return_value={"x": "not-an-int"})++ with pytest.raises(ProcessingError, match="LiteLLM typed generation failed"):+ llm.generate_typed("give me a point", _Point, max_retries=2)++ assert llm.generate_structured.call_count == 2+ # The retry must feed the validation error back into the prompt.+ first_prompt = llm.generate_structured.call_args_list[0].args[0]+ retry_prompt = llm.generate_structured.call_args_list[1].args[0]+ assert first_prompt == "give me a point"+ assert "did not match the required" in retry_prompt+ # Backs off between attempts (but not after the last one) - mirrors+ # BaseProvider's fallback loop instead of hammering the API immediately.+ assert sleep_calls == [1]diff --git a/tests/test_llm_openai.py b/tests/test_llm_openai.pynew file mode 100644index 00000000..0e59f5a4--- /dev/null+++ b/tests/test_llm_openai.py@@ -0,0 +1,80 @@+"""Tests for the OpenAI LLM provider wrapper (semantica.llms.OpenAI)."""++from unittest.mock import MagicMock++import pytest++from semantica.llms import OpenAI+from semantica.utils.exceptions import ProcessingError+++def test_construction_stores_model_and_api_key():+ llm = OpenAI(model="gpt-4", api_key="fake-key")+ assert llm.model == "gpt-4"+ assert llm.api_key == "fake-key"+++def test_is_available_false_with_no_key(monkeypatch):+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)+ llm = OpenAI(api_key=None)+ assert llm.is_available() is False+++def test_generate_raises_clear_error_when_unavailable(monkeypatch):+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)+ llm = OpenAI(api_key=None)+ with pytest.raises(ProcessingError, match="OpenAI provider not available"):+ llm.generate("hello")+++def test_generate_forwards_to_the_real_provider_when_available():+ llm = OpenAI(api_key="fake-key")+ llm.provider = MagicMock()+ llm.provider.is_available.return_value = True+ llm.provider.generate.return_value = "a fake response"++ result = llm.generate("hello", temperature=0.5)++ assert result == "a fake response"+ llm.provider.generate.assert_called_once_with("hello", temperature=0.5)+++def test_generate_structured_forwards_to_the_real_provider():+ llm = OpenAI(api_key="fake-key")+ llm.provider = MagicMock()+ llm.provider.is_available.return_value = True+ llm.provider.generate_structured.return_value = {"key": "value"}++ result = llm.generate_structured("hello")++ assert result == {"key": "value"}+ llm.provider.generate_structured.assert_called_once_with("hello")+++def test_generate_typed_forwards_schema_and_max_retries():+ llm = OpenAI(api_key="fake-key")+ llm.provider = MagicMock()+ llm.provider.is_available.return_value = True+ fake_schema = object()+ llm.provider.generate_typed.return_value = "typed result"++ result = llm.generate_typed("hello", fake_schema, max_retries=5)++ assert result == "typed result"+ llm.provider.generate_typed.assert_called_once_with(+ "hello", fake_schema, max_retries=5+ )+++def test_generate_structured_raises_clear_error_when_unavailable(monkeypatch):+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)+ llm = OpenAI(api_key=None)+ with pytest.raises(ProcessingError, match="OpenAI provider not available"):+ llm.generate_structured("hello")+++def test_generate_typed_raises_clear_error_when_unavailable(monkeypatch):+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)+ llm = OpenAI(api_key=None)+ with pytest.raises(ProcessingError, match="OpenAI provider not available"):+ llm.generate_typed("hello", object())