Skip to content

Graph-Native Infrastructure for Context and Accountable AI Systems

feat(llms): standardize generate_typed() across all provider wrappers (#1553)

Sakshi Jainc47ffae8ff94
diff --git a/docs/guides/llm-integrations.md b/docs/guides/llm-integrations.md
index 09458d58..8b6fd62d 100644
--- a/docs/guides/llm-integrations.md
+++ b/docs/guides/llm-integrations.md
@@ -3,11 +3,11 @@ title: "LLM Integrations"
description: "Connect Semantica to Groq, OpenAI, Anthropic, HuggingFace, Novita AI, and 100+ LLM providers through a unified interface."
---
-Semantica exposes a unified provider interface — a single `.generate()` method — across Groq, OpenAI, Anthropic Claude, HuggingFace, Novita AI, and 100+ providers via LiteLLM. Use it when you need to swap providers for latency, accuracy, cost, or data-residency reasons without touching application code.
+Semantica exposes a unified provider interface — the same `.generate()`, `.generate_structured()` and `.generate_typed()` methods — across Groq, OpenAI, Anthropic Claude, HuggingFace, Novita AI, and 100+ providers via LiteLLM. Use it when you need to swap providers for latency, accuracy, cost, or data-residency reasons without touching application code.
## What Are LLM Integrations?
-The `semantica.llms` module provides a unified interface for connecting to Large Language Model providers. Instead of learning different APIs for each provider, you use the same methods (`.generate()`, `.generate_structured()`) regardless of whether you're calling Groq, OpenAI, Anthropic, or local HuggingFace models.
+The `semantica.llms` module provides a unified interface for connecting to Large Language Model providers. Instead of learning different APIs for each provider, you use the same methods (`.generate()`, `.generate_structured()`, `.generate_typed()`) regardless of whether you're calling Groq, OpenAI, Anthropic, or local HuggingFace models.
**Unified interface across providers:** All LLM providers in Semantica expose identical methods, so switching from OpenAI to Anthropic requires changing only the provider constructor, not your application code.
@@ -19,7 +19,7 @@ The `semantica.llms` module provides a unified interface for connecting to Large
**Reduced vendor lock-in.** Avoid tying your application to a single LLM provider's API. If pricing changes or service availability issues arise, switching providers is straightforward.
-**Consistent APIs.** Use the same `.generate()` and `.generate_structured()` methods across all providers instead of learning provider-specific interfaces.
+**Consistent APIs.** Use the same `.generate()`, `.generate_structured()`, and `.generate_typed()` methods across all providers instead of learning provider-specific interfaces.
**Multi-provider workflows.** Run fast models for initial classification and expensive frontier models for complex reasoning in the same pipeline.
@@ -70,15 +70,16 @@ The unified interface means you can prototype with Groq for speed, validate accu
## The Shared Interface
-Every provider exposes the same two methods:
+Every provider exposes the same methods:
```python
provider.generate(prompt: str, **kwargs) -> str
-provider.generate_structured(prompt: str, **kwargs) -> dict
+provider.generate_structured(prompt: str, **kwargs) -> dict | list
+provider.generate_typed(prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs) -> BaseModel
provider.is_available() -> bool
```
-`generate()` returns a plain string. `generate_structured()` instructs the model to respond in JSON and returns a parsed `dict`. `is_available()` lets you health-check the provider before committing to a call — useful in retry logic and warm-up checks.
+`generate()` returns a plain string. `generate_structured()` instructs the model to respond in JSON and returns the parsed result — a `dict` for a top-level JSON object, or a `list` if the model returns a top-level JSON array. `generate_typed()` takes a Pydantic model, validates the model's output against it, and retries up to `max_retries` times with the validation error fed back into the prompt — reach for it when downstream code needs a guaranteed shape rather than best-effort JSON. `is_available()` lets you health-check the provider before committing to a call — useful in retry logic and warm-up checks.
This means every place in Semantica that accepts an LLM — `query_with_reasoning()`, semantic extraction, custom reasoning loops — accepts any of these providers interchangeably.
@@ -711,7 +712,7 @@ for src in best["sources"]:
**Using LLMs for deterministic pattern matching that regex can handle.** If your task is extracting email addresses, phone numbers, or other pattern-based entities, regular expressions are faster, cheaper, and more reliable than LLM extraction. Use LLMs when context, ambiguity, or domain knowledge matter for correct interpretation.
-**Not validating structured outputs.** The `generate_structured()` method returns parsed JSON, but LLMs can still produce malformed or incomplete structures. Always validate the returned dictionary against your expected schema before using the data downstream.
+**Not validating structured outputs.** The `generate_structured()` method returns parsed JSON (a dict, or a list for a top-level array), but LLMs can still produce malformed or incomplete structures. Validate the result against your expected schema before using it downstream — or use `generate_typed()`, which validates against a Pydantic model for you.
**Switching providers without testing prompt behavior.** Different models respond differently to the same prompt. A prompt optimized for GPT-4 may produce poor results with Llama or Claude. When switching providers, test your prompts and adjust temperature, instructions, or examples as needed.
diff --git a/semantica/llms/__init__.py b/semantica/llms/__init__.py
index 943a88e6..b31511ff 100644
--- a/semantica/llms/__init__.py
+++ b/semantica/llms/__init__.py
@@ -5,6 +5,12 @@ This module provides clean, intuitive imports for LLM providers used in Semantic
It wraps the underlying provider functionality from semantica.semantic_extract.providers
to provide a cleaner API.
+Every provider wrapper exposes the same generation interface:
+ - generate(prompt, **kwargs) -> str
+ - generate_structured(prompt, **kwargs) -> dict | list
+ - generate_typed(prompt, schema, max_retries=3, **kwargs) -> BaseModel
+ - is_available() -> bool
+
Supported Providers:
- Groq: Groq API for fast inference
- OpenAI: OpenAI API (GPT-3.5, GPT-4, etc.)
diff --git a/semantica/llms/anthropic.py b/semantica/llms/anthropic.py
index 573d761b..54cc330b 100644
--- a/semantica/llms/anthropic.py
+++ b/semantica/llms/anthropic.py
@@ -4,7 +4,9 @@ Anthropic LLM Provider
Wrapper for Anthropic Claude API provider with clean interface
"""
-from typing import Any, Dict, List, Optional, Union
+from typing import Any, Dict, List, Optional, Type, Union
+
+from pydantic import BaseModel
from ..semantic_extract.providers import AnthropicProvider
from ..utils.exceptions import ProcessingError
@@ -88,7 +90,9 @@ class Anthropic:
)
return self.provider.generate_structured(prompt, **kwargs)
- def generate_typed(self, prompt: str, schema: Any, max_retries: int = 3, **kwargs) -> Any:
+ def generate_typed(
+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs
+ ) -> BaseModel:
"""
Generate output validated against a Pydantic schema.
diff --git a/semantica/llms/deepseek.py b/semantica/llms/deepseek.py
index 545f764c..89faa18a 100644
--- a/semantica/llms/deepseek.py
+++ b/semantica/llms/deepseek.py
@@ -4,7 +4,9 @@ DeepSeek LLM Provider
Wrapper for DeepSeek API provider with clean interface.
"""
-from typing import Any, Dict, List, Optional, Union
+from typing import Any, Dict, List, Optional, Type, Union
+
+from pydantic import BaseModel
from ..semantic_extract.providers import DeepSeekProvider
from ..utils.exceptions import ProcessingError
@@ -88,7 +90,9 @@ class DeepSeek:
)
return self.provider.generate_structured(prompt, **kwargs)
- def generate_typed(self, prompt: str, schema: Any, max_retries: int = 3, **kwargs) -> Any:
+ def generate_typed(
+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs
+ ) -> BaseModel:
"""
Generate output validated against a Pydantic schema.
diff --git a/semantica/llms/gemini.py b/semantica/llms/gemini.py
index a8bfa163..83bf9df7 100644
--- a/semantica/llms/gemini.py
+++ b/semantica/llms/gemini.py
@@ -4,7 +4,9 @@ Gemini LLM Provider
Wrapper for Google Gemini API provider with clean interface.
"""
-from typing import Any, Dict, List, Optional, Union
+from typing import Any, Dict, List, Optional, Type, Union
+
+from pydantic import BaseModel
from ..semantic_extract.providers import GeminiProvider
from ..utils.exceptions import ProcessingError
@@ -88,7 +90,9 @@ class Gemini:
)
return self.provider.generate_structured(prompt, **kwargs)
- def generate_typed(self, prompt: str, schema: Any, max_retries: int = 3, **kwargs) -> Any:
+ def generate_typed(
+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs
+ ) -> BaseModel:
"""
Generate output validated against a Pydantic schema.
diff --git a/semantica/llms/groq.py b/semantica/llms/groq.py
index e3cbf43e..20fac876 100644
--- a/semantica/llms/groq.py
+++ b/semantica/llms/groq.py
@@ -4,7 +4,9 @@ Groq LLM Provider
Wrapper for Groq API provider with clean interface.
"""
-from typing import Any, Dict, Optional
+from typing import Any, Dict, List, Optional, Type, Union
+
+from pydantic import BaseModel
from ..semantic_extract.providers import GroqProvider
from ..utils.exceptions import ProcessingError
@@ -67,7 +69,7 @@ class Groq:
)
return self.provider.generate(prompt, **kwargs)
- def generate_structured(self, prompt: str, **kwargs) -> Dict[str, Any]:
+ def generate_structured(self, prompt: str, **kwargs) -> Union[Dict[str, Any], List[Any]]:
"""
Generate structured JSON output.
@@ -76,7 +78,8 @@ class Groq:
**kwargs: Generation options
Returns:
- Parsed JSON response as dictionary
+ Parsed JSON response. A dict for a top-level JSON object, or a
+ list if the model returns a top-level JSON array.
Raises:
ProcessingError: If provider is not available or parsing fails
@@ -87,3 +90,27 @@ class Groq:
)
return self.provider.generate_structured(prompt, **kwargs)
+ def generate_typed(
+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs
+ ) -> BaseModel:
+ """
+ Generate output validated against a Pydantic schema.
+
+ Args:
+ prompt: Input prompt text
+ schema: Pydantic model class to validate the output against
+ max_retries: Number of retries if validation fails (default: 3)
+ **kwargs: Generation options
+
+ Returns:
+ An instance of `schema`, populated from the model's response
+
+ Raises:
+ ProcessingError: If provider is not available or generation fails
+ """
+ if not self.is_available():
+ raise ProcessingError(
+ "Groq provider not available. Set GROQ_API_KEY or pass api_key."
+ )
+ return self.provider.generate_typed(prompt, schema, max_retries=max_retries, **kwargs)
+
diff --git a/semantica/llms/huggingface.py b/semantica/llms/huggingface.py
index bd5326ce..c7f162e0 100644
--- a/semantica/llms/huggingface.py
+++ b/semantica/llms/huggingface.py
@@ -4,7 +4,9 @@ HuggingFace LLM Provider
Wrapper for HuggingFace Transformers LLM provider with clean interface.
"""
-from typing import Any, Dict, Optional
+from typing import Any, Dict, List, Optional, Type, Union
+
+from pydantic import BaseModel
from ..semantic_extract.providers import HuggingFaceLLMProvider
from ..utils.exceptions import ProcessingError
@@ -78,7 +80,7 @@ class HuggingFaceLLM:
)
return self.provider.generate(prompt, **kwargs)
- def generate_structured(self, prompt: str, **kwargs) -> Dict[str, Any]:
+ def generate_structured(self, prompt: str, **kwargs) -> Union[Dict[str, Any], List[Any]]:
"""
Generate structured JSON output.
@@ -87,7 +89,8 @@ class HuggingFaceLLM:
**kwargs: Generation options
Returns:
- Parsed JSON response as dictionary
+ Parsed JSON response. A dict for a top-level JSON object, or a
+ list if the model returns a top-level JSON array.
Raises:
ProcessingError: If provider is not available or parsing fails
@@ -98,3 +101,27 @@ class HuggingFaceLLM:
)
return self.provider.generate_structured(prompt, **kwargs)
+ def generate_typed(
+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs
+ ) -> BaseModel:
+ """
+ Generate output validated against a Pydantic schema.
+
+ Args:
+ prompt: Input prompt text
+ schema: Pydantic model class to validate the output against
+ max_retries: Number of retries if validation fails (default: 3)
+ **kwargs: Generation options
+
+ Returns:
+ An instance of `schema`, populated from the model's response
+
+ Raises:
+ ProcessingError: If provider is not available or generation fails
+ """
+ if not self.is_available():
+ raise ProcessingError(
+ "HuggingFace LLM provider not available. Install transformers library."
+ )
+ return self.provider.generate_typed(prompt, schema, max_retries=max_retries, **kwargs)
+
diff --git a/semantica/llms/litellm.py b/semantica/llms/litellm.py
index 770db273..8c75e2e9 100644
--- a/semantica/llms/litellm.py
+++ b/semantica/llms/litellm.py
@@ -5,7 +5,10 @@ Wrapper for LiteLLM library that provides unified access to 100+ LLM providers.
Supports OpenAI, Anthropic, Groq, Azure, Bedrock, Vertex AI, and many more.
"""
-from typing import Any, Dict, List, Optional
+import time
+from typing import Any, Dict, List, Optional, Type, Union
+
+from pydantic import BaseModel
from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger
@@ -123,7 +126,7 @@ class LiteLLM:
logger.error(f"LiteLLM generation failed: {e}")
raise ProcessingError(f"LiteLLM generation failed: {e}")
- def generate_structured(self, prompt: str, **kwargs) -> Dict[str, Any]:
+ def generate_structured(self, prompt: str, **kwargs) -> Union[Dict[str, Any], List[Any]]:
"""
Generate structured JSON output.
@@ -132,7 +135,8 @@ class LiteLLM:
**kwargs: Generation options
Returns:
- Parsed JSON response as dictionary
+ Parsed JSON response. A dict for a top-level JSON object, or a
+ list if the model returns a top-level JSON array.
Raises:
ProcessingError: If provider is not available or parsing fails
@@ -189,3 +193,84 @@ class LiteLLM:
logger.error(f"LiteLLM structured generation failed: {e}")
raise ProcessingError(f"LiteLLM structured generation failed: {e}")
+ def generate_typed(
+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs
+ ) -> BaseModel:
+ """
+ Generate output validated against a Pydantic schema.
+
+ Uses the ``instructor`` library when it is installed and can be
+ initialised (schema and retries handled natively). Otherwise - instructor
+ missing, or its setup failing - falls back to a JSON-generation plus
+ Pydantic-validation retry loop on top of ``generate_structured()``. A
+ failure of the instructor completion request itself is raised, not
+ retried through the fallback.
+
+ Args:
+ prompt: Input prompt text
+ schema: Pydantic model class to validate the output against
+ max_retries: Number of retries if validation fails (default: 3)
+ **kwargs: Generation options
+
+ Returns:
+ An instance of `schema`, populated from the model's response
+
+ Raises:
+ ProcessingError: If provider is not available or generation fails
+ """
+ if not self.is_available():
+ raise ProcessingError(
+ "LiteLLM library not installed. Install with: pip install litellm"
+ )
+
+ options = {**self.config, **kwargs}
+
+ # Preferred path: instructor drives the schema and retries natively.
+ # Only an instructor setup failure falls through to the manual loop - a
+ # failure of the completion request itself (auth, transport, exhausted
+ # retries) is surfaced directly rather than retried a second way.
+ instructor_mod, instructor_available = safe_import("instructor")
+ if instructor_available:
+ client = None
+ try:
+ client = instructor_mod.from_litellm(completion)
+ except Exception as e:
+ logger.warning(
+ f"instructor is installed but could not be initialised for "
+ f"LiteLLM ({e}); falling back to manual validation loop."
+ )
+ if client is not None:
+ try:
+ return client.chat.completions.create(
+ model=self.model,
+ messages=[{"role": "user", "content": prompt}],
+ api_key=self.api_key,
+ response_model=schema,
+ max_retries=max_retries,
+ **options,
+ )
+ except Exception as e:
+ raise ProcessingError(
+ f"LiteLLM typed generation failed: {e}"
+ ) from e
+
+ # Fallback: generate JSON, validate against the schema, retry with feedback.
+ last_error = None
+ current_prompt = prompt
+ for _attempt in range(max_retries):
+ try:
+ data = self.generate_structured(current_prompt, **kwargs)
+ return schema.model_validate(data)
+ except Exception as e:
+ last_error = e
+ current_prompt = (
+ f"{prompt}\n\nThe previous response did not match the required "
+ f"schema:\n{e}\n\nReturn valid JSON that matches the schema."
+ )
+ if _attempt < max_retries - 1:
+ time.sleep(1)
+
+ raise ProcessingError(
+ f"LiteLLM typed generation failed after {max_retries} attempts: {last_error}"
+ )
+
diff --git a/semantica/llms/novita.py b/semantica/llms/novita.py
index ddada85e..d99ee789 100644
--- a/semantica/llms/novita.py
+++ b/semantica/llms/novita.py
@@ -4,7 +4,9 @@ Novita LLM Provider
Wrapper for Novita AI's OpenAI-compatible API with clean interface.
"""
-from typing import Any, Dict, List, Optional, Union
+from typing import Any, Dict, List, Optional, Type, Union
+
+from pydantic import BaseModel
from ..semantic_extract.providers import NovitaProvider
from ..utils.exceptions import ProcessingError
@@ -88,7 +90,9 @@ class Novita:
)
return self.provider.generate_structured(prompt, **kwargs)
- def generate_typed(self, prompt: str, schema: Any, max_retries: int = 3, **kwargs) -> Any:
+ def generate_typed(
+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs
+ ) -> BaseModel:
"""
Generate output validated against a Pydantic schema.
diff --git a/semantica/llms/ollama.py b/semantica/llms/ollama.py
index bc7f4aae..c6fb5788 100644
--- a/semantica/llms/ollama.py
+++ b/semantica/llms/ollama.py
@@ -4,7 +4,9 @@ Ollama LLM Provider
Wrapper for local Ollama models with clean interface.
"""
-from typing import Any, Dict, List, Union
+from typing import Any, Dict, List, Type, Union
+
+from pydantic import BaseModel
from ..semantic_extract.providers import OllamaProvider
from ..utils.exceptions import ProcessingError
@@ -92,7 +94,9 @@ class Ollama:
)
return self.provider.generate_structured(prompt, **kwargs)
- def generate_typed(self, prompt: str, schema: Any, max_retries: int = 3, **kwargs) -> Any:
+ def generate_typed(
+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs
+ ) -> BaseModel:
"""
Generate output validated against a Pydantic schema.
diff --git a/semantica/llms/openai.py b/semantica/llms/openai.py
index 15a3a8df..b8615d87 100644
--- a/semantica/llms/openai.py
+++ b/semantica/llms/openai.py
@@ -4,7 +4,9 @@ OpenAI LLM Provider
Wrapper for OpenAI API provider with clean interface.
"""
-from typing import Any, Dict, Optional
+from typing import Any, Dict, List, Optional, Type, Union
+
+from pydantic import BaseModel
from ..semantic_extract.providers import OpenAIProvider
from ..utils.exceptions import ProcessingError
@@ -67,7 +69,7 @@ class OpenAI:
)
return self.provider.generate(prompt, **kwargs)
- def generate_structured(self, prompt: str, **kwargs) -> Dict[str, Any]:
+ def generate_structured(self, prompt: str, **kwargs) -> Union[Dict[str, Any], List[Any]]:
"""
Generate structured JSON output.
@@ -76,7 +78,8 @@ class OpenAI:
**kwargs: Generation options
Returns:
- Parsed JSON response as dictionary
+ Parsed JSON response. A dict for a top-level JSON object, or a
+ list if the model returns a top-level JSON array.
Raises:
ProcessingError: If provider is not available or parsing fails
@@ -87,3 +90,27 @@ class OpenAI:
)
return self.provider.generate_structured(prompt, **kwargs)
+ def generate_typed(
+ self, prompt: str, schema: Type[BaseModel], max_retries: int = 3, **kwargs
+ ) -> BaseModel:
+ """
+ Generate output validated against a Pydantic schema.
+
+ Args:
+ prompt: Input prompt text
+ schema: Pydantic model class to validate the output against
+ max_retries: Number of retries if validation fails (default: 3)
+ **kwargs: Generation options
+
+ Returns:
+ An instance of `schema`, populated from the model's response
+
+ Raises:
+ ProcessingError: If provider is not available or generation fails
+ """
+ if not self.is_available():
+ raise ProcessingError(
+ "OpenAI provider not available. Set OPENAI_API_KEY or pass api_key."
+ )
+ return self.provider.generate_typed(prompt, schema, max_retries=max_retries, **kwargs)
+
diff --git a/tests/test_llm_groq.py b/tests/test_llm_groq.py
new file mode 100644
index 00000000..5245b376
--- /dev/null
+++ b/tests/test_llm_groq.py
@@ -0,0 +1,82 @@
+"""Tests for the Groq LLM provider wrapper (semantica.llms.Groq)."""
+
+from unittest.mock import MagicMock
+
+import pytest
+
+from semantica.llms import Groq
+from semantica.utils.exceptions import ProcessingError
+
+
+def test_construction_stores_model_and_api_key():
+ llm = Groq(model="llama-3.1-8b-instant", api_key="fake-key")
+ assert llm.model == "llama-3.1-8b-instant"
+ assert llm.api_key == "fake-key"
+
+
+def test_is_available_false_with_no_key(monkeypatch):
+ monkeypatch.delenv("GROQ_API_KEY", raising=False)
+ llm = Groq(api_key=None)
+ assert llm.is_available() is False
+
+
+def test_generate_raises_clear_error_when_unavailable(monkeypatch):
+ monkeypatch.delenv("GROQ_API_KEY", raising=False)
+ llm = Groq(api_key=None)
+ with pytest.raises(ProcessingError, match="Groq provider not available"):
+ llm.generate("hello")
+
+
+def test_generate_forwards_to_the_real_provider_when_available():
+ llm = Groq(api_key="fake-key")
+ llm.provider = MagicMock()
+ llm.provider.is_available.return_value = True
+ llm.provider.generate.return_value = "a fake response"
+
+ result = llm.generate("hello", temperature=0.5)
+
+ assert result == "a fake response"
+ llm.provider.generate.assert_called_once_with("hello", temperature=0.5)
+
+
+def test_generate_structured_forwards_to_the_real_provider():
+ llm = Groq(api_key="fake-key")
+ llm.provider = MagicMock()
+ llm.provider.is_available.return_value = True
+ llm.provider.generate_structured.return_value = {"key": "value"}
+
+ result = llm.generate_structured("hello")
+
+ assert result == {"key": "value"}
+ llm.provider.generate_structured.assert_called_once_with("hello")
+
+
+def test_generate_typed_forwards_schema_max_retries_and_extra_kwargs():
+ """Also verifies the **kwargs contract: anything beyond schema/max_retries
+ (e.g. temperature) must reach the underlying provider unchanged."""
+ llm = Groq(api_key="fake-key")
+ llm.provider = MagicMock()
+ llm.provider.is_available.return_value = True
+ fake_schema = object()
+ llm.provider.generate_typed.return_value = "typed result"
+
+ result = llm.generate_typed("hello", fake_schema, max_retries=5, temperature=0.5)
+
+ assert result == "typed result"
+ llm.provider.generate_typed.assert_called_once_with(
+ "hello", fake_schema, max_retries=5, temperature=0.5
+ )
+
+
+def test_generate_structured_raises_clear_error_when_unavailable(monkeypatch):
+ monkeypatch.delenv("GROQ_API_KEY", raising=False)
+ llm = Groq(api_key=None)
+ with pytest.raises(ProcessingError, match="Groq provider not available"):
+ llm.generate_structured("hello")
+
+
+def test_generate_typed_raises_clear_error_when_unavailable(monkeypatch):
+ monkeypatch.delenv("GROQ_API_KEY", raising=False)
+ llm = Groq(api_key=None)
+ with pytest.raises(ProcessingError, match="Groq provider not available"):
+ llm.generate_typed("hello", object())
diff --git a/tests/test_llm_huggingface.py b/tests/test_llm_huggingface.py
new file mode 100644
index 00000000..24bfd562
--- /dev/null
+++ b/tests/test_llm_huggingface.py
@@ -0,0 +1,79 @@
+"""Tests for the HuggingFace LLM provider wrapper (semantica.llms.HuggingFaceLLM).
+
+The wrapper builds a real ``HuggingFaceLLMProvider`` in ``__init__``, which would
+load a model from disk/network. These tests patch that class so we only exercise
+the wrapper's delegation and error handling.
+"""
+
+from unittest.mock import patch
+
+import pytest
+
+from semantica.utils.exceptions import ProcessingError
+
+PROVIDER_PATH = "semantica.llms.huggingface.HuggingFaceLLMProvider"
+
+
+def _make_wrapper(available=True):
+ from semantica.llms import HuggingFaceLLM
+
+ with patch(PROVIDER_PATH) as provider_cls:
+ provider_cls.return_value.is_available.return_value = available
+ return HuggingFaceLLM(model_name="gpt2")
+
+
+def test_construction_stores_model_name():
+ hf = _make_wrapper()
+ assert hf.model_name == "gpt2"
+ assert hf.model == "gpt2" # alias kept for cross-provider consistency
+
+
+def test_generate_raises_clear_error_when_unavailable():
+ hf = _make_wrapper(available=False)
+ with pytest.raises(ProcessingError, match="HuggingFace LLM provider not available"):
+ hf.generate("hello")
+
+
+def test_generate_forwards_to_the_real_provider_when_available():
+ hf = _make_wrapper()
+ hf.provider.generate.return_value = "a fake response"
+
+ result = hf.generate("hello", max_new_tokens=10)
+
+ assert result == "a fake response"
+ hf.provider.generate.assert_called_once_with("hello", max_new_tokens=10)
+
+
+def test_generate_structured_forwards_to_the_real_provider():
+ hf = _make_wrapper()
+ hf.provider.generate_structured.return_value = {"key": "value"}
+
+ result = hf.generate_structured("hello")
+
+ assert result == {"key": "value"}
+ hf.provider.generate_structured.assert_called_once_with("hello")
+
+
+def test_generate_typed_forwards_schema_and_max_retries():
+ hf = _make_wrapper()
+ fake_schema = object()
+ hf.provider.generate_typed.return_value = "typed result"
+
+ result = hf.generate_typed("hello", fake_schema, max_retries=5)
+
+ assert result == "typed result"
+ hf.provider.generate_typed.assert_called_once_with(
+ "hello", fake_schema, max_retries=5
+ )
+
+
+def test_generate_structured_raises_clear_error_when_unavailable():
+ hf = _make_wrapper(available=False)
+ with pytest.raises(ProcessingError, match="HuggingFace LLM provider not available"):
+ hf.generate_structured("hello")
+
+
+def test_generate_typed_raises_clear_error_when_unavailable():
+ hf = _make_wrapper(available=False)
+ with pytest.raises(ProcessingError, match="HuggingFace LLM provider not available"):
+ hf.generate_typed("hello", object())
diff --git a/tests/test_llm_litellm.py b/tests/test_llm_litellm.py
new file mode 100644
index 00000000..5c7294c8
--- /dev/null
+++ b/tests/test_llm_litellm.py
@@ -0,0 +1,190 @@
+"""Tests for the LiteLLM provider wrapper (semantica.llms.LiteLLM).
+
+Unlike the other wrappers, LiteLLM does not sit on a ``BaseProvider`` - it calls
+``litellm.completion`` directly - so ``generate_typed()`` has its own
+implementation: it uses ``instructor`` when that package is importable, and
+otherwise falls back to a validate-and-retry loop on top of
+``generate_structured()``. Which path runs depends only on whether ``instructor``
+imports, so every test here pins that explicitly via ``safe_import``.
+"""
+
+from unittest.mock import MagicMock
+
+import pytest
+from pydantic import BaseModel
+
+import semantica.llms.litellm as litellm_module
+from semantica.llms.litellm import LiteLLM
+from semantica.utils.exceptions import ProcessingError
+
+
+class _Point(BaseModel):
+ x: int
+ y: int
+
+
+def litellm_available(monkeypatch):
+ """Pretend the litellm library is installed."""
+ monkeypatch.setattr(litellm_module, "LITELLM_AVAILABLE", True)
+ monkeypatch.setattr(litellm_module, "completion", MagicMock(), raising=False)
+
+
+def without_instructor(monkeypatch):
+ """Force ``safe_import("instructor")`` to report unavailable.
+
+ Without this the selected code path would depend on whether ``instructor``
+ happens to be installed in the test environment (it is part of the repo's
+ all-extras set), so the manual-fallback tests would not reliably exercise
+ the fallback.
+ """
+ real = litellm_module.safe_import
+ monkeypatch.setattr(
+ litellm_module,
+ "safe_import",
+ lambda name, *a, **k: (None, False) if name == "instructor" else real(name, *a, **k),
+ )
+
+
+def _with_instructor(monkeypatch, fake_client):
+ """Point ``safe_import("instructor")`` at a fake instructor module."""
+ fake_instructor = MagicMock()
+ fake_instructor.from_litellm.return_value = fake_client
+ real = litellm_module.safe_import
+ monkeypatch.setattr(
+ litellm_module,
+ "safe_import",
+ lambda name, *a, **k: (fake_instructor, True)
+ if name == "instructor"
+ else real(name, *a, **k),
+ )
+ return fake_instructor
+
+
+def _with_instructor_init_failure(monkeypatch, error=None):
+ """Point ``safe_import("instructor")`` at a module whose ``from_litellm()`` raises.
+
+ Simulates instructor being importable but unable to wrap this LiteLLM
+ ``completion`` (e.g. an incompatible instructor/litellm version pairing).
+ """
+ fake_instructor = MagicMock()
+ fake_instructor.from_litellm.side_effect = error or RuntimeError("incompatible client")
+ real = litellm_module.safe_import
+ monkeypatch.setattr(
+ litellm_module,
+ "safe_import",
+ lambda name, *a, **k: (fake_instructor, True)
+ if name == "instructor"
+ else real(name, *a, **k),
+ )
+ return fake_instructor
+
+
+def test_construction_raises_without_litellm_installed(monkeypatch):
+ monkeypatch.setattr(litellm_module, "LITELLM_AVAILABLE", False)
+ with pytest.raises(ProcessingError, match="LiteLLM library not installed"):
+ LiteLLM(model="openai/gpt-4o")
+
+
+def test_construction_stores_model_and_api_key(litellm_available):
+ llm = LiteLLM(model="openai/gpt-4o", api_key="fake-key")
+ assert llm.model == "openai/gpt-4o"
+ assert llm.api_key == "fake-key"
+
+
+def test_generate_typed_raises_when_unavailable(litellm_available, monkeypatch):
+ llm = LiteLLM(model="openai/gpt-4o")
+ monkeypatch.setattr(litellm_module, "LITELLM_AVAILABLE", False)
+ with pytest.raises(ProcessingError, match="LiteLLM library not installed"):
+ llm.generate_typed("hello", _Point)
+
+
+# --- preferred path: instructor is available -------------------------------
+
+
+def test_generate_typed_prefers_instructor_when_available(litellm_available, monkeypatch):
+ fake_client = MagicMock()
+ fake_client.chat.completions.create.return_value = _Point(x=3, y=4)
+ _with_instructor(monkeypatch, fake_client)
+
+ llm = LiteLLM(model="openai/gpt-4o", api_key="k")
+ llm.generate_structured = MagicMock() # must not be touched on this path
+
+ result = llm.generate_typed("give me a point", _Point, max_retries=2)
+
+ assert result == _Point(x=3, y=4)
+ llm.generate_structured.assert_not_called()
+ _, kwargs = fake_client.chat.completions.create.call_args
+ assert kwargs["response_model"] is _Point
+ assert kwargs["max_retries"] == 2
+ assert kwargs["model"] == "openai/gpt-4o"
+
+
+def test_generate_typed_surfaces_instructor_completion_error(litellm_available, monkeypatch):
+ """A failing completion request is raised, not retried through the manual loop."""
+ fake_client = MagicMock()
+ fake_client.chat.completions.create.side_effect = RuntimeError("401 unauthorized")
+ _with_instructor(monkeypatch, fake_client)
+
+ llm = LiteLLM(model="openai/gpt-4o", api_key="k")
+ llm.generate_structured = MagicMock()
+
+ with pytest.raises(ProcessingError, match="LiteLLM typed generation failed"):
+ llm.generate_typed("give me a point", _Point)
+
+ llm.generate_structured.assert_not_called()
+
+
+def test_generate_typed_falls_back_when_instructor_init_fails(litellm_available, monkeypatch):
+ """Regression test: instructor is installed but from_litellm() raises - this
+ must fall through to the manual loop and still succeed, not propagate the
+ init failure or skip generation."""
+ fake_instructor = _with_instructor_init_failure(monkeypatch)
+
+ llm = LiteLLM(model="openai/gpt-4o")
+ llm.generate_structured = MagicMock(return_value={"x": 5, "y": 6})
+
+ result = llm.generate_typed("give me a point", _Point)
+
+ assert isinstance(result, _Point)
+ assert (result.x, result.y) == (5, 6)
+ fake_instructor.from_litellm.assert_called_once()
+ llm.generate_structured.assert_called_once()
+
+
+# --- fallback path: instructor is not available -----------------------------
+
+
+def test_generate_typed_validates_structured_output(litellm_available, without_instructor):
+ llm = LiteLLM(model="openai/gpt-4o")
+ llm.generate_structured = MagicMock(return_value={"x": 1, "y": 2})
+
+ result = llm.generate_typed("give me a point", _Point)
+
+ assert isinstance(result, _Point)
+ assert (result.x, result.y) == (1, 2)
+ llm.generate_structured.assert_called_once()
+
+
+def test_generate_typed_retries_then_raises_on_bad_output(
+ litellm_available, without_instructor, monkeypatch
+):
+ sleep_calls = []
+ monkeypatch.setattr(litellm_module.time, "sleep", sleep_calls.append)
+
+ llm = LiteLLM(model="openai/gpt-4o")
+ llm.generate_structured = MagicMock(return_value={"x": "not-an-int"})
+
+ with pytest.raises(ProcessingError, match="LiteLLM typed generation failed"):
+ llm.generate_typed("give me a point", _Point, max_retries=2)
+
+ assert llm.generate_structured.call_count == 2
+ # The retry must feed the validation error back into the prompt.
+ first_prompt = llm.generate_structured.call_args_list[0].args[0]
+ retry_prompt = llm.generate_structured.call_args_list[1].args[0]
+ assert first_prompt == "give me a point"
+ assert "did not match the required" in retry_prompt
+ # Backs off between attempts (but not after the last one) - mirrors
+ # BaseProvider's fallback loop instead of hammering the API immediately.
+ assert sleep_calls == [1]
diff --git a/tests/test_llm_openai.py b/tests/test_llm_openai.py
new file mode 100644
index 00000000..0e59f5a4
--- /dev/null
+++ b/tests/test_llm_openai.py
@@ -0,0 +1,80 @@
+"""Tests for the OpenAI LLM provider wrapper (semantica.llms.OpenAI)."""
+
+from unittest.mock import MagicMock
+
+import pytest
+
+from semantica.llms import OpenAI
+from semantica.utils.exceptions import ProcessingError
+
+
+def test_construction_stores_model_and_api_key():
+ llm = OpenAI(model="gpt-4", api_key="fake-key")
+ assert llm.model == "gpt-4"
+ assert llm.api_key == "fake-key"
+
+
+def test_is_available_false_with_no_key(monkeypatch):
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+ llm = OpenAI(api_key=None)
+ assert llm.is_available() is False
+
+
+def test_generate_raises_clear_error_when_unavailable(monkeypatch):
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+ llm = OpenAI(api_key=None)
+ with pytest.raises(ProcessingError, match="OpenAI provider not available"):
+ llm.generate("hello")
+
+
+def test_generate_forwards_to_the_real_provider_when_available():
+ llm = OpenAI(api_key="fake-key")
+ llm.provider = MagicMock()
+ llm.provider.is_available.return_value = True
+ llm.provider.generate.return_value = "a fake response"
+
+ result = llm.generate("hello", temperature=0.5)
+
+ assert result == "a fake response"
+ llm.provider.generate.assert_called_once_with("hello", temperature=0.5)
+
+
+def test_generate_structured_forwards_to_the_real_provider():
+ llm = OpenAI(api_key="fake-key")
+ llm.provider = MagicMock()
+ llm.provider.is_available.return_value = True
+ llm.provider.generate_structured.return_value = {"key": "value"}
+
+ result = llm.generate_structured("hello")
+
+ assert result == {"key": "value"}
+ llm.provider.generate_structured.assert_called_once_with("hello")
+
+
+def test_generate_typed_forwards_schema_and_max_retries():
+ llm = OpenAI(api_key="fake-key")
+ llm.provider = MagicMock()
+ llm.provider.is_available.return_value = True
+ fake_schema = object()
+ llm.provider.generate_typed.return_value = "typed result"
+
+ result = llm.generate_typed("hello", fake_schema, max_retries=5)
+
+ assert result == "typed result"
+ llm.provider.generate_typed.assert_called_once_with(
+ "hello", fake_schema, max_retries=5
+ )
+
+
+def test_generate_structured_raises_clear_error_when_unavailable(monkeypatch):
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+ llm = OpenAI(api_key=None)
+ with pytest.raises(ProcessingError, match="OpenAI provider not available"):
+ llm.generate_structured("hello")
+
+
+def test_generate_typed_raises_clear_error_when_unavailable(monkeypatch):
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+ llm = OpenAI(api_key=None)
+ with pytest.raises(ProcessingError, match="OpenAI provider not available"):
+ llm.generate_typed("hello", object())