Skip to content

Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.

[Feat]: Refactor mem0 OpenTelemetry Instrumentation (#815)

Patcher149239e70a84
diff --git a/contributors/framework-instrumentation-guide.md b/contributors/framework-instrumentation-guide.md
index 82ac1684..6463b5bd 100644
--- a/contributors/framework-instrumentation-guide.md
+++ b/contributors/framework-instrumentation-guide.md
@@ -23,12 +23,24 @@ Before starting, ensure you have:
- Access to competitor repositories for analysis
- Understanding of OpenTelemetry concepts
-**CRITICAL**: Always use local OpenLIT source code for testing:
+**CRITICAL**: Always use the venv in the root directory and test with test.py:
```bash
-# DON'T install OpenLIT - use local source
-PYTHONPATH=sdk/python/src python your_test_script.py
+# ALWAYS use the venv in project root
+source venv/bin/activate
+
+# ALWAYS name your test file test.py (mandatory naming convention)
+python test.py
+
+# Alternative approach with explicit sys.path (for test.py):
+sys.path.insert(0, "sdk/python/src")
```
+**MANDATORY Testing Standards:**
+- Test file must be named `test.py` (not test_framework.py or similar)
+- Always use `source venv/bin/activate` first
+- Place test.py in project root directory
+- Use sys.path.insert(0, "sdk/python/src") in test.py
+
## Phase 1: Competitive Analysis
### Step 1.1: Clone and Research Competitor Implementations
@@ -58,7 +70,8 @@ cd openllmetry/packages/openllmetry-instrumentation-{framework}
**Deep Analysis Checklist:**
- [ ] **Integration Pattern**: Function wrapping vs native integration (like TracingProcessor)
- [ ] **Span Structure**: How many spans do they create and why?
-- [ ] **Span Hierarchy**: Do they maintain proper parent-child relationships?
+- [ ] **🚨 CRITICAL: Span Hierarchy**: Do they maintain proper parent-child relationships? (MOST IMPORTANT)
+- [ ] **Threading Context**: How do they handle ThreadPoolExecutor and async contexts?
- [ ] **Span Naming**: What naming convention do they use?
- [ ] **Attributes**: What attributes do they capture? Check against semantic conventions
- [ ] **Content Capture**: Do they capture input/output content with MIME types?
@@ -67,6 +80,8 @@ cd openllmetry/packages/openllmetry-instrumentation-{framework}
- [ ] **Performance**: How much overhead do they add?
- [ ] **Coverage**: Which framework operations do they instrument?
+**⚠️ SPAN HIERARCHY IS CRITICAL**: This is the #1 issue that breaks observability. Frameworks often use ThreadPoolExecutor or async operations that break OpenTelemetry context propagation, resulting in "all root spans" instead of proper parent-child relationships.
+
### Step 1.2: Document Competitive Gaps and OpenLIT Advantages
Create detailed comparison:
@@ -106,35 +121,103 @@ framework_name/
β”œβ”€β”€ __init__.py # Instrumentation setup
└── callback_handler.py # Framework's callback interface
-# PATTERN 4: Standard Wrapper Pattern (CrewAI, Pydantic AI)
-# Example: Most agent frameworks
+# PATTERN 4: RECOMMENDED 4-File Structure (Mem0, CrewAI, Pydantic AI)
+# Example: Complex frameworks needing performance optimization
framework_name/
-β”œβ”€β”€ __init__.py # Instrumentation setup
-β”œβ”€β”€ framework_name.py # Sync wrappers
-β”œβ”€β”€ async_framework_name.py # Async wrappers
-└── utils.py # Shared utilities with context caching
+β”œβ”€β”€ __init__.py # Instrumentation setup with separated SYNC/ASYNC methods
+β”œβ”€β”€ framework_name.py # Sync wrappers with threading context fixes
+β”œβ”€β”€ async_framework_name.py # Async wrappers
+└── utils.py # Shared utilities with context caching and __slots__
+
+**🎯 PREFERRED STRUCTURE**: Use Pattern 4 for any framework with threading issues or performance needs.
```
-### Step 1.3: Performance Optimization Patterns
+### Step 1.3: Threading Context Propagation Issues (CRITICAL)
+
+**🚨 MOST COMMON FAILURE**: Frameworks using ThreadPoolExecutor break OpenTelemetry context propagation.
+
+**Example Problem (like mem0):**
+```python
+# mem0/memory/main.py - Line 257-261
+with concurrent.futures.ThreadPoolExecutor() as executor:
+ future1 = executor.submit(self._add_to_vector_store, messages, ...)
+ future2 = executor.submit(self._add_to_graph, messages, ...)
+ # ❌ This breaks context propagation - results in "all root spans"
+```
+
+**OpenLIT Solution Pattern:**
+```python
+# In utils.py - Threading context fix
+def patch_concurrent_futures_context(span_context):
+ """Patch ThreadPoolExecutor to propagate OpenTelemetry context."""
+ original_submit = concurrent.futures.ThreadPoolExecutor.submit
+
+ def patched_submit(self, fn, *args, **kwargs):
+ # Capture current context and propagate it
+ current_context = context.get_current()
+
+ def context_wrapper(*args, **kwargs):
+ token = context.attach(span_context)
+ try:
+ return fn(*args, **kwargs)
+ finally:
+ context.detach(token)
+
+ return original_submit(self, context_wrapper, *args, **kwargs)
+
+ concurrent.futures.ThreadPoolExecutor.submit = patched_submit
+ return lambda: setattr(concurrent.futures.ThreadPoolExecutor, "submit", original_submit)
+
+# In framework_name.py - Apply threading fix
+def framework_wrap(gen_ai_endpoint, ...):
+ def wrapper(wrapped, instance, args, kwargs):
+ with tracer.start_as_current_span(gen_ai_endpoint, kind=span_kind, context=current_context) as span:
+ span_context = set_span_in_context(span, context=current_context)
+
+ # βœ… Apply threading context fix for CLIENT operations
+ if span_kind == SpanKind.CLIENT:
+ restore_patch = patch_concurrent_futures_context(span_context)
+ try:
+ response = wrapped(*args, **kwargs)
+ finally:
+ restore_patch()
+ else:
+ response = wrapped(*args, **kwargs)
+```
+
+**πŸ”§ When to Apply Threading Fixes:**
+- Framework uses `concurrent.futures.ThreadPoolExecutor`
+- Framework uses `asyncio.run_in_executor`
+- You see "all root spans" in your hierarchy test
+- Child operations (OpenAI, Qdrant, etc.) appear as separate root spans
+
+### Step 1.4: Performance Optimization Patterns
**CRITICAL**: Study existing OpenLIT implementations for optimization patterns:
```python
# Study these proven patterns in existing instrumentations:
+# - Mem0: threading context propagation, __slots__ optimization
# - CrewAI: excellent agent patterns, lifecycle management
# - OpenAI Agents: great tool handling (processor.py pattern)
# - LangChain/LlamaIndex: mature framework patterns, utils.py caching
# - Pydantic AI: context caching with PydanticAIInstrumentationContext
-# Example: Context caching pattern from Pydantic AI
+# Example: Optimized context caching with __slots__
class FrameworkInstrumentationContext:
"""Context object to cache expensive extractions."""
+ __slots__ = ("instance", "args", "kwargs", "version", "environment",
+ "application_name", "_agent_name", "_model_name", "_tools", "_messages")
+
def __init__(self, instance, args, kwargs, version, environment, application_name):
self.instance = instance
self.args = args
self.kwargs = kwargs
+ self.version = version
+ self.environment = environment
+ self.application_name = application_name
# Cache expensive operations with lazy loading
self._agent_name = None
@@ -297,65 +380,89 @@ class SemanticConvention:
## Phase 4: Code Implementation
-### Step 4.1: Create Test Infrastructure First
+### Step 4.1: Create Span Hierarchy Test First (MANDATORY)
+
+**🚨 CRITICAL**: Create test.py (mandatory name) with span hierarchy analysis BEFORE any implementation:
-**Create span hierarchy test before implementing:**
```python
-# test_span_hierarchy.py - Always create this first
+# test.py - MANDATORY filename, must be in project root
+#!/usr/bin/env python3
+"""
+MANDATORY span hierarchy test for framework instrumentation.
+This MUST be created first and MUST verify proper parent-child relationships.
+"""
+
import sys
-sys.path.insert(0, 'sdk/python/src') # Use local source
+import os
+
+sys.path.insert(0, "sdk/python/src") # MANDATORY path setup
import openlit
-from opentelemetry.sdk.trace import TracerProvider
-from opentelemetry.sdk.trace.export import SimpleSpanProcessor
+import logging
-class CollectingSpanExporter:
- def __init__(self):
- self.spans = []
-
- def export(self, spans):
- for span in spans:
- self.spans.append({
- 'name': span.name,
- 'parent_id': format(span.parent.span_id, '016x') if span.parent else None,
- 'attributes': dict(span.attributes) if span.attributes else {}
- })
- return 0
+# Minimal logging to focus on span hierarchy
+logging.getLogger().setLevel(logging.ERROR)
-def print_span_hierarchy(spans):
- """Print spans in hierarchical tree structure"""
- # Build parent-child relationships
- root_spans = [s for s in spans if s['parent_id'] is None]
- child_map = {}
- for span in spans:
- if span['parent_id']:
- child_map.setdefault(span['parent_id'], []).append(span)
-
- def print_span(span, level=0):
- indent = " " * level
- prefix = "β”œβ”€β”€ " if level > 0 else ""
- print(f"{indent}{prefix}{span['name']}")
-
- span_id = format(int(span['parent_id'] or 0), '016x')
- for child in child_map.get(span_id, []):
- print_span(child, level + 1)
-
- for root in root_spans:
- print_span(root)
+def test_framework_hierarchy():
+ """
+ 🚨 CRITICAL TEST: Verify span hierarchy is correct, not "all root spans"
+ """
+ print("🧠 Framework Instrumentation Test")
+ print("=" * 40)
-def test_framework():
- collector = CollectingSpanExporter()
- tracer_provider = TracerProvider()
- tracer_provider.add_span_processor(SimpleSpanProcessor(collector))
-
+ # Initialize OpenLIT - use venv: source venv/bin/activate
+ print("βœ… Initializing OpenLIT...")
openlit.init(detailed_tracing=True)
-
- # Test framework operations here
- # ...
-
- print_span_hierarchy(collector.spans)
+
+ # Import and use framework
+ print("βœ… Testing framework operations...")
+ from your_framework import YourClass
+
+ instance = YourClass()
+
+ # Test operations that should create proper hierarchy
+ result1 = instance.main_operation("test data", user_id="test_user")
+ print(f"βœ… Main operation result: {result1}")
+
+ result2 = instance.secondary_operation("search query", user_id="test_user", limit=1)
+ print(f"βœ… Secondary operation result: {len(result2) if result2 else 0} items")
+
+ print("\n🎯 EXPECTED HIERARCHY:")
+ print("πŸ”Έ ROOT: main_operation")
+ print(" ↳ internal_operation_1")
+ print(" ↳ openai_call (or similar)")
+ print(" ↳ database_operation")
+ print("πŸ”Έ ROOT: secondary_operation")
+ print(" ↳ internal_operation_2")
+ print(" ↳ embedding_call")
+
+ print("\n⚠️ VERIFY: Check console output above for proper parent-child relationships")
+ print("❌ If you see 'all root spans', you have threading context propagation issues!")
+
+if __name__ == "__main__":
+ test_framework_hierarchy()
+```
+
+**πŸ”§ SPAN HIERARCHY DEBUGGING COMMANDS:**
+
+```bash
+# Run test.py and analyze hierarchy
+source venv/bin/activate # MANDATORY first step
+python test.py > spans.json 2>&1
+
+# Parse and analyze hierarchy (create this script)
+python3 -c "
+import json
+# ... hierarchy parsing script from mem0 experience ...
+# Shows proper tree structure or identifies 'all root spans' issue
+"
```
+**⚠️ HIERARCHY TESTING IS THE MOST IMPORTANT STEP**:
+- If hierarchy is broken (all root spans), nothing else matters
+- This is the #1 failure mode in framework instrumentation
+- Threading context propagation issues MUST be caught early
+
### Step 4.2: Implement Based on Integration Pattern
**For Native Integration:**
@@ -689,16 +796,27 @@ echo "Lines over 80 characters: $long_lines"
### Step 7.2: Comprehensive Pylint Error Handling
+**🚨 MANDATORY: 10.00/10 PYLINT SCORE REQUIRED**
+
**Run Pylint Check with Project Configuration:**
```bash
-# Use the project's pylintrc configuration
+# ALWAYS use venv first, then run from SDK directory
+source venv/bin/activate
cd sdk/python
-pylint src/openlit/instrumentation/{framework}/ --rcfile=.pylintrc
+python -m pylint src/openlit/instrumentation/{framework}/ --rcfile=.pylintrc
+
+# TARGET SCORE: 10.00/10 (not 9.5, must be perfect)
+```
-# Target scores:
-# - Individual files: 9.5-10.0/10
-# - Module overall: 9.0-9.5/10
-# - Perfect 10.0/10 is achievable by ignoring import errors
+**Run Script for Perfect 10.0/10:**
+```bash
+# Use the script pattern from mem0 success
+#!/bin/bash
+cd /Users/user/openlit # Adjust path
+source venv/bin/activate
+export PYTHONPATH="/Users/user/openlit/sdk/python/src:$PYTHONPATH"
+cd sdk/python
+python -m pylint src/openlit/instrumentation/{framework}/ --rcfile=.pylintrc
```
**Note**: The project's `.pylintrc` disables many common warnings like `broad-exception-caught`, `too-many-locals`, etc.
@@ -1276,16 +1394,22 @@ span.set_attribute(SemanticConvention.GEN_AI_CONTENT_COMPLETION, response)
### Success Criteria
**Instrumentation is complete when:**
-1. βœ… **Competitive Analysis**: Studied OpenInference, OpenLLMetry, AgentOps, LangFuse
-2. βœ… **Target SDK Study**: Cloned and analyzed framework's internal structure
-3. βœ… **Semantic Conventions**: Uses SemanticConvention extensively (no hardcoded strings)
-4. βœ… **Span Hierarchy**: Generates more/better spans than competitors
-5. βœ… **Naming Convention**: Follows `{operation_type} {operation_name}` format
-6. βœ… **Context Caching**: Implements performance optimization patterns
-7. βœ… **Business Intelligence**: Captures comprehensive cost/token/performance data
-8. βœ… **Performance**: Overhead <10% with benchmarking vs competitors
-9. βœ… **Error Resilience**: Handles framework version differences gracefully
-10. βœ… **Code Quality**: Achieves 9.5-10.0/10 pylint score
+1. 🚨 **CRITICAL: Proper Span Hierarchy**: NO "all root spans" - proper parent-child relationships maintained
+2. 🚨 **CRITICAL: Threading Context Fixes**: Handles ThreadPoolExecutor and async context propagation
+3. βœ… **Perfect Pylint Score**: Achieves exactly 10.00/10 (not 9.5, must be perfect)
+4. βœ… **Mandatory test.py**: Uses exactly "test.py" filename with venv testing
+5. βœ… **Competitive Analysis**: Studied OpenInference, OpenLLMetry, AgentOps, LangFuse
+6. βœ… **Target SDK Study**: Cloned and analyzed framework's internal structure
+7. βœ… **Semantic Conventions**: Uses SemanticConvention extensively (no hardcoded strings)
+8. βœ… **Superior Span Coverage**: Generates more/better spans than competitors
+9. βœ… **Naming Convention**: Follows `{operation_type} {operation_name}` format
+10. βœ… **4-File Structure**: Uses recommended Pattern 4 when needed for complex frameworks
+11. βœ… **Context Caching**: Implements performance optimization patterns with __slots__
+12. βœ… **Business Intelligence**: Captures comprehensive cost/token/performance data
+13. βœ… **Performance**: Overhead <10% with benchmarking vs competitors
+14. βœ… **Error Resilience**: Handles framework version differences gracefully
+
+**⚠️ PRIORITY ORDER**: Span hierarchy (#1) and threading fixes (#2) are MOST CRITICAL - everything else is secondary.
**OpenLIT Competitive Advantages Delivered:**
- 🎯 **Superior Business Intelligence**: Detailed cost tracking, token usage, performance metrics
diff --git a/otel-gpu-collector/collector.py b/otel-gpu-collector/collector.py
index 3a17761b..03c21c06 100644
--- a/otel-gpu-collector/collector.py
+++ b/otel-gpu-collector/collector.py
@@ -6,51 +6,57 @@ import json
import signal
import time
+
# Create a JSON logger
class JSONFormatter(logging.Formatter):
def format(self, record):
log_record = {
"level": record.levelname,
"message": record.getMessage(),
- "timestamp": time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(record.created)),
+ "timestamp": time.strftime(
+ "%Y-%m-%d %H:%M:%S", time.localtime(record.created)
+ ),
"name": record.name,
"application_name": record.application_name,
- "environment": record.environment
+ "environment": record.environment,
}
if record.exc_info:
log_record["exception"] = self.formatException(record.exc_info)
return json.dumps(log_record)
+
# Custom filter to add fixed information to log records
class ContextFilter(logging.Filter):
def __init__(self, application_name, environment):
super().__init__()
self.application_name = application_name
self.environment = environment
-
+
def filter(self, record):
record.application_name = self.application_name
record.environment = self.environment
return True
+
def setup_logger(application_name, environment):
logger = logging.getLogger("otel-gpu-collector")
logger.setLevel(logging.INFO)
-
+
# Log to stdout
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JSONFormatter())
-
+
logger.addHandler(handler)
-
+
# Add context filter to logger
logger.addFilter(ContextFilter(application_name, environment))
-
+
return logger
+
# Load environment variables
-application_name = os.getenv('GPU_APPLICATION_NAME', 'default')
-environment = os.getenv('GPU_ENVIRONMENT', 'default')
+application_name = os.getenv("GPU_APPLICATION_NAME", "default")
+environment = os.getenv("GPU_ENVIRONMENT", "default")
# Initialize logger
logger = setup_logger(application_name, environment)
@@ -58,30 +64,38 @@ logger = setup_logger(application_name, environment)
# Flag to control the infinite loop
keep_running = True
+
def signal_handler(sig, frame):
global keep_running
logger.info("Received termination signal", extra={"signal": sig})
keep_running = False
+
def main():
logger.info("Starting otel-gpu-collector")
try:
# Log environment variables
logger.info("Environment variables loaded")
-
+
# Initialize OpenLit with provided parameters
logger.info("Initializing GPU OpenTelemetry Instrumentation library OpenLIT")
- openlit.init(collect_gpu_stats=True, application_name=application_name, environment=environment)
+ openlit.init(
+ collect_gpu_stats=True,
+ application_name=application_name,
+ environment=environment,
+ )
logger.info("Initialization complete")
-
+
# Register signal handlers
signal.signal(signal.SIGTERM, signal_handler)
signal.signal(signal.SIGINT, signal_handler)
-
+
# Log that signal handlers are registered
- logger.info("Signal handlers registered", extra={"signals": ["SIGTERM", "SIGINT"]})
-
+ logger.info(
+ "Signal handlers registered", extra={"signals": ["SIGTERM", "SIGINT"]}
+ )
+
# Keep the script running indefinitely
global keep_running
while keep_running:
@@ -92,5 +106,6 @@ def main():
finally:
logger.info("Shutting down GPU Monitor Service")
+
if __name__ == "__main__":
- main()
\ No newline at end of file
+ main()
diff --git a/sdk/python/pyproject.toml b/sdk/python/pyproject.toml
index 77a3e46e..a1699c4a 100644
--- a/sdk/python/pyproject.toml
+++ b/sdk/python/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "openlit"
-version = "1.34.33"
+version = "1.34.34"
description = "OpenTelemetry-native Auto instrumentation library for monitoring LLM Applications and GPUs, facilitating the integration of observability into your GenAI-driven projects"
authors = ["OpenLIT"]
license = "Apache-2.0"
diff --git a/sdk/python/src/openlit/instrumentation/mem0/__init__.py b/sdk/python/src/openlit/instrumentation/mem0/__init__.py
index 32e94a2e..369cbae9 100644
--- a/sdk/python/src/openlit/instrumentation/mem0/__init__.py
+++ b/sdk/python/src/openlit/instrumentation/mem0/__init__.py
@@ -1,5 +1,7 @@
# pylint: disable=useless-return, bad-staticmethod-argument, disable=duplicate-code
-"""Initializer of Auto Instrumentation of mem0 Functions"""
+"""
+Optimized Auto Instrumentation of mem0 Functions following OpenLIT Framework Guide.
+"""
from typing import Collection
import importlib.metadata
@@ -7,82 +9,297 @@ from opentelemetry.instrumentation.instrumentor import BaseInstrumentor
from wrapt import wrap_function_wrapper
from openlit.instrumentation.mem0.mem0 import mem0_wrap
+from openlit.instrumentation.mem0.async_mem0 import async_mem0_wrap
_instruments = ("mem0ai >= 0.1.32",)
-WRAPPED_METHODS = [
+# Optimized method configuration with performance priorities
+SYNC_METHODS = [
+ # Memory initialization
+ {
+ "package": "mem0",
+ "object": "Memory.__init__",
+ "endpoint": "memory init",
+ "priority": "critical",
+ },
+ # Core Memory operations
{
"package": "mem0",
"object": "Memory.add",
- "endpoint": "mem0.memory_add",
- "wrapper": mem0_wrap,
+ "endpoint": "memory add",
+ "priority": "high",
},
{
"package": "mem0",
- "object": "Memory.get_all",
- "endpoint": "mem0.memory_get_all",
- "wrapper": mem0_wrap,
+ "object": "Memory.search",
+ "endpoint": "memory search",
+ "priority": "high",
},
{
"package": "mem0",
"object": "Memory.get",
- "endpoint": "mem0.memory_get",
- "wrapper": mem0_wrap,
+ "endpoint": "memory get",
+ "priority": "medium",
},
{
"package": "mem0",
- "object": "Memory.search",
- "endpoint": "mem0.memory_search",
- "wrapper": mem0_wrap,
+ "object": "Memory.get_all",
+ "endpoint": "memory get_all",
+ "priority": "medium",
},
{
"package": "mem0",
"object": "Memory.update",
- "endpoint": "mem0.memory_update",
- "wrapper": mem0_wrap,
+ "endpoint": "memory update",
+ "priority": "medium",
},
{
"package": "mem0",
- "object": "Memory.update",
- "endpoint": "mem0.memory_update",
- "wrapper": mem0_wrap,
+ "object": "Memory.delete",
+ "endpoint": "memory delete",
+ "priority": "medium",
+ },
+ {
+ "package": "mem0",
+ "object": "Memory.delete_all",
+ "endpoint": "memory delete_all",
+ "priority": "low",
+ },
+ {
+ "package": "mem0",
+ "object": "Memory.history",
+ "endpoint": "memory history",
+ "priority": "low",
+ },
+ {
+ "package": "mem0",
+ "object": "Memory.reset",
+ "endpoint": "memory reset",
+ "priority": "low",
+ },
+ {
+ "package": "mem0",
+ "object": "Memory.chat",
+ "endpoint": "memory chat",
+ "priority": "medium",
+ },
+ # Internal operations for detailed tracing
+ {
+ "package": "mem0.memory.main",
+ "object": "Memory._add_to_vector_store",
+ "endpoint": "memory add_to_vector_store",
+ "priority": "medium",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "Memory._add_to_graph",
+ "endpoint": "memory add_to_graph",
+ "priority": "medium",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "Memory._search_vector_store",
+ "endpoint": "memory search_vector_store",
+ "priority": "medium",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "Memory._get_all_from_vector_store",
+ "endpoint": "memory get_all_from_vector_store",
+ "priority": "low",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "Memory._create_memory",
+ "endpoint": "memory create_memory",
+ "priority": "low",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "Memory._update_memory",
+ "endpoint": "memory update_memory",
+ "priority": "low",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "Memory._delete_memory",
+ "endpoint": "memory delete_memory",
+ "priority": "low",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "Memory._create_procedural_memory",
+ "endpoint": "memory create_procedural_memory",
+ "priority": "low",
+ },
+]
+
+ASYNC_METHODS = [
+ # AsyncMemory initialization
+ {
+ "package": "mem0",
+ "object": "AsyncMemory.__init__",
+ "endpoint": "memory init",
+ "priority": "critical",
+ },
+ # Core AsyncMemory operations
+ {
+ "package": "mem0",
+ "object": "AsyncMemory.add",
+ "endpoint": "memory add",
+ "priority": "high",
+ },
+ {
+ "package": "mem0",
+ "object": "AsyncMemory.search",
+ "endpoint": "memory search",
+ "priority": "high",
+ },
+ {
+ "package": "mem0",
+ "object": "AsyncMemory.get",
+ "endpoint": "memory get",
+ "priority": "medium",
+ },
+ {
+ "package": "mem0",
+ "object": "AsyncMemory.get_all",
+ "endpoint": "memory get_all",
+ "priority": "medium",
+ },
+ {
+ "package": "mem0",
+ "object": "AsyncMemory.update",
+ "endpoint": "memory update",
+ "priority": "medium",
+ },
+ {
+ "package": "mem0",
+ "object": "AsyncMemory.delete",
+ "endpoint": "memory delete",
+ "priority": "medium",
+ },
+ {
+ "package": "mem0",
+ "object": "AsyncMemory.delete_all",
+ "endpoint": "memory delete_all",
+ "priority": "low",
+ },
+ {
+ "package": "mem0",
+ "object": "AsyncMemory.history",
+ "endpoint": "memory history",
+ "priority": "low",
+ },
+ {
+ "package": "mem0",
+ "object": "AsyncMemory.reset",
+ "endpoint": "memory reset",
+ "priority": "low",
+ },
+ {
+ "package": "mem0",
+ "object": "AsyncMemory.chat",
+ "endpoint": "memory chat",
+ "priority": "medium",
+ },
+ # Internal async operations for detailed tracing
+ {
+ "package": "mem0.memory.main",
+ "object": "AsyncMemory._add_to_graph",
+ "endpoint": "memory add_to_graph",
+ "priority": "medium",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "AsyncMemory._search_vector_store",
+ "endpoint": "memory search_vector_store",
+ "priority": "medium",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "AsyncMemory._get_all_from_vector_store",
+ "endpoint": "memory get_all_from_vector_store",
+ "priority": "low",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "AsyncMemory._create_memory",
+ "endpoint": "memory create_memory",
+ "priority": "low",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "AsyncMemory._update_memory",
+ "endpoint": "memory update_memory",
+ "priority": "low",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "AsyncMemory._delete_memory",
+ "endpoint": "memory delete_memory",
+ "priority": "low",
+ },
+ {
+ "package": "mem0.memory.main",
+ "object": "AsyncMemory._create_procedural_memory",
+ "endpoint": "memory create_procedural_memory",
+ "priority": "low",
},
]
class Mem0Instrumentor(BaseInstrumentor):
- """An instrumentor for mem0's client library."""
+ """Optimized instrumentor for mem0's client library."""
def instrumentation_dependencies(self) -> Collection[str]:
return _instruments
def _instrument(self, **kwargs):
+ """Optimized instrumentation with performance considerations."""
+ # Extract configuration
application_name = kwargs.get("application_name")
environment = kwargs.get("environment")
tracer = kwargs.get("tracer")
pricing_info = kwargs.get("pricing_info")
capture_message_content = kwargs.get("capture_message_content")
- version = importlib.metadata.version("mem0ai")
- for wrapped_method in WRAPPED_METHODS:
- wrap_package = wrapped_method.get("package")
- wrap_object = wrapped_method.get("object")
- gen_ai_endpoint = wrapped_method.get("endpoint")
- wrapper = wrapped_method.get("wrapper")
- wrap_function_wrapper(
- wrap_package,
- wrap_object,
- wrapper(
- gen_ai_endpoint,
- version,
- environment,
- application_name,
- tracer,
- pricing_info,
- capture_message_content,
- ),
- )
+ # Cache version lookup
+ try:
+ version = importlib.metadata.version("mem0ai")
+ except importlib.metadata.PackageNotFoundError:
+ version = "unknown"
+
+ # Pre-create wrapper args
+ wrapper_args = (
+ version,
+ environment,
+ application_name,
+ tracer,
+ pricing_info,
+ capture_message_content,
+ )
+
+ # Instrument sync methods
+ self._wrap_methods(SYNC_METHODS, mem0_wrap, wrapper_args)
+
+ # Instrument async methods
+ self._wrap_methods(ASYNC_METHODS, async_mem0_wrap, wrapper_args)
+
+ def _wrap_methods(self, methods, wrapper_func, wrapper_args):
+ """Efficiently wrap methods with error handling."""
+ for method_config in methods:
+ try:
+ wrap_function_wrapper(
+ method_config["package"],
+ method_config["object"],
+ wrapper_func(method_config["endpoint"], *wrapper_args),
+ )
+ except (ImportError, AttributeError):
+ # Gracefully handle missing methods in different mem0 versions
+ pass
@staticmethod
def _uninstrument(self, **kwargs):
- pass
+ """Uninstrument mem0."""
+ # Currently no cleanup needed
diff --git a/sdk/python/src/openlit/instrumentation/mem0/async_mem0.py b/sdk/python/src/openlit/instrumentation/mem0/async_mem0.py
new file mode 100644
index 00000000..214f3130
--- /dev/null
+++ b/sdk/python/src/openlit/instrumentation/mem0/async_mem0.py
@@ -0,0 +1,105 @@
+# pylint: disable=duplicate-code, broad-exception-caught, too-many-statements, unused-argument
+"""
+Optimized async mem0 instrumentation following OpenLIT Framework Guide.
+"""
+
+import time
+from opentelemetry.trace import SpanKind, Status, StatusCode, set_span_in_context
+from opentelemetry import context
+from opentelemetry.instrumentation.utils import _SUPPRESS_INSTRUMENTATION_KEY
+from openlit.__helpers import handle_exception
+from openlit.instrumentation.mem0.utils import (
+ Mem0Context,
+ set_mem0_span_attributes,
+ set_mem0_content_attributes,
+ logger,
+)
+
+
+def async_mem0_wrap(
+ gen_ai_endpoint,
+ version,
+ environment,
+ application_name,
+ tracer,
+ pricing_info,
+ capture_message_content,
+):
+ """
+ Optimized wrapper for async mem0 operations with proper hierarchy and performance.
+
+ Returns:
+ Wrapper function for async mem0 operations
+ """
+
+ def wrapper(wrapped, instance, args, kwargs):
+ """
+ Inner wrapper with comprehensive tracing for async operations.
+ """
+ # Suppression check
+ if context.get_value(_SUPPRESS_INSTRUMENTATION_KEY):
+ return wrapped(*args, **kwargs)
+
+ # Handle AsyncMemory.__init__ to create parent span for initialization operations
+ if gen_ai_endpoint == "memory init":
+ with tracer.start_as_current_span("memory init", kind=SpanKind.INTERNAL):
+ return wrapped(*args, **kwargs)
+
+ # Determine span type efficiently
+ is_internal = any(
+ marker in gen_ai_endpoint
+ for marker in ["_", "vector_store", "graph", "create_memory", "init"]
+ )
+ span_kind = SpanKind.INTERNAL if is_internal else SpanKind.CLIENT
+
+ # Create optimized context
+ ctx = Mem0Context(
+ instance, args, kwargs, version, environment, application_name
+ )
+ current_context = context.get_current()
+
+ with tracer.start_as_current_span(
+ gen_ai_endpoint, kind=span_kind, context=current_context
+ ) as span:
+ span_context = set_span_in_context(span, context=current_context)
+ token = context.attach(span_context)
+
+ start_time = time.perf_counter()
+
+ try:
+ # Execute async operation (no threading patches needed for async)
+ response = wrapped(*args, **kwargs)
+
+ # Calculate duration
+ duration = time.perf_counter() - start_time
+
+ # Set span attributes
+ set_mem0_span_attributes(span, gen_ai_endpoint, ctx, response)
+ span.set_attribute("gen_ai.client.operation.duration", duration)
+
+ # Set content attributes if enabled
+ set_mem0_content_attributes(
+ span, gen_ai_endpoint, ctx, response, capture_message_content
+ )
+
+ span.set_status(Status(StatusCode.OK))
+ return response
+
+ except Exception as e:
+ duration = time.perf_counter() - start_time
+ span.set_attribute("gen_ai.client.operation.duration", duration)
+
+ try:
+ set_mem0_span_attributes(span, gen_ai_endpoint, ctx)
+ except Exception as attr_error:
+ logger.debug(
+ "Failed to set span attributes on error: %s", attr_error
+ )
+
+ handle_exception(span, e)
+ raise
+
+ finally:
+ context.detach(token)
+
+ return wrapper
diff --git a/sdk/python/src/openlit/instrumentation/mem0/mem0.py b/sdk/python/src/openlit/instrumentation/mem0/mem0.py
index 1641d247..e6593566 100644
--- a/sdk/python/src/openlit/instrumentation/mem0/mem0.py
+++ b/sdk/python/src/openlit/instrumentation/mem0/mem0.py
@@ -1,20 +1,20 @@
# pylint: disable=duplicate-code, broad-exception-caught, too-many-statements, unused-argument
"""
-Module for monitoring mem0 applications.
+Optimized sync mem0 instrumentation following OpenLIT Framework Guide.
"""
-import logging
-from opentelemetry.trace import SpanKind, Status, StatusCode
-from opentelemetry.sdk.resources import (
- SERVICE_NAME,
- TELEMETRY_SDK_NAME,
- DEPLOYMENT_ENVIRONMENT,
-)
+import time
+from opentelemetry.trace import SpanKind, Status, StatusCode, set_span_in_context
+from opentelemetry import context
+from opentelemetry.instrumentation.utils import _SUPPRESS_INSTRUMENTATION_KEY
from openlit.__helpers import handle_exception
-from openlit.semcov import SemanticConvention
-
-# Initialize logger for logging potential issues and operations
-logger = logging.getLogger(__name__)
+from openlit.instrumentation.mem0.utils import (
+ patch_concurrent_futures_context,
+ Mem0Context,
+ set_mem0_span_attributes,
+ set_mem0_content_attributes,
+ logger,
+)
def mem0_wrap(
@@ -27,110 +27,87 @@ def mem0_wrap(
capture_message_content,
):
"""
- Creates a wrapper around a function call to trace and log its execution metrics.
-
- This function wraps any given function to measure its execution time,
- log its operation, and trace its execution using OpenTelemetry.
-
- Parameters:
- - gen_ai_endpoint (str): A descriptor or name for the endpoint being traced.
- - version (str): The version of the mem0 application.
- - environment (str): The deployment environment (e.g., 'production', 'development').
- - application_name (str): Name of the mem0 application.
- - tracer (opentelemetry.trace.Tracer): The tracer object used for OpenTelemetry tracing.
- - pricing_info (dict): Information about the pricing for internal metrics (currently not used).
- - capture_message_content (bool): Flag indicating whether to trace the content of the response.
+ Optimized wrapper for sync mem0 operations with proper hierarchy and performance.
Returns:
- - function: A higher-order function that takes a function 'wrapped' and returns
- a new function that wraps 'wrapped' with additional tracing and logging.
+ Wrapper function for mem0 operations
"""
def wrapper(wrapped, instance, args, kwargs):
"""
- An inner wrapper function that executes the wrapped function, measures execution
- time, and records trace data using OpenTelemetry.
-
- Parameters:
- - wrapped (Callable): The original function that this wrapper will execute.
- - instance (object): The instance to which the wrapped function belongs. This
- is used for instance methods. For static and classmethods,
- this may be None.
- - args (tuple): Positional arguments passed to the wrapped function.
- - kwargs (dict): Keyword arguments passed to the wrapped function.
-
- Returns:
- - The result of the wrapped function call.
-
- The wrapper initiates a span with the provided tracer, sets various attributes
- on the span based on the function's execution and response, and ensures
- errors are handled and logged appropriately.
+ Inner wrapper with comprehensive tracing and optimized performance.
"""
+ # Suppression check
+ if context.get_value(_SUPPRESS_INSTRUMENTATION_KEY):
+ return wrapped(*args, **kwargs)
+
+ # Handle Memory.__init__ to create parent span for initialization operations
+ if gen_ai_endpoint == "memory init":
+ with tracer.start_as_current_span("memory init", kind=SpanKind.INTERNAL):
+ return wrapped(*args, **kwargs)
+
+ # Determine span type efficiently
+ is_internal = any(
+ marker in gen_ai_endpoint
+ for marker in ["_", "vector_store", "graph", "create_memory", "init"]
+ )
+ span_kind = SpanKind.INTERNAL if is_internal else SpanKind.CLIENT
+
+ # Create optimized context
+ ctx = Mem0Context(
+ instance, args, kwargs, version, environment, application_name
+ )
+ current_context = context.get_current()
+
with tracer.start_as_current_span(
- gen_ai_endpoint, kind=SpanKind.CLIENT
+ gen_ai_endpoint, kind=span_kind, context=current_context
) as span:
- response = wrapped(*args, **kwargs)
+ span_context = set_span_in_context(span, context=current_context)
+ token = context.attach(span_context)
+
+ start_time = time.perf_counter()
try:
- span.set_attribute(TELEMETRY_SDK_NAME, "openlit")
- span.set_attribute(
- SemanticConvention.GEN_AI_SYSTEM,
- SemanticConvention.GEN_AI_SYSTEM_MEM0,
- )
- span.set_attribute(SemanticConvention.GEN_AI_ENDPOINT, gen_ai_endpoint)
- span.set_attribute(DEPLOYMENT_ENVIRONMENT, environment)
- span.set_attribute(
- SemanticConvention.GEN_AI_OPERATION,
- SemanticConvention.GEN_AI_OPERATION_TYPE_FRAMEWORK,
+ # Apply threading context fix for all CLIENT operations (mem0 operations use ThreadPoolExecutor)
+ if span_kind == SpanKind.CLIENT:
+ restore_patch = patch_concurrent_futures_context(span_context)
+ try:
+ response = wrapped(*args, **kwargs)
+ finally:
+ restore_patch()
+ else:
+ response = wrapped(*args, **kwargs)
+
+ # Calculate duration
+ duration = time.perf_counter() - start_time
+
+ # Set span attributes
+ set_mem0_span_attributes(span, gen_ai_endpoint, ctx, response)
+ span.set_attribute("gen_ai.client.operation.duration", duration)
+
+ # Set content attributes if enabled
+ set_mem0_content_attributes(
+ span, gen_ai_endpoint, ctx, response, capture_message_content
)
- span.set_attribute(SERVICE_NAME, application_name)
-
- if gen_ai_endpoint == "mem0.memory_add":
- span.set_attribute(
- SemanticConvention.DB_METADATA, str(kwargs.get("metadata", ""))
- )
- if response:
- span.set_attribute(
- SemanticConvention.GEN_AI_DATA_SOURCES, len(response)
- )
-
- elif gen_ai_endpoint == "mem0.memory_get_all":
- if response:
- span.set_attribute(
- SemanticConvention.GEN_AI_DATA_SOURCES, len(response)
- )
-
- elif gen_ai_endpoint == "mem0.memory_get":
- if response:
- span.set_attribute(
- SemanticConvention.GEN_AI_DATA_SOURCES, len(response)
- )
-
- elif gen_ai_endpoint == "mem0.memory_search":
- span.set_attribute(
- SemanticConvention.DB_STATEMENT, kwargs.get("query", "")
- )
-
- elif gen_ai_endpoint == "mem0.memory_update":
- span.set_attribute(
- SemanticConvention.DB_UPDATE_ID, kwargs.get("memory_id", "")
- )
-
- elif gen_ai_endpoint == "mem0.memory_delete":
- span.set_attribute(
- SemanticConvention.DB_DELETE_ID, kwargs.get("memory_id", "")
- )
span.set_status(Status(StatusCode.OK))
-
- # Return original response
return response
except Exception as e:
+ duration = time.perf_counter() - start_time
+ span.set_attribute("gen_ai.client.operation.duration", duration)
+
+ try:
+ set_mem0_span_attributes(span, gen_ai_endpoint, ctx)
+ except Exception as attr_error:
+ logger.debug(
+ "Failed to set span attributes on error: %s", attr_error
+ )
+
handle_exception(span, e)
- logger.error("Error in trace creation: %s", e)
+ raise
- # Return original response
- return response
+ finally:
+ context.detach(token)
return wrapper
diff --git a/sdk/python/src/openlit/instrumentation/mem0/utils.py b/sdk/python/src/openlit/instrumentation/mem0/utils.py
new file mode 100644
index 00000000..df135b8c
--- /dev/null
+++ b/sdk/python/src/openlit/instrumentation/mem0/utils.py
@@ -0,0 +1,301 @@
+# pylint: disable=duplicate-code
+"""
+Shared utilities for mem0 instrumentation.
+"""
+
+import concurrent.futures
+import functools
+import json
+import logging
+from typing import Any, Dict, Optional
+
+from opentelemetry import context
+from opentelemetry.sdk.resources import (
+ SERVICE_NAME,
+ TELEMETRY_SDK_NAME,
+ DEPLOYMENT_ENVIRONMENT,
+)
+from openlit.semcov import SemanticConvention
+
+# Initialize logger
+logger = logging.getLogger(__name__)
+
+
+# Threading context propagation utility
+def patch_concurrent_futures_context(current_context):
+ """
+ Patches ThreadPoolExecutor.submit to propagate OpenTelemetry context
+ across thread boundaries for proper mem0 span hierarchy.
+
+ Args:
+ current_context: OpenTelemetry context to propagate
+
+ Returns:
+ Function to restore original ThreadPoolExecutor.submit
+ """
+ original_submit = concurrent.futures.ThreadPoolExecutor.submit
+
+ def patched_submit(executor_self, fn, *args, **kwargs):
+ @functools.wraps(fn)
+ def context_wrapped_fn(*fn_args, **fn_kwargs):
+ token = context.attach(current_context)
+ try:
+ return fn(*fn_args, **fn_kwargs)
+ finally:
+ context.detach(token)
+
+ return original_submit(executor_self, context_wrapped_fn, *args, **kwargs)
+
+ concurrent.futures.ThreadPoolExecutor.submit = patched_submit
+ return lambda: setattr(
+ concurrent.futures.ThreadPoolExecutor, "submit", original_submit
+ )
+
+
+class Mem0Context:
+ """Optimized context object with lazy loading and __slots__ for performance."""
+
+ __slots__ = (
+ "instance",
+ "args",
+ "kwargs",
+ "version",
+ "environment",
+ "application_name",
+ "_user_id",
+ "_agent_id",
+ "_run_id",
+ "_metadata",
+ "_memory_type",
+ "_messages",
+ )
+
+ def __init__(self, instance, args, kwargs, version, environment, application_name):
+ self.instance = instance
+ self.args = args
+ self.kwargs = kwargs
+ self.version = version
+ self.environment = environment
+ self.application_name = application_name
+
+ # Lazy-loaded cached properties
+ self._user_id = None
+ self._agent_id = None
+ self._run_id = None
+ self._metadata = None
+ self._memory_type = None
+ self._messages = None
+
+ @property
+ def user_id(self) -> Optional[str]:
+ """Get user_id with lazy loading."""
+ if self._user_id is None:
+ self._user_id = self.kwargs.get("user_id")
+ return self._user_id
+
+ @property
+ def agent_id(self) -> Optional[str]:
+ """Get agent_id with lazy loading."""
+ if self._agent_id is None:
+ self._agent_id = self.kwargs.get("agent_id")
+ return self._agent_id
+
+ @property
+ def run_id(self) -> Optional[str]:
+ """Get run_id with lazy loading."""
+ if self._run_id is None:
+ self._run_id = self.kwargs.get("run_id")
+ return self._run_id
+
+ @property
+ def metadata(self) -> Optional[Dict[str, Any]]:
+ """Get metadata with lazy loading."""
+ if self._metadata is None:
+ self._metadata = self.kwargs.get("metadata")
+ return self._metadata
+
+ @property
+ def memory_type(self) -> Optional[str]:
+ """Get memory_type with lazy loading."""
+ if self._memory_type is None:
+ self._memory_type = self.kwargs.get("memory_type")
+ return self._memory_type
+
+ @property
+ def messages(self) -> Any:
+ """Get messages with lazy loading."""
+ if self._messages is None:
+ self._messages = self.args[0] if self.args else self.kwargs.get("messages")
+ return self._messages
+
+
+def set_mem0_span_attributes(
+ span, operation_name: str, ctx: Mem0Context, response: Any = None
+):
+ """Set optimized span attributes with batched operations."""
+
+ # Batch core attributes for performance
+ core_attributes = {
+ TELEMETRY_SDK_NAME: "openlit",
+ SemanticConvention.GEN_AI_SYSTEM: SemanticConvention.GEN_AI_SYSTEM_MEM0,
+ SemanticConvention.GEN_AI_ENDPOINT: operation_name,
+ DEPLOYMENT_ENVIRONMENT: ctx.environment,
+ SERVICE_NAME: ctx.application_name,
+ SemanticConvention.GEN_AI_SDK_VERSION: ctx.version,
+ }
+
+ # Set core attributes in batch
+ for key, value in core_attributes.items():
+ span.set_attribute(key, value)
+
+ # Determine operation type efficiently
+ is_internal = any(
+ marker in operation_name
+ for marker in ["_", "vector_store", "graph", "create_memory", "procedural"]
+ )
+
+ operation_type = (
+ SemanticConvention.GEN_AI_OPERATION_TYPE_FRAMEWORK
+ if is_internal
+ else SemanticConvention.GEN_AI_OPERATION_TYPE_MEMORY
+ )
+ span.set_attribute(SemanticConvention.GEN_AI_OPERATION, operation_type)
+
+ # Component type mapping for internal operations
+ if is_internal:
+ component_map = {
+ "vector_store": "vector_store",
+ "graph": "graph_store",
+ "create_memory": "memory_creation",
+ "procedural": "procedural_memory",
+ }
+ for key, component_type in component_map.items():
+ if key in operation_name:
+ span.set_attribute(
+ SemanticConvention.GEN_AI_FRAMEWORK_COMPONENT_TYPE, component_type
+ )
+ break
+
+ # Session attributes
+ session_attrs = {}
+ if ctx.user_id:
+ session_attrs[SemanticConvention.GEN_AI_USER_ID] = ctx.user_id
+ if ctx.agent_id:
+ session_attrs[SemanticConvention.GEN_AI_AGENT_ID] = ctx.agent_id
+ if ctx.run_id:
+ session_attrs[SemanticConvention.GEN_AI_RUN_ID] = ctx.run_id
+
+ for key, value in session_attrs.items():
+ span.set_attribute(key, value)
+
+ # Memory-specific attributes
+ if ctx.memory_type:
+ span.set_attribute(SemanticConvention.GEN_AI_MEMORY_TYPE, ctx.memory_type)
+
+ if ctx.metadata:
+ try:
+ span.set_attribute(
+ SemanticConvention.GEN_AI_MEMORY_METADATA, json.dumps(ctx.metadata)
+ )
+ except Exception:
+ span.set_attribute(
+ SemanticConvention.GEN_AI_MEMORY_METADATA, str(ctx.metadata)
+ )
+
+ # Operation-specific attributes
+ _set_operation_attributes(span, operation_name, ctx)
+
+ # Response attributes
+ _set_response_attributes(span, response)
+
+
+def _set_operation_attributes(span, operation_name: str, ctx: Mem0Context):
+ """Set operation-specific attributes efficiently."""
+ if "add" in operation_name:
+ if "infer" in ctx.kwargs:
+ span.set_attribute(
+ SemanticConvention.GEN_AI_MEMORY_INFER, ctx.kwargs["infer"]
+ )
+ if ctx.messages:
+ try:
+ count = (
+ len(ctx.messages) if isinstance(ctx.messages, (list, tuple)) else 1
+ )
+ span.set_attribute(SemanticConvention.GEN_AI_MEMORY_COUNT, count)
+ except Exception:
+ pass
+
+ elif "search" in operation_name:
+ search_attrs = {}
+ if "query" in ctx.kwargs:
+ search_attrs[SemanticConvention.GEN_AI_MEMORY_SEARCH_QUERY] = ctx.kwargs[
+ "query"
+ ]
+ if "limit" in ctx.kwargs:
+ search_attrs[SemanticConvention.GEN_AI_MEMORY_SEARCH_LIMIT] = ctx.kwargs[
+ "limit"
+ ]
+ if "threshold" in ctx.kwargs:
+ search_attrs[SemanticConvention.GEN_AI_MEMORY_SEARCH_THRESHOLD] = (
+ ctx.kwargs["threshold"]
+ )
+
+ for key, value in search_attrs.items():
+ span.set_attribute(key, value)
+
+ elif "update" in operation_name and "memory_id" in ctx.kwargs:
+ span.set_attribute(SemanticConvention.DB_UPDATE_ID, ctx.kwargs["memory_id"])
+
+ elif "delete" in operation_name and "memory_id" in ctx.kwargs:
+ span.set_attribute(SemanticConvention.DB_DELETE_ID, ctx.kwargs["memory_id"])
+
+ elif "get" in operation_name and "memory_id" in ctx.kwargs:
+ span.set_attribute(SemanticConvention.DB_OPERATION_ID, ctx.kwargs["memory_id"])
+
+
+def _set_response_attributes(span, response: Any):
+ """Set response attributes for business intelligence."""
+ if not response:
+ return
+
+ try:
+ if isinstance(response, (list, tuple)):
+ count = len(response)
+ elif isinstance(response, dict):
+ count = len(response.get("results", [])) if response.get("results") else 1
+ else:
+ count = 1
+
+ span.set_attribute(
+ SemanticConvention.GEN_AI_MEMORY_OPERATION_RESULT_COUNT, count
+ )
+ span.set_attribute(SemanticConvention.GEN_AI_DATA_SOURCES, count)
+ except Exception:
+ pass
+
+
+def set_mem0_content_attributes(
+ span, operation_name: str, ctx: Mem0Context, response: Any, capture_content: bool
+):
+ """Set content attributes if enabled."""
+ if not capture_content:
+ return
+
+ try:
+ if ctx.messages and "add" in operation_name:
+ content = (
+ json.dumps(ctx.messages)
+ if isinstance(ctx.messages, (list, dict))
+ else str(ctx.messages)
+ )
+ span.set_attribute(SemanticConvention.GEN_AI_CONTENT_PROMPT, content)
+
+ if response and "search" in operation_name:
+ content = (
+ json.dumps(response)
+ if isinstance(response, (dict, list))
+ else str(response)
+ )
+ span.set_attribute(SemanticConvention.GEN_AI_CONTENT_COMPLETION, content)
+ except Exception as e:
+ logger.debug("Failed to capture content: %s", e)
diff --git a/sdk/python/src/openlit/semcov/__init__.py b/sdk/python/src/openlit/semcov/__init__.py
index a0506d9b..aca5db79 100644
--- a/sdk/python/src/openlit/semcov/__init__.py
+++ b/sdk/python/src/openlit/semcov/__init__.py
@@ -540,6 +540,25 @@ class SemanticConvention:
# Session and conversation tracking
GEN_AI_SESSION_ID = "gen_ai.session.id"
GEN_AI_USER_ID = "gen_ai.user.id"
+ GEN_AI_RUN_ID = "gen_ai.run.id"
+
+ # Memory operation types
+ GEN_AI_OPERATION_TYPE_MEMORY = "memory"
+ GEN_AI_OPERATION_TYPE_MEMORY_ADD = "memory_add"
+ GEN_AI_OPERATION_TYPE_MEMORY_SEARCH = "memory_search"
+ GEN_AI_OPERATION_TYPE_MEMORY_GET = "memory_get"
+ GEN_AI_OPERATION_TYPE_MEMORY_UPDATE = "memory_update"
+ GEN_AI_OPERATION_TYPE_MEMORY_DELETE = "memory_delete"
+
+ # Memory-specific attributes
+ GEN_AI_MEMORY_TYPE = "gen_ai.memory.type"
+ GEN_AI_MEMORY_METADATA = "gen_ai.memory.metadata"
+ GEN_AI_MEMORY_INFER = "gen_ai.memory.infer"
+ GEN_AI_MEMORY_COUNT = "gen_ai.memory.count"
+ GEN_AI_MEMORY_SEARCH_QUERY = "gen_ai.memory.search.query"
+ GEN_AI_MEMORY_SEARCH_LIMIT = "gen_ai.memory.search.limit"
+ GEN_AI_MEMORY_SEARCH_THRESHOLD = "gen_ai.memory.search.threshold"
+ GEN_AI_MEMORY_OPERATION_RESULT_COUNT = "gen_ai.memory.operation.result_count"
# Agent lifecycle phases
GEN_AI_AGENT_LIFECYCLE_PHASE = "gen_ai.agent.lifecycle.phase"
diff --git a/sdk/python/tests/test_ai21.py b/sdk/python/tests/test_ai21.py
index 45e2a2e3..58eb7dcf 100644
--- a/sdk/python/tests/test_ai21.py
+++ b/sdk/python/tests/test_ai21.py
@@ -2,7 +2,7 @@
"""
This module contains tests for AI21 functionality using the AI21 Python library.
-Tests cover various API endpoints, including chat and embeddings.
+Tests cover various API endpoints, including chat and embeddings.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -24,12 +24,15 @@ sync_client = AI21Client()
async_client = AsyncAI21Client()
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-python-testing", application_name="openlit-python-ai21-test")
+openlit.init(
+ environment="openlit-python-testing", application_name="openlit-python-ai21-test"
+)
MESSAGES = [
ChatMessage(content="hi", role="user"),
]
+
def test_sync_ai21_chat():
"""
Tests synchronous chat.
@@ -45,6 +48,7 @@ def test_sync_ai21_chat():
)
assert isinstance(response.id, str)
+
def test_sync_ai21_chat_stream():
"""
Tests synchronous chat streaming.
@@ -63,6 +67,7 @@ def test_sync_ai21_chat_stream():
assert isinstance(response.id, str)
return
+
def test_sync_ai21_chat_rag():
"""
Tests synchronous chat rag.
@@ -82,6 +87,7 @@ def test_sync_ai21_chat_rag():
)
assert isinstance(response.id, str)
+
@pytest.mark.asyncio
async def test_async_ai21_test():
"""
diff --git a/sdk/python/tests/test_anthropic.py b/sdk/python/tests/test_anthropic.py
index 1ecbdc8d..b53f5745 100644
--- a/sdk/python/tests/test_anthropic.py
+++ b/sdk/python/tests/test_anthropic.py
@@ -2,7 +2,7 @@
"""
This module contains tests for Anthropic functionality using the Anthropic Python library.
-Tests cover various API endpoints, including chat.
+Tests cover various API endpoints, including chat.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -23,7 +23,11 @@ sync_client = Anthropic()
async_client = AsyncAnthropic()
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-python-testing", application_name="openlit-python-anthropic-test")
+openlit.init(
+ environment="openlit-python-testing",
+ application_name="openlit-python-anthropic-test",
+)
+
def test_sync_anthropic_messages():
"""
@@ -44,7 +48,7 @@ def test_sync_anthropic_messages():
],
model="claude-3-haiku-20240307",
)
- assert message.type == 'message'
+ assert message.type == "message"
# pylint: disable=broad-exception-caught
except Exception as e:
@@ -53,6 +57,7 @@ def test_sync_anthropic_messages():
else:
raise
+
@pytest.mark.asyncio
async def test_async_anthropic_messages():
"""
@@ -73,7 +78,7 @@ async def test_async_anthropic_messages():
],
model="claude-3-haiku-20240307",
)
- assert message.type == 'message'
+ assert message.type == "message"
# pylint: disable=broad-exception-caught
except Exception as e:
diff --git a/sdk/python/tests/test_assemblyai.py b/sdk/python/tests/test_assemblyai.py
index c4148c7a..efeecf40 100644
--- a/sdk/python/tests/test_assemblyai.py
+++ b/sdk/python/tests/test_assemblyai.py
@@ -2,7 +2,7 @@
"""
This module contains tests for Assembly AI functionality using the Assembly AI Python library.
-Tests cover various API endpoints.
+Tests cover various API endpoints.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -18,9 +18,10 @@ import openlit
# Initialize environment and application name for OpenLIT monitoring
openlit.init(
environment="openlit-python-testing",
- application_name="openlit-python-assemblyai-test"
+ application_name="openlit-python-assemblyai-test",
)
+
def test_sync_assemblyai_transcribe():
"""
Tests synchronous transcribe.
@@ -30,9 +31,7 @@ def test_sync_assemblyai_transcribe():
"""
try:
- audio_file = (
- "https://audio-samples.github.io/samples/mp3/blizzard_unconditional/sample-0.mp3"
- )
+ audio_file = "https://audio-samples.github.io/samples/mp3/blizzard_unconditional/sample-0.mp3"
transcriber = aai.Transcriber()
config = aai.TranscriptionConfig(speech_model=aai.SpeechModel.nano)
diff --git a/sdk/python/tests/test_astra.py b/sdk/python/tests/test_astra.py
index 1b216b9a..0129ed09 100644
--- a/sdk/python/tests/test_astra.py
+++ b/sdk/python/tests/test_astra.py
@@ -3,7 +3,7 @@
This module contains tests for AstraDB functionality using the AstraDB Python library.
Tests cover various API endpoints, including create_collection, add, query,
-upsert, update, get, peek and delete.
+upsert, update, get, peek and delete.
These tests validate integration with OpenLIT.
Note: Ensure the environment is properly configured for AstraDB access and OpenLIT monitoring
@@ -16,12 +16,13 @@ import openlit
# Initialize the client
client = DataAPIClient(os.getenv("ASTRA_DB_APPLICATION_TOKEN"))
-db = client.get_database_by_api_endpoint(
- os.getenv("ASTRA_DB_API_ENDPOINT")
-)
+db = client.get_database_by_api_endpoint(os.getenv("ASTRA_DB_API_ENDPOINT"))
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-python-testing", application_name="openlit-python-astra-test")
+openlit.init(
+ environment="openlit-python-testing", application_name="openlit-python-astra-test"
+)
+
def test_db_astra():
"""
@@ -40,31 +41,33 @@ def test_db_astra():
# assert collection.name == "openlit"
collection = db.get_collection("openlit")
- response = collection.insert_one({"summary": "I was flying", "$vector": [-0.4, 0.7, 0]})
+ response = collection.insert_one(
+ {"summary": "I was flying", "$vector": [-0.4, 0.7, 0]}
+ )
assert isinstance(response.inserted_id, str)
response = collection.insert_many(
- [
- {
- "tags": ["food"],
- "summary": "A dinner on the Moon",
- "$vector": [0.2, -0.3, -0.5],
- },
- {
- "summary": "Riding the waves",
- "tags": ["sport"],
- "$vector": [0, 0.2, 1],
- },
- {
- "summary": "Friendly aliens in town",
- "tags": ["scifi"],
- "$vector": [-0.3, 0, 0.8],
- },
- {
- "summary": "Meeting Beethoven at the dentist",
- "$vector": [0.2, 0.6, 0],
- },
- ],
+ [
+ {
+ "tags": ["food"],
+ "summary": "A dinner on the Moon",
+ "$vector": [0.2, -0.3, -0.5],
+ },
+ {
+ "summary": "Riding the waves",
+ "tags": ["sport"],
+ "$vector": [0, 0.2, 1],
+ },
+ {
+ "summary": "Friendly aliens in town",
+ "tags": ["scifi"],
+ "$vector": [-0.3, 0, 0.8],
+ },
+ {
+ "summary": "Meeting Beethoven at the dentist",
+ "$vector": [0.2, 0.6, 0],
+ },
+ ],
)
assert isinstance(response.inserted_ids, list)
@@ -96,11 +99,11 @@ def test_db_astra():
include_similarity=True,
)
for response in cursor:
- assert isinstance(response['summary'], str)
+ assert isinstance(response["summary"], str)
response = collection.replace_one(
- {"tag": {"$exists": True}}, # filter
- {"summary": "Pest"}, # replacement
+ {"tag": {"$exists": True}}, # filter
+ {"summary": "Pest"}, # replacement
)
assert isinstance(response.update_info["n"], int)
diff --git a/sdk/python/tests/test_azure_ai_inference.py b/sdk/python/tests/test_azure_ai_inference.py
index 7cd80743..4105fabc 100644
--- a/sdk/python/tests/test_azure_ai_inference.py
+++ b/sdk/python/tests/test_azure_ai_inference.py
@@ -37,6 +37,7 @@ sync_embed_client = EmbeddingsClient(
# Initialize environment and application name for OpenLIT monitoring
openlit.init(environment="openlit-testing", application_name="openlit-python-test")
+
def test_sync_chat_completions():
"""
Tests synchronous chat completions with the "microsoft/Phi-4-mini-reasoning" model.
@@ -53,9 +54,10 @@ def test_sync_chat_completions():
model="microsoft/Phi-4-mini-reasoning",
temperature=1.0,
max_tokens=1000,
- top_p=1.0
+ top_p=1.0,
)
- assert response["object"] == 'chat.completion'
+ assert response["object"] == "chat.completion"
+
def test_sync_embeddings():
"""
@@ -67,11 +69,13 @@ def test_sync_embeddings():
response = sync_embed_client.embed(
input=["LLM Observability", "Monitor GPUs"],
- model="openai/text-embedding-3-small"
+ model="openai/text-embedding-3-small",
)
- assert response.data[0]["object"] == 'embedding'
+ assert response.data[0]["object"] == "embedding"
+
from azure.ai.inference.aio import ChatCompletionsClient, EmbeddingsClient
+
async_chat_client = ChatCompletionsClient(
endpoint="https://models.github.ai/inference",
credential=AzureKeyCredential(os.getenv("AZURE_AI_INFERENCE_API_TOKEN")),
@@ -82,6 +86,7 @@ async_embed_client = EmbeddingsClient(
credential=AzureKeyCredential(os.getenv("AZURE_AI_INFERENCE_API_TOKEN")),
)
+
@pytest.mark.asyncio
async def test_async_chat_completions():
"""
@@ -99,9 +104,10 @@ async def test_async_chat_completions():
model="microsoft/Phi-4-mini-reasoning",
temperature=1.0,
max_tokens=1000,
- top_p=1.0
+ top_p=1.0,
)
- assert response["object"] == 'chat.completion'
+ assert response["object"] == "chat.completion"
+
@pytest.mark.asyncio
async def test_async_embeddings():
@@ -114,6 +120,6 @@ async def test_async_embeddings():
response = await async_embed_client.embed(
input=["LLM Observability", "Monitor GPUs"],
- model="openai/text-embedding-3-small"
+ model="openai/text-embedding-3-small",
)
- assert response.data[0]["object"] == 'embedding'
+ assert response.data[0]["object"] == "embedding"
diff --git a/sdk/python/tests/test_chroma.py b/sdk/python/tests/test_chroma.py
index 774f8a4c..0328eaf0 100644
--- a/sdk/python/tests/test_chroma.py
+++ b/sdk/python/tests/test_chroma.py
@@ -3,7 +3,7 @@
This module contains tests for ChromaDB functionality using the ChromaDB Python library.
Tests cover various API endpoints, including create_collection, add, query,
-upsert, update, get, peek and delete.
+upsert, update, get, peek and delete.
These tests validate integration with OpenLIT.
Note: Ensure the environment is properly configured for ChromaDB access and OpenLIT monitoring
@@ -19,6 +19,7 @@ chroma_client = chromadb.Client()
# Initialize environment and application name for OpenLIT monitoring
openlit.init(environment="openlit-testing", application_name="openlit-python-test")
+
def test_db_chroma():
"""
Tests basic operations within a ChromaDB collection.
@@ -35,7 +36,7 @@ def test_db_chroma():
expected document ids.
- A delete operation targets a specific document by id, and successful deletion
is implicitly verified by the absence of an error response.
-
+
The test ensures the basic CRUD operations perform as expected in ChromaDB.
Raises:
AssertionError: If the responses from ChromaDB operations do not meet the expected outcomes.
@@ -43,26 +44,20 @@ def test_db_chroma():
# Create a new collection named "openlit"
collection = chroma_client.create_collection(name="openlit")
- assert collection.name == 'openlit'
+ assert collection.name == "openlit"
# Add documents to the collection
db_add = collection.add(
documents=["This is a document", "This is another document"],
metadatas=[{"source": "my_source"}, {"source": "my_source"}],
- ids=["id1", "id2"]
+ ids=["id1", "id2"],
)
assert db_add is None
# Query the documents in the collection
- db_query = collection.query(
- query_texts=["This is a query document"],
- n_results=2
- )
- assert db_query["ids"] == [['id1', 'id2']]
+ db_query = collection.query(query_texts=["This is a query document"], n_results=2)
+ assert db_query["ids"] == [["id1", "id2"]]
# Delete a document from the collection
- db_delete = collection.delete(
- ids=["id2"],
- where={"source": "my_source"}
- )
+ db_delete = collection.delete(ids=["id2"], where={"source": "my_source"})
assert db_delete is None
diff --git a/sdk/python/tests/test_cohere.py b/sdk/python/tests/test_cohere.py
index 07bb430b..dd4240bc 100644
--- a/sdk/python/tests/test_cohere.py
+++ b/sdk/python/tests/test_cohere.py
@@ -2,7 +2,7 @@
"""
This module contains tests for Cohere functionality using the Cohere Python library.
-Tests cover various API endpoints, including chat and embeddings.
+Tests cover various API endpoints, including chat and embeddings.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -19,7 +19,9 @@ import openlit
sync_client = cohere.Client()
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-python-testing", application_name="openlit-python-cohere-test")
+openlit.init(
+ environment="openlit-python-testing", application_name="openlit-python-cohere-test"
+)
def test_embed():
diff --git a/sdk/python/tests/test_controlflow.py b/sdk/python/tests/test_controlflow.py
index 174a33cb..46e26830 100644
--- a/sdk/python/tests/test_controlflow.py
+++ b/sdk/python/tests/test_controlflow.py
@@ -2,7 +2,7 @@
"""
This module contains tests for ControlFlow functionality using the ControlFlow Python library.
-Tests cover various API endpoints, including chat and embeddings.
+Tests cover various API endpoints, including chat and embeddings.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -18,9 +18,10 @@ import openlit
# Initialize environment and application name for OpenLIT monitoring
openlit.init(
environment="openlit-python-testing",
- application_name="openlit-python-controlflow-test"
+ application_name="openlit-python-controlflow-test",
)
+
def test_sync_controlflow_agent():
"""
Tests Agent Creation.
@@ -36,6 +37,7 @@ def test_sync_controlflow_agent():
)
assert isinstance(agent, cf.Agent)
+
def test_sync_controlflow_task():
"""
Tests Task Creation.
diff --git a/sdk/python/tests/test_elevenlabs.py b/sdk/python/tests/test_elevenlabs.py
index b83d3e2c..55258003 100644
--- a/sdk/python/tests/test_elevenlabs.py
+++ b/sdk/python/tests/test_elevenlabs.py
@@ -2,7 +2,7 @@
"""
This module contains tests for ElevenLabs functionality using the ElevenLabs Python library.
-Tests cover various API endpoints.
+Tests cover various API endpoints.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -25,7 +25,7 @@ async_client = AsyncElevenLabs()
# Initialize environment and application name for OpenLIT monitoring
openlit.init(
environment="openlit-python-testing",
- application_name="openlit-python-elevenlabs-test"
+ application_name="openlit-python-elevenlabs-test",
)
@@ -53,6 +53,7 @@ def test_sync_elevenlabs_t2s():
else:
raise
+
@pytest.mark.asyncio
async def test_async_elevenlabs_t2s():
"""
diff --git a/sdk/python/tests/test_embedchain.py b/sdk/python/tests/test_embedchain.py
index e6e8a98a..6e432d3c 100644
--- a/sdk/python/tests/test_embedchain.py
+++ b/sdk/python/tests/test_embedchain.py
@@ -19,43 +19,40 @@ import openlit
# Initialize environment and application name for OpenLIT monitoring
openlit.init(
environment="openlit-python-testing",
- application_name="openlit-python-embedchain-test"
+ application_name="openlit-python-embedchain-test",
)
config = {
- 'app': {
- 'config': {
- 'name': 'openlit-testing'
- }
+ "app": {"config": {"name": "openlit-testing"}},
+ "llm": {
+ "provider": "openai",
+ "config": {
+ "model": "gpt-3.5-turbo",
+ "temperature": 0.5,
+ "max_tokens": 1,
+ "top_p": 1,
+ "stream": False,
+ "api_key": os.getenv("OPENAI_API_TOKEN"),
+ },
},
- 'llm': {
- 'provider': 'openai',
- 'config': {
- 'model': 'gpt-3.5-turbo',
- 'temperature': 0.5,
- 'max_tokens': 1,
- 'top_p': 1,
- 'stream': False,
- 'api_key': os.getenv("OPENAI_API_TOKEN")
- }
+ "vectordb": {
+ "provider": "chroma",
+ "config": {
+ "collection_name": "full-stack-app",
+ "dir": "db",
+ "allow_reset": True,
+ },
},
- 'vectordb': {
- 'provider': 'chroma',
- 'config': {
- 'collection_name': 'full-stack-app',
- 'dir': 'db',
- 'allow_reset': True
- }
+ "embedder": {
+ "provider": "openai",
+ "config": {
+ "model": "text-embedding-ada-002",
+ "api_key": os.getenv("OPENAI_API_TOKEN"),
+ },
},
- 'embedder': {
- 'provider': 'openai',
- 'config': {
- 'model': 'text-embedding-ada-002',
- 'api_key': os.getenv("OPENAI_API_TOKEN")
- }
- }
}
+
def test_embedchain():
"""
Verifies the functionality of Embedchain operations including document loading, text splitting,
@@ -76,6 +73,7 @@ def test_embedchain():
data_sources = app.get_data_sources()
assert isinstance(len(data_sources), int)
- evals = app.evaluate(["What is the net worth of Elon Musk?",
- "How many companies Elon Musk owns?"])
+ evals = app.evaluate(
+ ["What is the net worth of Elon Musk?", "How many companies Elon Musk owns?"]
+ )
assert isinstance(evals, dict)
diff --git a/sdk/python/tests/test_firecrawl.py b/sdk/python/tests/test_firecrawl.py
index 218f9ca3..732b0a16 100644
--- a/sdk/python/tests/test_firecrawl.py
+++ b/sdk/python/tests/test_firecrawl.py
@@ -2,7 +2,7 @@
"""
This module contains tests for FireCrawl functionality using the FireCrawl Python library.
-Tests cover various API endpoints, including chat and embeddings.
+Tests cover various API endpoints, including chat and embeddings.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -20,7 +20,11 @@ import openlit
sync_client = FirecrawlApp(api_key=os.getenv("FIRECRAWL_API_KEY"))
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-python-testing", application_name="openlit-python-firecrawl-test")
+openlit.init(
+ environment="openlit-python-testing",
+ application_name="openlit-python-firecrawl-test",
+)
+
def test_sync_scarpe_url():
"""
@@ -31,8 +35,8 @@ def test_sync_scarpe_url():
"""
response = sync_client.scrape_url(
- 'https://openlit.io',
- formats=['markdown', 'html'],
+ "https://openlit.io",
+ formats=["markdown", "html"],
)
assert response.success is True
diff --git a/sdk/python/tests/test_google_ai_studio.py b/sdk/python/tests/test_google_ai_studio.py
index 5b5360f7..5f674ebf 100644
--- a/sdk/python/tests/test_google_ai_studio.py
+++ b/sdk/python/tests/test_google_ai_studio.py
@@ -3,7 +3,7 @@
This module contains tests for Google AI Studio functionality using the google-generativeai
Python library.
-Tests cover various API endpoints, including chat.
+Tests cover various API endpoints, including chat.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -20,9 +20,7 @@ from google.genai import types
import openlit
# Initialize Google AI Studio client
-client = genai.Client(
- api_key=os.getenv("GOOGLE_AI_STUDIO_API_TOKEN")
-)
+client = genai.Client(api_key=os.getenv("GOOGLE_AI_STUDIO_API_TOKEN"))
contents = [
types.Content(
role="user",
@@ -36,7 +34,10 @@ generate_content_config = types.GenerateContentConfig(
)
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-testing", application_name="openlit-python-gemini-test")
+openlit.init(
+ environment="openlit-testing", application_name="openlit-python-gemini-test"
+)
+
def test_sync_generate_content():
"""
@@ -61,6 +62,7 @@ def test_sync_generate_content():
else:
raise
+
@pytest.mark.asyncio
async def test_async_generate_content():
"""
diff --git a/sdk/python/tests/test_groq.py b/sdk/python/tests/test_groq.py
index ed6bb84d..3fd8c389 100644
--- a/sdk/python/tests/test_groq.py
+++ b/sdk/python/tests/test_groq.py
@@ -2,7 +2,7 @@
"""
This module contains tests for Groq functionality using the Groq Python library.
-Tests cover various API endpoints, including chat.
+Tests cover various API endpoints, including chat.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -18,18 +18,15 @@ from groq import Groq, AsyncGroq
import openlit
# Initialize synchronous Groq client
-sync_client = Groq(
- api_key=os.getenv("GROQ_API_TOKEN")
-)
+sync_client = Groq(api_key=os.getenv("GROQ_API_TOKEN"))
# Initialize asynchronous Groq client
-async_client = AsyncGroq(
- api_key=os.getenv("GROQ_API_TOKEN")
-)
+async_client = AsyncGroq(api_key=os.getenv("GROQ_API_TOKEN"))
# Initialize environment and application name for OpenLIT monitoring
openlit.init(environment="openlit-testing", application_name="openlit-python-test")
+
def test_sync_groq_chat():
"""
Tests synchronous Chat Completions with the 'llama3-8b-8192' model.
@@ -48,9 +45,9 @@ def test_sync_groq_chat():
],
model="llama3-8b-8192",
max_tokens=1,
- stream = False,
+ stream=False,
)
- assert chat_completions_resp.object == 'chat.completion'
+ assert chat_completions_resp.object == "chat.completion"
# pylint: disable=broad-exception-caught
except Exception as e:
@@ -59,6 +56,7 @@ def test_sync_groq_chat():
else:
raise
+
@pytest.mark.asyncio
async def test_async_groq_chat():
"""
@@ -80,7 +78,7 @@ async def test_async_groq_chat():
max_tokens=1,
stream=False,
)
- assert chat_completions_resp.object == 'chat.completion'
+ assert chat_completions_resp.object == "chat.completion"
# pylint: disable=broad-exception-caught
except Exception as e:
diff --git a/sdk/python/tests/test_mistral.py b/sdk/python/tests/test_mistral.py
index 0904fe07..35e9d702 100644
--- a/sdk/python/tests/test_mistral.py
+++ b/sdk/python/tests/test_mistral.py
@@ -2,7 +2,7 @@
"""
This module contains tests for Mistral functionality using the Mistral Python library.
-Tests cover various API endpoints, including chat and embeddings.
+Tests cover various API endpoints, including chat and embeddings.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -23,7 +23,10 @@ client = Mistral(
)
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-python-testing", application_name="openlit-python-mistral-test")
+openlit.init(
+ environment="openlit-python-testing", application_name="openlit-python-mistral-test"
+)
+
def test_sync_mistral_chat():
"""
@@ -45,7 +48,8 @@ def test_sync_mistral_chat():
messages=messages,
max_tokens=1,
)
- assert message.object == 'chat.completion'
+ assert message.object == "chat.completion"
+
def test_sync_mistral_embeddings():
"""
@@ -56,10 +60,11 @@ def test_sync_mistral_embeddings():
"""
response = client.embeddings.create(
- model="mistral-embed",
- inputs=["Embed this sentence.", "OpenTelemetry LLM Observability"],
+ model="mistral-embed",
+ inputs=["Embed this sentence.", "OpenTelemetry LLM Observability"],
)
- assert response.object == 'list'
+ assert response.object == "list"
+
@pytest.mark.asyncio
async def test_async_mistral():
@@ -83,11 +88,11 @@ async def test_async_mistral():
messages=messages,
max_tokens=1,
)
- assert message.object == 'chat.completion'
+ assert message.object == "chat.completion"
# Tests asynchronous embedding creation with the 'mistral-embed' model.
response = await client.embeddings.create_async(
- model="mistral-embed",
- inputs=["Embed this sentence.", "Monitor LLM Applications"],
+ model="mistral-embed",
+ inputs=["Embed this sentence.", "Monitor LLM Applications"],
)
- assert response.object == 'list'
+ assert response.object == "list"
diff --git a/sdk/python/tests/test_openai.py b/sdk/python/tests/test_openai.py
index bea1cfba..fbaf5bb7 100644
--- a/sdk/python/tests/test_openai.py
+++ b/sdk/python/tests/test_openai.py
@@ -25,7 +25,10 @@ sync_client = OpenAI()
async_client = AsyncOpenAI()
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-python-testing", application_name="openlit-python-openai-test")
+openlit.init(
+ environment="openlit-python-testing", application_name="openlit-python-openai-test"
+)
+
def test_sync_openai_chat_completions():
"""
@@ -38,9 +41,10 @@ def test_sync_openai_chat_completions():
chat_completions_resp = sync_client.chat.completions.create(
model="gpt-3.5-turbo",
max_tokens=1,
- messages=[{"role": "user", "content": "What is LLM Observability?"}]
+ messages=[{"role": "user", "content": "What is LLM Observability?"}],
)
- assert chat_completions_resp.object == 'chat.completion'
+ assert chat_completions_resp.object == "chat.completion"
+
def test_sync_openai_chat_completions_parse():
"""
@@ -52,19 +56,26 @@ def test_sync_openai_chat_completions_parse():
class User(BaseModel):
"""A model to represent a user's details."""
+
name: str
age: int
-
chat_completions_resp = sync_client.beta.chat.completions.parse(
model="gpt-4o-mini",
messages=[
- {"role": "system", "content": "Extract the user's name and age from the following text."},
- {"role": "user", "content": "The user's name is John Doe and he is 30 years old."},
+ {
+ "role": "system",
+ "content": "Extract the user's name and age from the following text.",
+ },
+ {
+ "role": "user",
+ "content": "The user's name is John Doe and he is 30 years old.",
+ },
],
response_format=User, # Pass the Pydantic model as the response format
)
- assert chat_completions_resp.object == 'chat.completion'
+ assert chat_completions_resp.object == "chat.completion"
+
def test_sync_openai_embeddings():
"""
@@ -77,9 +88,10 @@ def test_sync_openai_embeddings():
embeddings_resp = sync_client.embeddings.create(
model="text-embedding-ada-002",
input="The quick brown fox jumped over the lazy dog",
- encoding_format="float"
+ encoding_format="float",
)
- assert embeddings_resp.data[0].object == 'embedding'
+ assert embeddings_resp.data[0].object == "embedding"
+
# def test_sync_openai_fine_tuning_job_creation():
# """
@@ -102,6 +114,7 @@ def test_sync_openai_embeddings():
# rate_limit_code = error_json['error']['code']
# print(rate_limit_code)
+
def test_sync_openai_image_generation():
"""
Tests synchronous image generation with the 'dall-e-2' model.
@@ -111,13 +124,11 @@ def test_sync_openai_image_generation():
"""
image_generation_resp = sync_client.images.generate(
- model='dall-e-2',
- prompt='Generate an image of a cat.',
- size='256x256',
- n=1
+ model="dall-e-2", prompt="Generate an image of a cat.", size="256x256", n=1
)
assert image_generation_resp.created is not None
+
def test_sync_openai_image_variations():
"""
Tests synchronous image variation creation with the 'dall-e-2' model
@@ -128,12 +139,13 @@ def test_sync_openai_image_variations():
image_variation_resp = sync_client.images.create_variation(
image=open("tests/test-image-for-openai.png", "rb"),
- model='dall-e-2',
+ model="dall-e-2",
n=1,
- size="256x256"
+ size="256x256",
)
assert image_variation_resp.created is not None
+
def test_sync_openai_audio_speech_create():
"""
Tests synchronous audio speech generation with the 'tts-1' model.
@@ -143,11 +155,11 @@ def test_sync_openai_audio_speech_create():
"""
audio_speech_resp = sync_client.audio.speech.create(
- model='tts-1',
- voice='alloy',
- input='LLM Observability!')
+ model="tts-1", voice="alloy", input="LLM Observability!"
+ )
assert audio_speech_resp is not None and isinstance(audio_speech_resp, object)
+
@pytest.mark.asyncio
async def test_async_openai_chat_completions():
"""
@@ -160,9 +172,10 @@ async def test_async_openai_chat_completions():
chat_completions_resp = await async_client.chat.completions.create(
model="gpt-3.5-turbo",
max_tokens=1,
- messages=[{"role": "user", "content": "What is LLM Observability?"}]
+ messages=[{"role": "user", "content": "What is LLM Observability?"}],
)
- assert chat_completions_resp.object == 'chat.completion'
+ assert chat_completions_resp.object == "chat.completion"
+
@pytest.mark.asyncio
async def test_async_openai_chat_completions_parse():
@@ -175,18 +188,26 @@ async def test_async_openai_chat_completions_parse():
class User(BaseModel):
"""A model to represent a user's details."""
+
name: str
age: int
chat_completions_resp = await async_client.beta.chat.completions.parse(
model="gpt-4o-mini",
messages=[
- {"role": "system", "content": "Extract the user's name and age from the following text."},
- {"role": "user", "content": "The user's name is John Doe and he is 30 years old."},
+ {
+ "role": "system",
+ "content": "Extract the user's name and age from the following text.",
+ },
+ {
+ "role": "user",
+ "content": "The user's name is John Doe and he is 30 years old.",
+ },
],
response_format=User, # Pass the Pydantic model as the response format
)
- assert chat_completions_resp.object == 'chat.completion'
+ assert chat_completions_resp.object == "chat.completion"
+
# @pytest.mark.asyncio
# async def test_async_openai_embeddings():
diff --git a/sdk/python/tests/test_qdrant.py b/sdk/python/tests/test_qdrant.py
index a38e6e6b..2393493b 100644
--- a/sdk/python/tests/test_qdrant.py
+++ b/sdk/python/tests/test_qdrant.py
@@ -3,7 +3,7 @@
This module contains tests for Qdrant functionality using the Qdrant Python library.
Tests cover various API endpoints, including create_collection, add, query,
-upsert, update, get and delete.
+upsert, update, get and delete.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -26,9 +26,12 @@ client = QdrantClient(
)
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-python-testing", application_name="openlit-python-qdrant-test")
+openlit.init(
+ environment="openlit-python-testing", application_name="openlit-python-qdrant-test"
+)
COLLECTION_NAME = "openlit"
+
def test_db_qdrant():
"""
Tests basic operations within a Qdrant collection.
@@ -45,7 +48,7 @@ def test_db_qdrant():
expected document ids.
- A delete operation targets a specific document by id, and successful deletion
is implicitly verified by the absence of an error response.
-
+
The test ensures the basic CRUD operations perform as expected in ChromaDB.
Raises:
AssertionError: If the responses from ChromaDB operations do not meet the expected outcomes.
@@ -61,165 +64,179 @@ def test_db_qdrant():
# Upsert to the collection
upsert = client.upsert(
- collection_name=COLLECTION_NAME,
- wait=True,
- points=[
- PointStruct(id=1, vector=[0.05, 0.61, 0.76, 0.74], payload={"city": "Berlin"}),
- PointStruct(id=2, vector=[0.19, 0.81, 0.75, 0.11], payload={"city": "London"}),
- PointStruct(id=3, vector=[0.36, 0.55, 0.47, 0.94], payload={"city": "Moscow"}),
- PointStruct(id=4, vector=[0.18, 0.01, 0.85, 0.80], payload={"city": "New York"}),
- PointStruct(id=5, vector=[0.24, 0.18, 0.22, 0.44], payload={"city": "Beijing"}),
- PointStruct(id=6, vector=[0.35, 0.08, 0.11, 0.44], payload={"city": "Mumbai"}),
- ],
+ collection_name=COLLECTION_NAME,
+ wait=True,
+ points=[
+ PointStruct(
+ id=1, vector=[0.05, 0.61, 0.76, 0.74], payload={"city": "Berlin"}
+ ),
+ PointStruct(
+ id=2, vector=[0.19, 0.81, 0.75, 0.11], payload={"city": "London"}
+ ),
+ PointStruct(
+ id=3, vector=[0.36, 0.55, 0.47, 0.94], payload={"city": "Moscow"}
+ ),
+ PointStruct(
+ id=4, vector=[0.18, 0.01, 0.85, 0.80], payload={"city": "New York"}
+ ),
+ PointStruct(
+ id=5, vector=[0.24, 0.18, 0.22, 0.44], payload={"city": "Beijing"}
+ ),
+ PointStruct(
+ id=6, vector=[0.35, 0.08, 0.11, 0.44], payload={"city": "Mumbai"}
+ ),
+ ],
)
- assert upsert.status == 'completed'
+ assert upsert.status == "completed"
# Set Payload to the collection
set_payload = client.set_payload(
- collection_name=COLLECTION_NAME,
- payload={
- "city": "Vienna",
- },
- points=[1],
+ collection_name=COLLECTION_NAME,
+ payload={
+ "city": "Vienna",
+ },
+ points=[1],
)
- assert set_payload.status == 'completed'
+ assert set_payload.status == "completed"
# Overwrite Payload to the collection
overwrite_payload = client.overwrite_payload(
- collection_name=COLLECTION_NAME,
- payload={
- "city": "Toronto",
- },
- points=[1],
+ collection_name=COLLECTION_NAME,
+ payload={
+ "city": "Toronto",
+ },
+ points=[1],
)
- assert overwrite_payload.status == 'completed'
+ assert overwrite_payload.status == "completed"
# Clear Payload to the collection
clear_payload = client.clear_payload(
collection_name=COLLECTION_NAME,
points_selector=[1],
)
- assert clear_payload.status == 'completed'
+ assert clear_payload.status == "completed"
# Delete Payload to the collection
delete_payload = client.delete_payload(
- collection_name=COLLECTION_NAME,
- keys=["city"],
- points=[2],
+ collection_name=COLLECTION_NAME,
+ keys=["city"],
+ points=[2],
)
- assert delete_payload.status == 'completed'
+ assert delete_payload.status == "completed"
# Upload Points to the collection
upload_points = client.upload_points(
- collection_name=COLLECTION_NAME,
- points=[
- models.PointStruct(
- id=1,
- payload={
- "city": "Toronto",
- },
- vector=[0.9, 0.1, 0.1, 0.2],
- )
- ],
- max_retries=3,
+ collection_name=COLLECTION_NAME,
+ points=[
+ models.PointStruct(
+ id=1,
+ payload={
+ "city": "Toronto",
+ },
+ vector=[0.9, 0.1, 0.1, 0.2],
+ )
+ ],
+ max_retries=3,
)
assert upload_points is None
# Update Vectors in the collection
update_vectors = client.update_vectors(
- collection_name=COLLECTION_NAME,
- points=[
- models.PointVectors(
- id=1,
- vector=[0.9, 0.1, 0.1, 0.2],
- )
- ],
+ collection_name=COLLECTION_NAME,
+ points=[
+ models.PointVectors(
+ id=1,
+ vector=[0.9, 0.1, 0.1, 0.2],
+ )
+ ],
)
- assert update_vectors.status == 'completed'
+ assert update_vectors.status == "completed"
# Delete Points in the collection
delt = client.delete(
- collection_name=COLLECTION_NAME,
- points_selector=models.PointIdsList(
- points=[2],
- ),
+ collection_name=COLLECTION_NAME,
+ points_selector=models.PointIdsList(
+ points=[2],
+ ),
)
- assert delt.status == 'completed'
+ assert delt.status == "completed"
# Retrieve vectors from the collection
retrieve = client.retrieve(
- collection_name=COLLECTION_NAME,
- ids=[4],
+ collection_name=COLLECTION_NAME,
+ ids=[4],
)
assert isinstance(retrieve, list)
client.create_payload_index(
- collection_name=COLLECTION_NAME,
- field_name="city",
- field_schema="keyword",
+ collection_name=COLLECTION_NAME,
+ field_name="city",
+ field_schema="keyword",
)
# Scroll vectors from the collection
scroll = client.scroll(
- collection_name=COLLECTION_NAME,
- scroll_filter=models.Filter(
- must=[
- models.FieldCondition(key="city", match=models.MatchValue(value="Toronto")),
- ]
- ),
- limit=1,
- with_payload=True,
- with_vectors=False,
+ collection_name=COLLECTION_NAME,
+ scroll_filter=models.Filter(
+ must=[
+ models.FieldCondition(
+ key="city", match=models.MatchValue(value="Toronto")
+ ),
+ ]
+ ),
+ limit=1,
+ with_payload=True,
+ with_vectors=False,
)
assert isinstance(scroll, tuple)
# Search vectors from the collection
search = client.search(
- collection_name=COLLECTION_NAME,
- query_filter=models.Filter(
- must=[
- models.FieldCondition(
- key="city",
- match=models.MatchValue(
- value="London",
- ),
- )
- ]
- ),
- search_params=models.SearchParams(hnsw_ef=128, exact=False),
- query_vector=[0.2, 0.1, 0.9, 0.7],
- limit=3,
+ collection_name=COLLECTION_NAME,
+ query_filter=models.Filter(
+ must=[
+ models.FieldCondition(
+ key="city",
+ match=models.MatchValue(
+ value="London",
+ ),
+ )
+ ]
+ ),
+ search_params=models.SearchParams(hnsw_ef=128, exact=False),
+ query_vector=[0.2, 0.1, 0.9, 0.7],
+ limit=3,
)
assert isinstance(search, list)
# Search groups from the collection
search_groups = client.search_groups(
- collection_name=COLLECTION_NAME,
- query_vector=[1.1, 1.2, 1.3, 1.4],
- group_by="city",
- limit=4,
- group_size=2,
+ collection_name=COLLECTION_NAME,
+ query_vector=[1.1, 1.2, 1.3, 1.4],
+ group_by="city",
+ limit=4,
+ group_size=2,
)
assert isinstance(search_groups.groups, list)
# Get Recommened vectors from the collection
recommend = client.recommend(
- collection_name=COLLECTION_NAME,
- positive=[1, 3],
- negative=[4, [0.2, 0.3, 0.4, 0.5]],
- strategy=models.RecommendStrategy.AVERAGE_VECTOR,
- query_filter=models.Filter(
- must=[
- models.FieldCondition(
- key="city",
- match=models.MatchValue(
- value="London",
- ),
- )
- ]
- ),
- limit=3,
+ collection_name=COLLECTION_NAME,
+ positive=[1, 3],
+ negative=[4, [0.2, 0.3, 0.4, 0.5]],
+ strategy=models.RecommendStrategy.AVERAGE_VECTOR,
+ query_filter=models.Filter(
+ must=[
+ models.FieldCondition(
+ key="city",
+ match=models.MatchValue(
+ value="London",
+ ),
+ )
+ ]
+ ),
+ limit=3,
)
assert isinstance(recommend, list)
diff --git a/sdk/python/tests/test_reka.py b/sdk/python/tests/test_reka.py
index 729ecf8d..f6f8e137 100644
--- a/sdk/python/tests/test_reka.py
+++ b/sdk/python/tests/test_reka.py
@@ -2,7 +2,7 @@
"""
This module contains tests for Reka AI functionality using the Reka Python library.
-Tests cover various API endpoints, including chat.
+Tests cover various API endpoints, including chat.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -24,7 +24,10 @@ sync_client = Reka()
async_client = AsyncReka()
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-python-testing", application_name="openlit-python-reka-test")
+openlit.init(
+ environment="openlit-python-testing", application_name="openlit-python-reka-test"
+)
+
def test_sync_reka_chat():
"""
@@ -45,7 +48,7 @@ def test_sync_reka_chat():
model="reka-core",
max_tokens=1,
)
- assert response.model == 'reka-core'
+ assert response.model == "reka-core"
# pylint: disable=broad-exception-caught
except Exception as e:
@@ -54,6 +57,7 @@ def test_sync_reka_chat():
else:
raise
+
@pytest.mark.asyncio
async def test_async_reka_chat():
"""
@@ -74,7 +78,7 @@ async def test_async_reka_chat():
model="reka-core",
max_tokens=1,
)
- assert response.model == 'reka-core'
+ assert response.model == "reka-core"
# pylint: disable=broad-exception-caught
except Exception as e:
diff --git a/sdk/python/tests/test_together.py b/sdk/python/tests/test_together.py
index 57104682..4096f8ce 100644
--- a/sdk/python/tests/test_together.py
+++ b/sdk/python/tests/test_together.py
@@ -1,7 +1,7 @@
"""
This module contains tests for Together functionality using the Together Python library.
-Tests cover various API endpoints, including chat and embeddings.
+Tests cover various API endpoints, including chat and embeddings.
These tests validate integration with OpenLIT.
Environment Variables:
@@ -22,7 +22,11 @@ sync_client = Together()
async_client = AsyncTogether()
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-python-testing", application_name="openlit-python-together-test")
+openlit.init(
+ environment="openlit-python-testing",
+ application_name="openlit-python-together-test",
+)
+
def test_sync_together_chat():
"""
@@ -33,18 +37,15 @@ def test_sync_together_chat():
"""
try:
- response = sync_client.chat.completions.create(
+ response = sync_client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=[
- {
- "role": "user",
- "content": "Hi"
- },
+ {"role": "user", "content": "Hi"},
],
max_tokens=1,
stream=False,
)
- assert response.model == 'meta-llama/Llama-3.3-70B-Instruct-Turbo'
+ assert response.model == "meta-llama/Llama-3.3-70B-Instruct-Turbo"
except Exception as e:
if "credit_limit" in str(e).lower():
@@ -54,6 +55,7 @@ def test_sync_together_chat():
else:
raise
+
def test_sync_together_image():
"""
Tests synchronous image generate.
@@ -70,7 +72,7 @@ def test_sync_together_image():
height=768,
n=1,
)
- assert response.model == 'black-forest-labs/FLUX.1-dev'
+ assert response.model == "black-forest-labs/FLUX.1-dev"
except Exception as e:
if "credit_limit" in str(e).lower():
@@ -80,6 +82,7 @@ def test_sync_together_image():
else:
raise
+
@pytest.mark.asyncio
async def test_async_together_chat():
"""
@@ -90,18 +93,15 @@ async def test_async_together_chat():
"""
try:
- response = await async_client.chat.completions.create(
+ response = await async_client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=[
- {
- "role": "user",
- "content": "Hi"
- },
+ {"role": "user", "content": "Hi"},
],
max_tokens=1,
stream=False,
)
- assert response.model == 'meta-llama/Llama-3.3-70B-Instruct-Turbo'
+ assert response.model == "meta-llama/Llama-3.3-70B-Instruct-Turbo"
except Exception as e:
if "credit_limit" in str(e).lower():
@@ -111,6 +111,7 @@ async def test_async_together_chat():
else:
raise
+
@pytest.mark.asyncio
async def test_async_together_image():
"""
@@ -128,7 +129,7 @@ async def test_async_together_image():
height=768,
n=1,
)
- assert response.model == 'black-forest-labs/FLUX.1-dev'
+ assert response.model == "black-forest-labs/FLUX.1-dev"
except Exception as e:
if "credit_limit" in str(e).lower():
diff --git a/sdk/python/tests/test_transformers.py b/sdk/python/tests/test_transformers.py
index feeb9bd6..0a7fe863 100644
--- a/sdk/python/tests/test_transformers.py
+++ b/sdk/python/tests/test_transformers.py
@@ -2,7 +2,7 @@
"""
This module contains tests for Text Generation functionality in HuggingFace Transformers library.
-Tests cover the usage of the Transformers' pipeline for generating text.
+Tests cover the usage of the Transformers' pipeline for generating text.
These tests validate integration with OpenLIT.
Note: Ensure the environment is properly configured for Transformers and OpenLIT monitoring
@@ -15,18 +15,22 @@ import openlit
# Initialize environment and application name for OpenLIT monitoring
openlit.init(environment="openlit-testing", application_name="openlit-python-test")
+
def test_text_transformers():
"""
Test text generation capabilities from HuggingFace Transformers library.
"""
- pipe= pipeline(task="text-generation", model="Qwen/Qwen2.5-1.5B")
+ pipe = pipeline(task="text-generation", model="Qwen/Qwen2.5-1.5B")
response = pipe("LLM Observability")
assert isinstance(response[0]["generated_text"], str)
chat = [
- {"role": "system", "content": "You are an OpenTelemetry AI Observability expert"},
- {"role": "user", "content": "What is Agent Observability?"}
+ {
+ "role": "system",
+ "content": "You are an OpenTelemetry AI Observability expert",
+ },
+ {"role": "user", "content": "What is Agent Observability?"},
]
response = pipe(chat, max_new_tokens=100)
diff --git a/sdk/python/tests/test_xai.py b/sdk/python/tests/test_xai.py
index 5b7eea86..d87b8d1e 100644
--- a/sdk/python/tests/test_xai.py
+++ b/sdk/python/tests/test_xai.py
@@ -31,7 +31,10 @@ async_client = AsyncOpenAI(
)
# Initialize environment and application name for OpenLIT monitoring
-openlit.init(environment="openlit-python-testing", application_name="openlit-python-xai-test")
+openlit.init(
+ environment="openlit-python-testing", application_name="openlit-python-xai-test"
+)
+
def test_sync_xai_chat_completions():
"""
@@ -49,7 +52,7 @@ def test_sync_xai_chat_completions():
],
max_tokens=1,
)
- assert response.object == 'chat.completion'
+ assert response.object == "chat.completion"
# pylint: disable=broad-exception-caught
except Exception as e:
@@ -58,6 +61,7 @@ def test_sync_xai_chat_completions():
else:
raise
+
@pytest.mark.asyncio
async def test_async_xai_chat_completions():
"""
@@ -75,7 +79,7 @@ async def test_async_xai_chat_completions():
],
max_tokens=1,
)
- assert response.object == 'chat.completion'
+ assert response.object == "chat.completion"
# pylint: disable=broad-exception-caught
except Exception as e:
diff --git a/src/client/package-lock.json b/src/client/package-lock.json
index fc6e519b..7765f50f 100644
--- a/src/client/package-lock.json
+++ b/src/client/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "openlit",
- "version": "1.14.1",
+ "version": "1.14.2",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "openlit",
- "version": "1.14.1",
+ "version": "1.14.2",
"dependencies": {
"@anthropic-ai/sdk": "^0.21.1",
"@clickhouse/client": "^1.1.0",
diff --git a/src/client/package.json b/src/client/package.json
index 12582d0d..e48ccbcc 100644
--- a/src/client/package.json
+++ b/src/client/package.json
@@ -1,6 +1,6 @@
{
"name": "openlit",
- "version": "1.14.1",
+ "version": "1.14.2",
"private": true,
"scripts": {
"dev": "next dev",
diff --git a/src/client/scripts/evaluation/evaluate.py b/src/client/scripts/evaluation/evaluate.py
index ba146a80..293b0170 100644
--- a/src/client/scripts/evaluation/evaluate.py
+++ b/src/client/scripts/evaluation/evaluate.py
@@ -100,8 +100,10 @@ import litellm
import json
import sys
-def get_system_prompt(threshold_score: float,
- prompt: str, contexts: str, response: str) -> str:
+
+def get_system_prompt(
+ threshold_score: float, prompt: str, contexts: str, response: str
+) -> str:
return f"""
Output Requirement: List of JSON Objects in JSON array
@@ -192,7 +194,7 @@ def llm_response(prompt: str, model: str, api_key=str):
{"role": "user", "content": prompt},
],
temperature=0.0,
- response_format={ "type": "json_object" }
+ response_format={"type": "json_object"},
)
try:
@@ -201,17 +203,23 @@ def llm_response(prompt: str, model: str, api_key=str):
parsed_content = json.loads(content)
return parsed_content
except Exception as e:
- return {
- "success": False,
- "error": str(e)
- }
-
-def measure(api_key: [str], model: Optional[str] = "openai/gpt-4o",
- prompt: Optional[str] = None,
- contexts: Optional[List[str]] = None,
- response: Optional[str] = None, threshold_score: Optional[float] = 0.5):
-
- llm_prompt = get_system_prompt(threshold_score=threshold_score, prompt=prompt, contexts=contexts, response=response)
+ return {"success": False, "error": str(e)}
+
+
+def measure(
+ api_key: [str],
+ model: Optional[str] = "openai/gpt-4o",
+ prompt: Optional[str] = None,
+ contexts: Optional[List[str]] = None,
+ response: Optional[str] = None,
+ threshold_score: Optional[float] = 0.5,
+):
+ llm_prompt = get_system_prompt(
+ threshold_score=threshold_score,
+ prompt=prompt,
+ contexts=contexts,
+ response=response,
+ )
response = llm_response(llm_prompt, model, api_key)
return response
@@ -221,31 +229,26 @@ def main():
try:
# Read JSON input from stdin
input_json = json.loads(sys.argv[1])
-
+
# Call measure function with parameters from JSON
result = measure(
- api_key=input_json.get('api_key'),
- model=input_json.get('model'),
- prompt=input_json.get('prompt'),
- contexts=input_json.get('contexts'),
- response=input_json.get('response'),
- threshold_score=input_json.get('threshold_score', 0.5)
+ api_key=input_json.get("api_key"),
+ model=input_json.get("model"),
+ prompt=input_json.get("prompt"),
+ contexts=input_json.get("contexts"),
+ response=input_json.get("response"),
+ threshold_score=input_json.get("threshold_score", 0.5),
)
-
+
# print(input_json)
# Print result as JSON string
print(json.dumps(result))
except json.JSONDecodeError:
- print(json.dumps({
- "success": False,
- "error": "Invalid JSON input"
- }))
+ print(json.dumps({"success": False, "error": "Invalid JSON input"}))
except Exception as e:
- print(json.dumps({
- "success": False,
- "error": str(e)
- }))
+ print(json.dumps({"success": False, "error": str(e)}))
+
if __name__ == "__main__":
main()
diff --git a/temp_analysis/logfire b/temp_analysis/logfire
deleted file mode 160000
index da7c0802..00000000
--- a/temp_analysis/logfire
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit da7c0802ac6dbe93a315c0accb0b0efe8d84d17f
diff --git a/temp_analysis/openinference b/temp_analysis/openinference
deleted file mode 160000
index 0864c13f..00000000
--- a/temp_analysis/openinference
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit 0864c13fdfa9e289468ac0a79a2860a155be46de
diff --git a/temp_analysis/openllmetry b/temp_analysis/openllmetry
deleted file mode 160000
index a7509447..00000000
--- a/temp_analysis/openllmetry
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit a750944729721f6b0e4f8873a69635c5a564c9a2
diff --git a/temp_analysis/pydantic-ai b/temp_analysis/pydantic-ai
deleted file mode 160000
index babc23b3..00000000
--- a/temp_analysis/pydantic-ai
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit babc23b30a94cbfa7d6ee9cebba25309b0cf8360