Skip to content

Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.

[Feat]: Add OpenTelemetry Instrumentation for monitoring Azure AI Inference models and GitHub Models (#408)

patcher94696f4d5d4af
diff --git a/docs/images/docs-github-models-banner.gif b/docs/images/docs-github-models-banner.gif
new file mode 100644
index 00000000..e8f06ecf
Binary files /dev/null and b/docs/images/docs-github-models-banner.gif differ
diff --git a/docs/latest/integrations/azure-ai-inference.mdx b/docs/latest/integrations/azure-ai-inference.mdx
new file mode 100644
index 00000000..40553cfb
--- /dev/null
+++ b/docs/latest/integrations/azure-ai-inference.mdx
@@ -0,0 +1,78 @@
+---
+title: 'Monitor Azure AI Inference using OpenTelemetry'
+sidebarTitle: 'Azure AI Inference'
+---
+
+<Frame>
+ <img src="/images/docs-azure-banner.gif" />
+</Frame>
+
+OpenLIT uses OpenTelemetry Auto-Instrumentation to help you monitor LLM applications built using Azure Azure AI Inference models. This includes tracking performance, token usage, costs, and how users interact with the application.
+
+Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up.
+
+The integration is compatible with
+- azure-ai-inference Python SDK client `>=1.0.0b4`
+
+## Get Started
+
+<Steps>
+ <Step title="Install OpenLIT">
+ Open your command line or terminal and run:
+ ```shell
+ pip install openlit
+ ```
+ </Step>
+ <Step title="Initialize OpenLIT in your Application">
+ You can set up OpenLIT in your application using either function arguments directly in your code or by using environment variables.
+ <Tabs>
+ <Tab title="Setup using function arguments">
+
+ Add the following two lines to your application code:
+
+ ```python
+ import openlit
+
+ openlit.init(
+ otlp_endpoint="YOUR_OTEL_ENDPOINT",
+ )
+ ```
+
+ Replace:
+ 1. `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` if you are using OpenLIT and a local OTel Collector.
+ </Tab>
+ <Tab title="Setup using Environment Variables">
+
+ Add the following two lines to your application code:
+
+ ```python
+ import openlit
+
+ openlit.init()
+ ```
+
+ Then, configure the your OTLP endpoint using environment variable:
+ ```shell
+ export OTEL_EXPORTER_OTLP_ENDPOINT = "YOUR_OTEL_ENDPOINT"
+ ```
+
+ Replace:
+ 1. `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` if you are using OpenLIT and a local OTel Collector.
+ </Tab>
+ </Tabs>
+ To send metrics and traces to other Observability tools, refer to the [Connections Guide](/latest/connections/intro).
+
+ For more advanced configurations and application use cases, visit the [OpenLIT Python repository](https://github.com/openlit/openlit/tree/main/sdk/python).
+ </Step>
+</Steps>
+
+---
+
+<CardGroup cols={2}>
+<Card title="Connections" href="/latest/connections/intro" icon='link'>
+Connect to your existing Observablity Stack
+</Card>
+<Card title="SDK configuration" href="/latest/sdk-configuration" icon='file-code'>
+Documentation of the configuration options for the OpenLIT SDK.
+</Card>
+</CardGroup>
\ No newline at end of file
diff --git a/docs/latest/integrations/github-models.mdx b/docs/latest/integrations/github-models.mdx
new file mode 100644
index 00000000..95ec76a8
--- /dev/null
+++ b/docs/latest/integrations/github-models.mdx
@@ -0,0 +1,78 @@
+---
+title: 'Monitor GitHub Models using OpenTelemetry'
+sidebarTitle: 'GitHub Models'
+---
+
+<Frame>
+ <img src="/images/docs-github-models-banner.gif" />
+</Frame>
+
+OpenLIT uses OpenTelemetry Auto-Instrumentation to help you monitor LLM applications built using GitHub AI Models. This includes tracking performance, token usage, costs, and how users interact with the application.
+
+Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up.
+
+The integration is compatible with
+- azure-ai-inference Python SDK client `>=1.0.0b4`
+
+## Get Started
+
+<Steps>
+ <Step title="Install OpenLIT">
+ Open your command line or terminal and run:
+ ```shell
+ pip install openlit
+ ```
+ </Step>
+ <Step title="Initialize OpenLIT in your Application">
+ You can set up OpenLIT in your application using either function arguments directly in your code or by using environment variables.
+ <Tabs>
+ <Tab title="Setup using function arguments">
+
+ Add the following two lines to your application code:
+
+ ```python
+ import openlit
+
+ openlit.init(
+ otlp_endpoint="YOUR_OTEL_ENDPOINT",
+ )
+ ```
+
+ Replace:
+ 1. `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` if you are using OpenLIT and a local OTel Collector.
+ </Tab>
+ <Tab title="Setup using Environment Variables">
+
+ Add the following two lines to your application code:
+
+ ```python
+ import openlit
+
+ openlit.init()
+ ```
+
+ Then, configure the your OTLP endpoint using environment variable:
+ ```shell
+ export OTEL_EXPORTER_OTLP_ENDPOINT = "YOUR_OTEL_ENDPOINT"
+ ```
+
+ Replace:
+ 1. `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` if you are using OpenLIT and a local OTel Collector.
+ </Tab>
+ </Tabs>
+ To send metrics and traces to other Observability tools, refer to the [Connections Guide](/latest/connections/intro).
+
+ For more advanced configurations and application use cases, visit the [OpenLIT Python repository](https://github.com/openlit/openlit/tree/main/sdk/python).
+ </Step>
+</Steps>
+
+---
+
+<CardGroup cols={2}>
+<Card title="Connections" href="/latest/connections/intro" icon='link'>
+Connect to your existing Observablity Stack
+</Card>
+<Card title="SDK configuration" href="/latest/sdk-configuration" icon='file-code'>
+Documentation of the configuration options for the OpenLIT SDK.
+</Card>
+</CardGroup>
\ No newline at end of file
diff --git a/docs/latest/integrations/introduction.mdx b/docs/latest/integrations/introduction.mdx
index 92110885..c73fc1ba 100644
--- a/docs/latest/integrations/introduction.mdx
+++ b/docs/latest/integrations/introduction.mdx
@@ -282,6 +282,15 @@ Start integrating and monitoring your applications with OpenLIT effortlessly. Ch
iconType="duotone"
>
</Card>
+ <Card
+ title="GitHub Models"
+ href="/latest/integrations/github-models"
+ icon={
+ <svg viewBox="0 0 122 122" xmlns="http://www.w3.org/2000/svg"><path fill-rule="evenodd" clip-rule="evenodd" d="M48.854 0C21.839 0 0 22 0 49.217c0 21.756 13.993 40.172 33.405 46.69 2.427.49 3.316-1.059 3.316-2.362 0-1.141-.08-5.052-.08-9.127-13.59 2.934-16.42-5.867-16.42-5.867-2.184-5.704-5.42-7.17-5.42-7.17-4.448-3.015.324-3.015.324-3.015 4.934.326 7.523 5.052 7.523 5.052 4.367 7.496 11.404 5.378 14.235 4.074.404-3.178 1.699-5.378 3.074-6.6-10.839-1.141-22.243-5.378-22.243-24.283 0-5.378 1.94-9.778 5.014-13.2-.485-1.222-2.184-6.275.486-13.038 0 0 4.125-1.304 13.426 5.052a46.97 46.97 0 0 1 12.214-1.63c4.125 0 8.33.571 12.213 1.63 9.302-6.356 13.427-5.052 13.427-5.052 2.67 6.763.97 11.816.485 13.038 3.155 3.422 5.015 7.822 5.015 13.2 0 18.905-11.404 23.06-22.324 24.283 1.78 1.548 3.316 4.481 3.316 9.126 0 6.6-.08 11.897-.08 13.526 0 1.304.89 2.853 3.316 2.364 19.412-6.52 33.405-24.935 33.405-46.691C97.707 22 75.788 0 48.854 0z" fill="#24292f"/></svg>
+ }
+ iconType="duotone"
+ >
+ </Card>
<Card
title="Azure OpenAI"
href="/latest/integrations/azure-openai"
@@ -291,6 +300,15 @@ Start integrating and monitoring your applications with OpenLIT effortlessly. Ch
iconType="duotone"
>
</Card>
+ <Card
+ title="Azure AI Inference"
+ href="/latest/integrations/azure-ai-inference"
+ icon={
+ <svg viewBox="0 0 512 512" xmlns="http://www.w3.org/2000/svg" fill-rule="evenodd" clip-rule="evenodd" stroke-linejoin="round" stroke-miterlimit="2"><g fill-rule="nonzero"><path d="M52.091 10.225h40.684L50.541 135.361a6.5 6.5 0 01-6.146 4.412H12.732c-3.553 0-6.477-2.923-6.477-6.476 0-.704.115-1.403.34-2.07L45.944 14.638a6.501 6.501 0 016.147-4.415v.002z" fill="url(#prefix___Linear1)" transform="translate(2.076 1.626) scale(3.37462)"/><path d="M377.371 319.374H159.644c-5.527 0-10.076 4.549-10.076 10.077 0 2.794 1.164 5.466 3.206 7.37l139.901 130.577a21.986 21.986 0 0015.004 5.91H430.96l-53.589-153.934z" fill="#0078d4"/><path d="M52.091 10.225a6.447 6.447 0 00-6.161 4.498L6.644 131.12a6.457 6.457 0 00-.38 2.185c0 3.548 2.92 6.468 6.469 6.468H45.23a6.95 6.95 0 005.328-4.531l7.834-23.089 27.985 26.102a6.622 6.622 0 004.165 1.518h36.395l-15.962-45.615-46.533.011 28.48-83.944H52.091z" fill="url(#prefix___Linear2)" transform="translate(2.076 1.626) scale(3.37462)"/><path d="M104.055 14.631a6.492 6.492 0 00-6.138-4.406H52.575a6.493 6.493 0 016.138 4.406l39.35 116.594c.225.668.34 1.367.34 2.072 0 3.554-2.924 6.478-6.478 6.478h45.344c3.553-.001 6.476-2.925 6.476-6.478 0-.705-.115-1.404-.34-2.072l-39.35-116.594z" fill="url(#prefix___Linear3)" transform="translate(2.076 1.626) scale(3.37462)"/></g><defs><linearGradient id="prefix___Linear1" x1="0" y1="0" x2="1" y2="0" gradientUnits="userSpaceOnUse" gradientTransform="rotate(108.701 26.35 33.911) scale(131.7791)"><stop offset="0" stop-color="#114a8b"/><stop offset="1" stop-color="#0669bc"/></linearGradient><linearGradient id="prefix___Linear2" x1="0" y1="0" x2="1" y2="0" gradientUnits="userSpaceOnUse" gradientTransform="rotate(161.318 33.644 45.587) scale(10.31703)"><stop offset="0" stop-opacity=".3"/><stop offset=".07" stop-opacity=".2"/><stop offset=".32" stop-opacity=".1"/><stop offset=".62" stop-opacity=".05"/><stop offset="1" stop-opacity="0"/></linearGradient><linearGradient id="prefix___Linear3" x1="0" y1="0" x2="1" y2="0" gradientUnits="userSpaceOnUse" gradientTransform="rotate(69.426 25.69 62.036) scale(131.9816)"><stop offset="0" stop-color="#3ccbf4"/><stop offset="1" stop-color="#2892df"/></linearGradient></defs></svg>
+ }
+ iconType="duotone"
+ >
+ </Card>
<Card
title="Mistral"
href="/latest/integrations/mistral"
@@ -925,6 +943,9 @@ Start integrating and monitoring your applications with OpenLIT effortlessly. Ch
| GPT4All | LLM | ✔️ | ❌ |
| vLLM | LLM | ✔️ | ❌ |
| Azure OpenAI | LLM | ✔️ | ❌ |
+| Azure AI Inference | LLM | ✔️ | ❌ |
+| Google AI Studio | LLM | ✔️ | ❌ |
+| GitHub Models | LLM | ✔️ | ❌ |
| Mistral | LLM | ✔️ | ❌ |
| HuggingFace | LLM | ✔️ | ❌ |
| Amazon Bedrock | LLM | ✔️ | ❌ |
diff --git a/docs/mint.json b/docs/mint.json
index 705dcf9a..8427ec24 100644
--- a/docs/mint.json
+++ b/docs/mint.json
@@ -71,7 +71,9 @@
"latest/integrations/gpt4all",
"latest/integrations/cohere",
"latest/integrations/mistral",
+ "latest/integrations/github-models",
"latest/integrations/azure-openai",
+ "latest/integrations/azure-ai-inference",
"latest/integrations/huggingface",
"latest/integrations/bedrock",
"latest/integrations/vertexai",
diff --git a/sdk/python/README.md b/sdk/python/README.md
index b4ecc100..36153a3c 100644
--- a/sdk/python/README.md
+++ b/sdk/python/README.md
@@ -27,23 +27,25 @@ This project adheres to the [Semantic Conventions](https://github.com/open-telem
## Auto Instrumentation Capabilities
-| LLMs | Vector DBs | Frameworks | GPUs |
-|-----------------------------------------------------------------|----------------------------------------------|----------------------------------------------|---------------|
-| [✅ OpenAI](https://docs.openlit.io/latest/integrations/openai) | [✅ ChromaDB](https://docs.openlit.io/latest/integrations/chromadb) | [✅ Langchain](https://docs.openlit.io/latest/integrations/langchain) | [✅ NVIDIA GPUs](https://docs.openlit.io/latest/integrations/nvidia-gpu) |
-| [✅ Ollama](https://docs.openlit.io/latest/integrations/ollama) | [✅ Pinecone](https://docs.openlit.io/latest/integrations/pinecone) | [✅ LiteLLM](https://docs.openlit.io/latest/integrations/litellm) | |
-| [✅ Anthropic](https://docs.openlit.io/latest/integrations/anthropic) | [✅ Qdrant](https://docs.openlit.io/latest/integrations/qdrant) | [✅ LlamaIndex](https://docs.openlit.io/latest/integrations/llama-index) | |
-| [✅ GPT4All](https://docs.openlit.io/latest/integrations/gpt4all) | [✅ Milvus](https://docs.openlit.io/latest/integrations/milvus) | [✅ Haystack](https://docs.openlit.io/latest/integrations/haystack) | |
-| [✅ Cohere](https://docs.openlit.io/latest/integrations/cohere) | | [✅ EmbedChain](https://docs.openlit.io/latest/integrations/embedchain) | |
-| [✅ Mistral](https://docs.openlit.io/latest/integrations/mistral) | | [✅ Guardrails](https://docs.openlit.io/latest/integrations/guardrails) | |
-| [✅ Azure OpenAI](https://docs.openlit.io/latest/integrations/azure-openai) | | | |
-| [✅ HuggingFace Transformers](https://docs.openlit.io/latest/integrations/huggingface) | | | |
-| [✅ Amazon Bedrock](https://docs.openlit.io/latest/integrations/bedrock) | | | |
-| [✅ Vertex AI](https://docs.openlit.io/latest/integrations/vertexai) | | | |
-| [✅ Groq](https://docs.openlit.io/latest/integrations/groq) | | | |
-| [✅ ElevenLabs](https://docs.openlit.io/latest/integrations/elevenlabs) | | | |
-| [✅ vLLM](https://docs.openlit.io/latest/integrations/vllm) | | | |
-| [✅ OLA Krutrim](https://docs.openlit.io/latest/integrations/krutrim) | | | |
-| [✅ Google AI Studio](https://docs.openlit.io/latest/integrations/google-ai-studio) | | | |
+| LLMs | Vector DBs | Frameworks | GPUs |
+|--------------------------------------------------------------------------|----------------------------------------------|----------------------------------------------|---------------|
+| [✅ OpenAI](https://docs.openlit.io/latest/integrations/openai) | [✅ ChromaDB](https://docs.openlit.io/latest/integrations/chromadb) | [✅ Langchain](https://docs.openlit.io/latest/integrations/langchain) | [✅ NVIDIA GPUs](https://docs.openlit.io/latest/integrations/nvidia-gpu) |
+| [✅ Ollama](https://docs.openlit.io/latest/integrations/ollama) | [✅ Pinecone](https://docs.openlit.io/latest/integrations/pinecone) | [✅ LiteLLM](https://docs.openlit.io/latest/integrations/litellm) | |
+| [✅ Anthropic](https://docs.openlit.io/latest/integrations/anthropic) | [✅ Qdrant](https://docs.openlit.io/latest/integrations/qdrant) | [✅ LlamaIndex](https://docs.openlit.io/latest/integrations/llama-index) | |
+| [✅ GPT4All](https://docs.openlit.io/latest/integrations/gpt4all) | [✅ Milvus](https://docs.openlit.io/latest/integrations/milvus) | [✅ Haystack](https://docs.openlit.io/latest/integrations/haystack) | |
+| [✅ Cohere](https://docs.openlit.io/latest/integrations/cohere) | | [✅ EmbedChain](https://docs.openlit.io/latest/integrations/embedchain) | |
+| [✅ Mistral](https://docs.openlit.io/latest/integrations/mistral) | | [✅ Guardrails](https://docs.openlit.io/latest/integrations/guardrails) | |
+| [✅ Azure OpenAI](https://docs.openlit.io/latest/integrations/azure-openai) | | | |
+| [✅ Azure AI Inference](https://docs.openlit.io/latest/integrations/azure-ai-inference) | | | |
+| [✅ GitHub AI Models](https://docs.openlit.io/latest/integrations/github-models) | | | |
+| [✅ HuggingFace Transformers](https://docs.openlit.io/latest/integrations/huggingface) | | | |
+| [✅ Amazon Bedrock](https://docs.openlit.io/latest/integrations/bedrock) | | | |
+| [✅ Vertex AI](https://docs.openlit.io/latest/integrations/vertexai) | | | |
+| [✅ Groq](https://docs.openlit.io/latest/integrations/groq) | | | |
+| [✅ ElevenLabs](https://docs.openlit.io/latest/integrations/elevenlabs) | | | |
+| [✅ vLLM](https://docs.openlit.io/latest/integrations/vllm) | | | |
+| [✅ OLA Krutrim](https://docs.openlit.io/latest/integrations/krutrim) | | | |
+| [✅ Google AI Studio](https://docs.openlit.io/latest/integrations/google-ai-studio) | | | |
## Supported Destinations
- [✅ OpenTelemetry Collector](https://docs.openlit.io/latest/connections/otelcol)
diff --git a/sdk/python/pyproject.toml b/sdk/python/pyproject.toml
index 57fe9e15..931a6560 100644
--- a/sdk/python/pyproject.toml
+++ b/sdk/python/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "openlit"
-version = "1.21.0"
+version = "1.22.0"
description = "OpenTelemetry-native Auto instrumentation library for monitoring LLM Applications and GPUs, facilitating the integration of observability into your GenAI-driven projects"
authors = ["OpenLIT"]
repository = "https://github.com/openlit/openlit/tree/main/openlit/python"
diff --git a/sdk/python/src/openlit/__init__.py b/sdk/python/src/openlit/__init__.py
index ba257b74..bc1c648c 100644
--- a/sdk/python/src/openlit/__init__.py
+++ b/sdk/python/src/openlit/__init__.py
@@ -34,6 +34,7 @@ from openlit.instrumentation.gpt4all import GPT4AllInstrumentor
from openlit.instrumentation.elevenlabs import ElevenLabsInstrumentor
from openlit.instrumentation.vllm import VLLMInstrumentor
from openlit.instrumentation.google_ai_studio import GoogleAIStudioInstrumentor
+from openlit.instrumentation.azure_ai_inference import AzureAIInferenceInstrumentor
from openlit.instrumentation.langchain import LangChainInstrumentor
from openlit.instrumentation.llamaindex import LlamaIndexInstrumentor
from openlit.instrumentation.haystack import HaystackInstrumentor
@@ -198,6 +199,7 @@ def init(environment="default", application_name="default", tracer=None, otlp_en
"elevenlabs": "elevenlabs",
"vllm": "vllm",
"google-ai-studio": "google.generativeai",
+ "azure-ai-inference": "azure.ai.inference",
"langchain": "langchain",
"llama_index": "llama_index",
"haystack": "haystack",
@@ -276,6 +278,7 @@ def init(environment="default", application_name="default", tracer=None, otlp_en
"elevenlabs": ElevenLabsInstrumentor(),
"vllm": VLLMInstrumentor(),
"google-ai-studio": GoogleAIStudioInstrumentor(),
+ "azure-ai-inference": AzureAIInferenceInstrumentor(),
"langchain": LangChainInstrumentor(),
"llama_index": LlamaIndexInstrumentor(),
"haystack": HaystackInstrumentor(),
diff --git a/sdk/python/src/openlit/instrumentation/azure_ai_inference/__init__.py b/sdk/python/src/openlit/instrumentation/azure_ai_inference/__init__.py
new file mode 100644
index 00000000..81fcf234
--- /dev/null
+++ b/sdk/python/src/openlit/instrumentation/azure_ai_inference/__init__.py
@@ -0,0 +1,71 @@
+# pylint: disable=useless-return, bad-staticmethod-argument, disable=duplicate-code
+"""Initializer of Auto Instrumentation of Azure AI Inference Functions"""
+
+from typing import Collection
+import importlib.metadata
+from opentelemetry.instrumentation.instrumentor import BaseInstrumentor
+from wrapt import wrap_function_wrapper
+
+from openlit.instrumentation.azure_ai_inference.azure_ai_inference import (
+ complete, embedding
+)
+
+from openlit.instrumentation.azure_ai_inference.async_azure_ai_inference import (
+ async_complete, async_embedding
+)
+
+_instruments = ("azure-ai-inference >= 1.0.0b4",)
+
+class AzureAIInferenceInstrumentor(BaseInstrumentor):
+ """
+ An instrumentor for azure-ai-inference's client library.
+ """
+
+ def instrumentation_dependencies(self) -> Collection[str]:
+ return _instruments
+
+ def _instrument(self, **kwargs):
+ application_name = kwargs.get("application_name", "default_application")
+ environment = kwargs.get("environment", "default_environment")
+ tracer = kwargs.get("tracer")
+ metrics = kwargs.get("metrics_dict")
+ pricing_info = kwargs.get("pricing_info", {})
+ trace_content = kwargs.get("trace_content", False)
+ disable_metrics = kwargs.get("disable_metrics")
+ version = importlib.metadata.version("azure-ai-inference")
+
+ # sync generate
+ wrap_function_wrapper(
+ "azure.ai.inference",
+ "ChatCompletionsClient.complete",
+ complete("azure_ai.complete", version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics),
+ )
+
+ # sync embedding
+ wrap_function_wrapper(
+ "azure.ai.inference",
+ "EmbeddingsClient.embed",
+ embedding("azure_ai.embed", version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics),
+ )
+
+ # async generate
+ wrap_function_wrapper(
+ "azure.ai.inference.aio",
+ "ChatCompletionsClient.complete",
+ async_complete("azure_ai.complete", version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics),
+ )
+
+ # async embedding
+ wrap_function_wrapper(
+ "azure.ai.inference.aio",
+ "EmbeddingsClient.embed",
+ async_embedding("azure_ai.embed", version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics),
+ )
+
+ def _uninstrument(self, **kwargs):
+ # Proper uninstrumentation logic to revert patched methods
+ pass
diff --git a/sdk/python/src/openlit/instrumentation/azure_ai_inference/async_azure_ai_inference.py b/sdk/python/src/openlit/instrumentation/azure_ai_inference/async_azure_ai_inference.py
new file mode 100644
index 00000000..0c60be52
--- /dev/null
+++ b/sdk/python/src/openlit/instrumentation/azure_ai_inference/async_azure_ai_inference.py
@@ -0,0 +1,432 @@
+# pylint: disable=duplicate-code, broad-exception-caught, too-many-statements, unused-argument, possibly-used-before-assignment, protected-access
+"""
+Module for monitoring Azure AI Inference API calls.
+"""
+
+import logging
+from opentelemetry.trace import SpanKind, Status, StatusCode
+from opentelemetry.sdk.resources import TELEMETRY_SDK_NAME
+from openlit.__helpers import (
+ handle_exception,
+ get_chat_model_cost,
+ get_embed_model_cost,
+ general_tokens
+)
+from openlit.semcov import SemanticConvetion
+
+# Initialize logger for logging potential issues and operations
+logger = logging.getLogger(__name__)
+
+def async_complete(gen_ai_endpoint, version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics):
+ """
+ Generates a telemetry wrapper for chat to collect metrics.
+
+ Args:
+ gen_ai_endpoint: Endpoint identifier for logging and tracing.
+ version: Version of the monitoring package.
+ environment: Deployment environment (e.g., production, staging).
+ application_name: Name of the application using the Azure AI Inference API.
+ tracer: OpenTelemetry tracer for creating spans.
+ pricing_info: Information used for calculating the cost of Azure AI Inference usage.
+ trace_content: Flag indicating whether to trace the actual content.
+
+ Returns:
+ A function that wraps the chat method to add telemetry.
+ """
+
+ async def wrapper(wrapped, instance, args, kwargs):
+ """
+ Wraps the 'chat' API call to add telemetry.
+
+ This collects metrics such as execution time, cost, and token usage, and handles errors
+ gracefully, adding details to the trace for observability.
+
+ Args:
+ wrapped: The original 'chat' method to be wrapped.
+ instance: The instance of the class where the original method is defined.
+ args: Positional arguments for the 'chat' method.
+ kwargs: Keyword arguments for the 'chat' method.
+
+ Returns:
+ The response from the original 'chat' method.
+ """
+ # pylint: disable=no-else-return
+ if kwargs.get("stream", False) is True:
+ # Special handling for streaming response to accommodate the nature of data flow
+ async def stream_generator():
+ with tracer.start_as_current_span(gen_ai_endpoint, kind= SpanKind.CLIENT) as span:
+ # Placeholder for aggregating streaming response
+ llmresponse = ""
+
+ # Loop through streaming events capturing relevant details
+ async for chunk in await wrapped(*args, **kwargs):
+ if chunk.choices:
+ # Collect message IDs and aggregated response from events
+ content = chunk.choices[0].delta.content
+ if content:
+ llmresponse += content
+
+ yield chunk
+ response_id = chunk.id
+
+ # Handling exception ensure observability without disrupting operation
+ try:
+ # Format 'messages' into a single string
+ message_prompt = kwargs.get("messages", "")
+ formatted_messages = []
+ for message in message_prompt:
+ role = message["role"]
+ content = message["content"]
+
+ if isinstance(content, list):
+ content_str = ", ".join(
+ # pylint: disable=line-too-long
+ f'{item["type"]}: {item["text"] if "text" in item else item["image_url"]}'
+ if "type" in item else f'text: {item["text"]}'
+ for item in content
+ )
+ formatted_messages.append(f"{role}: {content_str}")
+ else:
+ formatted_messages.append(f"{role}: {content}")
+ prompt = "\n".join(formatted_messages)
+
+ model = kwargs.get("model", "phi3-mini-4k")
+
+ # Calculate tokens using input prompt and aggregated response
+ input_tokens = general_tokens(prompt)
+ output_tokens = general_tokens(llmresponse)
+
+ total_tokens = input_tokens + output_tokens
+ # Calculate cost of the operation
+ cost = get_chat_model_cost(model,
+ pricing_info, input_tokens,
+ output_tokens)
+
+ # Set base span attribues
+ span.set_attribute(TELEMETRY_SDK_NAME, "openlit")
+ span.set_attribute(SemanticConvetion.GEN_AI_SYSTEM,
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE)
+ span.set_attribute(SemanticConvetion.GEN_AI_TYPE,
+ SemanticConvetion.GEN_AI_TYPE_CHAT)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENDPOINT,
+ gen_ai_endpoint)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENVIRONMENT,
+ environment)
+ span.set_attribute(SemanticConvetion.GEN_AI_APPLICATION_NAME,
+ application_name)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MODEL,
+ model)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_IS_STREAM,
+ True)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_USER,
+ kwargs.get("user", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TOP_P,
+ kwargs.get("top_p", 1.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MAX_TOKENS,
+ kwargs.get("max_tokens", -1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TEMPERATURE,
+ kwargs.get("temperature", 1.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_PRESENCE_PENALTY,
+ kwargs.get("presence_penalty", 0.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_FREQUENCY_PENALTY,
+ kwargs.get("frequency_penalty", 0.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_SEED,
+ kwargs.get("seed", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_ID,
+ response_id)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ input_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COMPLETION_TOKENS,
+ output_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ total_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+
+
+ if trace_content:
+ span.add_event(
+ name=SemanticConvetion.GEN_AI_CONTENT_PROMPT_EVENT,
+ attributes={
+ SemanticConvetion.GEN_AI_CONTENT_PROMPT: prompt,
+ },
+ )
+ span.add_event(
+ name=SemanticConvetion.GEN_AI_CONTENT_COMPLETION_EVENT,
+ attributes={
+ SemanticConvetion.GEN_AI_CONTENT_COMPLETION: llmresponse,
+ },
+ )
+
+ span.set_status(Status(StatusCode.OK))
+
+ if disable_metrics is False:
+ attributes = {
+ TELEMETRY_SDK_NAME:
+ "openlit",
+ SemanticConvetion.GEN_AI_APPLICATION_NAME:
+ application_name,
+ SemanticConvetion.GEN_AI_SYSTEM:
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE,
+ SemanticConvetion.GEN_AI_ENVIRONMENT:
+ environment,
+ SemanticConvetion.GEN_AI_TYPE:
+ SemanticConvetion.GEN_AI_TYPE_CHAT,
+ SemanticConvetion.GEN_AI_REQUEST_MODEL:
+ model
+ }
+
+ metrics["genai_requests"].add(1, attributes)
+ metrics["genai_total_tokens"].add(
+ total_tokens, attributes
+ )
+ metrics["genai_completion_tokens"].add(output_tokens, attributes)
+ metrics["genai_prompt_tokens"].add(input_tokens, attributes)
+ metrics["genai_cost"].record(cost, attributes)
+
+ except Exception as e:
+ handle_exception(span, e)
+ logger.error("Error in trace creation: %s", e)
+
+ return stream_generator()
+ else:
+ # pylint: disable=line-too-long
+ with tracer.start_as_current_span(gen_ai_endpoint, kind= SpanKind.CLIENT) as span:
+ response = await wrapped(*args, **kwargs)
+
+ # print(instance._system_instruction.__dict__["_pb"].parts[0].text)
+ try:
+ # Format 'messages' into a single string
+ message_prompt = kwargs.get("messages", "")
+ formatted_messages = []
+ for message in message_prompt:
+ role = message["role"]
+ content = message["content"]
+
+ if isinstance(content, list):
+ content_str = ", ".join(
+ # pylint: disable=line-too-long
+ f'{item["type"]}: {item["text"] if "text" in item else item["image_url"]}'
+ if "type" in item else f'text: {item["text"]}'
+ for item in content
+ )
+ formatted_messages.append(f"{role}: {content_str}")
+ else:
+ formatted_messages.append(f"{role}: {content}")
+ prompt = "\n".join(formatted_messages)
+
+ model = kwargs.get("model", "phi3-mini-4k")
+
+ # Set base span attribues
+ span.set_attribute(TELEMETRY_SDK_NAME, "openlit")
+ span.set_attribute(SemanticConvetion.GEN_AI_SYSTEM,
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE)
+ span.set_attribute(SemanticConvetion.GEN_AI_TYPE,
+ SemanticConvetion.GEN_AI_TYPE_CHAT)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENDPOINT,
+ gen_ai_endpoint)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENVIRONMENT,
+ environment)
+ span.set_attribute(SemanticConvetion.GEN_AI_APPLICATION_NAME,
+ application_name)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MODEL,
+ model)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_IS_STREAM,
+ False)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_USER,
+ kwargs.get("user", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TOP_P,
+ kwargs.get("top_p", 1.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MAX_TOKENS,
+ kwargs.get("max_tokens", -1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TEMPERATURE,
+ kwargs.get("temperature", 1.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_PRESENCE_PENALTY,
+ kwargs.get("presence_penalty", 0.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_FREQUENCY_PENALTY,
+ kwargs.get("frequency_penalty", 0.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_SEED,
+ kwargs.get("seed", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_ID,
+ response.id)
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_FINISH_REASON,
+ [response.choices[0]["finish_reason"]])
+
+ if trace_content:
+ span.add_event(
+ name=SemanticConvetion.GEN_AI_CONTENT_PROMPT_EVENT,
+ attributes={
+ SemanticConvetion.GEN_AI_CONTENT_PROMPT: prompt,
+ },
+ )
+ span.add_event(
+ name=SemanticConvetion.GEN_AI_CONTENT_COMPLETION_EVENT,
+ attributes={
+ SemanticConvetion.GEN_AI_CONTENT_COMPLETION: response.choices[0].message.content,
+ },
+ )
+
+ input_tokens = response.usage.prompt_tokens
+ output_tokens = response.usage.completion_tokens
+ total_tokens = response.usage.total_tokens
+ # Calculate cost of the operation
+ cost = get_chat_model_cost(model,
+ pricing_info, input_tokens, output_tokens)
+
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ input_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COMPLETION_TOKENS,
+ output_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ total_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+
+ span.set_status(Status(StatusCode.OK))
+
+ if disable_metrics is False:
+ attributes = {
+ TELEMETRY_SDK_NAME:
+ "openlit",
+ SemanticConvetion.GEN_AI_APPLICATION_NAME:
+ application_name,
+ SemanticConvetion.GEN_AI_SYSTEM:
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE,
+ SemanticConvetion.GEN_AI_ENVIRONMENT:
+ environment,
+ SemanticConvetion.GEN_AI_TYPE:
+ SemanticConvetion.GEN_AI_TYPE_CHAT,
+ SemanticConvetion.GEN_AI_REQUEST_MODEL:
+ model
+ }
+
+ metrics["genai_requests"].add(1, attributes)
+ metrics["genai_total_tokens"].add(total_tokens, attributes)
+ metrics["genai_completion_tokens"].add(output_tokens, attributes)
+ metrics["genai_prompt_tokens"].add(input_tokens, attributes)
+ metrics["genai_cost"].record(cost, attributes)
+
+ # Return original response
+ return response
+
+ except Exception as e:
+ handle_exception(span, e)
+ logger.error("Error in trace creation: %s", e)
+
+ # Return original response
+ return response
+
+ return wrapper
+
+def async_embedding(gen_ai_endpoint, version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics):
+ """
+ Generates a telemetry wrapper for embeddings to collect metrics.
+
+ Args:
+ gen_ai_endpoint: Endpoint identifier for logging and tracing.
+ version: Version of the monitoring package.
+ environment: Deployment environment (e.g., production, staging).
+ application_name: Name of the application using the Azure AI Inference API.
+ tracer: OpenTelemetry tracer for creating spans.
+ pricing_info: Information used for calculating the cost of Azure AI Inference usage.
+ trace_content: Flag indicating whether to trace the actual content.
+
+ Returns:
+ A function that wraps the embeddings method to add telemetry.
+ """
+
+ async def wrapper(wrapped, instance, args, kwargs):
+ """
+ Wraps the 'embeddings' API call to add telemetry.
+
+ This collects metrics such as execution time, cost, and token usage, and handles errors
+ gracefully, adding details to the trace for observability.
+
+ Args:
+ wrapped: The original 'embeddings' method to be wrapped.
+ instance: The instance of the class where the original method is defined.
+ args: Positional arguments for the 'embeddings' method.
+ kwargs: Keyword arguments for the 'embeddings' method.
+
+ Returns:
+ The response from the original 'embeddings' method.
+ """
+
+ with tracer.start_as_current_span(gen_ai_endpoint, kind= SpanKind.CLIENT) as span:
+ response = await wrapped(*args, **kwargs)
+
+ try:
+ # Calculate cost of the operation
+ cost = get_embed_model_cost(kwargs.get("model", "text-embedding-ada-002"),
+ pricing_info, response.usage.prompt_tokens)
+
+ # Set Span attributes
+ span.set_attribute(TELEMETRY_SDK_NAME, "openlit")
+ span.set_attribute(SemanticConvetion.GEN_AI_SYSTEM,
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE)
+ span.set_attribute(SemanticConvetion.GEN_AI_TYPE,
+ SemanticConvetion.GEN_AI_TYPE_EMBEDDING)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENDPOINT,
+ gen_ai_endpoint)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENVIRONMENT,
+ environment)
+ span.set_attribute(SemanticConvetion.GEN_AI_APPLICATION_NAME,
+ application_name)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MODEL,
+ kwargs.get("model", "text-embedding-ada-002"))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_EMBEDDING_FORMAT,
+ kwargs.get("encoding_format", "float"))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_EMBEDDING_DIMENSION,
+ kwargs.get("dimensions", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_USER,
+ kwargs.get("user", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ response.usage.prompt_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ response.usage.total_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+ if trace_content:
+ span.add_event(
+ name=SemanticConvetion.GEN_AI_CONTENT_PROMPT_EVENT,
+ attributes={
+ SemanticConvetion.GEN_AI_CONTENT_PROMPT: kwargs.get("input", ""),
+ },
+ )
+
+ span.set_status(Status(StatusCode.OK))
+
+ if disable_metrics is False:
+ attributes = {
+ TELEMETRY_SDK_NAME:
+ "openlit",
+ SemanticConvetion.GEN_AI_APPLICATION_NAME:
+ application_name,
+ SemanticConvetion.GEN_AI_SYSTEM:
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE,
+ SemanticConvetion.GEN_AI_ENVIRONMENT:
+ environment,
+ SemanticConvetion.GEN_AI_TYPE:
+ SemanticConvetion.GEN_AI_TYPE_EMBEDDING,
+ SemanticConvetion.GEN_AI_REQUEST_MODEL:
+ kwargs.get("model", "text-embedding-ada-002")
+ }
+
+ metrics["genai_requests"].add(1, attributes)
+ metrics["genai_total_tokens"].add(response.usage.total_tokens, attributes)
+ metrics["genai_prompt_tokens"].add(response.usage.prompt_tokens, attributes)
+ metrics["genai_cost"].record(cost, attributes)
+
+ # Return original response
+ return response
+
+ except Exception as e:
+ handle_exception(span, e)
+ logger.error("Error in trace creation: %s", e)
+
+ # Return original response
+ return response
+
+ return wrapper
diff --git a/sdk/python/src/openlit/instrumentation/azure_ai_inference/azure_ai_inference.py b/sdk/python/src/openlit/instrumentation/azure_ai_inference/azure_ai_inference.py
new file mode 100644
index 00000000..77c9e8b3
--- /dev/null
+++ b/sdk/python/src/openlit/instrumentation/azure_ai_inference/azure_ai_inference.py
@@ -0,0 +1,432 @@
+# pylint: disable=duplicate-code, broad-exception-caught, too-many-statements, unused-argument, possibly-used-before-assignment, protected-access
+"""
+Module for monitoring Azure AI Inference API calls.
+"""
+
+import logging
+from opentelemetry.trace import SpanKind, Status, StatusCode
+from opentelemetry.sdk.resources import TELEMETRY_SDK_NAME
+from openlit.__helpers import (
+ handle_exception,
+ get_chat_model_cost,
+ get_embed_model_cost,
+ general_tokens
+)
+from openlit.semcov import SemanticConvetion
+
+# Initialize logger for logging potential issues and operations
+logger = logging.getLogger(__name__)
+
+def complete(gen_ai_endpoint, version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics):
+ """
+ Generates a telemetry wrapper for chat to collect metrics.
+
+ Args:
+ gen_ai_endpoint: Endpoint identifier for logging and tracing.
+ version: Version of the monitoring package.
+ environment: Deployment environment (e.g., production, staging).
+ application_name: Name of the application using the Azure AI Inference API.
+ tracer: OpenTelemetry tracer for creating spans.
+ pricing_info: Information used for calculating the cost of Azure AI Inference usage.
+ trace_content: Flag indicating whether to trace the actual content.
+
+ Returns:
+ A function that wraps the chat method to add telemetry.
+ """
+
+ def wrapper(wrapped, instance, args, kwargs):
+ """
+ Wraps the 'chat' API call to add telemetry.
+
+ This collects metrics such as execution time, cost, and token usage, and handles errors
+ gracefully, adding details to the trace for observability.
+
+ Args:
+ wrapped: The original 'chat' method to be wrapped.
+ instance: The instance of the class where the original method is defined.
+ args: Positional arguments for the 'chat' method.
+ kwargs: Keyword arguments for the 'chat' method.
+
+ Returns:
+ The response from the original 'chat' method.
+ """
+ # pylint: disable=no-else-return
+ if kwargs.get("stream", False) is True:
+ # Special handling for streaming response to accommodate the nature of data flow
+ def stream_generator():
+ with tracer.start_as_current_span(gen_ai_endpoint, kind= SpanKind.CLIENT) as span:
+ # Placeholder for aggregating streaming response
+ llmresponse = ""
+
+ # Loop through streaming events capturing relevant details
+ for chunk in wrapped(*args, **kwargs):
+ if chunk.choices:
+ # Collect message IDs and aggregated response from events
+ content = chunk.choices[0].delta.content
+ if content:
+ llmresponse += content
+
+ yield chunk
+ response_id = chunk.id
+
+ # Handling exception ensure observability without disrupting operation
+ try:
+ # Format 'messages' into a single string
+ message_prompt = kwargs.get("messages", "")
+ formatted_messages = []
+ for message in message_prompt:
+ role = message["role"]
+ content = message["content"]
+
+ if isinstance(content, list):
+ content_str = ", ".join(
+ # pylint: disable=line-too-long
+ f'{item["type"]}: {item["text"] if "text" in item else item["image_url"]}'
+ if "type" in item else f'text: {item["text"]}'
+ for item in content
+ )
+ formatted_messages.append(f"{role}: {content_str}")
+ else:
+ formatted_messages.append(f"{role}: {content}")
+ prompt = "\n".join(formatted_messages)
+
+ model = kwargs.get("model", "phi3-mini-4k")
+
+ # Calculate tokens using input prompt and aggregated response
+ input_tokens = general_tokens(prompt)
+ output_tokens = general_tokens(llmresponse)
+
+ total_tokens = input_tokens + output_tokens
+ # Calculate cost of the operation
+ cost = get_chat_model_cost(model,
+ pricing_info, input_tokens,
+ output_tokens)
+
+ # Set base span attribues
+ span.set_attribute(TELEMETRY_SDK_NAME, "openlit")
+ span.set_attribute(SemanticConvetion.GEN_AI_SYSTEM,
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE)
+ span.set_attribute(SemanticConvetion.GEN_AI_TYPE,
+ SemanticConvetion.GEN_AI_TYPE_CHAT)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENDPOINT,
+ gen_ai_endpoint)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENVIRONMENT,
+ environment)
+ span.set_attribute(SemanticConvetion.GEN_AI_APPLICATION_NAME,
+ application_name)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MODEL,
+ model)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_IS_STREAM,
+ True)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_USER,
+ kwargs.get("user", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TOP_P,
+ kwargs.get("top_p", 1.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MAX_TOKENS,
+ kwargs.get("max_tokens", -1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TEMPERATURE,
+ kwargs.get("temperature", 1.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_PRESENCE_PENALTY,
+ kwargs.get("presence_penalty", 0.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_FREQUENCY_PENALTY,
+ kwargs.get("frequency_penalty", 0.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_SEED,
+ kwargs.get("seed", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_ID,
+ response_id)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ input_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COMPLETION_TOKENS,
+ output_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ total_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+
+
+ if trace_content:
+ span.add_event(
+ name=SemanticConvetion.GEN_AI_CONTENT_PROMPT_EVENT,
+ attributes={
+ SemanticConvetion.GEN_AI_CONTENT_PROMPT: prompt,
+ },
+ )
+ span.add_event(
+ name=SemanticConvetion.GEN_AI_CONTENT_COMPLETION_EVENT,
+ attributes={
+ SemanticConvetion.GEN_AI_CONTENT_COMPLETION: llmresponse,
+ },
+ )
+
+ span.set_status(Status(StatusCode.OK))
+
+ if disable_metrics is False:
+ attributes = {
+ TELEMETRY_SDK_NAME:
+ "openlit",
+ SemanticConvetion.GEN_AI_APPLICATION_NAME:
+ application_name,
+ SemanticConvetion.GEN_AI_SYSTEM:
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE,
+ SemanticConvetion.GEN_AI_ENVIRONMENT:
+ environment,
+ SemanticConvetion.GEN_AI_TYPE:
+ SemanticConvetion.GEN_AI_TYPE_CHAT,
+ SemanticConvetion.GEN_AI_REQUEST_MODEL:
+ model
+ }
+
+ metrics["genai_requests"].add(1, attributes)
+ metrics["genai_total_tokens"].add(
+ total_tokens, attributes
+ )
+ metrics["genai_completion_tokens"].add(output_tokens, attributes)
+ metrics["genai_prompt_tokens"].add(input_tokens, attributes)
+ metrics["genai_cost"].record(cost, attributes)
+
+ except Exception as e:
+ handle_exception(span, e)
+ logger.error("Error in trace creation: %s", e)
+
+ return stream_generator()
+ else:
+ # pylint: disable=line-too-long
+ with tracer.start_as_current_span(gen_ai_endpoint, kind= SpanKind.CLIENT) as span:
+ response = wrapped(*args, **kwargs)
+
+ # print(instance._system_instruction.__dict__["_pb"].parts[0].text)
+ try:
+ # Format 'messages' into a single string
+ message_prompt = kwargs.get("messages", "")
+ formatted_messages = []
+ for message in message_prompt:
+ role = message["role"]
+ content = message["content"]
+
+ if isinstance(content, list):
+ content_str = ", ".join(
+ # pylint: disable=line-too-long
+ f'{item["type"]}: {item["text"] if "text" in item else item["image_url"]}'
+ if "type" in item else f'text: {item["text"]}'
+ for item in content
+ )
+ formatted_messages.append(f"{role}: {content_str}")
+ else:
+ formatted_messages.append(f"{role}: {content}")
+ prompt = "\n".join(formatted_messages)
+
+ model = kwargs.get("model", "phi3-mini-4k")
+
+ # Set base span attribues
+ span.set_attribute(TELEMETRY_SDK_NAME, "openlit")
+ span.set_attribute(SemanticConvetion.GEN_AI_SYSTEM,
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE)
+ span.set_attribute(SemanticConvetion.GEN_AI_TYPE,
+ SemanticConvetion.GEN_AI_TYPE_CHAT)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENDPOINT,
+ gen_ai_endpoint)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENVIRONMENT,
+ environment)
+ span.set_attribute(SemanticConvetion.GEN_AI_APPLICATION_NAME,
+ application_name)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MODEL,
+ model)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_IS_STREAM,
+ False)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_USER,
+ kwargs.get("user", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TOP_P,
+ kwargs.get("top_p", 1.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MAX_TOKENS,
+ kwargs.get("max_tokens", -1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TEMPERATURE,
+ kwargs.get("temperature", 1.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_PRESENCE_PENALTY,
+ kwargs.get("presence_penalty", 0.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_FREQUENCY_PENALTY,
+ kwargs.get("frequency_penalty", 0.0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_SEED,
+ kwargs.get("seed", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_ID,
+ response.id)
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_FINISH_REASON,
+ [response.choices[0]["finish_reason"]])
+
+ if trace_content:
+ span.add_event(
+ name=SemanticConvetion.GEN_AI_CONTENT_PROMPT_EVENT,
+ attributes={
+ SemanticConvetion.GEN_AI_CONTENT_PROMPT: prompt,
+ },
+ )
+ span.add_event(
+ name=SemanticConvetion.GEN_AI_CONTENT_COMPLETION_EVENT,
+ attributes={
+ SemanticConvetion.GEN_AI_CONTENT_COMPLETION: response.choices[0].message.content,
+ },
+ )
+
+ input_tokens = response.usage.prompt_tokens
+ output_tokens = response.usage.completion_tokens
+ total_tokens = response.usage.total_tokens
+ # Calculate cost of the operation
+ cost = get_chat_model_cost(model,
+ pricing_info, input_tokens, output_tokens)
+
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ input_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COMPLETION_TOKENS,
+ output_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ total_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+
+ span.set_status(Status(StatusCode.OK))
+
+ if disable_metrics is False:
+ attributes = {
+ TELEMETRY_SDK_NAME:
+ "openlit",
+ SemanticConvetion.GEN_AI_APPLICATION_NAME:
+ application_name,
+ SemanticConvetion.GEN_AI_SYSTEM:
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE,
+ SemanticConvetion.GEN_AI_ENVIRONMENT:
+ environment,
+ SemanticConvetion.GEN_AI_TYPE:
+ SemanticConvetion.GEN_AI_TYPE_CHAT,
+ SemanticConvetion.GEN_AI_REQUEST_MODEL:
+ model
+ }
+
+ metrics["genai_requests"].add(1, attributes)
+ metrics["genai_total_tokens"].add(total_tokens, attributes)
+ metrics["genai_completion_tokens"].add(output_tokens, attributes)
+ metrics["genai_prompt_tokens"].add(input_tokens, attributes)
+ metrics["genai_cost"].record(cost, attributes)
+
+ # Return original response
+ return response
+
+ except Exception as e:
+ handle_exception(span, e)
+ logger.error("Error in trace creation: %s", e)
+
+ # Return original response
+ return response
+
+ return wrapper
+
+def embedding(gen_ai_endpoint, version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics):
+ """
+ Generates a telemetry wrapper for embeddings to collect metrics.
+
+ Args:
+ gen_ai_endpoint: Endpoint identifier for logging and tracing.
+ version: Version of the monitoring package.
+ environment: Deployment environment (e.g., production, staging).
+ application_name: Name of the application using the Azure AI Inference API.
+ tracer: OpenTelemetry tracer for creating spans.
+ pricing_info: Information used for calculating the cost of Azure AI Inference usage.
+ trace_content: Flag indicating whether to trace the actual content.
+
+ Returns:
+ A function that wraps the embeddings method to add telemetry.
+ """
+
+ def wrapper(wrapped, instance, args, kwargs):
+ """
+ Wraps the 'embeddings' API call to add telemetry.
+
+ This collects metrics such as execution time, cost, and token usage, and handles errors
+ gracefully, adding details to the trace for observability.
+
+ Args:
+ wrapped: The original 'embeddings' method to be wrapped.
+ instance: The instance of the class where the original method is defined.
+ args: Positional arguments for the 'embeddings' method.
+ kwargs: Keyword arguments for the 'embeddings' method.
+
+ Returns:
+ The response from the original 'embeddings' method.
+ """
+
+ with tracer.start_as_current_span(gen_ai_endpoint, kind= SpanKind.CLIENT) as span:
+ response = wrapped(*args, **kwargs)
+
+ try:
+ # Calculate cost of the operation
+ cost = get_embed_model_cost(kwargs.get("model", "text-embedding-ada-002"),
+ pricing_info, response.usage.prompt_tokens)
+
+ # Set Span attributes
+ span.set_attribute(TELEMETRY_SDK_NAME, "openlit")
+ span.set_attribute(SemanticConvetion.GEN_AI_SYSTEM,
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE)
+ span.set_attribute(SemanticConvetion.GEN_AI_TYPE,
+ SemanticConvetion.GEN_AI_TYPE_EMBEDDING)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENDPOINT,
+ gen_ai_endpoint)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENVIRONMENT,
+ environment)
+ span.set_attribute(SemanticConvetion.GEN_AI_APPLICATION_NAME,
+ application_name)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MODEL,
+ kwargs.get("model", "text-embedding-ada-002"))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_EMBEDDING_FORMAT,
+ kwargs.get("encoding_format", "float"))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_EMBEDDING_DIMENSION,
+ kwargs.get("dimensions", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_USER,
+ kwargs.get("user", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ response.usage.prompt_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ response.usage.total_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+ if trace_content:
+ span.add_event(
+ name=SemanticConvetion.GEN_AI_CONTENT_PROMPT_EVENT,
+ attributes={
+ SemanticConvetion.GEN_AI_CONTENT_PROMPT: kwargs.get("input", ""),
+ },
+ )
+
+ span.set_status(Status(StatusCode.OK))
+
+ if disable_metrics is False:
+ attributes = {
+ TELEMETRY_SDK_NAME:
+ "openlit",
+ SemanticConvetion.GEN_AI_APPLICATION_NAME:
+ application_name,
+ SemanticConvetion.GEN_AI_SYSTEM:
+ SemanticConvetion.GEN_AI_SYSTEM_AZURE_AI_INFERENCE,
+ SemanticConvetion.GEN_AI_ENVIRONMENT:
+ environment,
+ SemanticConvetion.GEN_AI_TYPE:
+ SemanticConvetion.GEN_AI_TYPE_EMBEDDING,
+ SemanticConvetion.GEN_AI_REQUEST_MODEL:
+ kwargs.get("model", "text-embedding-ada-002")
+ }
+
+ metrics["genai_requests"].add(1, attributes)
+ metrics["genai_total_tokens"].add(response.usage.total_tokens, attributes)
+ metrics["genai_prompt_tokens"].add(response.usage.prompt_tokens, attributes)
+ metrics["genai_cost"].record(cost, attributes)
+
+ # Return original response
+ return response
+
+ except Exception as e:
+ handle_exception(span, e)
+ logger.error("Error in trace creation: %s", e)
+
+ # Return original response
+ return response
+
+ return wrapper
diff --git a/sdk/python/src/openlit/instrumentation/google_ai_studio/__init__.py b/sdk/python/src/openlit/instrumentation/google_ai_studio/__init__.py
index 3e966409..897f8e9a 100644
--- a/sdk/python/src/openlit/instrumentation/google_ai_studio/__init__.py
+++ b/sdk/python/src/openlit/instrumentation/google_ai_studio/__init__.py
@@ -32,7 +32,7 @@ class GoogleAIStudioInstrumentor(BaseInstrumentor):
pricing_info = kwargs.get("pricing_info", {})
trace_content = kwargs.get("trace_content", False)
disable_metrics = kwargs.get("disable_metrics")
- version = importlib.metadata.version("ollama")
+ version = importlib.metadata.version("google-generativeai")
# sync generate
wrap_function_wrapper(
diff --git a/sdk/python/src/openlit/semcov/__init__.py b/sdk/python/src/openlit/semcov/__init__.py
index e127d667..e8f8377f 100644
--- a/sdk/python/src/openlit/semcov/__init__.py
+++ b/sdk/python/src/openlit/semcov/__init__.py
@@ -103,6 +103,7 @@ class SemanticConvetion:
GEN_AI_SYSTEM_ELEVENLABS = "elevenlabs"
GEN_AI_SYSTEM_VLLM = "vLLM"
GEN_AI_SYSTEM_GOOGLE_AI_STUDIO = "google-ai-studio"
+ GEN_AI_SYSTEM_AZURE_AI_INFERENCE = "azure-ai-inference"
GEN_AI_SYSTEM_LANGCHAIN = "langchain"
GEN_AI_SYSTEM_LLAMAINDEX = "llama_index"
GEN_AI_SYSTEM_HAYSTACK = "haystack"
diff --git a/sdk/python/tests/requirements.txt b/sdk/python/tests/requirements.txt
index 865f4a6c..9de965e8 100644
--- a/sdk/python/tests/requirements.txt
+++ b/sdk/python/tests/requirements.txt
@@ -17,6 +17,7 @@ google-cloud-aiplatform >= 1.38.1
groq >= 0.5.0
elevenlabs >= 1.4.0
google-generativeai >= 0.7.2
+azure-ai-inference >= 1.0.0b4
# Vector DBs
chromadb>=0.4.0
@@ -37,6 +38,7 @@ bs4
tiktoken>=0.7.0
tensorflow>=2.16.1
tf_keras>=2.16.0
+aiohttp >= 3.10.5
# OpenTelemetry
opentelemetry-api>=1.24.0
diff --git a/sdk/python/tests/test_azure_ai_inference.py b/sdk/python/tests/test_azure_ai_inference.py
new file mode 100644
index 00000000..d137a820
--- /dev/null
+++ b/sdk/python/tests/test_azure_ai_inference.py
@@ -0,0 +1,119 @@
+# pylint: disable=duplicate-code, consider-using-with, no-name-in-module, reimported, wrong-import-order, wrong-import-position
+"""
+This module contains tests for Azure AI Inference functionality using the
+Azure AI Inference Python library.
+
+Tests cover various API endpoints, including completions, chat completions,
+embeddings, fine-tuning job creation, image generation, image variation creation,
+and audio speech generation. These tests validate integration with OpenLIT.
+
+Environment Variables:
+ - AZURE_AI_INFERENCE_API_TOKEN: Azure AI Inference API key for authentication.
+
+Note: Ensure the environment is properly configured for Azure AI Inference access
+and OpenLIT monitoring prior to running these tests.
+"""
+
+import os
+import pytest
+import openlit
+from azure.ai.inference import ChatCompletionsClient, EmbeddingsClient
+from azure.ai.inference.models import SystemMessage
+from azure.ai.inference.models import UserMessage
+from azure.core.credentials import AzureKeyCredential
+
+# Initialize synchronous Azure AI Inference Chat client
+sync_chat_client = ChatCompletionsClient(
+ endpoint="https://models.inference.ai.azure.com",
+ credential=AzureKeyCredential(os.getenv("AZURE_AI_INFERENCE_API_TOKEN")),
+)
+
+# Initialize synchronous Azure AI Inference Chat client
+sync_embed_client = EmbeddingsClient(
+ endpoint="https://models.inference.ai.azure.com",
+ credential=AzureKeyCredential(os.getenv("AZURE_AI_INFERENCE_API_TOKEN")),
+)
+
+# Initialize environment and application name for OpenLIT monitoring
+openlit.init(environment="openlit-testing", application_name="openlit-python-test")
+
+def test_sync_chat_completions():
+ """
+ Tests synchronous chat completions with the "Phi-3-mini-4k-instruct" model.
+
+ Raises:
+ AssertionError: If the chat completion response object is not as expected.
+ """
+
+ response = sync_chat_client.complete(
+ messages=[
+ SystemMessage(content="You are a helpful assistant."),
+ UserMessage(content="sync non-streaming"),
+ ],
+ model="Phi-3-mini-4k-instruct",
+ temperature=1.0,
+ max_tokens=1000,
+ top_p=1.0
+ )
+ assert response["object"] == 'chat.completion'
+
+def test_sync_embeddings():
+ """
+ Tests synchronous embedding creation with the "cohere-embed-v3-english" model.
+
+ Raises:
+ AssertionError: If the embedding response object is not as expected.
+ """
+
+ response = sync_embed_client.embed(
+ input=["LLM Observability", "Monitor GPUs"],
+ model="cohere-embed-v3-english"
+ )
+ assert response.data[0]["object"] == 'embedding'
+
+from azure.ai.inference.aio import ChatCompletionsClient, EmbeddingsClient
+async_chat_client = ChatCompletionsClient(
+ endpoint="https://models.inference.ai.azure.com",
+ credential=AzureKeyCredential(os.getenv("AZURE_AI_INFERENCE_API_TOKEN")),
+)
+
+async_embed_client = EmbeddingsClient(
+ endpoint="https://models.inference.ai.azure.com",
+ credential=AzureKeyCredential(os.getenv("AZURE_AI_INFERENCE_API_TOKEN")),
+)
+
+async def test_async_chat_completions():
+ """
+ Tests synchronous chat completions with the "Phi-3-mini-4k-instruct" model.
+
+ Raises:
+ AssertionError: If the chat completion response object is not as expected.
+ """
+
+ response = await async_chat_client.complete(
+ messages=[
+ SystemMessage(content="You are a helpful assistant."),
+ UserMessage(content="sync non-streaming"),
+ ],
+ model="Phi-3-mini-4k-instruct",
+ temperature=1.0,
+ max_tokens=1000,
+ top_p=1.0
+ )
+ assert response["object"] == 'chat.completion'
+
+async def test_async_embeddings():
+ """
+ Tests synchronous embedding creation with the "cohere-embed-v3-english" model.
+
+ Raises:
+ AssertionError: If the embedding response object is not as expected.
+ """
+
+ response = await async_embed_client.embed(
+ input=["LLM Observability", "Monitor GPUs"],
+ model="cohere-embed-v3-english"
+ )
+ assert response.data[0]["object"] == 'embedding'