Skip to content

Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.

[Feat]: Add OpenTelemetry instrumentation for Monitoring LLM Applications using Groq (#229)

patcher9c8a12bbe0aea
diff --git a/.github/workflows/python-tests.yml b/.github/workflows/python-tests.yml
index 2f531767..551e66d1 100644
--- a/.github/workflows/python-tests.yml
+++ b/.github/workflows/python-tests.yml
@@ -19,6 +19,7 @@ env:
ANTHROPIC_API_TOKEN: ${{ secrets.ANTHROPIC_API_TOKEN }}
MISTRAL_API_TOKEN: ${{ secrets.MISTRAL_API_TOKEN }}
PINECONE_API_TOKEN: ${{ secrets.PINECONE_API_TOKEN }}
+ GROQ_API_TOKEN: ${{ secrets.GROQ_API_TOKEN }}
jobs:
build:
diff --git a/docs/images/docs-groq-banner.gif b/docs/images/docs-groq-banner.gif
new file mode 100644
index 00000000..419418df
Binary files /dev/null and b/docs/images/docs-groq-banner.gif differ
diff --git a/docs/latest/integrations/groq.mdx b/docs/latest/integrations/groq.mdx
new file mode 100644
index 00000000..bbdd2444
--- /dev/null
+++ b/docs/latest/integrations/groq.mdx
@@ -0,0 +1,52 @@
+---
+title: 'Monitor Groq using OpenTelemetry'
+sidebarTitle: 'Groq'
+---
+
+<Frame>
+ <img src="/images/docs-groq-banner.gif" />
+</Frame>
+
+OpenLIT SDKs provide OpenTelemetry Auto-Instrumentation for monitoring LLM Application using Groq which includes performance, token and cost usage and user interactions.
+
+The integration is compatible with
+- Groq Python SDK client `>= 0.5.0`
+
+## Get Started
+
+<Steps>
+ <Step title="Install the OpenLIT Python SDK using pip">
+ ```shell
+ pip install openlit
+ ```
+ </Step>
+ <Step title="Add the following two lines to your application code:">
+ Integrating the OpenLIT into LLM applications is straightforward. Start monitoring for your LLM Application with just **one line of code**:
+
+ ```python
+ import openlit
+
+ openlit.init(otlp_endpoint="otlp_backend_endpoint", otlp_headers="otlp_backend_token")
+ ```
+
+ Alternatively, you can configure the OTLP endpoint by setting the `OTEL_EXPORTER_OTLP_ENDPOINT` and the OTLP headers by setting the Appli`OTEL_EXPORTER_OTLP_HEADERS` environmentcation variable as recommended in the OpenTelemetry documentation.
+
+ If your aim is primarily to test or develop without sending telemetry data to an external backend, you can initialize OpenLIT without any parameters. This approach directs all telemetry output to your console, enabling quick diagnostics and local debugging:
+
+ ```python
+ import openlit
+
+ openlit.init()
+ ```
+
+ Refer to the OpenLIT [Python SDK repository](https://github.com/openlit/openlit/tree/main/sdk/python) for more advanced configurations and use cases.
+ </Step>
+</Steps>
+
+---
+
+<CardGroup cols={2}>
+<Card title="Connections" href="/latest/connections/intro" icon='link'>
+Connect to your existing Observablity Stack
+</Card>
+</CardGroup>
\ No newline at end of file
diff --git a/docs/latest/integrations/haystack.mdx b/docs/latest/integrations/haystack.mdx
index b4b4d1fc..2b276f5e 100644
--- a/docs/latest/integrations/haystack.mdx
+++ b/docs/latest/integrations/haystack.mdx
@@ -12,21 +12,6 @@ OpenLIT SDKs provide OpenTelemetry Auto-Instrumentation for monitoring LLM Appli
## Get Started
<Steps>
- <Step title="Disable Haystack default tracing">
- To disable default tracing available in Haystack, there are two options:
-
- Set the environment variable `HAYSTACK_AUTO_TRACE_ENABLED` to `false` when running your Haystack application
-
- or
-
- Disable tracing in Python:
-
- ```Python
- from haystack.tracing import disable_tracing
-
- disable_tracing()
- ```
- </Step>
<Step title="Install the OpenLIT Python SDK using pip">
```shell
pip install openlit
diff --git a/docs/latest/integrations/introduction.mdx b/docs/latest/integrations/introduction.mdx
index 7f3ba575..7e467624 100644
--- a/docs/latest/integrations/introduction.mdx
+++ b/docs/latest/integrations/introduction.mdx
@@ -165,6 +165,18 @@ Start integrating and monitoring your applications with OpenLIT effortlessly. Ch
iconType="duotone"
>
</Card>
+ <Card
+ title="Groq"
+ href="/latest/integrations/groq"
+ icon={
+ <svg version="1.1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 225 225">
+ <path d="M0 0 C0.77751617 0.00087616 1.55503235 0.00175232 2.35610962 0.00265503 C20.60349771 0.07056781 35.58989245 3.13332863 51.9375 11.375 C52.54045898 11.67390137 53.14341797 11.97280273 53.76464844 12.28076172 C63.71468409 17.29920578 71.89838612 23.73584203 79.9375 31.375 C80.46617676 31.87628418 80.99485352 32.37756836 81.53955078 32.89404297 C94.25524732 45.20328352 103.50482279 60.55291589 108.9375 77.375 C109.34871094 78.58542969 109.75992188 79.79585938 110.18359375 81.04296875 C113.34570596 91.6366393 113.36024244 102.47605705 113.4375 113.4375 C113.44803406 114.1954184 113.45856812 114.95333679 113.46942139 115.73422241 C113.78011784 144.69590579 102.61213812 171.18127346 82.484375 191.953125 C63.6237976 210.85195915 37.60487103 225.30797858 10.328125 225.61328125 C9.60810806 225.62312042 8.88809113 225.63295959 8.14625549 225.64309692 C-13.55255984 225.87871084 -31.44666577 224.26435285 -51.0625 214.375 C-51.66545898 214.07609863 -52.26841797 213.77719727 -52.88964844 213.46923828 C-78.8802558 200.36050042 -98.15534701 176.61693057 -107.50390625 149.25390625 C-116.80002598 120.83072845 -114.45810514 88.06090266 -101.109375 61.40576172 C-86.3048308 32.68743181 -61.55688318 13.35266467 -31.234375 3.19921875 C-20.81895566 0.49274894 -10.70423246 -0.02937254 0 0 Z " fill="#F0553A" transform="translate(112.0625,-0.375)"/>
+ <path d="M0 0 C10.00535849 6.25334906 16.78611255 15.19162612 19.57647705 26.7093811 C20.32827419 30.77546709 20.32550971 34.824414 20.3359375 38.94921875 C20.3459227 40.35093071 20.3459227 40.35093071 20.35610962 41.78096008 C20.36618632 43.75003101 20.37092819 45.71913574 20.37060547 47.68823242 C20.37496247 50.66188535 20.41129098 53.63403236 20.44921875 56.60742188 C20.54079179 71.99137078 19.25010928 84.30263269 8.22265625 95.86328125 C-0.26778382 103.97173834 -8.89793523 106.71273877 -20.43041992 106.52368164 C-30.35916159 106.0696585 -36.85958963 102.82995774 -44 96 C-42.61811547 92.78956118 -41.07578097 90.95063635 -38.375 88.75 C-37.74335937 88.22921875 -37.11171875 87.7084375 -36.4609375 87.171875 C-35.73777344 86.59179688 -35.73777344 86.59179688 -35 86 C-34.42894531 86.3609375 -33.85789062 86.721875 -33.26953125 87.09375 C-26.1741928 91.15467502 -20.1031195 92.19309735 -12 91 C-4.64533384 88.97670807 -0.93610714 85.32962417 2.77359009 78.8208313 C4.82184031 74.10970799 4.83990951 69.69638249 4.76171875 64.62109375 C4.76508743 63.56898209 4.76845612 62.51687042 4.77192688 61.43287659 C4.77162975 59.22351153 4.7565437 57.0141031 4.72729492 54.80493164 C4.68755591 51.44223136 4.7066432 48.0834705 4.73242188 44.72070312 C4.72486295 42.56248486 4.71390968 40.40427525 4.69921875 38.24609375 C4.70620285 37.25210068 4.71318695 36.2581076 4.72038269 35.23399353 C4.58466601 28.45791259 3.1334789 23.44966219 -1 18 C-7.05144758 12.49656886 -13.07792569 10.51932219 -21.21484375 10.68359375 C-27.84719534 11.43705866 -32.81657343 14.88692308 -37 20 C-41.15590093 25.96814626 -42.1445345 31.83421877 -41 39 C-38.53248368 46.39204888 -34.87897569 51.44565868 -27.92578125 55.0859375 C-23.7105358 56.40284996 -19.58070203 56.46774892 -15.1875 56.625 C-14.30384766 56.66367188 -13.42019531 56.70234375 -12.50976562 56.7421875 C-10.34017028 56.83600784 -8.17011453 56.91907326 -6 57 C-6 61.95 -6 66.9 -6 72 C-21.88140531 72.40376454 -32.63717119 70.84252573 -44.75 59.75 C-53.05121186 51.10129556 -54.46291245 40.99230499 -54.27734375 29.53515625 C-53.8707883 19.68276025 -49.05406991 11.74364719 -41.96484375 5.08984375 C-29.64652968 -5.14282984 -14.51859451 -7.59537583 0 0 Z " fill="#FEFAFA" transform="translate(131,63)"/>
+ </svg>
+ }
+ iconType="duotone"
+ >
+ </Card>
</CardGroup>
### VectorDBs
diff --git a/docs/mint.json b/docs/mint.json
index 8e05d4dc..a7b782f9 100644
--- a/docs/mint.json
+++ b/docs/mint.json
@@ -66,7 +66,8 @@
"latest/integrations/azure-openai",
"latest/integrations/huggingface",
"latest/integrations/bedrock",
- "latest/integrations/vertexai"
+ "latest/integrations/vertexai",
+ "latest/integrations/groq"
]
},
{
diff --git a/sdk/python/README.md b/sdk/python/README.md
index ecd756db..2d853065 100644
--- a/sdk/python/README.md
+++ b/sdk/python/README.md
@@ -32,6 +32,7 @@ This project adheres to the [Semantic Conventions](https://github.com/open-telem
- [✅ HuggingFace Transformers](https://docs.openlit.io/latest/integrations/huggingface)
- [✅ Amazon Bedrock](https://docs.openlit.io/latest/integrations/bedrock)
- [✅ Vertex AI](https://docs.openlit.io/latest/integrations/vertexai)
+- [✅ Groq](https://docs.openlit.io/latest/integrations/groq)
### Vector DBs
- [✅ ChromaDB](https://docs.openlit.io/latest/integrations/chromadb)
diff --git a/sdk/python/pyproject.toml b/sdk/python/pyproject.toml
index 51ef93d2..4ef9a297 100644
--- a/sdk/python/pyproject.toml
+++ b/sdk/python/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "openlit"
-version = "1.3.0"
+version = "1.4.0"
description = "OpenTelemetry-native Auto instrumentation library for monitoring LLM Applications, facilitating the integration of observability into your GenAI-driven projects"
authors = ["OpenLIT"]
repository = "https://github.com/openlit/openlit/tree/main/openlit/python"
diff --git a/sdk/python/src/openlit/__init__.py b/sdk/python/src/openlit/__init__.py
index fc5838cc..8643c10f 100644
--- a/sdk/python/src/openlit/__init__.py
+++ b/sdk/python/src/openlit/__init__.py
@@ -19,6 +19,7 @@ from openlit.instrumentation.cohere import CohereInstrumentor
from openlit.instrumentation.mistral import MistralInstrumentor
from openlit.instrumentation.bedrock import BedrockInstrumentor
from openlit.instrumentation.vertexai import VertexAIInstrumentor
+from openlit.instrumentation.groq import GroqInstrumentor
from openlit.instrumentation.langchain import LangChainInstrumentor
from openlit.instrumentation.llamaindex import LlamaIndexInstrumentor
from openlit.instrumentation.haystack import HaystackInstrumentor
@@ -151,6 +152,7 @@ def init(environment="default", application_name="default", tracer=None, otlp_en
"mistral": "mistralai",
"bedrock": "boto3",
"vertexai": "vertexai",
+ "groq": "groq",
"langchain": "langchain",
"llama_index": "llama_index",
"haystack": "haystack",
@@ -201,6 +203,7 @@ def init(environment="default", application_name="default", tracer=None, otlp_en
"mistral": MistralInstrumentor(),
"bedrock": BedrockInstrumentor(),
"vertexai": VertexAIInstrumentor(),
+ "groq": GroqInstrumentor(),
"langchain": LangChainInstrumentor(),
"llama_index": LlamaIndexInstrumentor(),
"haystack": HaystackInstrumentor(),
diff --git a/sdk/python/src/openlit/instrumentation/groq/__init__.py b/sdk/python/src/openlit/instrumentation/groq/__init__.py
new file mode 100644
index 00000000..69148a71
--- /dev/null
+++ b/sdk/python/src/openlit/instrumentation/groq/__init__.py
@@ -0,0 +1,50 @@
+# pylint: disable=useless-return, bad-staticmethod-argument, disable=duplicate-code
+"""Initializer of Auto Instrumentation of Groq Functions"""
+
+from typing import Collection
+import importlib.metadata
+from opentelemetry.instrumentation.instrumentor import BaseInstrumentor
+from wrapt import wrap_function_wrapper
+
+from openlit.instrumentation.groq.groq import chat
+from openlit.instrumentation.groq.async_groq import async_chat
+
+_instruments = ("groq >= 0.5.0",)
+
+class GroqInstrumentor(BaseInstrumentor):
+ """
+ An instrumentor for Groq's client library.
+ """
+
+ def instrumentation_dependencies(self) -> Collection[str]:
+ return _instruments
+
+ def _instrument(self, **kwargs):
+ application_name = kwargs.get("application_name", "default_application")
+ environment = kwargs.get("environment", "default_environment")
+ tracer = kwargs.get("tracer")
+ metrics = kwargs.get("metrics_dict")
+ pricing_info = kwargs.get("pricing_info", {})
+ trace_content = kwargs.get("trace_content", False)
+ disable_metrics = kwargs.get("disable_metrics")
+ version = importlib.metadata.version("groq")
+
+ #sync
+ wrap_function_wrapper(
+ "groq.resources.chat.completions",
+ "Completions.create",
+ chat("groq.chat.completions", version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics),
+ )
+
+ #async
+ wrap_function_wrapper(
+ "groq.resources.chat.completions",
+ "AsyncCompletions.create",
+ async_chat("groq.chat.completions", version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics),
+ )
+
+ def _uninstrument(self, **kwargs):
+ # Proper uninstrumentation logic to revert patched methods
+ pass
diff --git a/sdk/python/src/openlit/instrumentation/groq/async_groq.py b/sdk/python/src/openlit/instrumentation/groq/async_groq.py
new file mode 100644
index 00000000..b0fdeefa
--- /dev/null
+++ b/sdk/python/src/openlit/instrumentation/groq/async_groq.py
@@ -0,0 +1,331 @@
+# pylint: disable=duplicate-code, broad-exception-caught, too-many-statements, unused-argument, used-before-assignment, too-many-branches
+"""
+Module for monitoring Groq API calls.
+"""
+
+import logging
+from opentelemetry.trace import SpanKind, Status, StatusCode
+from opentelemetry.sdk.resources import TELEMETRY_SDK_NAME
+from openlit.__helpers import get_chat_model_cost, handle_exception
+from openlit.semcov import SemanticConvetion
+
+# Initialize logger for logging potential issues and operations
+logger = logging.getLogger(__name__)
+
+def async_chat(gen_ai_endpoint, version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics):
+ """
+ Generates a telemetry wrapper for chat completions to collect metrics.
+
+ Args:
+ gen_ai_endpoint: Endpoint identifier for logging and tracing.
+ version: Version of the monitoring package.
+ environment: Deployment environment (e.g., production, staging).
+ application_name: Name of the application using the Groq API.
+ tracer: OpenTelemetry tracer for creating spans.
+ pricing_info: Information used for calculating the cost of Groq usage.
+ trace_content: Flag indicating whether to trace the actual content.
+
+ Returns:
+ A function that wraps the chat completions method to add telemetry.
+ """
+
+ async def wrapper(wrapped, instance, args, kwargs):
+ """
+ Wraps the 'chat.completions' API call to add telemetry.
+
+ This collects metrics such as execution time, cost, and token usage, and handles errors
+ gracefully, adding details to the trace for observability.
+
+ Args:
+ wrapped: The original 'chat.completions' method to be wrapped.
+ instance: The instance of the class where the original method is defined.
+ args: Positional arguments for the 'chat.completions' method.
+ kwargs: Keyword arguments for the 'chat.completions' method.
+
+ Returns:
+ The response from the original 'chat.completions' method.
+ """
+
+ # Check if streaming is enabled for the API call
+ streaming = kwargs.get("stream", False)
+
+ # pylint: disable=no-else-return
+ if streaming:
+ # Special handling for streaming response to accommodate the nature of data flow
+ async def stream_generator():
+ with tracer.start_as_current_span(gen_ai_endpoint, kind= SpanKind.CLIENT) as span:
+ # Placeholder for aggregating streaming response
+ llmresponse = ""
+
+ # Loop through streaming events capturing relevant details
+ async for chunk in await wrapped(*args, **kwargs):
+ # Collect message IDs and aggregated response from events
+ if len(chunk.choices) > 0:
+ # pylint: disable=line-too-long
+ if hasattr(chunk.choices[0], "delta") and hasattr(chunk.choices[0].delta, "content"):
+ content = chunk.choices[0].delta.content
+ if content:
+ llmresponse += content
+ if chunk.x_groq is not None and chunk.x_groq.usage is not None:
+ prompt_tokens = chunk.x_groq.usage.prompt_tokens
+ completion_tokens = chunk.x_groq.usage.completion_tokens
+ total_tokens = chunk.x_groq.usage.total_tokens
+ response_id = chunk.x_groq.id
+ yield chunk
+
+ # Handling exception ensure observability without disrupting operation
+ try:
+ # Format 'messages' into a single string
+ message_prompt = kwargs.get("messages", "")
+ formatted_messages = []
+ for message in message_prompt:
+ role = message["role"]
+ content = message["content"]
+
+ if isinstance(content, list):
+ content_str = ", ".join(
+ # pylint: disable=line-too-long
+ f'{item["type"]}: {item["text"] if "text" in item else item["image_url"]}'
+ if "type" in item else f'text: {item["text"]}'
+ for item in content
+ )
+ formatted_messages.append(f"{role}: {content_str}")
+ else:
+ formatted_messages.append(f"{role}: {content}")
+ prompt = "\n".join(formatted_messages)
+
+ # Calculate cost of the operation
+ cost = get_chat_model_cost(kwargs.get("model", "gpt-3.5-turbo"),
+ pricing_info, prompt_tokens,
+ completion_tokens)
+
+ # Set Span attributes
+ span.set_attribute(TELEMETRY_SDK_NAME, "openlit")
+ span.set_attribute(SemanticConvetion.GEN_AI_SYSTEM,
+ SemanticConvetion.GEN_AI_SYSTEM_GROQ)
+ span.set_attribute(SemanticConvetion.GEN_AI_TYPE,
+ SemanticConvetion.GEN_AI_TYPE_CHAT)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENDPOINT,
+ gen_ai_endpoint)
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_ID,
+ response_id)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENVIRONMENT,
+ environment)
+ span.set_attribute(SemanticConvetion.GEN_AI_APPLICATION_NAME,
+ application_name)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MODEL,
+ kwargs.get("model", "gpt-3.5-turbo"))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_USER,
+ kwargs.get("user", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TOP_P,
+ kwargs.get("top_p", 1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MAX_TOKENS,
+ kwargs.get("max_tokens", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TEMPERATURE,
+ kwargs.get("temperature", 1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_PRESENCE_PENALTY,
+ kwargs.get("presence_penalty", 0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_FREQUENCY_PENALTY,
+ kwargs.get("frequency_penalty", 0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_SEED,
+ kwargs.get("seed", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_IS_STREAM,
+ True)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ prompt_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COMPLETION_TOKENS,
+ completion_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ prompt_tokens + completion_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+ if trace_content:
+ span.set_attribute(SemanticConvetion.GEN_AI_CONTENT_PROMPT,
+ prompt)
+ span.set_attribute(SemanticConvetion.GEN_AI_CONTENT_COMPLETION,
+ llmresponse)
+
+ span.set_status(Status(StatusCode.OK))
+
+ if disable_metrics is False:
+ attributes = {
+ TELEMETRY_SDK_NAME:
+ "openlit",
+ SemanticConvetion.GEN_AI_APPLICATION_NAME:
+ application_name,
+ SemanticConvetion.GEN_AI_SYSTEM:
+ SemanticConvetion.GEN_AI_SYSTEM_GROQ,
+ SemanticConvetion.GEN_AI_ENVIRONMENT:
+ environment,
+ SemanticConvetion.GEN_AI_TYPE:
+ SemanticConvetion.GEN_AI_TYPE_CHAT,
+ SemanticConvetion.GEN_AI_REQUEST_MODEL:
+ kwargs.get("model", "gpt-3.5-turbo")
+ }
+
+ metrics["genai_requests"].add(1, attributes)
+ metrics["genai_total_tokens"].add(
+ total_tokens, attributes
+ )
+ metrics["genai_completion_tokens"].add(completion_tokens, attributes)
+ metrics["genai_prompt_tokens"].add(prompt_tokens, attributes)
+ metrics["genai_cost"].record(cost, attributes)
+
+ except Exception as e:
+ handle_exception(span, e)
+ logger.error("Error in trace creation: %s", e)
+
+ return stream_generator()
+
+ # Handling for non-streaming responses
+ else:
+ # pylint: disable=line-too-long
+ with tracer.start_as_current_span(gen_ai_endpoint, kind= SpanKind.CLIENT) as span:
+ response = await wrapped(*args, **kwargs)
+
+ try:
+ # Format 'messages' into a single string
+ message_prompt = kwargs.get("messages", "")
+ formatted_messages = []
+ for message in message_prompt:
+ role = message["role"]
+ content = message["content"]
+
+ if isinstance(content, list):
+ content_str = ", ".join(
+ # pylint: disable=line-too-long
+ f'{item["type"]}: {item["text"] if "text" in item else item["image_url"]}'
+ if "type" in item else f'text: {item["text"]}'
+ for item in content
+ )
+ formatted_messages.append(f"{role}: {content_str}")
+ else:
+ formatted_messages.append(f"{role}: {content}")
+ prompt = "\n".join(formatted_messages)
+
+ # Set base span attribues
+ span.set_attribute(TELEMETRY_SDK_NAME, "openlit")
+ span.set_attribute(SemanticConvetion.GEN_AI_SYSTEM,
+ SemanticConvetion.GEN_AI_SYSTEM_GROQ)
+ span.set_attribute(SemanticConvetion.GEN_AI_TYPE,
+ SemanticConvetion.GEN_AI_TYPE_CHAT)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENDPOINT,
+ gen_ai_endpoint)
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_ID,
+ response.x_groq["id"])
+ span.set_attribute(SemanticConvetion.GEN_AI_ENVIRONMENT,
+ environment)
+ span.set_attribute(SemanticConvetion.GEN_AI_APPLICATION_NAME,
+ application_name)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MODEL,
+ kwargs.get("model", "llama3-8b-8192"))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TOP_P,
+ kwargs.get("top_p", 1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MAX_TOKENS,
+ kwargs.get("max_tokens", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_USER,
+ kwargs.get("name", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TEMPERATURE,
+ kwargs.get("temperature", 1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_PRESENCE_PENALTY,
+ kwargs.get("presence_penalty", 0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_FREQUENCY_PENALTY,
+ kwargs.get("frequency_penalty", 0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_SEED,
+ kwargs.get("seed", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_IS_STREAM,
+ False)
+ if trace_content:
+ span.set_attribute(SemanticConvetion.GEN_AI_CONTENT_PROMPT,
+ prompt)
+
+ # Set span attributes when tools is not passed to the function call
+ if "tools" not in kwargs:
+ # Calculate cost of the operation
+ cost = get_chat_model_cost(kwargs.get("model", "llama3-8b-8192"),
+ pricing_info, response.usage.prompt_tokens,
+ response.usage.completion_tokens)
+
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ response.usage.prompt_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COMPLETION_TOKENS,
+ response.usage.completion_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ response.usage.total_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_FINISH_REASON,
+ response.choices[0].finish_reason)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+
+ # Set span attributes for when n = 1 (default)
+ if "n" not in kwargs or kwargs["n"] == 1:
+ if trace_content:
+ span.set_attribute(SemanticConvetion.GEN_AI_CONTENT_COMPLETION,
+ response.choices[0].message.content)
+
+ # Set span attributes for when n > 0
+ else:
+ i = 0
+ while i < kwargs["n"] and trace_content is True:
+ attribute_name = f"gen_ai.content.completion.{i}"
+ span.set_attribute(attribute_name,
+ response.choices[i].message.content)
+ i += 1
+
+ # Return original response
+ return response
+
+ # Set span attributes when tools is passed to the function call
+ elif "tools" in kwargs:
+ # Calculate cost of the operation
+ cost = get_chat_model_cost(kwargs.get("model", "gpt-3.5-turbo"),
+ pricing_info, response.usage.prompt_tokens,
+ response.usage.completion_tokens)
+
+ span.set_attribute(SemanticConvetion.GEN_AI_CONTENT_COMPLETION,
+ "Function called with tools")
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ response.usage.prompt_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COMPLETION_TOKENS,
+ response.usage.completion_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ response.usage.total_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+
+ span.set_status(Status(StatusCode.OK))
+
+ if disable_metrics is False:
+ attributes = {
+ TELEMETRY_SDK_NAME:
+ "openlit",
+ SemanticConvetion.GEN_AI_APPLICATION_NAME:
+ application_name,
+ SemanticConvetion.GEN_AI_SYSTEM:
+ SemanticConvetion.GEN_AI_SYSTEM_GROQ,
+ SemanticConvetion.GEN_AI_ENVIRONMENT:
+ environment,
+ SemanticConvetion.GEN_AI_TYPE:
+ SemanticConvetion.GEN_AI_TYPE_CHAT,
+ SemanticConvetion.GEN_AI_REQUEST_MODEL:
+ kwargs.get("model", "gpt-3.5-turbo")
+ }
+
+ metrics["genai_requests"].add(1, attributes)
+ metrics["genai_total_tokens"].add(response.usage.total_tokens, attributes)
+ metrics["genai_completion_tokens"].add(response.usage.completion_tokens, attributes)
+ metrics["genai_prompt_tokens"].add(response.usage.prompt_tokens, attributes)
+ metrics["genai_cost"].record(cost, attributes)
+
+ # Return original response
+ return response
+
+ except Exception as e:
+ handle_exception(span, e)
+ logger.error("Error in trace creation: %s", e)
+
+ # Return original response
+ return response
+
+ return wrapper
diff --git a/sdk/python/src/openlit/instrumentation/groq/groq.py b/sdk/python/src/openlit/instrumentation/groq/groq.py
new file mode 100644
index 00000000..e478cbb1
--- /dev/null
+++ b/sdk/python/src/openlit/instrumentation/groq/groq.py
@@ -0,0 +1,331 @@
+# pylint: disable=duplicate-code, broad-exception-caught, too-many-statements, unused-argument, used-before-assignment, too-many-branches
+"""
+Module for monitoring Groq API calls.
+"""
+
+import logging
+from opentelemetry.trace import SpanKind, Status, StatusCode
+from opentelemetry.sdk.resources import TELEMETRY_SDK_NAME
+from openlit.__helpers import get_chat_model_cost, handle_exception
+from openlit.semcov import SemanticConvetion
+
+# Initialize logger for logging potential issues and operations
+logger = logging.getLogger(__name__)
+
+def chat(gen_ai_endpoint, version, environment, application_name,
+ tracer, pricing_info, trace_content, metrics, disable_metrics):
+ """
+ Generates a telemetry wrapper for chat completions to collect metrics.
+
+ Args:
+ gen_ai_endpoint: Endpoint identifier for logging and tracing.
+ version: Version of the monitoring package.
+ environment: Deployment environment (e.g., production, staging).
+ application_name: Name of the application using the Groq API.
+ tracer: OpenTelemetry tracer for creating spans.
+ pricing_info: Information used for calculating the cost of Groq usage.
+ trace_content: Flag indicating whether to trace the actual content.
+
+ Returns:
+ A function that wraps the chat completions method to add telemetry.
+ """
+
+ def wrapper(wrapped, instance, args, kwargs):
+ """
+ Wraps the 'chat.completions' API call to add telemetry.
+
+ This collects metrics such as execution time, cost, and token usage, and handles errors
+ gracefully, adding details to the trace for observability.
+
+ Args:
+ wrapped: The original 'chat.completions' method to be wrapped.
+ instance: The instance of the class where the original method is defined.
+ args: Positional arguments for the 'chat.completions' method.
+ kwargs: Keyword arguments for the 'chat.completions' method.
+
+ Returns:
+ The response from the original 'chat.completions' method.
+ """
+
+ # Check if streaming is enabled for the API call
+ streaming = kwargs.get("stream", False)
+
+ # pylint: disable=no-else-return
+ if streaming:
+ # Special handling for streaming response to accommodate the nature of data flow
+ def stream_generator():
+ with tracer.start_as_current_span(gen_ai_endpoint, kind= SpanKind.CLIENT) as span:
+ # Placeholder for aggregating streaming response
+ llmresponse = ""
+
+ # Loop through streaming events capturing relevant details
+ for chunk in wrapped(*args, **kwargs):
+ # Collect message IDs and aggregated response from events
+ if len(chunk.choices) > 0:
+ # pylint: disable=line-too-long
+ if hasattr(chunk.choices[0], "delta") and hasattr(chunk.choices[0].delta, "content"):
+ content = chunk.choices[0].delta.content
+ if content:
+ llmresponse += content
+ if chunk.x_groq is not None and chunk.x_groq.usage is not None:
+ prompt_tokens = chunk.x_groq.usage.prompt_tokens
+ completion_tokens = chunk.x_groq.usage.completion_tokens
+ total_tokens = chunk.x_groq.usage.total_tokens
+ response_id = chunk.x_groq.id
+ yield chunk
+
+ # Handling exception ensure observability without disrupting operation
+ try:
+ # Format 'messages' into a single string
+ message_prompt = kwargs.get("messages", "")
+ formatted_messages = []
+ for message in message_prompt:
+ role = message["role"]
+ content = message["content"]
+
+ if isinstance(content, list):
+ content_str = ", ".join(
+ # pylint: disable=line-too-long
+ f'{item["type"]}: {item["text"] if "text" in item else item["image_url"]}'
+ if "type" in item else f'text: {item["text"]}'
+ for item in content
+ )
+ formatted_messages.append(f"{role}: {content_str}")
+ else:
+ formatted_messages.append(f"{role}: {content}")
+ prompt = "\n".join(formatted_messages)
+
+ # Calculate cost of the operation
+ cost = get_chat_model_cost(kwargs.get("model", "gpt-3.5-turbo"),
+ pricing_info, prompt_tokens,
+ completion_tokens)
+
+ # Set Span attributes
+ span.set_attribute(TELEMETRY_SDK_NAME, "openlit")
+ span.set_attribute(SemanticConvetion.GEN_AI_SYSTEM,
+ SemanticConvetion.GEN_AI_SYSTEM_GROQ)
+ span.set_attribute(SemanticConvetion.GEN_AI_TYPE,
+ SemanticConvetion.GEN_AI_TYPE_CHAT)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENDPOINT,
+ gen_ai_endpoint)
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_ID,
+ response_id)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENVIRONMENT,
+ environment)
+ span.set_attribute(SemanticConvetion.GEN_AI_APPLICATION_NAME,
+ application_name)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MODEL,
+ kwargs.get("model", "gpt-3.5-turbo"))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_USER,
+ kwargs.get("user", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TOP_P,
+ kwargs.get("top_p", 1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MAX_TOKENS,
+ kwargs.get("max_tokens", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TEMPERATURE,
+ kwargs.get("temperature", 1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_PRESENCE_PENALTY,
+ kwargs.get("presence_penalty", 0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_FREQUENCY_PENALTY,
+ kwargs.get("frequency_penalty", 0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_SEED,
+ kwargs.get("seed", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_IS_STREAM,
+ True)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ prompt_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COMPLETION_TOKENS,
+ completion_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ prompt_tokens + completion_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+ if trace_content:
+ span.set_attribute(SemanticConvetion.GEN_AI_CONTENT_PROMPT,
+ prompt)
+ span.set_attribute(SemanticConvetion.GEN_AI_CONTENT_COMPLETION,
+ llmresponse)
+
+ span.set_status(Status(StatusCode.OK))
+
+ if disable_metrics is False:
+ attributes = {
+ TELEMETRY_SDK_NAME:
+ "openlit",
+ SemanticConvetion.GEN_AI_APPLICATION_NAME:
+ application_name,
+ SemanticConvetion.GEN_AI_SYSTEM:
+ SemanticConvetion.GEN_AI_SYSTEM_GROQ,
+ SemanticConvetion.GEN_AI_ENVIRONMENT:
+ environment,
+ SemanticConvetion.GEN_AI_TYPE:
+ SemanticConvetion.GEN_AI_TYPE_CHAT,
+ SemanticConvetion.GEN_AI_REQUEST_MODEL:
+ kwargs.get("model", "gpt-3.5-turbo")
+ }
+
+ metrics["genai_requests"].add(1, attributes)
+ metrics["genai_total_tokens"].add(
+ total_tokens, attributes
+ )
+ metrics["genai_completion_tokens"].add(completion_tokens, attributes)
+ metrics["genai_prompt_tokens"].add(prompt_tokens, attributes)
+ metrics["genai_cost"].record(cost, attributes)
+
+ except Exception as e:
+ handle_exception(span, e)
+ logger.error("Error in trace creation: %s", e)
+
+ return stream_generator()
+
+ # Handling for non-streaming responses
+ else:
+ # pylint: disable=line-too-long
+ with tracer.start_as_current_span(gen_ai_endpoint, kind= SpanKind.CLIENT) as span:
+ response = wrapped(*args, **kwargs)
+
+ try:
+ # Format 'messages' into a single string
+ message_prompt = kwargs.get("messages", "")
+ formatted_messages = []
+ for message in message_prompt:
+ role = message["role"]
+ content = message["content"]
+
+ if isinstance(content, list):
+ content_str = ", ".join(
+ # pylint: disable=line-too-long
+ f'{item["type"]}: {item["text"] if "text" in item else item["image_url"]}'
+ if "type" in item else f'text: {item["text"]}'
+ for item in content
+ )
+ formatted_messages.append(f"{role}: {content_str}")
+ else:
+ formatted_messages.append(f"{role}: {content}")
+ prompt = "\n".join(formatted_messages)
+
+ # Set base span attribues
+ span.set_attribute(TELEMETRY_SDK_NAME, "openlit")
+ span.set_attribute(SemanticConvetion.GEN_AI_SYSTEM,
+ SemanticConvetion.GEN_AI_SYSTEM_GROQ)
+ span.set_attribute(SemanticConvetion.GEN_AI_TYPE,
+ SemanticConvetion.GEN_AI_TYPE_CHAT)
+ span.set_attribute(SemanticConvetion.GEN_AI_ENDPOINT,
+ gen_ai_endpoint)
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_ID,
+ response.x_groq["id"])
+ span.set_attribute(SemanticConvetion.GEN_AI_ENVIRONMENT,
+ environment)
+ span.set_attribute(SemanticConvetion.GEN_AI_APPLICATION_NAME,
+ application_name)
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MODEL,
+ kwargs.get("model", "llama3-8b-8192"))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TOP_P,
+ kwargs.get("top_p", 1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_MAX_TOKENS,
+ kwargs.get("max_tokens", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_USER,
+ kwargs.get("name", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_TEMPERATURE,
+ kwargs.get("temperature", 1))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_PRESENCE_PENALTY,
+ kwargs.get("presence_penalty", 0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_FREQUENCY_PENALTY,
+ kwargs.get("frequency_penalty", 0))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_SEED,
+ kwargs.get("seed", ""))
+ span.set_attribute(SemanticConvetion.GEN_AI_REQUEST_IS_STREAM,
+ False)
+ if trace_content:
+ span.set_attribute(SemanticConvetion.GEN_AI_CONTENT_PROMPT,
+ prompt)
+
+ # Set span attributes when tools is not passed to the function call
+ if "tools" not in kwargs:
+ # Calculate cost of the operation
+ cost = get_chat_model_cost(kwargs.get("model", "llama3-8b-8192"),
+ pricing_info, response.usage.prompt_tokens,
+ response.usage.completion_tokens)
+
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ response.usage.prompt_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COMPLETION_TOKENS,
+ response.usage.completion_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ response.usage.total_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_RESPONSE_FINISH_REASON,
+ response.choices[0].finish_reason)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+
+ # Set span attributes for when n = 1 (default)
+ if "n" not in kwargs or kwargs["n"] == 1:
+ if trace_content:
+ span.set_attribute(SemanticConvetion.GEN_AI_CONTENT_COMPLETION,
+ response.choices[0].message.content)
+
+ # Set span attributes for when n > 0
+ else:
+ i = 0
+ while i < kwargs["n"] and trace_content is True:
+ attribute_name = f"gen_ai.content.completion.{i}"
+ span.set_attribute(attribute_name,
+ response.choices[i].message.content)
+ i += 1
+
+ # Return original response
+ return response
+
+ # Set span attributes when tools is passed to the function call
+ elif "tools" in kwargs:
+ # Calculate cost of the operation
+ cost = get_chat_model_cost(kwargs.get("model", "gpt-3.5-turbo"),
+ pricing_info, response.usage.prompt_tokens,
+ response.usage.completion_tokens)
+
+ span.set_attribute(SemanticConvetion.GEN_AI_CONTENT_COMPLETION,
+ "Function called with tools")
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_PROMPT_TOKENS,
+ response.usage.prompt_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COMPLETION_TOKENS,
+ response.usage.completion_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_TOTAL_TOKENS,
+ response.usage.total_tokens)
+ span.set_attribute(SemanticConvetion.GEN_AI_USAGE_COST,
+ cost)
+
+ span.set_status(Status(StatusCode.OK))
+
+ if disable_metrics is False:
+ attributes = {
+ TELEMETRY_SDK_NAME:
+ "openlit",
+ SemanticConvetion.GEN_AI_APPLICATION_NAME:
+ application_name,
+ SemanticConvetion.GEN_AI_SYSTEM:
+ SemanticConvetion.GEN_AI_SYSTEM_GROQ,
+ SemanticConvetion.GEN_AI_ENVIRONMENT:
+ environment,
+ SemanticConvetion.GEN_AI_TYPE:
+ SemanticConvetion.GEN_AI_TYPE_CHAT,
+ SemanticConvetion.GEN_AI_REQUEST_MODEL:
+ kwargs.get("model", "gpt-3.5-turbo")
+ }
+
+ metrics["genai_requests"].add(1, attributes)
+ metrics["genai_total_tokens"].add(response.usage.total_tokens, attributes)
+ metrics["genai_completion_tokens"].add(response.usage.completion_tokens, attributes)
+ metrics["genai_prompt_tokens"].add(response.usage.prompt_tokens, attributes)
+ metrics["genai_cost"].record(cost, attributes)
+
+ # Return original response
+ return response
+
+ except Exception as e:
+ handle_exception(span, e)
+ logger.error("Error in trace creation: %s", e)
+
+ # Return original response
+ return response
+
+ return wrapper
diff --git a/sdk/python/src/openlit/otel/tracing.py b/sdk/python/src/openlit/otel/tracing.py
index 04a40036..3594dbc1 100644
--- a/sdk/python/src/openlit/otel/tracing.py
+++ b/sdk/python/src/openlit/otel/tracing.py
@@ -40,6 +40,9 @@ def setup_tracing(application_name, environment, tracer, otlp_endpoint, otlp_hea
global TRACER_SET
try:
+ #Disable Haystack Auto Tracing
+ os.environ["HAYSTACK_AUTO_TRACE_ENABLED"] = "false"
+
if not TRACER_SET:
# Create a resource with the service name attribute.
resource = Resource(attributes={
diff --git a/sdk/python/src/openlit/semcov/__init__.py b/sdk/python/src/openlit/semcov/__init__.py
index 1c2dfc9b..a9425f58 100644
--- a/sdk/python/src/openlit/semcov/__init__.py
+++ b/sdk/python/src/openlit/semcov/__init__.py
@@ -87,6 +87,7 @@ class SemanticConvetion:
GEN_AI_SYSTEM_MISTRAL = "mistral"
GEN_AI_SYSTEM_BEDROCK = "bedrock"
GEN_AI_SYSTEM_VERTEXAI = "vertexai"
+ GEN_AI_SYSTEM_GROQ = "groq"
GEN_AI_SYSTEM_LANGCHAIN = "langchain"
GEN_AI_SYSTEM_LLAMAINDEX = "llama_index"
GEN_AI_SYSTEM_HAYSTACK = "haystack"
diff --git a/sdk/python/tests/requirements.txt b/sdk/python/tests/requirements.txt
index a25d0900..370a350c 100644
--- a/sdk/python/tests/requirements.txt
+++ b/sdk/python/tests/requirements.txt
@@ -13,6 +13,7 @@ transformers>=4.39.3
boto3>=1.34.93
botocore>=1.34.93
google-cloud-aiplatform >= 1.38.1
+groq >= 0.5.0
# Vector DBs
chromadb>=0.4.0
diff --git a/sdk/python/tests/test_groq.py b/sdk/python/tests/test_groq.py
new file mode 100644
index 00000000..53df5dba
--- /dev/null
+++ b/sdk/python/tests/test_groq.py
@@ -0,0 +1,86 @@
+# pylint: disable=duplicate-code, no-name-in-module
+"""
+This module contains tests for Groq functionality using the Groq Python library.
+
+Tests cover various API endpoints, including chat.
+These tests validate integration with OpenLIT.
+
+Environment Variables:
+ - GROQ_API_TOKEN: Groq API api_key for authentication.
+
+Note: Ensure the environment is properly configured for Groq access and OpenLIT monitoring
+prior to running these tests.
+"""
+
+import os
+import pytest
+from groq import Groq, AsyncGroq
+import openlit
+
+# Initialize synchronous Groq client
+sync_client = Groq(
+ api_key=os.getenv("GROQ_API_TOKEN")
+)
+
+# Initialize asynchronous Groq client
+async_client = AsyncGroq(
+ api_key=os.getenv("GROQ_API_TOKEN")
+)
+
+# Initialize environment and application name for OpenLIT monitoring
+openlit.init(environment="openlit-testing", application_name="openlit-python-test")
+
+def test_sync_groq_chat():
+ """
+ Tests synchronous Chat Completions with the 'llama3-8b-8192' model.
+
+ Raises:
+ AssertionError: If the Chat Completions response object is not as expected.
+ """
+
+ try:
+ chat_completions_resp = sync_client.chat.completions.create(
+ messages=[
+ {
+ "role": "user",
+ "content": "Monitor LLM Applications",
+ }
+ ],
+ model="llama3-8b-8192",
+ max_tokens=1,
+ stream = False,
+ )
+ assert chat_completions_resp.object == 'chat.completion'
+
+ # pylint: disable=broad-exception-caught
+ except Exception as e:
+ if "rate limit" in str(e).lower():
+ print("Rate Limited:", e)
+
+async def test_async_groq_chat():
+ """
+ Tests synchronous Chat Completions with the 'claude-3-haiku-20240307' model.
+
+ Raises:
+ AssertionError: If the Chat Completions response object is not as expected.
+ """
+
+ try:
+ chat_completions_resp = await async_client.chat.completions.create(
+ messages=[
+ {
+ "role": "user",
+ "content": "What is LLM Observability?",
+ }
+ ],
+ model="llama3-8b-8192",
+ max_tokens=1,
+ stream=False,
+ )
+ assert chat_completions_resp.object == 'chat.completion'
+
+ # pylint: disable=broad-exception-caught
+ except Exception as e:
+ if "rate limit" in str(e).lower():
+ print("Rate Limited:", e)