Skip to content

Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.

[Feat]: Add OpenTelemetry Instrumentation for monitoring NVIDIA NIM (#410)

patcher9d8e175a15c4f
diff --git a/docs/images/docs-nvidia-gpu-banner.gif b/docs/images/docs-nvidia-banner.gif
similarity index 100%
rename from docs/images/docs-nvidia-gpu-banner.gif
rename to docs/images/docs-nvidia-banner.gif
diff --git a/docs/latest/integrations/introduction.mdx b/docs/latest/integrations/introduction.mdx
index c73fc1ba..6d257ea9 100644
--- a/docs/latest/integrations/introduction.mdx
+++ b/docs/latest/integrations/introduction.mdx
@@ -445,6 +445,17 @@ Start integrating and monitoring your applications with OpenLIT effortlessly. Ch
iconType="duotone"
>
</Card>
+ <Card
+ title="NVIDIA NIM"
+ href="/latest/integrations/nvidia-nim"
+ iconType="duotone"
+ icon={
+ <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 60 60">
+ <path d="M23.862 23.46v-3.816l1.13-.047c10.46-.33 17.313 8.998 17.313 8.998s-7.396 10.27-15.335 10.27a9.73 9.73 0 0 1-3.086-.495v-11.59c4.075.495 4.9 2.285 7.326 6.36l5.44-4.57s-3.98-5.206-10.67-5.206c-.707-.024-1.413.024-2.12.094m0-12.626v5.7l1.13-.07c14.534-.495 24.026 11.92 24.026 11.92S38.136 41.622 26.806 41.622c-.99 0-1.955-.094-2.92-.26v3.533c.8.094 1.625.165 2.426.165 10.553 0 18.185-5.394 25.58-11.754 1.225.99 6.242 3.368 7.28 4.405-7.02 5.89-23.39 10.623-32.67 10.623a23.24 23.24 0 0 1-2.591-.141v4.97H64v-42.33zm0 27.536v3.015C14.1 39.644 11.4 29.49 11.4 29.49s4.688-5.182 12.46-6.03v3.298h-.024c-4.075-.495-7.28 3.32-7.28 3.32s1.814 6.43 7.302 8.29M6.548 29.067s5.77-8.527 17.337-9.422v-3.11C11.07 17.572 0 28.408 0 28.408s6.266 18.138 23.862 19.787v-3.298c-12.908-1.602-17.313-15.83-17.313-15.83z" fill="#76b900"/>
+ </svg>
+ }
+ >
+ </Card>
<Card
title="ElevenLabs"
href="/latest/integrations/elevenlabs"
@@ -946,6 +957,7 @@ Start integrating and monitoring your applications with OpenLIT effortlessly. Ch
| Azure AI Inference | LLM | ✔️ | ❌ |
| Google AI Studio | LLM | ✔️ | ❌ |
| GitHub Models | LLM | ✔️ | ❌ |
+| NVIDIA NIM | LLM | ✔️ | ❌ |
| Mistral | LLM | ✔️ | ❌ |
| HuggingFace | LLM | ✔️ | ❌ |
| Amazon Bedrock | LLM | ✔️ | ❌ |
diff --git a/docs/latest/integrations/nvidia-gpu.mdx b/docs/latest/integrations/nvidia-gpu.mdx
index 36b052b1..848c097b 100644
--- a/docs/latest/integrations/nvidia-gpu.mdx
+++ b/docs/latest/integrations/nvidia-gpu.mdx
@@ -4,7 +4,7 @@ sidebarTitle: 'NVIDIA GPUs'
---
<Frame>
- <img src="/images/docs-nvidia-gpu-banner.gif" />
+ <img src="/images/docs-nvidia-banner.gif" />
</Frame>
OpenLIT uses OpenTelemetry to help you monitor NVIDIA GPUs. This includes tracking GPU metrics like utilization, temperature, memory usage and power consumption.
diff --git a/docs/latest/integrations/nvidia-nim.mdx b/docs/latest/integrations/nvidia-nim.mdx
new file mode 100644
index 00000000..f88ed9d6
--- /dev/null
+++ b/docs/latest/integrations/nvidia-nim.mdx
@@ -0,0 +1,115 @@
+---
+title: 'Monitor NVIDIA NIM using OpenTelemetry'
+sidebarTitle: 'NVIDIA NIM'
+---
+
+<Frame>
+ <img src="/images/docs-nvidia-banner.gif" />
+</Frame>
+
+OpenLIT uses OpenTelemetry Auto-Instrumentation to help you monitor LLM applications built using NVIDIA NIM models. This includes tracking performance, token usage, costs, and how users interact with the application.
+
+Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up.
+
+The integration is compatible with
+- OpenAI Python SDK client `>=1.13.0`
+- OpenAI Typescript SDK client `>=1.13.0`
+
+## Get Started
+
+<Steps>
+ <Step title="Install OpenLIT">
+ Open your command line or terminal and run:
+ <Tabs>
+ <Tab title="Python">
+ ```shell
+ pip install openlit
+ ```
+ </Tab>
+ <Tab title="Typescript">
+ ```shell
+ npm install openlit
+ ```
+ </Tab>
+ </Tabs>
+
+ </Step>
+ <Step title="Initialize OpenLIT in your Application">
+ You can set up OpenLIT in your application using either function arguments directly in your code or by using environment variables.
+ <Tabs>
+ <Tab title="Setup using function arguments">
+ Add the following two lines to your application code:
+ <Tabs>
+ <Tab title="Python">
+ ```python
+ import openlit
+
+ openlit.init(
+ otlp_endpoint="YOUR_OTEL_ENDPOINT",
+ )
+ ```
+
+ </Tab>
+ <Tab title="Typescript">
+ <Warning>Typescript sdk for openai doesn't support metrics tracking for now!</Warning>
+
+ ```typescript
+ import openlit from "openlit"
+
+ openlit.init({ otlpEndpoint: "YOUR_OTEL_ENDPOINT" })
+ ```
+ </Tab>
+ </Tabs>
+
+
+ Replace:
+ 1. `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` if you are using OpenLIT and a local OTel Collector.
+
+ </Tab>
+ <Tab title="Setup using Environment Variables">
+
+ Add the following two lines to your application code:
+
+ <Tabs>
+ <Tab title="Python">
+ ```python
+ import openlit
+
+ openlit.init()
+ ```
+
+ </Tab>
+ <Tab title="Typescript">
+ ```typescript
+ import openlit from "openlit"
+
+ openlit.init()
+ ```
+ </Tab>
+ </Tabs>
+
+ Then, configure the your OTLP endpoint using environment variable:
+ ```shell
+ export OTEL_EXPORTER_OTLP_ENDPOINT = "YOUR_OTEL_ENDPOINT"
+ ```
+
+ Replace:
+ 1. `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` if you are using OpenLIT and a local OTel Collector.
+ </Tab>
+ </Tabs>
+ To send metrics and traces to other Observability tools, refer to the [Connections Guide](/latest/connections/intro).
+
+ For more advanced configurations and application use cases, visit the [OpenLIT Python repository](https://github.com/openlit/openlit/tree/main/sdk/python) or [OpenLIT Typescript repository](https://github.com/openlit/openlit/tree/main/sdk/typescript).
+ </Step>
+</Steps>
+
+---
+
+<CardGroup cols={2}>
+<Card title="Connections" href="/latest/connections/intro" icon='link'>
+Connect to your existing Observablity Stack
+</Card>
+<Card title="SDK configuration" href="/latest/sdk-configuration" icon='file-code'>
+Documentation of the configuration options for the OpenLIT SDK.
+</Card>
+</CardGroup>
\ No newline at end of file
diff --git a/docs/mint.json b/docs/mint.json
index 8427ec24..4a8ffd28 100644
--- a/docs/mint.json
+++ b/docs/mint.json
@@ -79,6 +79,7 @@
"latest/integrations/vertexai",
"latest/integrations/google-ai-studio",
"latest/integrations/groq",
+ "latest/integrations/nvidia-nim",
"latest/integrations/elevenlabs",
"latest/integrations/krutrim"
]
diff --git a/sdk/python/README.md b/sdk/python/README.md
index 36153a3c..39a74f63 100644
--- a/sdk/python/README.md
+++ b/sdk/python/README.md
@@ -46,6 +46,7 @@ This project adheres to the [Semantic Conventions](https://github.com/open-telem
| [✅ vLLM](https://docs.openlit.io/latest/integrations/vllm) | | | |
| [✅ OLA Krutrim](https://docs.openlit.io/latest/integrations/krutrim) | | | |
| [✅ Google AI Studio](https://docs.openlit.io/latest/integrations/google-ai-studio) | | | |
+| [✅ NVIDIA NIM](https://docs.openlit.io/latest/integrations/nvidia-nim) | | | |
## Supported Destinations
- [✅ OpenTelemetry Collector](https://docs.openlit.io/latest/connections/otelcol)
diff --git a/sdk/python/pyproject.toml b/sdk/python/pyproject.toml
index 931a6560..6cd523b2 100644
--- a/sdk/python/pyproject.toml
+++ b/sdk/python/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "openlit"
-version = "1.22.0"
+version = "1.22.1"
description = "OpenTelemetry-native Auto instrumentation library for monitoring LLM Applications and GPUs, facilitating the integration of observability into your GenAI-driven projects"
authors = ["OpenLIT"]
repository = "https://github.com/openlit/openlit/tree/main/openlit/python"
diff --git a/sdk/python/src/openlit/__init__.py b/sdk/python/src/openlit/__init__.py
index bc1c648c..732d98aa 100644
--- a/sdk/python/src/openlit/__init__.py
+++ b/sdk/python/src/openlit/__init__.py
@@ -140,12 +140,17 @@ def instrument_if_available(
):
"""Instruments the specified instrumentor if its library is available."""
if instrumentor_name in disabled_instrumentors:
+ logger.info("Instrumentor %s is disabled", instrumentor_name)
return
module_name = module_name_map.get(instrumentor_name)
- if not module_name or find_spec(module_name) is not None:
- try:
+ if not module_name:
+ logger.error("No module mapping for %s", instrumentor_name)
+ return
+
+ try:
+ if find_spec(module_name) is not None:
instrumentor_instance.instrument(
environment=config.environment,
application_name=config.application_name,
@@ -155,10 +160,12 @@ def instrument_if_available(
metrics_dict=config.metrics_dict,
disable_metrics=config.disable_metrics,
)
-
- # pylint: disable=broad-exception-caught
- except Exception as e:
- logger.error("Failed to instrument %s: %s", instrumentor_name, e)
+ logger.info("Instrumented %s", instrumentor_name)
+ else:
+ # pylint: disable=line-too-long
+ logger.info("Library for %s (%s) not found. Skipping instrumentation", instrumentor_name, module_name)
+ except Exception as e:
+ logger.error("Failed to instrument %s: %s", instrumentor_name, e)
def init(environment="default", application_name="default", tracer=None, otlp_endpoint=None,
otlp_headers=None, disable_batch=False, trace_content=True, disabled_instrumentors=None,
@@ -184,7 +191,7 @@ def init(environment="default", application_name="default", tracer=None, otlp_en
collect_gpu_stats (bool): Flag to enable or disable GPU metrics collection.
"""
disabled_instrumentors = disabled_instrumentors if disabled_instrumentors else []
- # Check for invalid instrumentor names
+ logger.info("Starting openLIT initialization...")
module_name_map = {
"openai": "openai",
@@ -215,9 +222,7 @@ def init(environment="default", application_name="default", tracer=None, otlp_en
name for name in disabled_instrumentors if name not in module_name_map
]
for invalid_name in invalid_instrumentors:
- logger.warning(
- "Invalid instrumentor name detected and ignored: '%s'", invalid_name
- )
+ logger.warning("Invalid instrumentor name detected and ignored: '%s'", invalid_name)
try:
# Retrieve or create the single configuration instance.
@@ -237,7 +242,7 @@ def init(environment="default", application_name="default", tracer=None, otlp_en
logger.error("openLIT tracing setup failed. Tracing will not be available.")
return
- # Setup meter and receive metrics_dict instead of meter
+ # Setup meter and receive metrics_dict instead of meter.
metrics_dict = setup_meter(
application_name=application_name,
environment=environment,
@@ -293,14 +298,13 @@ def init(environment="default", application_name="default", tracer=None, otlp_en
# Initialize and instrument only the enabled instrumentors
for name, instrumentor in instrumentor_instances.items():
instrument_if_available(name, instrumentor, config,
- disabled_instrumentors, module_name_map)
+ disabled_instrumentors, module_name_map)
- if (disable_metrics is False) and (collect_gpu_stats is True):
+ if not disable_metrics and collect_gpu_stats:
NvidiaGPUInstrumentor().instrument(
environment=config.environment,
application_name=config.application_name,
)
-
except Exception as e:
logger.error("Error during openLIT initialization: %s", e)