Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
TypeScript agent-harnessagent-harness-engineeringai-engineeringai-observabilityamd-gpuclaude-codeclickhousecodexcoding-agentcoding-agentscursorgenaigpu-monitoringllm-observabilityllmopsnvidia-smiobservabilityopen-sourceopenaiopentelemetry
[Feat]: Add OpenTelemetry instrumentation for monitoring GPUs (#275)
patcher97640735810f7
diff --git a/README.md b/README.mdindex 66c03ae5..72c6d1db 100644--- a/README.md+++ b/README.md@@ -1,6 +1,6 @@<div align="center"><img src="https://github.com/openlit/.github/blob/main/profile/assets/wide-logo-no-bg.png?raw=true" alt="OpenLIT Logo" width="30%"><h1>-OpenTelemetry-native Observability & Evals for LLMs</h1>+OpenTelemetry-native Observability and Evals for LLMs & GPUs</h1>**[Documentation](https://docs.openlit.io/) | [Quickstart](#-getting-started) | [Python SDK](https://github.com/openlit/openlit/tree/main/sdk/python)**@@ -18,7 +18,9 @@ OpenTelemetry-native Observability & Evals for LLMs</h1>-OpenLIT is an **OpenTelemetry-native** GenAI and LLM Application Observability tool. It's designed to make the integration process of observability into GenAI projects as easy as pie – literally, with just **a single line of code**. Whether you're working with popular LLM Libraries such as OpenAI and HuggingFace or leveraging vector databases like ChromaDB, OpenLIT ensures your applications are monitored seamlessly, providing critical insights to improve performance and reliability.+**OpenLIT** is an **OpenTelemetry-native** tool designed to help developers gain insights into the performance of their LLM applications in production. It automatically collects LLM input and output metadata, and monitors GPU performance for self-hosted LLMs.++OpenLIT makes integrating observability into GenAI projects effortless with just **a single line of code**. Whether you're working with popular LLM providers such as OpenAI and HuggingFace, or leveraging vector databases like ChromaDB, OpenLIT ensures your applications are monitored seamlessly, providing critical insights including GPU performance stats for self-hosted LLMs to improve performance and reliability.This project proudly follows the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) of the OpenTelemetry community, consistently updating to align with the latest standards in observability.@@ -69,7 +71,7 @@ pip install openlit```### Step 3: Initialize OpenLIT in your Application-Integrating the OpenLIT into LLM applications is straightforward. Start monitoring for your LLM Application with just **two lines of code**:+Integrating OpenLIT into LLM applications is straightforward. Start monitoring for your LLM Application with just **two lines of code**:```pythonimport openlit@@ -149,7 +151,7 @@ Your input helps us grow and improve, and we're here to support you every step o## 💚 Community & Support-Connect with the OpenLIT community and maintainers for support, discussions, and updates:+Connect with OpenLIT community and maintainers for support, discussions, and updates:- 🌟 If you like it, Leave a star on our [GitHub](https://github.com/openlit/openlit/)- 🌍 Join our [Slack](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ) or [Discord](https://discord.gg/rjvTm6zd) community for live interactions and questions.diff --git a/docs/images/docs-main-banner.jpeg b/docs/images/docs-main-banner.jpegindex be7a622f..ecfcedf4 100644Binary files a/docs/images/docs-main-banner.jpeg and b/docs/images/docs-main-banner.jpeg differdiff --git a/docs/images/docs-nvidia-gpu-banner.gif b/docs/images/docs-nvidia-gpu-banner.gifnew file mode 100644index 00000000..6f9ffabaBinary files /dev/null and b/docs/images/docs-nvidia-gpu-banner.gif differdiff --git a/docs/latest/connections/grafanacloud.mdx b/docs/latest/connections/grafanacloud.mdxindex 3f79b115..47d11822 100644--- a/docs/latest/connections/grafanacloud.mdx+++ b/docs/latest/connections/grafanacloud.mdx@@ -94,7 +94,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"editable": true,"fiscalYearStartMonth": 0,"graphTooltip": 1,- "id": 12,+ "id": 14,"links": [{"asDropdown": false,@@ -140,10 +140,10 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"showLineNumbers": false,"showMiniMap": false},- "content": "---\n# GenAI Observability\n\nThis dashboard displays the usage of Large Language Models (LLM) and Vector Databases, tracking OpenTelemetry Traces and Metrics sent using [OpenLIT](https://github.com/openlit/openlit).\n\n---",+ "content": "---\n# GenAI Observability\n\nThis dashboard displays the usage stats of LLMs, Vector Databases and GPUs, tracking OpenTelemetry Traces and Metrics sent using [OpenLIT](https://github.com/openlit/openlit).\n\n---","mode": "markdown"},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","transparent": true,"type": "text"},@@ -193,6 +193,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"graphMode": "area","justifyMode": "auto","orientation": "auto",+ "percentChangeColorMode": "standard","reduceOptions": {"calcs": ["lastNotNull"@@ -204,7 +205,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"textMode": "value_and_name","wideLayout": true},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -272,6 +273,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"graphMode": "area","justifyMode": "auto","orientation": "auto",+ "percentChangeColorMode": "standard","reduceOptions": {"calcs": ["lastNotNull"@@ -283,7 +285,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"textMode": "value_and_name","wideLayout": true},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -347,6 +349,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"graphMode": "area","justifyMode": "auto","orientation": "auto",+ "percentChangeColorMode": "standard","reduceOptions": {"calcs": ["lastNotNull"@@ -358,7 +361,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"textMode": "auto","wideLayout": true},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -422,6 +425,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"graphMode": "area","justifyMode": "auto","orientation": "auto",+ "percentChangeColorMode": "standard","reduceOptions": {"calcs": ["lastNotNull"@@ -433,7 +437,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"textMode": "auto","wideLayout": true},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -496,6 +500,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"graphMode": "area","justifyMode": "auto","orientation": "auto",+ "percentChangeColorMode": "standard","reduceOptions": {"calcs": ["lastNotNull"@@ -507,7 +512,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"textMode": "auto","wideLayout": true},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -571,6 +576,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"graphMode": "area","justifyMode": "auto","orientation": "auto",+ "percentChangeColorMode": "standard","reduceOptions": {"calcs": ["lastNotNull"@@ -582,7 +588,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"textMode": "auto","wideLayout": true},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -650,6 +656,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"graphMode": "none","justifyMode": "auto","orientation": "auto",+ "percentChangeColorMode": "standard","reduceOptions": {"calcs": ["lastNotNull"@@ -661,7 +668,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"textMode": "auto","wideLayout": true},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -834,7 +841,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"sizing": "auto","valueMode": "text"},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -862,7 +869,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"type": "prometheus","uid": "${metricsDatasource}"},- "description": "This panel displays the distribution of GenAI requests across different platforms such as OpenAI, Cohere, Anthropic, etc. It shows where requests are being sent, giving insights into platform popularity and usage patterns. Monitoring this helps in understanding platform preferences, and it can guide strategic decisions for integration or diversification of GenAI services.",+ "description": "This panel displays the distribution of GenAI requests based on the environment, such as production, development, testing, etc. It shows where the requests are being utilized, providing a clear picture of operational focus and deployment strategies. Tracking this helps in aligning resources appropriately and optimizing the performance across different environments.","fieldConfig": {"defaults": {"color": {@@ -887,7 +894,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"x": 14,"y": 17},- "id": 16,+ "id": 15,"options": {"legend": {"displayMode": "list",@@ -916,7 +923,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl},"disableTextWrap": false,"editorMode": "code",- "expr": "sum by(gen_ai_system) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "expr": "sum by(gen_ai_environment) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})","fullMetaSearch": false,"includeNullMetadata": true,"instant": false,@@ -926,7 +933,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"useBackend": false}],- "title": "GenAI Requests by Platform",+ "title": "GenAI Requests by Environment","type": "piechart"},{@@ -934,7 +941,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"type": "prometheus","uid": "${metricsDatasource}"},- "description": "This panel displays the breakdown of GenAI requests by type, such as Chat, Embedding, Image, Audio, and Fine Tuning. It reveals the diversity in usage, highlighting which types of requests are most common. Understanding this distribution assists in optimizing resources for the most demanded services and planning for future needs based on usage trends.",+ "description": "This panel displays the distribution of GenAI requests based on the application, such as production, development, testing, etc. It shows where the requests are being utilized, providing a clear picture of operational focus and deployment strategies. Tracking this helps in aligning resources appropriately and optimizing the performance across different environments.","fieldConfig": {"defaults": {"color": {@@ -959,7 +966,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"x": 19,"y": 17},- "id": 13,+ "id": 27,"options": {"legend": {"displayMode": "list",@@ -988,7 +995,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl},"disableTextWrap": false,"editorMode": "code",- "expr": "sum by(gen_ai_type) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "expr": "sum by(gen_ai_application) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})","fullMetaSearch": false,"includeNullMetadata": true,"instant": false,@@ -998,7 +1005,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"useBackend": false}],- "title": "GenAI Requests by Type",+ "title": "GenAI Requests by Application","type": "piechart"},{@@ -1006,7 +1013,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"type": "prometheus","uid": "${metricsDatasource}"},- "description": "This panel displays the distribution of GenAI requests based on the environment, such as production, development, testing, etc. It shows where the requests are being utilized, providing a clear picture of operational focus and deployment strategies. Tracking this helps in aligning resources appropriately and optimizing the performance across different environments.",+ "description": "This panel displays the distribution of GenAI requests across different platforms such as OpenAI, Cohere, Anthropic, etc. It shows where requests are being sent, giving insights into platform popularity and usage patterns. Monitoring this helps in understanding platform preferences, and it can guide strategic decisions for integration or diversification of GenAI services.","fieldConfig": {"defaults": {"color": {@@ -1031,7 +1038,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"x": 14,"y": 22},- "id": 15,+ "id": 16,"options": {"legend": {"displayMode": "list",@@ -1060,7 +1067,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl},"disableTextWrap": false,"editorMode": "code",- "expr": "sum by(gen_ai_environment) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "expr": "sum by(gen_ai_system) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})","fullMetaSearch": false,"includeNullMetadata": true,"instant": false,@@ -1070,7 +1077,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"useBackend": false}],- "title": "GenAI Requests by Environment",+ "title": "GenAI Requests by Platform","type": "piechart"},{@@ -1078,7 +1085,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"type": "prometheus","uid": "${metricsDatasource}"},- "description": "This panel displays the distribution of GenAI requests based on the application, such as production, development, testing, etc. It shows where the requests are being utilized, providing a clear picture of operational focus and deployment strategies. Tracking this helps in aligning resources appropriately and optimizing the performance across different environments.",+ "description": "This panel displays the breakdown of GenAI requests by type, such as Chat, Embedding, Image, Audio, and Fine Tuning. It reveals the diversity in usage, highlighting which types of requests are most common. Understanding this distribution assists in optimizing resources for the most demanded services and planning for future needs based on usage trends.","fieldConfig": {"defaults": {"color": {@@ -1103,7 +1110,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"x": 19,"y": 22},- "id": 27,+ "id": 13,"options": {"legend": {"displayMode": "list",@@ -1132,7 +1139,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl},"disableTextWrap": false,"editorMode": "code",- "expr": "sum by(gen_ai_application) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "expr": "sum by(gen_ai_type) (gen_ai_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})","fullMetaSearch": false,"includeNullMetadata": true,"instant": false,@@ -1142,7 +1149,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"useBackend": false}],- "title": "GenAI Requests by Application",+ "title": "GenAI Requests by Type","type": "piechart"},{@@ -1284,7 +1291,7 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"type": "timeseries"},{- "collapsed": false,+ "collapsed": true,"gridPos": {"h": 1,"w": 24,@@ -1292,16 +1299,539 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"y": 35},"id": 10,- "panels": [],+ "panels": [+ {+ "datasource": {+ "type": "tempo",+ "uid": "${traceDatasource}"+ },+ "description": "This panel displays a detailed table of GenAI request traces, providing comprehensive insights into each request's timing, source, type, and outcome. It allows for the in-depth analysis of individual requests, facilitating troubleshooting, performance monitoring, and understanding user interactions with GenAI services. Tracking this helps in identifying patterns, potential issues, and opportunities for optimization.",+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "continuous-BlPu"+ },+ "custom": {+ "align": "auto",+ "cellOptions": {+ "type": "color-background"+ },+ "inspect": false+ },+ "mappings": [],+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "green"+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 16,+ "w": 24,+ "x": 0,+ "y": 36+ },+ "id": 8,+ "options": {+ "cellHeight": "sm",+ "footer": {+ "countRows": false,+ "fields": "",+ "reducer": [+ "sum"+ ],+ "show": false+ },+ "showHeader": true+ },+ "pluginVersion": "11.1.0-71516",+ "targets": [+ {+ "datasource": {+ "type": "tempo",+ "uid": "${traceDatasource}"+ },+ "filters": [+ {+ "id": "e7e29fde",+ "operator": "!=",+ "scope": "span",+ "tag": "gen_ai.operation.name",+ "value": [+ "vectordb"+ ],+ "valueType": "string"+ },+ {+ "id": "status",+ "operator": "=",+ "scope": "intrinsic",+ "tag": "status",+ "value": "ok",+ "valueType": "keyword"+ }+ ],+ "limit": 20,+ "query": "{span.gen_ai.operation.name!=\"vectordb\" && status=ok && span.gen_ai.application_name=~\"$application\" && span.gen_ai.environment=~\"$environment\"} | select(span.gen_ai.prompt, span.gen_ai.completion, span.gen_ai.usage.cost)",+ "queryType": "traceql",+ "refId": "A",+ "tableType": "traces"+ }+ ],+ "transparent": true,+ "type": "table"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "description": "",+ "gridPos": {+ "h": 2,+ "w": 24,+ "x": 0,+ "y": 52+ },+ "id": 25,+ "options": {+ "code": {+ "language": "plaintext",+ "showLineNumbers": false,+ "showMiniMap": false+ },+ "content": "### VectorDB\n\n---",+ "mode": "markdown"+ },+ "pluginVersion": "11.1.0-71516",+ "transparent": true,+ "type": "text"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "description": "This panel displays the breakdown of database requests based on the type of operation, such as add, updates, deletes, and queries. It provides insights into the operational dynamics of database interactions, highlighting the most common actions performed. Understanding this distribution helps in optimizing database performance and planning for capacity based on operational needs.",+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "continuous-BlYlRd"+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "green"+ },+ {+ "color": "red",+ "value": 80+ }+ ]+ },+ "unit": "none"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 9,+ "w": 14,+ "x": 0,+ "y": 54+ },+ "id": 20,+ "options": {+ "displayMode": "gradient",+ "maxVizHeight": 300,+ "minVizHeight": 16,+ "minVizWidth": 8,+ "namePlacement": "top",+ "orientation": "horizontal",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "showUnfilled": true,+ "sizing": "auto",+ "valueMode": "text"+ },+ "pluginVersion": "11.1.0-71516",+ "targets": [+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "code",+ "expr": "sum by(db_operation) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "fullMetaSearch": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "__auto",+ "range": true,+ "refId": "A",+ "useBackend": false+ }+ ],+ "title": "DB Requests by Operation",+ "transparent": true,+ "type": "bargauge"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "description": "This panel displays the distribution of requests across different VectorDB systems, such as Chroma, Pinecone, etc. It highlights which VectorDB services are most frequently used, providing insights into system preference and usage patterns. Monitoring this helps in evaluating the performance and scalability of each VectorDB system, guiding strategic decisions for technology adoption and integration.",+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ }+ },+ "mappings": []+ },+ "overrides": []+ },+ "gridPos": {+ "h": 5,+ "w": 10,+ "x": 14,+ "y": 54+ },+ "id": 18,+ "options": {+ "legend": {+ "displayMode": "list",+ "placement": "bottom",+ "showLegend": true+ },+ "pieType": "pie",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "tooltip": {+ "mode": "single",+ "sort": "none"+ }+ },+ "pluginVersion": "11.1.0-69950",+ "targets": [+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "code",+ "expr": "sum by(db_system) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "fullMetaSearch": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "__auto",+ "range": true,+ "refId": "A",+ "useBackend": false+ }+ ],+ "title": "DB Requests by System",+ "type": "piechart"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "description": "This panel displays the breakdown of DB requests by application name, offering insights into which applications are leveraging VectorDB services the most. This identification helps in understanding application-specific demand, guiding resource allocation, and supporting targeted optimization efforts for enhanced efficiency in application performance.\"",+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ }+ },+ "mappings": [],+ "min": 0,+ "unit": "none"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 4,+ "w": 5,+ "x": 14,+ "y": 59+ },+ "id": 19,+ "options": {+ "legend": {+ "displayMode": "list",+ "placement": "bottom",+ "showLegend": false+ },+ "pieType": "pie",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "tooltip": {+ "mode": "single",+ "sort": "none"+ }+ },+ "pluginVersion": "11.1.0-69372",+ "targets": [+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "builder",+ "expr": "sum by(gen_ai_application_name) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "fullMetaSearch": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "__auto",+ "range": true,+ "refId": "A",+ "useBackend": false+ }+ ],+ "title": "DB Requests by Application",+ "type": "piechart"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "description": "This panel displays the breakdown of DB requests by application name, offering insights into which applications are leveraging VectorDB services the most. This identification helps in understanding application-specific demand, guiding resource allocation, and supporting targeted optimization efforts for enhanced efficiency in application performance.\"",+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic"+ },+ "custom": {+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ }+ },+ "mappings": [],+ "min": 0,+ "unit": "none"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 4,+ "w": 5,+ "x": 19,+ "y": 59+ },+ "id": 26,+ "options": {+ "legend": {+ "displayMode": "list",+ "placement": "bottom",+ "showLegend": false+ },+ "pieType": "pie",+ "reduceOptions": {+ "calcs": [+ "lastNotNull"+ ],+ "fields": "",+ "values": false+ },+ "tooltip": {+ "mode": "single",+ "sort": "none"+ }+ },+ "pluginVersion": "11.1.0-69372",+ "targets": [+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "builder",+ "expr": "sum by(gen_ai_application_name) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "fullMetaSearch": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "__auto",+ "range": true,+ "refId": "A",+ "useBackend": false+ }+ ],+ "title": "DB Requests by Application",+ "type": "piechart"+ }+ ],"title": "GenAI Requests","type": "row"},+ {+ "collapsed": true,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 36+ },+ "id": 12,+ "panels": [+ {+ "datasource": {+ "type": "tempo",+ "uid": "${traceDatasource}"+ },+ "description": "This panel displays a table with detailed traces of VectorDB requests, offering insights into specifics like request timing, source, operation type, and results. It's designed for thorough examination of VectorDB interactions, enabling precise troubleshooting, performance evaluation, and user behavior understanding. By monitoring these traces, users can spot trends, diagnose problems, and improve operational efficiency in handling VectorDB requests.",+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic-by-name"+ },+ "custom": {+ "align": "auto",+ "cellOptions": {+ "type": "color-background"+ },+ "inspect": false+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "green"+ },+ {+ "color": "red",+ "value": 80+ }+ ]+ },+ "unit": "s"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 15,+ "w": 24,+ "x": 0,+ "y": 37+ },+ "id": 9,+ "options": {+ "cellHeight": "sm",+ "footer": {+ "countRows": false,+ "fields": "",+ "reducer": [+ "sum"+ ],+ "show": false+ },+ "showHeader": true+ },+ "pluginVersion": "11.1.0-71516",+ "targets": [+ {+ "datasource": {+ "type": "tempo",+ "uid": "${traceDatasource}"+ },+ "filters": [+ {+ "id": "e7e29fde",+ "operator": "=",+ "scope": "span",+ "tag": "gen_ai.operation.name",+ "value": [+ "vectordb"+ ],+ "valueType": "string"+ },+ {+ "id": "status",+ "operator": "=",+ "scope": "intrinsic",+ "tag": "status",+ "value": "ok",+ "valueType": "keyword"+ }+ ],+ "limit": 20,+ "query": "{span.gen_ai.operation.name=\"vectordb\" && status=ok && span.gen_ai.application_name=~\"$application\" && span.gen_ai.environment=~\"$environment\"}",+ "queryType": "traceql",+ "refId": "A",+ "tableType": "traces"+ }+ ],+ "transparent": true,+ "type": "table"+ }+ ],+ "title": "VectorDB Requests",+ "type": "row"+ },+ {+ "collapsed": false,+ "gridPos": {+ "h": 1,+ "w": 24,+ "x": 0,+ "y": 37+ },+ "id": 28,+ "panels": [],+ "title": "GPU",+ "type": "row"+ },{"datasource": {- "type": "tempo",- "uid": "${traceDatasource}"+ "type": "prometheus",+ "uid": "${metricsDatasource}"},- "description": "This panel displays a detailed table of GenAI request traces, providing comprehensive insights into each request's timing, source, type, and outcome. It allows for the in-depth analysis of individual requests, facilitating troubleshooting, performance monitoring, and understanding user interactions with GenAI services. Tracking this helps in identifying patterns, potential issues, and opportunities for optimization.","fieldConfig": {"defaults": {"color": {@@ -1310,10 +1840,14 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"custom": {"align": "auto","cellOptions": {+ "applyToRow": true,+ "mode": "gradient","type": "color-background"},- "inspect": false+ "filterable": false,+ "inspect": true},+ "displayName": "GPU","mappings": [],"thresholds": {"mode": "absolute",@@ -1321,64 +1855,79 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl{"color": "green","value": null+ },+ {+ "color": "red",+ "value": 80}]},- "unit": "s"+ "unit": "string"},"overrides": []},"gridPos": {- "h": 16,+ "h": 2,"w": 24,"x": 0,- "y": 36+ "y": 38},- "id": 8,+ "id": 43,"options": {- "cellHeight": "sm",+ "cellHeight": "lg","footer": {"countRows": false,+ "enablePagination": false,"fields": "","reducer": ["sum"],"show": false},- "showHeader": true+ "showHeader": false},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {- "type": "tempo",- "uid": "${traceDatasource}"+ "type": "prometheus",+ "uid": "${metricsDatasource}"},- "filters": [- {- "id": "e7e29fde",- "operator": "!=",- "scope": "span",- "tag": "gen_ai.operation.name",- "value": [- "vectordb"- ],- "valueType": "string"+ "disableTextWrap": false,+ "editorMode": "code",+ "expr": "gpu_fan_speed{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",+ "format": "table",+ "fullMetaSearch": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "__auto",+ "range": true,+ "refId": "A",+ "useBackend": false+ }+ ],+ "transformations": [+ {+ "id": "organize",+ "options": {+ "excludeByName": {+ "Time": true,+ "Value": true,+ "__name__": true,+ "gen_ai_application_name": true,+ "gen_ai_environment": true,+ "gpu_index": true,+ "gpu_uuid": true,+ "job": true,+ "telemetry_sdk_name": true},- {- "id": "status",- "operator": "=",- "scope": "intrinsic",- "tag": "status",- "value": "ok",- "valueType": "keyword"+ "includeByName": {},+ "indexByName": {},+ "renameByName": {+ "gpu_name": "Name",+ "gpu_uuid": "UUID"}- ],- "limit": 20,- "query": "{span.gen_ai.operation.name!=\"vectordb\" && status=ok && span.gen_ai.application_name=~\"$application\" && span.gen_ai.environment=~\"$environment\"} | select(span.gen_ai.prompt, span.gen_ai.completion, span.gen_ai.usage.cost)",- "queryType": "traceql",- "refId": "A",- "tableType": "traces"+ }}],"transparent": true,@@ -1389,71 +1938,41 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"type": "prometheus","uid": "${metricsDatasource}"},- "description": "",- "gridPos": {- "h": 2,- "w": 24,- "x": 0,- "y": 52- },- "id": 25,- "options": {- "code": {- "language": "plaintext",- "showLineNumbers": false,- "showMiniMap": false- },- "content": "### VectorDB\n\n---",- "mode": "markdown"- },- "pluginVersion": "11.1.0-69950",- "transparent": true,- "type": "text"- },- {- "datasource": {- "type": "prometheus",- "uid": "${metricsDatasource}"- },- "description": "This panel displays the breakdown of database requests based on the type of operation, such as add, updates, deletes, and queries. It provides insights into the operational dynamics of database interactions, highlighting the most common actions performed. Understanding this distribution helps in optimizing database performance and planning for capacity based on operational needs.",+ "description": "This panel displays the LLM Request Rate ","fieldConfig": {"defaults": {"color": {- "mode": "continuous-BlYlRd"+ "fixedColor": "blue",+ "mode": "continuous-GrYlRd"},"mappings": [],"min": 0,"thresholds": {"mode": "absolute","steps": [- {- "color": "green",- "value": null- },{"color": "red",- "value": 80+ "value": null}]},- "unit": "none"+ "unit": "percent"},"overrides": []},"gridPos": {- "h": 9,- "w": 14,+ "h": 3,+ "w": 6,"x": 0,- "y": 54+ "y": 40},- "id": 20,+ "id": 34,"options": {- "displayMode": "gradient",- "maxVizHeight": 300,- "minVizHeight": 16,- "minVizWidth": 8,- "namePlacement": "top",- "orientation": "horizontal",+ "colorMode": "background",+ "graphMode": "area",+ "justifyMode": "auto",+ "orientation": "auto",+ "percentChangeColorMode": "inverted","reduceOptions": {"calcs": ["lastNotNull"@@ -1461,11 +1980,11 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"fields": "","values": false},- "showUnfilled": true,- "sizing": "auto",- "valueMode": "text"+ "showPercentChange": true,+ "textMode": "value_and_name",+ "wideLayout": true},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -1474,56 +1993,59 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl},"disableTextWrap": false,"editorMode": "code",- "expr": "sum by(db_operation) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "expr": "avg(gpu_utilization{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})","fullMetaSearch": false,"includeNullMetadata": true,"instant": false,- "legendFormat": "__auto",+ "legendFormat": "Average GPU Utilization","range": true,"refId": "A","useBackend": false}],- "title": "DB Requests by Operation","transparent": true,- "type": "bargauge"+ "type": "stat"},{"datasource": {"type": "prometheus","uid": "${metricsDatasource}"},- "description": "This panel displays the distribution of requests across different VectorDB systems, such as Chroma, Pinecone, etc. It highlights which VectorDB services are most frequently used, providing insights into system preference and usage patterns. Monitoring this helps in evaluating the performance and scalability of each VectorDB system, guiding strategic decisions for technology adoption and integration.",+ "description": "","fieldConfig": {"defaults": {"color": {- "mode": "palette-classic"+ "fixedColor": "blue",+ "mode": "continuous-GrYlRd"},- "custom": {- "hideFrom": {- "legend": false,- "tooltip": false,- "viz": false- }+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "red",+ "value": null+ }+ ]},- "mappings": []+ "unit": "celsius"},"overrides": []},"gridPos": {- "h": 5,- "w": 10,- "x": 14,- "y": 54+ "h": 3,+ "w": 6,+ "x": 6,+ "y": 40},- "id": 18,+ "id": 36,"options": {- "legend": {- "displayMode": "list",- "placement": "bottom",- "showLegend": true- },- "pieType": "pie",+ "colorMode": "background",+ "graphMode": "area",+ "justifyMode": "auto",+ "orientation": "auto",+ "percentChangeColorMode": "inverted","reduceOptions": {"calcs": ["lastNotNull"@@ -1531,12 +2053,11 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"fields": "","values": false},- "tooltip": {- "mode": "single",- "sort": "none"- }+ "showPercentChange": true,+ "textMode": "value_and_name",+ "wideLayout": true},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -1545,57 +2066,59 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl},"disableTextWrap": false,"editorMode": "code",- "expr": "sum by(db_system) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "expr": "avg(gpu_temperature{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})","fullMetaSearch": false,"includeNullMetadata": true,"instant": false,- "legendFormat": "__auto",+ "legendFormat": "Average GPU Temperature","range": true,"refId": "A","useBackend": false}],- "title": "DB Requests by System",- "type": "piechart"+ "transparent": true,+ "type": "stat"},{"datasource": {"type": "prometheus","uid": "${metricsDatasource}"},- "description": "This panel displays the breakdown of DB requests by application name, offering insights into which applications are leveraging VectorDB services the most. This identification helps in understanding application-specific demand, guiding resource allocation, and supporting targeted optimization efforts for enhanced efficiency in application performance.\"",+ "description": "This panel displays the LLM Request Rate ","fieldConfig": {"defaults": {"color": {- "mode": "palette-classic"- },- "custom": {- "hideFrom": {- "legend": false,- "tooltip": false,- "viz": false- }+ "fixedColor": "blue",+ "mode": "continuous-GrYlRd"},"mappings": [],"min": 0,- "unit": "none"+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "red",+ "value": null+ }+ ]+ },+ "unit": "watt"},"overrides": []},"gridPos": {- "h": 4,- "w": 5,- "x": 14,- "y": 59+ "h": 3,+ "w": 6,+ "x": 12,+ "y": 40},- "id": 19,+ "id": 35,"options": {- "legend": {- "displayMode": "list",- "placement": "bottom",- "showLegend": false- },- "pieType": "pie",+ "colorMode": "background",+ "graphMode": "area",+ "justifyMode": "auto",+ "orientation": "auto",+ "percentChangeColorMode": "standard","reduceOptions": {"calcs": ["lastNotNull"@@ -1603,12 +2126,11 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"fields": "","values": false},- "tooltip": {- "mode": "single",- "sort": "none"- }+ "showPercentChange": true,+ "textMode": "value_and_name",+ "wideLayout": true},- "pluginVersion": "11.1.0-69372",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -1616,58 +2138,60 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"uid": "${metricsDatasource}"},"disableTextWrap": false,- "editorMode": "builder",- "expr": "sum by(gen_ai_application_name) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "editorMode": "code",+ "expr": "avg(gpu_power_draw{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})","fullMetaSearch": false,"includeNullMetadata": true,"instant": false,- "legendFormat": "__auto",+ "legendFormat": "Average GPU Power Draw","range": true,"refId": "A","useBackend": false}],- "title": "DB Requests by Application",- "type": "piechart"+ "transparent": true,+ "type": "stat"},{"datasource": {"type": "prometheus","uid": "${metricsDatasource}"},- "description": "This panel displays the breakdown of DB requests by application name, offering insights into which applications are leveraging VectorDB services the most. This identification helps in understanding application-specific demand, guiding resource allocation, and supporting targeted optimization efforts for enhanced efficiency in application performance.\"",+ "description": "This panel displays the LLM Request Rate ","fieldConfig": {"defaults": {"color": {- "mode": "palette-classic"- },- "custom": {- "hideFrom": {- "legend": false,- "tooltip": false,- "viz": false- }+ "fixedColor": "blue",+ "mode": "continuous-GrYlRd"},"mappings": [],"min": 0,- "unit": "none"+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "red",+ "value": null+ }+ ]+ },+ "unit": "percent"},"overrides": []},"gridPos": {- "h": 4,- "w": 5,- "x": 19,- "y": 59+ "h": 3,+ "w": 6,+ "x": 18,+ "y": 40},- "id": 26,+ "id": 37,"options": {- "legend": {- "displayMode": "list",- "placement": "bottom",- "showLegend": false- },- "pieType": "pie",+ "colorMode": "background",+ "graphMode": "area",+ "justifyMode": "auto",+ "orientation": "auto",+ "percentChangeColorMode": "inverted","reduceOptions": {"calcs": ["lastNotNull"@@ -1675,12 +2199,11 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"fields": "","values": false},- "tooltip": {- "mode": "single",- "sort": "none"- }+ "showPercentChange": true,+ "textMode": "value_and_name",+ "wideLayout": true},- "pluginVersion": "11.1.0-69372",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {@@ -1688,52 +2211,67 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"uid": "${metricsDatasource}"},"disableTextWrap": false,- "editorMode": "builder",- "expr": "sum by(gen_ai_application_name) (db_requests_total{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})",+ "editorMode": "code",+ "expr": "avg(gpu_memory_used{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}/gpu_memory_available{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"})","fullMetaSearch": false,"includeNullMetadata": true,"instant": false,- "legendFormat": "__auto",+ "legendFormat": "Average GPU Memory Usage","range": true,"refId": "A","useBackend": false}],- "title": "DB Requests by Application",- "type": "piechart"- },- {- "collapsed": false,- "gridPos": {- "h": 1,- "w": 24,- "x": 0,- "y": 63- },- "id": 12,- "panels": [],- "title": "VectorDB Requests",- "type": "row"+ "transparent": true,+ "type": "stat"},{"datasource": {- "type": "tempo",- "uid": "${traceDatasource}"+ "type": "prometheus",+ "uid": "${metricsDatasource}"},- "description": "This panel displays a table with detailed traces of VectorDB requests, offering insights into specifics like request timing, source, operation type, and results. It's designed for thorough examination of VectorDB interactions, enabling precise troubleshooting, performance evaluation, and user behavior understanding. By monitoring these traces, users can spot trends, diagnose problems, and improve operational efficiency in handling VectorDB requests.","fieldConfig": {"defaults": {"color": {"mode": "palette-classic-by-name"},"custom": {- "align": "auto",- "cellOptions": {- "type": "color-background"+ "axisBorderShow": true,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "Percentage",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "drawStyle": "line",+ "fillOpacity": 1,+ "gradientMode": "hue",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineStyle": {+ "fill": "solid"+ },+ "lineWidth": 1,+ "pointSize": 5,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "auto",+ "spanNulls": 3600000,+ "stacking": {+ "group": "A",+ "mode": "none"},- "inspect": false+ "thresholdsStyle": {+ "mode": "off"+ }},"mappings": [],+ "max": 100,"min": 0,"thresholds": {"mode": "absolute",@@ -1748,65 +2286,577 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl}]},- "unit": "s"+ "unit": "percent"},"overrides": []},"gridPos": {- "h": 15,- "w": 24,+ "h": 9,+ "w": 12,"x": 0,- "y": 64+ "y": 43},- "id": 9,+ "id": 38,"options": {- "cellHeight": "sm",- "footer": {- "countRows": false,- "fields": "",- "reducer": [- "sum"+ "legend": {+ "calcs": [+ "lastNotNull",+ "mean",+ "max"],- "show": false+ "displayMode": "table",+ "placement": "bottom",+ "showLegend": true},- "showHeader": true+ "tooltip": {+ "mode": "single",+ "sort": "none"+ }},- "pluginVersion": "11.1.0-69950",+ "pluginVersion": "11.1.0-71516","targets": [{"datasource": {- "type": "tempo",- "uid": "${traceDatasource}"+ "type": "prometheus",+ "uid": "${metricsDatasource}"},- "filters": [- {- "id": "e7e29fde",- "operator": "=",- "scope": "span",- "tag": "gen_ai.operation.name",- "value": [- "vectordb"- ],- "valueType": "string"- },- {- "id": "status",- "operator": "=",- "scope": "intrinsic",- "tag": "status",- "value": "ok",- "valueType": "keyword"- }- ],- "limit": 20,- "query": "{span.gen_ai.operation.name=\"vectordb\" && status=ok && span.gen_ai.application_name=~\"$application\" && span.gen_ai.environment=~\"$environment\"}",- "queryType": "traceql",+ "disableTextWrap": false,+ "editorMode": "code",+ "expr": "gpu_utilization{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",+ "fullMetaSearch": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",+ "range": true,"refId": "A",- "tableType": "traces"- }- ],- "transparent": true,- "type": "table"+ "useBackend": false+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "builder",+ "expr": "gpu_dec_utilization{telemetry_sdk_name=\"openlit\"}",+ "fullMetaSearch": false,+ "hide": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",+ "range": true,+ "refId": "B",+ "useBackend": false+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "builder",+ "expr": "gpu_enc_utilization{telemetry_sdk_name=\"openlit\"}",+ "fullMetaSearch": false,+ "hide": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",+ "range": true,+ "refId": "C",+ "useBackend": false+ }+ ],+ "title": "GPU Utilization",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic-by-name"+ },+ "custom": {+ "axisBorderShow": true,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "Temperature (°C)",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "drawStyle": "line",+ "fillOpacity": 1,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineStyle": {+ "fill": "solid"+ },+ "lineWidth": 1,+ "pointSize": 5,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "auto",+ "spanNulls": true,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "max": 100,+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "green",+ "value": null+ },+ {+ "color": "red",+ "value": 80+ }+ ]+ },+ "unit": "celsius"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 9,+ "w": 12,+ "x": 12,+ "y": 43+ },+ "id": 39,+ "options": {+ "legend": {+ "calcs": [+ "lastNotNull",+ "mean",+ "max"+ ],+ "displayMode": "table",+ "placement": "bottom",+ "showLegend": true+ },+ "tooltip": {+ "mode": "single",+ "sort": "none"+ }+ },+ "pluginVersion": "11.1.0-71516",+ "targets": [+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "code",+ "expr": "gpu_temperature{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",+ "fullMetaSearch": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",+ "range": true,+ "refId": "A",+ "useBackend": false+ }+ ],+ "title": "GPU Temperature",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic-by-name"+ },+ "custom": {+ "axisBorderShow": true,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "drawStyle": "bars",+ "fillOpacity": 50,+ "gradientMode": "hue",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "linear",+ "lineStyle": {+ "fill": "solid"+ },+ "lineWidth": 1,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "always",+ "spanNulls": 3600000,+ "stacking": {+ "group": "A",+ "mode": "percent"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "green",+ "value": null+ },+ {+ "color": "red",+ "value": 80+ }+ ]+ },+ "unit": "decmbytes"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 9,+ "w": 12,+ "x": 0,+ "y": 52+ },+ "id": 40,+ "options": {+ "legend": {+ "calcs": [+ "lastNotNull",+ "mean",+ "max"+ ],+ "displayMode": "table",+ "placement": "bottom",+ "showLegend": true+ },+ "tooltip": {+ "mode": "single",+ "sort": "none"+ }+ },+ "pluginVersion": "11.1.0-71516",+ "targets": [+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "code",+ "expr": "gpu_memory_used{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",+ "fullMetaSearch": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "Memory Used",+ "range": true,+ "refId": "A",+ "useBackend": false+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "builder",+ "expr": "gpu_memory_free{telemetry_sdk_name=\"openlit\"}",+ "fullMetaSearch": false,+ "hide": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "Memory Free",+ "range": true,+ "refId": "C",+ "useBackend": false+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "builder",+ "expr": "gpu_memory_total{telemetry_sdk_name=\"openlit\"}",+ "fullMetaSearch": false,+ "hide": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "Memory Total",+ "range": true,+ "refId": "B",+ "useBackend": false+ }+ ],+ "title": "GPU Memory Usage",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic-by-name"+ },+ "custom": {+ "axisBorderShow": true,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "Fan Speed (0-100)",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "drawStyle": "line",+ "fillOpacity": 1,+ "gradientMode": "opacity",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "smooth",+ "lineStyle": {+ "fill": "solid"+ },+ "lineWidth": 1,+ "pointSize": 5,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "auto",+ "spanNulls": true,+ "stacking": {+ "group": "A",+ "mode": "none"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "max": 100,+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "green",+ "value": null+ },+ {+ "color": "red",+ "value": 80+ }+ ]+ },+ "unit": "none"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 9,+ "w": 12,+ "x": 12,+ "y": 52+ },+ "id": 41,+ "options": {+ "legend": {+ "calcs": [+ "lastNotNull",+ "mean",+ "max"+ ],+ "displayMode": "table",+ "placement": "bottom",+ "showLegend": true+ },+ "tooltip": {+ "mode": "single",+ "sort": "none"+ }+ },+ "pluginVersion": "11.1.0-71516",+ "targets": [+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "code",+ "expr": "gpu_fan_speed{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",+ "fullMetaSearch": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "{{gpu_uuid}}-{{gpu_name}}",+ "range": true,+ "refId": "A",+ "useBackend": false+ }+ ],+ "title": "GPU Fan Speed",+ "type": "timeseries"+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "fieldConfig": {+ "defaults": {+ "color": {+ "mode": "palette-classic-by-name"+ },+ "custom": {+ "axisBorderShow": true,+ "axisCenteredZero": false,+ "axisColorMode": "text",+ "axisLabel": "",+ "axisPlacement": "auto",+ "barAlignment": 0,+ "drawStyle": "bars",+ "fillOpacity": 50,+ "gradientMode": "hue",+ "hideFrom": {+ "legend": false,+ "tooltip": false,+ "viz": false+ },+ "insertNulls": false,+ "lineInterpolation": "linear",+ "lineStyle": {+ "fill": "solid"+ },+ "lineWidth": 1,+ "pointSize": 3,+ "scaleDistribution": {+ "type": "linear"+ },+ "showPoints": "always",+ "spanNulls": 3600000,+ "stacking": {+ "group": "A",+ "mode": "percent"+ },+ "thresholdsStyle": {+ "mode": "off"+ }+ },+ "mappings": [],+ "min": 0,+ "thresholds": {+ "mode": "absolute",+ "steps": [+ {+ "color": "green",+ "value": null+ },+ {+ "color": "red",+ "value": 80+ }+ ]+ },+ "unit": "watt"+ },+ "overrides": []+ },+ "gridPos": {+ "h": 7,+ "w": 24,+ "x": 0,+ "y": 61+ },+ "id": 42,+ "options": {+ "legend": {+ "calcs": [+ "lastNotNull",+ "mean",+ "max"+ ],+ "displayMode": "table",+ "placement": "bottom",+ "showLegend": true+ },+ "tooltip": {+ "mode": "single",+ "sort": "none"+ }+ },+ "pluginVersion": "11.1.0-71516",+ "targets": [+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "code",+ "expr": "gpu_power_draw{telemetry_sdk_name=\"openlit\", gen_ai_application_name=~\"$application\", gen_ai_environment=~\"$environment\"}",+ "fullMetaSearch": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "Power Draw",+ "range": true,+ "refId": "A",+ "useBackend": false+ },+ {+ "datasource": {+ "type": "prometheus",+ "uid": "${metricsDatasource}"+ },+ "disableTextWrap": false,+ "editorMode": "builder",+ "expr": "gpu_power_limit{telemetry_sdk_name=\"openlit\"}",+ "fullMetaSearch": false,+ "hide": false,+ "includeNullMetadata": true,+ "instant": false,+ "legendFormat": "Power Limit",+ "range": true,+ "refId": "C",+ "useBackend": false+ }+ ],+ "title": "GPU Memory Usage",+ "type": "timeseries"}],"refresh": "",@@ -1814,7 +2864,8 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl"tags": ["LLM","VectorDB",- "GenAI"+ "GenAI",+ "GPU"],"templating": {"list": [@@ -1913,15 +2964,14 @@ To send OpenTelemetry metrics and traces generated by OpenLIT from your LLM Appl]},"time": {- "from": "now-24h",+ "from": "now-1h","to": "now"},- "timeRangeUpdatedDuringEditOrView": false,"timepicker": {},"timezone": "browser","title": "GenAI Observability","uid": "cdiz9piuoa3gge",- "version": 15,+ "version": 2,"weekStart": ""}```diff --git a/docs/latest/features/metrics.mdx b/docs/latest/features/metrics.mdxindex 0a4feebb..ba13f13e 100644--- a/docs/latest/features/metrics.mdx+++ b/docs/latest/features/metrics.mdx@@ -62,6 +62,23 @@ This section outlines the OpenTelemetry metrics collected by **OpenLIT** from ap|-------------------------|--------------------------------------------|-------|-------------|--------------------------------------------------------------|| `db.total.requests` | Number of requests to VectorDBs. | `1` | Counter | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment` |+### GPU Metrics++| Metric Name | Description | Unit | Type | Attributes |+|-------------------------------------|------------------------------------------|------------|-------|------------------------------------------------------------------------------------------------------------|+| `gpu.utilization_percentage` | GPU Utilization in percentage | `percent` | Gauge | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment`, `gpu_index`, `gpu_name`, `gpu_uuid` |+| `gpu.enc.utilization_percentage` | GPU encoder Utilization in percentage | `percent` | Gauge | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment`, `gpu_index`, `gpu_name`, `gpu_uuid` |+| `gpu.dec.utilization_percentage` | GPU decoder Utilization in percentage | `percent` | Gauge | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment`, `gpu_index`, `gpu_name`, `gpu_uuid` |+| `gpu.temperature` | GPU Temperature in Celsius | `Celcius` | Gauge | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment`, `gpu_index`, `gpu_name`, `gpu_uuid` |+| `gpu.fan_speed` | GPU Fan Speed (0-100) as an integer | `Integer` | Gauge | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment`, `gpu_index`, `gpu_name`, `gpu_uuid` |+| `gpu.memory.available` | Available GPU Memory in MB | `MB` | Gauge | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment`, `gpu_index`, `gpu_name`, `gpu_uuid` |+| `gpu.memory.total` | Total GPU Memory in MB | `MB` | Gauge | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment`, `gpu_index`, `gpu_name`, `gpu_uuid` |+| `gpu.memory.used` | Used GPU Memory in MB | `MB` | Gauge | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment`, `gpu_index`, `gpu_name`, `gpu_uuid` |+| `gpu.memory.free` | Free GPU Memory in MB | `MB` | Gauge | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment`, `gpu_index`, `gpu_name`, `gpu_uuid` |+| `gpu.power.draw` | GPU Power Draw in Watts | `Watt` | Gauge | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment`, `gpu_index`, `gpu_name`, `gpu_uuid` |+| `gpu.power.limit` | GPU Power Limit in Watts | `Watt` | Gauge | `telemetry.sdk.name`, `gen_ai.application_name`, `gen_ai.environment`, `gpu_index`, `gpu_name`, `gpu_uuid` |++<CardGroup cols={2}><Card title="Integrations" href="/latest/integrations/introduction" icon='circle-nodes'>Integrate your LLM Stack with OpenLITdiff --git a/docs/latest/integrations/introduction.mdx b/docs/latest/integrations/introduction.mdxindex 5eb7d761..e209f697 100644--- a/docs/latest/integrations/introduction.mdx+++ b/docs/latest/integrations/introduction.mdx@@ -707,6 +707,21 @@ Start integrating and monitoring your applications with OpenLIT effortlessly. Ch</Card></CardGroup>+### GPUs++<CardGroup>+ <Card+ title="NVIDIA GPUs"+ href="/latest/integrations/nvidia-gpu"+ iconType="duotone"+ icon={+ <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 60 60">+ <path d="M23.862 23.46v-3.816l1.13-.047c10.46-.33 17.313 8.998 17.313 8.998s-7.396 10.27-15.335 10.27a9.73 9.73 0 0 1-3.086-.495v-11.59c4.075.495 4.9 2.285 7.326 6.36l5.44-4.57s-3.98-5.206-10.67-5.206c-.707-.024-1.413.024-2.12.094m0-12.626v5.7l1.13-.07c14.534-.495 24.026 11.92 24.026 11.92S38.136 41.622 26.806 41.622c-.99 0-1.955-.094-2.92-.26v3.533c.8.094 1.625.165 2.426.165 10.553 0 18.185-5.394 25.58-11.754 1.225.99 6.242 3.368 7.28 4.405-7.02 5.89-23.39 10.623-32.67 10.623a23.24 23.24 0 0 1-2.591-.141v4.97H64v-42.33zm0 27.536v3.015C14.1 39.644 11.4 29.49 11.4 29.49s4.688-5.182 12.46-6.03v3.298h-.024c-4.075-.495-7.28 3.32-7.28 3.32s1.814 6.43 7.302 8.29M6.548 29.067s5.77-8.527 17.337-9.422v-3.11C11.07 17.572 0 28.408 0 28.408s6.266 18.138 23.862 19.787v-3.298c-12.908-1.602-17.313-15.83-17.313-15.83z" fill="#76b900"/>+ </svg>+ }+ >+ </Card>+</CardGroup>---<CardGroup cols={2}>diff --git a/docs/latest/integrations/nvidia-gpu.mdx b/docs/latest/integrations/nvidia-gpu.mdxnew file mode 100644index 00000000..73c11fb6--- /dev/null+++ b/docs/latest/integrations/nvidia-gpu.mdx@@ -0,0 +1,76 @@+---+title: 'Monitor NVIDIA GPUs using OpenTelemetry'+sidebarTitle: 'NVIDIA GPUs'+---++<Frame>+ <img src="/images/docs-nvidia-gpu-banner.gif" />+</Frame>++OpenLIT uses OpenTelemetry to help you monitor NVIDIA GPUs. This includes tracking GPU metrics like utilization, temperature, memory usage and power consumption.++Auto-instrumentation means you don't have to set up monitoring manually. By simply adding the OpenLIT in your application, all the necessary monitoring configurations are automatically set up for you.++## Get Started++<Steps>+ <Step title="Install OpenLIT">+ Open your command line or terminal and run:+ ```shell+ pip install openlit+ ```+ </Step>+ <Step title="Initialize OpenLIT in your Application">+ You can set up the OpenLIT in your application using either function arguments directly in your code or by using environment variables.+ <Tabs>+ <Tab title="Setup using function arguments">++ Add the following two lines to your application code:++ ```python+ import openlit++ openlit.init(+ otlp_endpoint="YOUR_OTEL_ENDPOINT",+ collect_gpu_stats=True+ )+ ```++ Replace:+ 1. `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` if you are using OpenLIT and a local OTel Collector.+ </Tab>+ <Tab title="Setup using Environment Variables">++ Add the following two lines to your application code:++ ```python+ import openlit++ openlit.init(collect_gpu_stats=True)+ ```++ Then, configure the your OTLP endpoint using environment variable:+ ```shell+ export OTEL_EXPORTER_OTLP_ENDPOINT = "YOUR_OTEL_ENDPOINT"+ ```++ Replace:+ 1. `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` if you are using OpenLIT and a local OTel Collector.+ </Tab>+ </Tabs>+ To send metrics to other Observability tools, refer to the [Connections Guide](/latest/connections/intro).++ For more advanced configurations and application use cases, visit the [OpenLIT Python repository](https://github.com/openlit/openlit/tree/main/sdk/python).+ </Step>+</Steps>++---++<CardGroup cols={2}>+<Card title="Connections" href="/latest/connections/intro" icon='link'>+Connect to your existing Observablity Stack+</Card>+<Card title="SDK configuration" href="/latest/sdk-configuration" icon='file-code'>+Documentation of the configuration options for the OpenLIT SDK.+</Card>+</CardGroup>\ No newline at end of filediff --git a/docs/latest/introduction.mdx b/docs/latest/introduction.mdxindex f2610e0a..ff0befa7 100644--- a/docs/latest/introduction.mdx+++ b/docs/latest/introduction.mdx@@ -1,16 +1,18 @@---title: What is OpenLIT?-description: 'OpenTelemetry-native LLM Application Observability'+description: 'OpenTelemetry-native Observability and Evaluations for LLMs and GPUs'---<Frame><img src="/images/docs-main-banner.jpeg" /></Frame>+**OpenLIT** is an **OpenTelemetry-native** tool designed to help developers gain insights into the performance of their LLM applications in production. It automatically collects LLM input and output metadata, and monitors GPU performance for self-hosted LLMs.-OpenLIT is an **OpenTelemetry-native** GenAI and LLM Application Observability tool. It's designed to make the integration process of observability into GenAI projects as easy as pie – literally, with just **a single line of code**. Whether you're working with popular LLM Libraries such as OpenAI and HuggingFace or leveraging vector databases like ChromaDB, OpenLIT ensures your applications are monitored seamlessly, providing critical insights to improve performance and reliability.+OpenLIT makes integrating observability into GenAI projects effortless with just **a single line of code**. Whether you're working with popular LLM providers such as OpenAI and HuggingFace, or leveraging vector databases like ChromaDB, OpenLIT ensures your applications are monitored seamlessly, providing critical insights including GPU performance stats for self-hosted LLMs to improve performance and reliability.This project proudly follows the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) of the OpenTelemetry community, consistently updating to align with the latest standards in observability.+## What is LIT?`LIT` stands for **Learning and Inference Tool**, which is a visual and interactive tool designed for understanding AI models and visualizing data.The term `LIT` was introduced by [Google](https://developers.google.com/machine-learning/glossary#learning-interpretability-tool-lit).diff --git a/docs/latest/quickstart.mdx b/docs/latest/quickstart.mdxindex e9dfb329..32e4c0ec 100644--- a/docs/latest/quickstart.mdx+++ b/docs/latest/quickstart.mdx@@ -39,12 +39,12 @@ flowchart TB;</Step></Steps></Step>- <Step title="Install the OpenLIT SDK">+ <Step title="Install OpenLIT SDK">```shellpip install openlit```</Step>- <Step title="Initialize the OpenLIT in Your Application">+ <Step title="Initialize OpenLIT in Your Application">Add the following two lines to your application code:<Tabs><Tab title="Setup using function arguments">@@ -115,7 +115,7 @@ flowchart TB;```</Tab></Tabs>- Refer to the OpenLIT [Python SDK repository](https://github.com/openlit/openlit/tree/main/sdk/python) for more advanced configurations and use cases.+ Refer to OpenLIT [Python SDK repository](https://github.com/openlit/openlit/tree/main/sdk/python) for more advanced configurations and use cases.</Step><Step title="Visualize and Analyze">With the LLM Observability data now being collected and sent to OpenLIT, the next step is to visualize and analyze this data to get insights into your LLM application's performance, behavior, and identify areas of improvement.diff --git a/docs/latest/sdk-configuration.mdx b/docs/latest/sdk-configuration.mdxindex 6032f413..21e60284 100644--- a/docs/latest/sdk-configuration.mdx+++ b/docs/latest/sdk-configuration.mdx@@ -16,6 +16,7 @@ Below is a detailed overview of the configuration options available, allowing yo| `otlp_headers` | Defines headers for the OTLP exporter, useful for backends requiring authentication. | `None` || `disable_batch` | A flag to disable batch span processing, favoring immediate dispatch. | `False` || `trace_content` | Enables tracing of content for deeper insights. | `True` |-| `disabled_instrumentors`| List of instrumentors to disable. | `None` |+| `disabled_instrumentors`| List of instrumentors to disable. | `None` || `disable_metrics` | If set, disables the collection of metrics. | `False` |-| `pricing_json` | URL or file path of the pricing JSON file. | `https://github.com/openlit/openlit/blob/main/assets/pricing.json` |\ No newline at end of file+| `pricing_json` | URL or file path of the pricing JSON file. | `https://github.com/openlit/openlit/blob/main/assets/pricing.json` |+| `collect_gpu_stats` | Flag to enable or disable GPU metrics collection. | `False` |\ No newline at end of filediff --git a/docs/mint.json b/docs/mint.jsonindex 502e70fa..644be248 100644--- a/docs/mint.json+++ b/docs/mint.json@@ -98,6 +98,13 @@"latest/integrations/haystack","latest/integrations/embedchain"]+ },+ {+ "group": "GPUs",+ "version": "latest",+ "pages": [+ "latest/integrations/nvidia-gpu"+ ]}]},diff --git a/sdk/python/README.md b/sdk/python/README.mdindex 809cb879..4d2d6c55 100644--- a/sdk/python/README.md+++ b/sdk/python/README.md@@ -160,6 +160,7 @@ Below is a detailed overview of the configuration options available, allowing yo| `disabled_instrumentors`| List of instrumentors to disable. | `None` | No || `disable_metrics` | If set, disables the collection of metrics. | `False` | No || `pricing_json` | URL or file path of the pricing JSON file. | `https://github.com/openlit/openlit/blob/main/assets/pricing.json` | No |+| `collect_gpu_stats` | Flag to enable or disable GPU metrics collection. | `False` | No |## 🌱 Contributingdiff --git a/sdk/python/pyproject.toml b/sdk/python/pyproject.tomlindex 39144ea8..e2917951 100644--- a/sdk/python/pyproject.toml+++ b/sdk/python/pyproject.toml@@ -11,7 +11,9 @@ keywords = ["OpenTelemetry", "otel", "otlp","llm", "tracing", "openai", "anthrop[tool.poetry.dependencies]python = "^3.7.1"requests = "^2.26.0"+schedule = "^1.2.2"tiktoken = "^0.6.0"+gpustat = "^1.1.1"boto3 = "^1.34.0"botocore = "^1.34.0"opentelemetry-api = "^1.24.0"diff --git a/sdk/python/src/openlit/__init__.py b/sdk/python/src/openlit/__init__.pyindex 4851a14e..0ae2ceb0 100644--- a/sdk/python/src/openlit/__init__.py+++ b/sdk/python/src/openlit/__init__.py@@ -40,6 +40,7 @@ from openlit.instrumentation.pinecone import PineconeInstrumentorfrom openlit.instrumentation.qdrant import QdrantInstrumentorfrom openlit.instrumentation.milvus import MilvusInstrumentorfrom openlit.instrumentation.transformers import TransformersInstrumentor+from openlit.instrumentation.gpu import NvidiaGPUInstrumentor# Set up logging for error and information messages.logger = logging.getLogger(__name__)@@ -155,20 +156,9 @@ def instrument_if_available(except Exception as e:logger.error("Failed to instrument %s: %s", instrumentor_name, e)--def init(- environment="default",- application_name="default",- tracer=None,- otlp_endpoint=None,- otlp_headers=None,- disable_batch=False,- trace_content=True,- disabled_instrumentors=None,- meter=None,- disable_metrics=False,- pricing_json=None,-):+def init(environment="default", application_name="default", tracer=None, otlp_endpoint=None,+ otlp_headers=None, disable_batch=False, trace_content=True, disabled_instrumentors=None,+ meter=None, disable_metrics=False, pricing_json=None, collect_gpu_stats=False):"""Initializes the openLIT configuration and setups tracing.@@ -185,8 +175,9 @@ def init(disable_batch (bool): Flag to disable batch span processing (Optional).trace_content (bool): Flag to trace content (Optional).disabled_instrumentors (List[str]): Optional. List of instrumentor names to disable.- disable_metrics (bool): Flag to disable metrics (Optional)- pricing_json(str): File path or url to the pricing json (Optional)+ disable_metrics (bool): Flag to disable metrics (Optional).+ pricing_json(str): File path or url to the pricing json (Optional).+ collect_gpu_stats (bool): Flag to enable or disable GPU metrics collection."""disabled_instrumentors = disabled_instrumentors if disabled_instrumentors else []# Check for invalid instrumentor names@@ -289,8 +280,13 @@ def init(# Initialize and instrument only the enabled instrumentorsfor name, instrumentor in instrumentor_instances.items():- instrument_if_available(- name, instrumentor, config, disabled_instrumentors, module_name_map+ instrument_if_available(name, instrumentor, config,+ disabled_instrumentors, module_name_map)++ if (disable_metrics is False) and (collect_gpu_stats is True):+ NvidiaGPUInstrumentor().instrument(+ environment=config.environment,+ application_name=config.application_name,)except Exception as e:diff --git a/sdk/python/src/openlit/instrumentation/gpu/__init__.py b/sdk/python/src/openlit/instrumentation/gpu/__init__.pynew file mode 100644index 00000000..fc7698ae--- /dev/null+++ b/sdk/python/src/openlit/instrumentation/gpu/__init__.py@@ -0,0 +1,132 @@+# pylint: disable=useless-return, bad-staticmethod-argument, duplicate-code, import-outside-toplevel, broad-exception-caught, unused-argument+"""Initializer of Auto Instrumentation of GPU Metrics"""++from typing import Collection, Iterable+import logging+from functools import partial++from opentelemetry.instrumentation.instrumentor import BaseInstrumentor+from opentelemetry.sdk.resources import TELEMETRY_SDK_NAME+from opentelemetry.metrics import get_meter, CallbackOptions, Observation++from openlit.semcov import SemanticConvetion++# Initialize logger for logging potential issues and operations+logger = logging.getLogger(__name__)++class NvidiaGPUInstrumentor(BaseInstrumentor):+ """+ An instrumentor for collecting NVIDIA GPU metrics.+ """++ def instrumentation_dependencies(self) -> Collection[str]:+ return []++ def _instrument(self, **kwargs):++ application_name = kwargs.get("application_name", "default")+ environment = kwargs.get("environment", "default")++ meter = get_meter(+ __name__,+ "0.1.0",+ schema_url="https://opentelemetry.io/schemas/1.11.0",+ )++ def check_and_record(value):+ return value if value is not None else 0++ meter.create_observable_gauge(+ name=SemanticConvetion.GPU_UTILIZATION,+ callbacks=[partial(self._collect_metric, environment,+ application_name, check_and_record, "utilization")],+ description="GPU Utilization",+ )+ meter.create_observable_gauge(+ name=SemanticConvetion.GPU_UTILIZATION_ENC,+ callbacks=[partial(self._collect_metric, environment,+ application_name, check_and_record, "utilization_enc")],+ description="GPU Encoder Utilization",+ )+ meter.create_observable_gauge(+ name=SemanticConvetion.GPU_UTILIZATION_DEC,+ callbacks=[partial(self._collect_metric, environment,+ application_name, check_and_record, "utilization_dec")],+ description="GPU Decoder Utilization",+ )+ meter.create_observable_gauge(+ name=SemanticConvetion.GPU_TEMPERATURE,+ callbacks=[partial(self._collect_metric, environment,+ application_name, check_and_record, "temperature")],+ description="GPU Temperature",+ )+ meter.create_observable_gauge(+ name=SemanticConvetion.GPU_FAN_SPEED,+ callbacks=[partial(self._collect_metric, environment,+ application_name, check_and_record, "fan_speed")],+ description="GPU Fan Speed",+ )+ meter.create_observable_gauge(+ name=SemanticConvetion.GPU_MEMORY_AVAILABLE,+ callbacks=[partial(self._collect_metric, environment,+ application_name, check_and_record, "memory_available")],+ description="GPU Memory Available",+ )+ meter.create_observable_gauge(+ name=SemanticConvetion.GPU_MEMORY_TOTAL,+ callbacks=[partial(self._collect_metric, environment,+ application_name, check_and_record, "memory_total")],+ description="GPU Memory Total",+ )+ meter.create_observable_gauge(+ name=SemanticConvetion.GPU_MEMORY_USED,+ callbacks=[partial(self._collect_metric, environment,+ application_name, check_and_record, "memory_used")],+ description="GPU Memory Used",+ )+ meter.create_observable_gauge(+ name=SemanticConvetion.GPU_MEMORY_FREE,+ callbacks=[partial(self._collect_metric, environment,+ application_name, check_and_record, "memory_free")],+ description="GPU Memory Free",+ )+ meter.create_observable_gauge(+ name=SemanticConvetion.GPU_POWER_DRAW,+ callbacks=[partial(self._collect_metric, environment,+ application_name, check_and_record, "power_draw")],+ description="GPU Power Draw",+ )+ meter.create_observable_gauge(+ name=SemanticConvetion.GPU_POWER_LIMIT,+ callbacks=[partial(self._collect_metric, environment,+ application_name, check_and_record, "power_limit")],+ description="GPU Power Limit",+ )++ def _uninstrument(self, **kwargs):+ # Proper uninstrumentation logic to revert patched methods+ pass++ def _collect_metric(self, environment, application_name,+ check_and_record, metric_name,+ options: CallbackOptions) -> Iterable[Observation]:++ import gpustat++ try:+ gpu_stats = gpustat.GPUStatCollection.new_query()++ for gpu in gpu_stats.gpus:+ attributes = {+ TELEMETRY_SDK_NAME: "openlit",+ SemanticConvetion.GEN_AI_APPLICATION_NAME: application_name,+ SemanticConvetion.GEN_AI_ENVIRONMENT: environment,+ SemanticConvetion.GPU_INDEX: gpu.index,+ SemanticConvetion.GPU_UUID: gpu.uuid,+ SemanticConvetion.GPU_NAME: gpu.name,+ }++ yield Observation(check_and_record(getattr(gpu, metric_name, 0)), attributes)++ except Exception as e:+ logger.error("Error in GPU metrics collection: %s", e)diff --git a/sdk/python/src/openlit/semcov/__init__.py b/sdk/python/src/openlit/semcov/__init__.pyindex ada6e25e..e9958802 100644--- a/sdk/python/src/openlit/semcov/__init__.py+++ b/sdk/python/src/openlit/semcov/__init__.py@@ -145,3 +145,20 @@ class SemanticConvetion:DB_SYSTEM_PINECONE = "pinecone"DB_SYSTEM_QDRANT = "qdrant"DB_SYSTEM_MILVUS = "milvus"++ # GPU+ GPU_INDEX = "gpu.index"+ GPU_UUID = "gpu.uuid"+ GPU_NAME = "gpu.name"++ GPU_UTILIZATION = "gpu.utilization"+ GPU_UTILIZATION_ENC = "gpu.enc.utilization"+ GPU_UTILIZATION_DEC = "gpu.dec.utilization"+ GPU_TEMPERATURE = "gpu.temperature"+ GPU_FAN_SPEED = "gpu.fan_speed"+ GPU_MEMORY_AVAILABLE = "gpu.memory.available"+ GPU_MEMORY_TOTAL = "gpu.memory.total"+ GPU_MEMORY_USED = "gpu.memory.used"+ GPU_MEMORY_FREE = "gpu.memory.free"+ GPU_POWER_DRAW = "gpu.power.draw"+ GPU_POWER_LIMIT = "gpu.power.limit"diff --git a/sdk/python/tests/requirements.txt b/sdk/python/tests/requirements.txtindex a3d6a345..de84b351 100644--- a/sdk/python/tests/requirements.txt+++ b/sdk/python/tests/requirements.txt@@ -2,6 +2,7 @@pytest>=7.4.0pytest-asyncio>=0.23.6requests>=2.31.0+schedule>=1.2.2pylint>=3.0.2# Gen AI@@ -13,7 +14,7 @@ transformers>=4.39.3boto3>=1.34.93botocore>=1.34.93google-cloud-aiplatform >= 1.38.1-groq >= 0.5.0+groq>= 0.5.0# Vector DBschromadb>=0.4.0@@ -26,6 +27,9 @@ langchain>=0.1.16langchainhub>=0.1.16embedchain>=0.1.104+# GPU+gpustat>=1.1.1+# Utilitiesbs4tiktoken>=0.1.1