Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
TypeScript agent-harnessagent-harness-engineeringai-engineeringai-observabilityamd-gpuclaude-codeclickhousecodexcoding-agentcoding-agentscursorgenaigpu-monitoringllm-observabilityllmopsnvidia-smiobservabilityopen-sourceopenaiopentelemetry
[Feat]: Update READMEs to account for new features and changes (#469)
Patcher04bfce2830be
diff --git a/README.md b/README.mdindex 24165c9c..490382c9 100644--- a/README.md+++ b/README.md@@ -1,18 +1,21 @@<div align="center"><img src="https://github.com/openlit/.github/blob/main/profile/assets/wide-logo-no-bg.png?raw=true" alt="OpenLIT Logo" width="30%">-<h3>OpenTelemetry-native</h3>-<h1>Observability & Evals for hosted and on-prem LLMs</h1>-**[Documentation](https://docs.openlit.io/) | [Quickstart](#-getting-started) | [Python SDK](https://github.com/openlit/openlit/tree/main/sdk/python)**+#### Observability, Exceptions, Prompts, Vault, Playground-[](https://github.com/openlit/openlit)+# Open Source Platform for AI Engineering++**[Documentation](https://docs.openlit.io/) | [Quickstart](-getting-started-with-llm-observability) | [Python SDK](https://github.com/openlit/openlit/tree/main/sdk/python) | [Typescript SDK](https://github.com/openlit/openlit/tree/main/sdk/typescript) |**++**[Roadmap](#️-roadmap) | [Feature Request](https://github.com/openlit/openlit/issues/new?assignees=&labels=%3Araised_hand%3A+Up+for+Grabs%2C+%3Arocket%3A+Feature&projects=&template=feature-request.md&title=%5BFeat%5D%3A) | [Report a Bug](https://github.com/openlit/openlit/issues/new?assignees=&labels=%3Abug%3A+Bug%2C+%3Araised_hand%3A+Up+for+Grabs&projects=&template=bug.md&title=%5BBug%5D%3A)**++[](https://openlit.io/)[](https://github.com/openlit/openlit/blob/main/LICENSE)[](https://pepy.tech/project/openlit)[](https://github.com/openlit/openlit/pulse)[](https://github.com/openlit/openlit/graphs/contributors)[](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ)-[](https://discord.gg/f6mwYAXv)[](https://twitter.com/openlit_io)</div>@@ -20,21 +23,27 @@-**OpenLIT** is an **OpenTelemetry-native** tool designed to help developers gain insights into the performance of their LLM applications in production. It automatically collects LLM input and output metadata, and monitors GPU performance for self-hosted LLMs.+**OpenLIT** allows you to simplify your AI development workflow, especially for Generative AI and LLMs. It streamlines essential tasks like experimenting with LLMs, organizing and versioning prompts, and securely handling API keys. With just one line of code, you can enable **OpenTelemetry-native** observability, offering full-stack monitoring that includes LLMs, vector databases, and GPUs. This enables developers to confidently build AI features and applications, transitioning smoothly from testing to production.-OpenLIT makes integrating observability into GenAI projects effortless with just **a single line of code**. Whether you're working with popular LLM providers such as OpenAI and HuggingFace, or leveraging vector databases like ChromaDB, OpenLIT ensures your applications are monitored seamlessly, providing critical insights including GPU performance stats for self-hosted LLMs to improve performance and reliability.+This project proudly follows and maintains the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) with the OpenTelemetry community, consistently updating to align with the latest standards in Observability.-This project proudly follows the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) of the OpenTelemetry community, consistently updating to align with the latest standards in observability.+## ⚡ Features-## What is LIT?-`LIT` stands for **Learning and Inference Tool**, which is a visual and interactive tool designed for understanding AI models and visualizing data. The term `LIT` was introduced by [Google](https://developers.google.com/machine-learning/glossary#learning-interpretability-tool-lit).+- 📈 **Analytics Dashboard**: Monitor your AI application's health and performance with detailed dashboards that track metrics, costs, and user interactions, providing a clear view of overall efficiency.-## ⚡ Features-- **Advanced Monitoring of LLM and VectorDB Performance**: OpenLIT offers automatic instrumentation that generates traces and metrics, providing insights into the performance and costs of your LLM and VectorDB usage. This helps you analyze how your applications perform in different environments, such as production, enabling you to optimize resource ussage and scale efficiently.-- **Cost Tracking for Custom and Fine-Tuned Models**: OpenLIT enables you to tailor cost tracking for specific models by using a custom JSON file. This feature allows for precise budgeting and ensures that cost estimations are perfectly aligned with your project needs.-- **OpenTelemetry-native & vendor-neutral SDKs**: OpenLIT is built with native support for OpenTelemetry, making it blend seamlessly with your projects. This vendor-neutral approach reduces barriers to integration, making OpenLIT an intuitive part of your software stack rather than an additional complexity.+- 🔌 **OpenTelemetry-native Observability SDKs**: Vendor-neutral SDKs to send traces and metrics to your existing observability tools.++- 💲 **Cost Tracking for Custom and Fine-Tuned Models**: Tailor cost estimations for specific models using custom pricing files for precise budgeting.-## 🚀 Getting Started+- 🔔 **Exceptions Monitoring Dashboard**: Quickly spot and resolve issues by tracking common exceptions and errors with a dedicated monitoring dashboard.++- 💭 **Prompt Management**: Manage and version prompts using Prompt Hub for consistent and easy access across applications.++- 🔑 **API Keys and Secrets Management**: Securely handle your API keys and secrets centrally, avoiding insecure practices.++- 🎮 **Experiemnt with different LLMs**: Use OpenGround to explore, test and compare various LLMs side by side.++## 🚀 Getting Started with LLM Observability```mermaidflowchart TB;@@ -55,15 +64,23 @@ flowchart TB;### Step 1: Deploy OpenLIT Stack1. Git Clone OpenLIT Repository++ Open your command line or terminal and run:+```shellgit clone [email protected]:openlit/openlit.git```-2. Start Docker Compose+2. Self-host using Docker++ Deploy and run OpenLIT with the following command:+```shelldocker compose up -d```+> For instructions on installing in Kubernetes using Helm, refer to the [Kubernetes Helm installation guide](https://docs.openlit.io/latest/installation#kubernetes).+### Step 2: Install OpenLIT SDKOpen your command line or terminal and run:@@ -72,8 +89,11 @@ Open your command line or terminal and run:pip install openlit```+> For instructions on using the TypeScript SDK, visit the [TypeScript SDK Installation guide](https://github.com/openlit/openlit/tree/main/sdk/typescript#-installation).+### Step 3: Initialize OpenLIT in your Application-Integrating OpenLIT into LLM applications is straightforward. Start monitoring for your LLM Application with just **two lines of code**:++Integrate OpenLIT into your AI applications by adding the following lines to your code.```pythonimport openlit@@ -81,11 +101,14 @@ import openlitopenlit.init()```-To forward telemetry data to an HTTP OTLP endpoint, such as the OpenTelemetry Collector, set the `otlp_endpoint` parameter with the desired endpoint. Alternatively, you can configure the endpoint by setting the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable as recommended in the OpenTelemetry documentation.+Configure the telemetry data destination as follows:-> 💡 Info: If you dont provide `otlp_endpoint` function argument or set the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable, The OpenLIT SDK directs the trace directly to your console, which can be useful during development.+| Purpose | Parameter/Environment Variable | For Sending to OpenLIT |+|-------------------------------------------|--------------------------------------------------|--------------------------------|+| Send data to an HTTP OTLP endpoint | `otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` | `"http://127.0.0.1:4318"` |+| Authenticate telemetry backends | `otlp_headers` or `OTEL_EXPORTER_OTLP_HEADERS` | Not required by default |-To send telemetry to OpenTelemetry backends requiring authentication, set the `otlp_headers` parameter with its desired value. Alternatively, you can configure the endpoint by setting the `OTEL_EXPORTER_OTLP_HEADERS` environment variable as recommended in the OpenTelemetry documentation.+> 💡 Info: If the `otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` is not provided, the OpenLIT SDK will output traces directly to your console, which is recommended during the development phase.#### Example@@ -132,7 +155,7 @@ To send telemetry to OpenTelemetry backends requiring authentication, set the `o---### Step 4: Visualize and Optimize!-With the LLM Observability data now being collected and sent to OpenLIT, the next step is to visualize and analyze this data to get insights into your LLM application's performance, behavior, and identify areas of improvement.+With the Observability data now being collected and sent to OpenLIT, the next step is to visualize and analyze this data to get insights into your AI application's performance, behavior, and identify areas of improvement.Just head over to OpenLIT at `127.0.0.1:3000` on your browser to start exploring. You can login using the default credentials- **Email**: `[email protected]`@@ -141,6 +164,24 @@ Just head over to OpenLIT at `127.0.0.1:3000` on your browser to start exploring+## 🛣️ Roadmap++We are dedicated to continuously improving OpenLIT. Here's a look at what's been accomplished and what's on the horizon:++| Feature | Status |+|----------------------------------------------------------------------------------------------|---------------|+| [OpenTelemetry-native Observability SDK for Tracing and Metrics](https://github.com/openlit/openlit/tree/text-upgrade/sdk/python) | ✅ Completed |+| [OpenTelemetry-native GPU Monitoring](https://docs.openlit.io/latest/features/gpu) | ✅ Completed |+| [Exceptions and Error Monitoring]() | ✅ Completed |+| [Prompt Hub for Managing and Versioning Prompts](https://docs.openlit.io/latest/features/prompt-hub) | ✅ Completed |+| [OpenGround for Testing and Comparing LLMs]() | ✅ Completed |+| [Vault for Central Management of LLM API Keys and Secrets](https://docs.openlit.io/latest/features/vault) | ✅ Completed |+| [Cost Tracking for Custom Models](https://docs.openlit.io/latest/features/pricing) | ✅ Completed |+| [Auto-Evaluation Metrics Based on Usage](https://github.com/openlit/openlit/issues/470) | 🔜 Coming Soon |+| [Human Feedback for LLM Events](https://github.com/openlit/openlit/issues/471) | 🔜 Coming Soon |+| [Dataset Generation Based on LLM Events](https://github.com/openlit/openlit/issues/472) | 🔜 Coming Soon |+| [Real-Time Guardrails Implementation]() | 📝 Planned |+## 🌱 ContributingWhether it's big or small, we love contributions 💚. Check out our [Contribution guide](./CONTRIBUTING.md) to get started@@ -166,7 +207,3 @@ Connect with OpenLIT community and maintainers for support, discussions, and upd## LicenseOpenLIT is available under the [Apache-2.0 license](LICENSE).--## Visualize! Analyze! Optimize!--Join us on this voyage to reshape the future of AI Observability. Share your thoughts, suggest features, and explore contributions. Engage with us on [GitHub](https://github.com/openlit/openlit) and be part of OpenLIT's community-led innovation.diff --git a/otel-gpu-collector/README.md b/otel-gpu-collector/README.mdindex ea987d0c..a02b55fb 100644--- a/otel-gpu-collector/README.md+++ b/otel-gpu-collector/README.md@@ -2,35 +2,36 @@<img src="https://github.com/openlit/.github/blob/main/profile/assets/wide-logo-no-bg.png?raw=true" alt="OpenLIT Logo" width="30%"><h1>OpenTelemetry GPU Collector</h1>-**[Documentation](https://docs.openlit.io/) | [Quickstart](#-getting-started) | [Python SDK](https://github.com/openlit/openlit/tree/main/sdk/python) | [Metrics](#metrics)**+**[Documentation](https://docs.openlit.io/latest/features/gpu) | [Quickstart](-getting-started-with-gpu-monitoring) | [Python SDK](https://github.com/openlit/openlit/tree/main/sdk/python) | [Metrics](#metrics)**-[](https://github.com/openlit/openlit)+**[Roadmap](#️-roadmap) | [Feature Request](https://github.com/openlit/openlit/issues/new?assignees=&labels=%3Araised_hand%3A+Up+for+Grabs%2C+%3Arocket%3A+Feature&projects=&template=feature-request.md&title=%5BFeat%5D%3A) | [Report a Bug](https://github.com/openlit/openlit/issues/new?assignees=&labels=%3Abug%3A+Bug%2C+%3Araised_hand%3A+Up+for+Grabs&projects=&template=bug.md&title=%5BBug%5D%3A)**++[](https://openlit.io/)[](https://github.com/openlit/openlit/blob/main/LICENSE)[](https://github.com/openlit/openlit/pulse)[](https://github.com/openlit/openlit/graphs/contributors)[](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ)-[](https://discord.gg/rjvTm6zd)[](https://twitter.com/openlit_io)</div>-OpenTelemetry GPU Collector is a lightweight, efficient tool designed to collect GPU performancemetrics and send them to an OpenTelemetry-compatible endpoint for monitoring and observability. This tool is particularly useful for monitoring GPUs in high-performance computing environments, AI/ML tasks, and LLMs.+OpenTelemetry GPU Collector is a lightweight, efficient COLLECTOR designed to collect GPU performance metrics and send them to an OpenTelemetry-compatible endpoint for monitoring and observability. This tool is particularly useful for monitoring GPUs in high-performance computing environments, AI/ML tasks, and LLMs.-## Features+## ⚡ Features- Collects detailed GPU performance metrics- OpenTelemetry-native- Lightweight and efficient-- Supports multiple GPU and system architectures+- Supports NVIDIA GPUs-## Quick Start+## 🚀 Getting Started with GPU Monitoring### Prerequisites-- Docker installed on your GPU system+- Docker installed on your system-### Installation+### Step 1: Pull the Docker ImageYou can quickly start using the OTel GPU Collector by pulling the Docker image:@@ -38,7 +39,7 @@ You can quickly start using the OTel GPU Collector by pulling the Docker image:docker pull ghcr.io/openlit/otel-gpu-collector:latest```-### Running the Container+### Step 2: Run the ContainerHere's a quick example showing how to run the container with the required environment variables:@@ -51,7 +52,7 @@ docker run --gpus all \ghcr.io/openlit/otel-gpu-collector:latest```-**Note:** If you've deployed **'OpenLIT'** using [Docker Compose](https://github.com/openlit/openlit/blob/main/docker-compose.yml), make sure to use the host's IP address or add OTel GPU Collector to the [Docker Compose](https://github.com/openlit/openlit/blob/main/docker-compose.yml):+**Note:** If you've deployed **OpenLIT** using [Docker Compose](https://github.com/openlit/openlit/blob/main/docker-compose.yml), make sure to use the host's IP address or add OTel GPU Collector to the [Docker Compose](https://github.com/openlit/openlit/blob/main/docker-compose.yml):<details><summary>Docker Compose: Add the following config under `services`</summary>@@ -126,11 +127,25 @@ For more details, check out the [OpenLIT documentation](https://docs.openlit.io/To build the Docker image yourself, you can clone the repository and execute the following commands:```sh+# Clone the OpenLIT repository and set directorygit clone https://github.com/openlit/openlit.gitcd otel-gpu-collector++# Build the Docker imagedocker build -t otel-gpu-collector .```+## 🛣️ Roadmap++We are dedicated to continuously improving OpenTelemetry GPU Collector. Here's a look at what's been accomplished and what's on the horizon:++| Feature | Status |+|----------------------------------------------------------------------------------------------|---------------|+| [OpenTelmetry metrics for NVIDIA GPUs using `gpustat`]() | ✅ Completed |+| [OpenTelmetry metrics for AMD GPUs]() | 🔜 Coming Soon |+| [OpenTelmetry metrics for NVIDIA GPUs using `nvidia-smi`]() | 🔜 Coming Soon |++## 🌱 Contributing@@ -153,8 +168,4 @@ Connect with OpenLIT community and maintainers for support, discussions, and upd## License-OpenLIT is available under the [Apache-2.0 license](LICENSE).--## Visualize! Analyze! Optimize!--Join us on this voyage to reshape the future of AI Observability. Share your thoughts, suggest features, and explore contributions. Engage with us on [GitHub](https://github.com/openlit/openlit) and be part of OpenLIT's community-led innovation.+OpenTelemetry GPU Collector is built and maintained by OpenLIT under the [Apache-2.0 license](LICENSE).diff --git a/sdk/python/README.md b/sdk/python/README.mdindex d4d51821..9e6d7b38 100644--- a/sdk/python/README.md+++ b/sdk/python/README.md@@ -1,8 +1,9 @@<div align="center">-<img src="https://github.com/openlit/.github/blob/main/profile/assets/wide-logo-no-bg.png?raw=true" alt="OpenLIT Logo" width="30%"><h1>-OpenTelemetry Auto-Instrumentation for GenAI & LLM Applications</h1>+<img src="https://github.com/openlit/.github/blob/main/profile/assets/wide-logo-no-bg.png?raw=true" alt="OpenLIT Logo" width="30%">+<h3>OpenTelemetry-native</h3>+<h1>AI Observability, Monitoring and Evaluation Framework</h1>-**[Documentation](https://docs.openlit.io/) | [Quickstart](#-getting-started) | [Python SDK](https://github.com/openlit/openlit/tree/main/sdk/python)**+**[Documentation](https://docs.openlit.io/) | [Quickstart](#-getting-started-with-llm-observability) | [Roadmap](#️-roadmap) | [Feature Request](https://github.com/openlit/openlit/issues/new?assignees=&labels=%3Araised_hand%3A+Up+for+Grabs%2C+%3Arocket%3A+Feature&projects=&template=feature-request.md&title=%5BFeat%5D%3A) | [Report a Bug](https://github.com/openlit/openlit/issues/new?assignees=&labels=%3Abug%3A+Bug%2C+%3Araised_hand%3A+Up+for+Grabs&projects=&template=bug.md&title=%5BBug%5D%3A)**[](https://github.com/openlit/openlit)[](https://github.com/openlit/openlit/blob/main/LICENSE)@@ -11,7 +12,6 @@ OpenTelemetry Auto-Instrumentation for GenAI & LLM Applications</h1>[](https://github.com/openlit/openlit/graphs/contributors)[](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ)-[](https://discord.gg/CQnXwNT3)[](https://twitter.com/openlit_io)@@ -19,11 +19,16 @@ OpenTelemetry Auto-Instrumentation for GenAI & LLM Applications</h1></div>-OpenLIT Python SDK is an **OpenTelemetry-native** Auto instrumentation library for monitoring LLM Applications, facilitating the integration of observability into your GenAI-driven projects. Designed with simplicity and efficiency, OpenLIT offers the ability to embed observability into your GenAI-driven projects effortlessly using just **a single line of code**.+OpenLIT SDK is a monitoring framework built on top of **OpenTelemetry** that gives your complete Observability for your AI stack, from LLMs to vector databases and GPUs, with just one line of code with tracing and metrics. It also allows you to send the generated traces and metrics to your existing monitoring tools like Grafana, New Relic, and more.-Whether you're directly using LLM Libraries like OpenAI, Anthropic or building complex RAG Agents using Langchain, OpenLIT seamlessly integrates observability into your applications, ensuring enhanced performance and reliability across diverse scenarios.+This project proudly follows and maintains the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) with the OpenTelemetry community, consistently updating to align with the latest standards in Observability.-This project adheres to the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) proposed by the OpenTelemetry community. You can check out the current definitions [here](src/openlit/semcov/__init__.py).+## ⚡ Features++- 🔎 **Auto Instrumentation**: Works with 30+ LLM providers, vector databases, and GPUs with just one line of code.+- 🔭 **OpenTelemetry-Native Observability SDKs**: Vendor-neutral SDKs that can send traces and metrics to your existing observability tool like Prometheus and Jaeger.+- 💲 **Cost Tracking for Custom and Fine-Tuned Models**: Pass custom pricing files for accurate budgeting of custom and fine-tuned models.+- 🚀 **Suppport for OpenLIT Features**: Includes suppprt for prompt management and secrets management features available in OpenLIT.## Auto Instrumentation Capabilities@@ -69,9 +74,9 @@ This project adheres to the [Semantic Conventions](https://github.com/open-telempip install openlit```-## 🚀 Getting Started+## 🚀 Getting Started with LLM Observability-### Step 1: Install OpenLIT+### Step 1: Install OpenLIT SDKOpen your command line or terminal and run:@@ -80,7 +85,7 @@ pip install openlit```### Step 2: Initialize OpenLIT in your Application-Integrating the OpenLIT into LLM applications is straightforward. Start monitoring for your LLM Application with just **two lines of code**:+Integrate OpenLIT into your AI applications by adding the following lines to your code.```pythonimport openlit@@ -88,11 +93,14 @@ import openlitopenlit.init()```-To forward telemetry data to an HTTP OTLP endpoint, such as the OpenTelemetry Collector, set the `otlp_endpoint` parameter with the desired endpoint. Alternatively, you can configure the endpoint by setting the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable as recommended in the OpenTelemetry documentation.+Configure the telemetry data destination as follows:-> 💡 Info: If you dont provide `otlp_endpoint` function argument or set the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable, OpenLIT directs the trace directly to your console, which can be useful during development.+| Purpose | Parameter/Environment Variable | For Sending to OpenLIT |+|-------------------------------------------|--------------------------------------------------|--------------------------------|+| Send data to an HTTP OTLP endpoint | `otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` | `"http://127.0.0.1:4318"` |+| Authenticate telemetry backends | `otlp_headers` or `OTEL_EXPORTER_OTLP_HEADERS` | Not required by default |-To send telemetry to OpenTelemetry backends requiring authentication, set the `otlp_headers` parameter with its desired value. Alternatively, you can configure the endpoint by setting the `OTEL_EXPORTER_OTLP_HEADERS` environment variable as recommended in the OpenTelemetry documentation.+> 💡 Info: If the `otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` is not provided, the OpenLIT SDK will output traces directly to your console, which is recommended during the development phase.#### Example@@ -143,19 +151,22 @@ To send telemetry to OpenTelemetry backends requiring authentication, set the `o---### Step 3: Visualize and Optimize!-With the LLM Observability data now being collected and sent to OpenLIT, the next step is to visualize and analyze this data to get insights into your LLM application’s performance, behavior, and identify areas of improvement.+Now that your LLM observability data is being collected and sent to configured OpenTelemetry destination, the next step is to visualize and analyze this data. This will help you understand your LLM application's performance and behavior and identify where it can be improved.-To begin exploring your LLM Application's performance data within the OpenLIT, please see the [Quickstart Guide](https://docs.openlit.io/latest/quickstart).+If you want to use OpenLIT's Observability Dashboard to monitor LLM usage—like cost, tokens, and user interactions—please check out our [Quickstart Guide](https://docs.openlit.io/latest/quickstart).-If you want to integrate and send metrics and traces to your existing observability tools, refer to our [Connections Guide](https://docs.openlit.io/latest/connections/intro) for detailed instructions.+If you're sending metrics and traces to other observability tools, take a look at our [Connections Guide](https://docs.openlit.io/latest/connections/intro) to start using a pre-built dashboard we have created for these tools.-### Configuration+## Configuration++### Observability - `openlit.init()`Below is a detailed overview of the configuration options available, allowing you to adjust OpenLIT's behavior and functionality to align with your specific observability needs:+| Argument | Description | Default Value | Required ||-------------------------|-----------------------------------------------------------------------------------------------|----------------|----------|| `environment` | The deployment environment of the application. | `"default"` | Yes |@@ -171,6 +182,48 @@ Below is a detailed overview of the configuration options available, allowing yo| `pricing_json` | URL or file path of the pricing JSON file. | `https://github.com/openlit/openlit/blob/main/assets/pricing.json` | No || `collect_gpu_stats` | Flag to enable or disable GPU metrics collection. | `False` | No |+### OpenLIT Prompt Hub - `openlit.get_prompt()`++Below are the parameters for use with the SDK for OpenLIT Prompt Hub for prompt management:++| Parameter | Description |+|-------------------|------------------------------------------------------------------------------------------------------------------------------------|+| `url` | Sets the OpenLIT URL. Defaults to the `OPENLIT_URL` environment variable. |+| `api_key` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |+| `name` | Sets the name to fetch a unique prompt. Use this or `prompt_id`. |+| `prompt_id` | Sets the ID to fetch a unique prompt. Use this or `name`. Optional |+| `version` | Set to `True` to get the prompt with variable substitution.. Optional |+| `shouldCompile` | Boolean value that compiles the prompt using the provided variables. Optional |+| `variables` | Sets the variables for prompt compilation. Optional |+| `meta_properties` | Sets the meta-properties for storing in the prompt's access history metadata. Optional |++### OpenLIT Vault - `openlit.get_secrets()`++Below are the parameters for use with the SDK for OpenLIT Vault for secret management:++| Parameter | Description |+|-------------------|------------------------------------------------------------------------------------------------------------------------------------|+| `url` | Sets the Openlit URL. Defaults to the `OPENLIT_URL` environment variable. |+| `api_key` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |+| `key` | Sets the key to fetch a specific secret. Optional |+| `should_set_env` | Boolean value that sets all the secrets as environment variables for the application. Optional |+| `tags` | Sets the tags for fetching only the secrets that have the mentioned tags assigned. Optional |+++## 🛣️ Roadmap++We are dedicated to continuously improving OpenLIT SDKs. Here's a look at what's been accomplished and what's on the horizon:++| Feature | Status |+|----------------------------------------------------------------------------------------------|---------------|+| [OpenTelmetry auto-instrumentation for LLM Providers like OpenAI, Anthropic]() | ✅ Completed |+| [OpenTelmetry auto-instrumentation for Vector databases like Pinecone, Chroma]() | ✅ Completed |+| [OpenTelmetry auto-instrumentation for LLM Frameworks like LangChain, LlamaIndex]() | ✅ Completed |+| [OpenTelemetry-native auto-instrumentation for NVIDIA GPU Monitoring](https://docs.openlit.io/latest/features/gpu) | ✅ Completed |+| [OpenTelmetry auto-instrumentation for Agent Frameworks like CrewAI, DsPy]() | 🔜 Coming Soon |+| [Evaluation Metrics]() | 🔜 Coming Soon |++## 🌱 ContributingWhether it's big or small, we love contributions 💚. Check out our [Contribution guide](../../CONTRIBUTING.md) to get starteddiff --git a/sdk/python/pyproject.toml b/sdk/python/pyproject.tomlindex 8a804429..b554340b 100644--- a/sdk/python/pyproject.toml+++ b/sdk/python/pyproject.toml@@ -1,6 +1,6 @@[tool.poetry]name = "openlit"-version = "1.26.0"+version = "1.26.1"description = "OpenTelemetry-native Auto instrumentation library for monitoring LLM Applications and GPUs, facilitating the integration of observability into your GenAI-driven projects"authors = ["OpenLIT"]repository = "https://github.com/openlit/openlit/tree/main/openlit/python"diff --git a/sdk/typescript/README.md b/sdk/typescript/README.mdindex f6537eac..e16f5f85 100644--- a/sdk/typescript/README.md+++ b/sdk/typescript/README.md@@ -1,8 +1,9 @@<div align="center">-<img src="https://github.com/openlit/.github/blob/main/profile/assets/wide-logo-no-bg.png?raw=true" alt="OpenLIT Logo" width="30%"><h1>-OpenTelemetry Auto-Instrumentation for GenAI & LLM Applications</h1>+<img src="https://github.com/openlit/.github/blob/main/profile/assets/wide-logo-no-bg.png?raw=true" alt="OpenLIT Logo" width="30%">+<h3>OpenTelemetry-native</h3>+<h1>AI Observability, Monitoring and Evaluation Framework</h1>-**[Documentation](https://docs.openlit.io/) | [Quickstart](#-getting-started) | [Typescript SDK](https://github.com/openlit/openlit/tree/main/sdk/typescript)**+**[Documentation](https://docs.openlit.io/) | [Quickstart](#-getting-started-with-llm-observability) | [Roadmap](#️-roadmap) | [Feature Request](https://github.com/openlit/openlit/issues/new?assignees=&labels=%3Araised_hand%3A+Up+for+Grabs%2C+%3Arocket%3A+Feature&projects=&template=feature-request.md&title=%5BFeat%5D%3A) | [Report a Bug](https://github.com/openlit/openlit/issues/new?assignees=&labels=%3Abug%3A+Bug%2C+%3Araised_hand%3A+Up+for+Grabs&projects=&template=bug.md&title=%5BBug%5D%3A)**[](https://github.com/openlit/openlit)[](https://github.com/openlit/openlit/blob/main/LICENSE)@@ -11,18 +12,23 @@ OpenTelemetry Auto-Instrumentation for GenAI & LLM Applications</h1>[](https://github.com/openlit/openlit/graphs/contributors)[](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ)-[](https://discord.gg/CQnXwNT3)[](https://twitter.com/openlit_io)</div>-OpenLIT Typescript SDK is an **OpenTelemetry-native** Auto instrumentation library for monitoring LLM Applications, facilitating the integration of observability into your GenAI-driven projects. Designed with simplicity and efficiency, OpenLIT offers the ability to embed observability into your GenAI-driven projects effortlessly using just **a single line of code**.+OpenLIT SDK is a monitoring framework built on top of **OpenTelemetry** that gives your complete Observability for your AI stack, from LLMs to vector databases, with just one line of code with tracing and metrics. It also allows you to send the generated traces and metrics to your existing monitoring tools like Grafana, New Relic, and more.-Whether you're directly using LLM Libraries like OpenAI or Anthropic, OpenLIT seamlessly integrates observability into your applications, ensuring enhanced performance and reliability across diverse scenarios.+This project proudly follows and maintains the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) with the OpenTelemetry community, consistently updating to align with the latest standards in Observability.++## ⚡ Features++- 🔎 **Auto Instrumentation**: Works with 30+ LLM providers and vector databases with just one line of code.+- 🔭 **OpenTelemetry-Native Observability SDKs**: Vendor-neutral SDKs that can send traces and metrics to your existing observability tool like Prometheus and Jaeger.+- 💲 **Cost Tracking for Custom and Fine-Tuned Models**: Pass custom pricing files for accurate budgeting of custom and fine-tuned models.+- 🚀 **Suppport for OpenLIT Features**: Includes suppprt for prompt management and secrets management features available in OpenLIT.-This project adheres to the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) proposed by the OpenTelemetry community. You can check out the current definitions [here](src/semantic-convention.ts).## Auto Instrumentation Capabilities@@ -30,6 +36,7 @@ This project adheres to the [Semantic Conventions](https://github.com/open-telem| --------------------------------------------------------------------- || [✅ OpenAI](https://docs.openlit.io/latest/integrations/openai) | [✅ ChromaDB](https://docs.openlit.io/latest/integrations/chromadb) | [✅ LiteLLM](https://docs.openlit.io/latest/integrations/litellm) | || [✅ Anthropic](https://docs.openlit.io/latest/integrations/anthropic) |+| [✅ Cohere](https://docs.openlit.io/latest/integrations/cohere) |## Supported Destinations@@ -65,7 +72,7 @@ npm install openlit### Step 2: Initialize OpenLIT in your Application-Integrating the OpenLIT into LLM applications is straightforward. Start monitoring for your LLM Application with just **two lines of code**:+Integrate OpenLIT into your AI applications by adding the following lines to your code.```typescriptimport Openlit from 'openlit';@@ -73,11 +80,14 @@ import Openlit from 'openlit';Openlit.init();```-To forward telemetry data to an HTTP OTLP endpoint, such as the OpenTelemetry Collector, set the `otlpEndpoint` parameter with the desired endpoint. Alternatively, you can configure the endpoint by setting the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable as recommended in the OpenTelemetry documentation.+Configure the telemetry data destination as follows:-> 💡 Info: If you dont provide `otlpEndpoint` function argument or set the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable, OpenLIT directs the trace directly to your console, which can be useful during development.+| Purpose | Parameter/Environment Variable | For Sending to OpenLIT |+|-------------------------------------------|--------------------------------------------------|--------------------------------|+| Send data to an HTTP OTLP endpoint | `otlpEndpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` | `"http://127.0.0.1:4318"` |+| Authenticate telemetry backends | `otlpHeaders` or `OTEL_EXPORTER_OTLP_HEADERS` | Not required by default |-To send telemetry to OpenTelemetry backends requiring authentication, set the `otlpHeaders` parameter with its desired value. Alternatively, you can configure the endpoint by setting the `OTEL_EXPORTER_OTLP_HEADERS` environment variable as recommended in the OpenTelemetry documentation.+> 💡 Info: If the `otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` is not provided, the OpenLIT SDK will output traces directly to your console, which is recommended during the development phase.#### Example@@ -129,16 +139,17 @@ export OTEL_EXPORTER_OTLP_HEADERS = "YOUR_OTEL_ENDPOINT_AUTH"---### Step 3: Visualize and Optimize!+Now that your LLM observability data is being collected and sent to configured OpenTelemetry destination, the next step is to visualize and analyze this data. This will help you understand your LLM application's performance and behavior and identify where it can be improved.-With the LLM Observability data now being collected and sent to OpenLIT, the next step is to visualize and analyze this data to get insights into your LLM application’s performance, behavior, and identify areas of improvement.+If you want to use OpenLIT's Observability Dashboard to monitor LLM usage—like cost, tokens, and user interactions—please check out our [Quickstart Guide](https://docs.openlit.io/latest/quickstart).-To begin exploring your LLM Application's performance data within the OpenLIT, please see the [Quickstart Guide](https://docs.openlit.io/latest/quickstart).--If you want to integrate and send metrics and traces to your existing observability tools, refer to our [Connections Guide](https://docs.openlit.io/latest/connections/intro) for detailed instructions.+If you're sending metrics and traces to other observability tools, take a look at our [Connections Guide](https://docs.openlit.io/latest/connections/intro) to start using a pre-built dashboard we have created for these tools.-### Configuration+## Configuration++### Observability - `Openlit.init()`Below is a detailed overview of the configuration options available, allowing you to adjust OpenLIT's behavior and functionality to align with your specific observability needs:@@ -155,6 +166,36 @@ Below is a detailed overview of the configuration options available, allowing yo| `instrumentations` | Object of instrumentation modules for manual patching | `undefined` | No || `pricing_json` | URL or file path of the pricing JSON file. | `https://github.com/openlit/openlit/blob/main/assets/pricing.json` | No |+### OpenLIT Prompt Hub - `openlit.get_prompt()`++Below are the parameters for use with the SDK for OpenLIT Prompt Hub for prompt management:++| Parameter | Description |+|------------------|------------------------------------------------------------------------------------------------------------------------------------|+| `url` | Sets the OpenLIT URL. Defaults to the `OPENLIT_URL` environment variable or `http://127.0.0.1:3000` if not set. |+| `apiKey` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |+| `name` | Sets the name to fetch a unique prompt. Use this or `promptId`. |+| `promptId` | Sets the ID to fetch a unique prompt. Use this or `name`. Optional |+| `version` | Sets the version to retrieve a specific prompt. Optional |+| `shouldCompile` | Boolean value that compiles the prompt using the provided variables. Optional |+| `variables` | Sets the variables for prompt compilation. Optional |+| `metaProperties` | Sets the meta-properties for storing in the prompt's access history metadata. |+++### OpenLIT Vault - `openlit.get_secrets()`++Below are the parameters for use with the SDK for OpenLIT Vault for secret management:++| Parameter | Description |+|------------------|------------------------------------------------------------------------------------------------------------------------------------|+| `url` | Sets the Openlit URL. Defaults to the `OPENLIT_URL` environment variable or `http://127.0.0.1:3000` if not set. |+| `apiKey` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |+| `key` | Sets the key to fetch a specific secret. Optional |+| `tags` | Sets the tags for fetching only the secrets that have the mentioned tags assigned. Optional |+| `shouldSetEnv` | Boolean value that sets all the secrets as environment variables for the application. Optional |+++## 🌱 ContributingWhether it's big or small, we love contributions 💚. Check out our [Contribution guide](../../CONTRIBUTING.md) to get starteddiff --git a/src/client/README.md b/src/client/README.mdindex 42d70fd4..c38cccd6 100644--- a/src/client/README.md+++ b/src/client/README.md@@ -1,28 +1,51 @@-# OpenLIT Client+<div align="center">+<img src="https://github.com/openlit/.github/blob/main/profile/assets/wide-logo-no-bg.png?raw=true" alt="OpenLIT Logo" width="30%">-[](https://github.com/openlit/openlit)-[](https://github.com/openlit/openlit/blob/main/LICENSE)-[](https://github.com/openlit/openlit/tags)+#### Observability, Exceptions, Prompts, Vault, Playground------+# Open Source Platform for AI Engineering-OpenLIT Client serves as the frontend UI for displaying visualizations and observability data for Large Language Models (LLM), drawing data from ClickHouse. The platform leverages the power of modern web technologies like Next.js and TailwindCSS to offer an intuitive and responsive user experience.+**[Documentation](https://docs.openlit.io/) | [Quickstart](-getting-started-with-llm-observability) | [Python SDK](https://github.com/openlit/openlit/tree/main/sdk/python) | [Typescript SDK](https://github.com/openlit/openlit/tree/main/sdk/typescript) |**-## Features+**[Roadmap](#️-roadmap) | [Feature Request](https://github.com/openlit/openlit/issues/new?assignees=&labels=%3Araised_hand%3A+Up+for+Grabs%2C+%3Arocket%3A+Feature&projects=&template=feature-request.md&title=%5BFeat%5D%3A) | [Report a Bug](https://github.com/openlit/openlit/issues/new?assignees=&labels=%3Abug%3A+Bug%2C+%3Araised_hand%3A+Up+for+Grabs&projects=&template=bug.md&title=%5BBug%5D%3A)**-- **Comprehensive Analytics Dashboard**: Easily monitor Large Language Model performance, user activity, and critical metrics related to costs and tokens in real-time.-- **User Management System**: Offers built-in user management to ensure secure access control and smooth authentication processes.-- **Versatile Data Source Connectivity**: Seamlessly adjust and integrate your ClickHouse database details directly within the UI. This flexibility enables quick switches between databases to keep your OpenLIT Client running efficiently.+[](https://openlit.io/)+[](https://github.com/openlit/openlit/blob/main/LICENSE)+[](https://pepy.tech/project/openlit)+[](https://github.com/openlit/openlit/pulse)+[](https://github.com/openlit/openlit/graphs/contributors)-## 🚀 Quick Start+[](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ)+[](https://twitter.com/openlit_io)+</div>++---++++**OpenLIT** allows you to simplify your AI development workflow, especially for Generative AI and LLMs. It streamlines essential tasks like experimenting with LLMs, organizing and versioning prompts, and securely handling API keys. With just one line of code, you can enable **OpenTelemetry-native** observability, offering full-stack monitoring that includes LLMs, vector databases, and GPUs. This enables developers to confidently build AI features and applications, transitioning smoothly from testing to production.++This project proudly follows and maintains the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) with the OpenTelemetry community, consistently updating to align with the latest standards in Observability.++## ⚡ Features++- 📈 **Analytics Dashboard**: Monitor your AI application's health and performance with detailed dashboards that track metrics, costs, and user interactions, providing a clear view of overall efficiency.++- 🔌 **OpenTelemetry-native Observability SDKs**: Vendor-neutral SDKs to send traces and metrics to your existing observability tools.++- 💲 **Cost Tracking for Custom and Fine-Tuned Models**: Tailor cost estimations for specific models using custom pricing files for precise budgeting.++- 🔔 **Exceptions Monitoring Dashboard**: Quickly spot and resolve issues by tracking common exceptions and errors with a dedicated monitoring dashboard.++- 💭 **Prompt Management**: Manage and version prompts using Prompt Hub for consistent and easy access across applications.-Follow the steps below to get OpenLIT Client running in your environment. Both Docker and manual installation options are provided.+- 🔑 **API Keys and Secrets Management**: Securely handle your API keys and secrets centrally, avoiding insecure practices.+- 🎮 **Experiemnt with different LLMs**: Use OpenGround to explore, test and compare various LLMs side by side.++## 🚀 Quick Start++Follow the steps below to get OpenLIT running in your environment. Both Docker and manual installation options are provided.### Docker@@ -115,18 +138,46 @@ To configure OpenLIT Client, you can pass the following environment values, eachFor more detailed information on configuration options and additional settings, please visit the OpenLIT documentation page: [OpenLIT Configuration Details](https://docs.openlit.io/latest/configuration).-## Security+## 🛣️ Roadmap-We take security seriously. OpenLIT Client incorporates best practices for authentication, authorization, and secure communication to ensure data privacy and protection.+We are dedicated to continuously improving OpenLIT. Here's a look at what's been accomplished and what's on the horizon:-## Contributing+| Feature | Status |+|----------------------------------------------------------------------------------------------|---------------|+| [OpenTelemetry-native Observability SDK for Tracing and Metrics](https://github.com/openlit/openlit/tree/text-upgrade/sdk/python) | ✅ Completed |+| [OpenTelemetry-native GPU Monitoring](https://docs.openlit.io/latest/features/gpu) | ✅ Completed |+| [Exceptions and Error Monitoring]() | ✅ Completed |+| [Prompt Hub for Managing and Versioning Prompts](https://docs.openlit.io/latest/features/prompt-hub) | ✅ Completed |+| [OpenGround for Testing and Comparing LLMs]() | ✅ Completed |+| [Vault for Central Management of LLM API Keys and Secrets](https://docs.openlit.io/latest/features/vault) | ✅ Completed |+| [Cost Tracking for Custom Models](https://docs.openlit.io/latest/features/pricing) | ✅ Completed |+| [Auto-Evaluation Metrics Based on Usage](https://github.com/openlit/openlit/issues/470) | 🔜 Coming Soon |+| [Human Feedback for LLM Events](https://github.com/openlit/openlit/issues/471) | 🔜 Coming Soon |+| [Dataset Generation Based on LLM Events](https://github.com/openlit/openlit/issues/472) | 🔜 Coming Soon |+| [Real-Time Guardrails Implementation]() | 📝 Planned |-Contributions to OpenLIT Client are greatly appreciated. Whether you have suggestions for bug fixes, improvements, or new features, please see [CONTRIBUTING](../../CONTRIBUTING.md) for more details on submitting pull requests or opening issues.+## 🌱 Contributing-## License+Whether it's big or small, we love contributions 💚. Check out our [Contribution guide](./CONTRIBUTING.md) to get started++Unsure where to start? Here are a few ways to get involved:++- Join our [Slack](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ) or [Discord](https://discord.gg/rjvTm6zd) community to discuss ideas, share feedback, and connect with both our team and the wider OpenLIT community.-OpenLIT Client is available under the [Apache-2.0 license](../../LICENSE).+Your input helps us grow and improve, and we're here to support you every step of the way.-## Support+[](https://www.producthunt.com/posts/openlit?embed=true&utm_source=badge-featured&utm_medium=badge&utm_souce=badge-openlit)+<a href="https://fazier.com/launches/openlit-2" target="_blank" rel="noopener noreferrer"><img src="https://fazier.com/api/v1/public/badges/embed_image.svg?launch_id=779&badge_type=daily" width="270" alt="Example Image" class="d-inline-block mt-3 p-3 rounded img-fluid" /></a>++## 💚 Community & Support++Connect with OpenLIT community and maintainers for support, discussions, and updates:++- 🌟 If you like it, Leave a star on our [GitHub](https://github.com/openlit/openlit/)+- 🌍 Join our [Slack](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ) or [Discord](https://discord.gg/CQnXwNT3) community for live interactions and questions.+- 🐞 Report bugs on our [GitHub Issues](https://github.com/openlit/openlit/issues) to help us improve OpenLIT.+- 𝕏 Follow us on [X](https://twitter.com/openlit_io) for the latest updates and news.++## License-For support, issues, or feature requests, please submit an issue through the [GitHub issues](https://github.com/openlit/openlit/issues) page for this repository.+OpenLIT is available under the [Apache-2.0 license](LICENSE).diff --git a/src/client/package-lock.json b/src/client/package-lock.jsonindex 66a69b4f..958c975d 100644--- a/src/client/package-lock.json+++ b/src/client/package-lock.json@@ -1,6 +1,6 @@{"name": "openlit",- "version": "1.11.0",+ "version": "1.11.1","lockfileVersion": 3,"requires": true,"packages": {diff --git a/src/client/package.json b/src/client/package.jsonindex f7d152f3..0ace6132 100644--- a/src/client/package.json+++ b/src/client/package.json@@ -1,6 +1,6 @@{"name": "openlit",- "version": "1.11.0",+ "version": "1.11.1","private": true,"scripts": {"dev": "next dev",