Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
TypeScript agent-harnessagent-harness-engineeringai-engineeringai-observabilityamd-gpuclaude-codeclickhousecodexcoding-agentcoding-agentscursorgenaigpu-monitoringllm-observabilityllmopsnvidia-smiobservabilityopen-sourceopenaiopentelemetry
[Feat]: Update Docs to represent all new features (#477)
Patcher3db31af043fb
diff --git a/README.md b/README.mdindex 636f79b0..1fca70bd 100644--- a/README.md+++ b/README.md@@ -35,7 +35,7 @@ This project proudly follows and maintains the [Semantic Conventions](https://gi- ๐ฒ **Cost Tracking for Custom and Fine-Tuned Models**: Tailor cost estimations for specific models using custom pricing files for precise budgeting.-- ๐ **Exceptions Monitoring Dashboard**: Quickly spot and resolve issues by tracking common exceptions and errors with a dedicated monitoring dashboard.+- ๐ **Exceptions Monitoring Dashboard**: Quickly spot and resolve issues by tracking common exceptions and errors with a dedicated monitoring dashboard.- ๐ญ **Prompt Management**: Manage and version prompts using Prompt Hub for consistent and easy access across applications.@@ -105,8 +105,8 @@ Configure the telemetry data destination as follows:| Purpose | Parameter/Environment Variable | For Sending to OpenLIT ||-------------------------------------------|--------------------------------------------------|--------------------------------|-| Send data to an HTTP OTLP endpoint | `otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` | `"http://127.0.0.1:4318"` |-| Authenticate telemetry backends | `otlp_headers` or `OTEL_EXPORTER_OTLP_HEADERS` | Not required by default |+| Send data to an HTTP OTLP endpoint | `otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` | `"http://127.0.0.1:4318"` |+| Authenticate telemetry backends | `otlp_headers` or `OTEL_EXPORTER_OTLP_HEADERS` | Not required by default |> ๐ก Info: If the `otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT` is not provided, the OpenLIT SDK will output traces directly to your console, which is recommended during the development phase.@@ -172,9 +172,9 @@ We are dedicated to continuously improving OpenLIT. Here's a look at what's been|----------------------------------------------------------------------------------------------|---------------|| [OpenTelemetry-native Observability SDK for Tracing and Metrics](https://github.com/openlit/openlit/tree/text-upgrade/sdk/python) | โ Completed || [OpenTelemetry-native GPU Monitoring](https://docs.openlit.io/latest/features/gpu) | โ Completed |-| [Exceptions and Error Monitoring]() | โ Completed |+| [Exceptions and Error Monitoring](https://docs.openlit.io/latest/features/exceptions) | โ Completed || [Prompt Hub for Managing and Versioning Prompts](https://docs.openlit.io/latest/features/prompt-hub) | โ Completed |-| [OpenGround for Testing and Comparing LLMs]() | โ Completed |+| [OpenGround for Testing and Comparing LLMs](https://docs.openlit.io/latest/features/openground) | โ Completed || [Vault for Central Management of LLM API Keys and Secrets](https://docs.openlit.io/latest/features/vault) | โ Completed || [Cost Tracking for Custom Models](https://docs.openlit.io/latest/features/pricing) | โ Completed || [Auto-Evaluation Metrics Based on Usage](https://github.com/openlit/openlit/issues/470) | ๐ Coming Soon |diff --git a/docs/.DS_Store b/docs/.DS_Storeindex a18d2840..c31a2e81 100644Binary files a/docs/.DS_Store and b/docs/.DS_Store differdiff --git a/docs/images/docs-openground-1.png b/docs/images/docs-openground-1.pngnew file mode 100644index 00000000..8312aa3eBinary files /dev/null and b/docs/images/docs-openground-1.png differdiff --git a/docs/images/docs-openground-2.png b/docs/images/docs-openground-2.pngnew file mode 100644index 00000000..9e15c70eBinary files /dev/null and b/docs/images/docs-openground-2.png differdiff --git a/docs/images/docs-openground-3.png b/docs/images/docs-openground-3.pngnew file mode 100644index 00000000..afb7ee9dBinary files /dev/null and b/docs/images/docs-openground-3.png differdiff --git a/docs/images/docs-openground-4.png b/docs/images/docs-openground-4.pngnew file mode 100644index 00000000..ecb633b9Binary files /dev/null and b/docs/images/docs-openground-4.png differdiff --git a/docs/images/docs-openground-5.png b/docs/images/docs-openground-5.pngnew file mode 100644index 00000000..5971e242Binary files /dev/null and b/docs/images/docs-openground-5.png differdiff --git a/docs/latest/features/exceptions.mdx b/docs/latest/features/exceptions.mdxnew file mode 100644index 00000000..e4aa8cce--- /dev/null+++ b/docs/latest/features/exceptions.mdx@@ -0,0 +1,29 @@+---+title: 'Exceptions Monitoring'+description: "Record Exceptions Automatically and See detailed Stack Traces"+---++OpenLIT effortlessly integrates exception monitoring into your AI engineering workflow, empowering you to automatically track exceptions in Python and TypeScript applications. This integration provides vital insights into application errors without requiring extensive code changes.++## Features++- **Automatic Exception Monitoring**: With our SDKs for Python and TypeScript, monitor exceptions seamlessly without altering your application code base significantly. This feature simplifies error tracking and debugging.++- **Detailed Stacktraces**: Access comprehensive stacktrace information for all caught exceptions, providing insight into where things went wrong.++- **Integration with Traces**: Leverage OpenTelemetry-powered trace data to capture exceptions within request flows. By visualizing exceptions on trace graphs, you'll gain richer context for debugging, understanding precisely where exceptions occur in your execution path.++## Get Started++1. **Log in** to the OpenLIT.+2. Go to the **Exceptions** page.+3. View all the exceptions and error traces and debug.++<CardGroup cols={2}>+<Card title="Integrations" href="/latest/integrations/introduction" icon='circle-nodes'>+Integrate your AI Stack with OpenLIT+</Card>+<Card title="Connections" href="/latest/connections/intro" icon='link'>+Connect to your existing Observablity Stack+</Card>+</CardGroup>\ No newline at end of filediff --git a/docs/latest/features/gpu.mdx b/docs/latest/features/gpu.mdxindex 9f55772d..f836d935 100644--- a/docs/latest/features/gpu.mdx+++ b/docs/latest/features/gpu.mdx@@ -1,6 +1,7 @@---title: 'GPU Performance Monitoring'sidebarTitle: 'GPU Monitoring'+description: "Monitor NVIDIA GPUs with key metrics like usage, temperature, and power using OpenTelemetry"---OpenLIT uses OpenTelemetry to help you monitor NVIDIA GPUs. This includes tracking GPU metrics like utilization, temperature, memory usage and power consumption.@@ -97,6 +98,46 @@ OpenLIT uses OpenTelemetry to help you monitor NVIDIA GPUs. This includes trackiFor more advanced configurations of the collector, visit the [OTel GPU Collector repository](https://github.com/openlit/openlit/tree/main/otel-gpu-collector/).+ **Note:** If you've deployed **OpenLIT** using [Docker Compose](https://github.com/openlit/openlit/blob/main/docker-compose.yml), make sure to use the host's IP address or add OTel GPU Collector to the [Docker Compose](https://github.com/openlit/openlit/blob/main/docker-compose.yml):++ <AccordionGroup>+ <Accordion title="Docker Compose: Add the following config under `services`">++ ```yaml+ otel-gpu-collector:+ image: ghcr.io/openlit/otel-gpu-collector:latest+ environment:+ GPU_APPLICATION_NAME: 'chatbot'+ GPU_ENVIRONMENT: 'staging'+ OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4318"+ device_requests:+ - driver: nvidia+ count: all+ capabilities: [gpu]+ depends_on:+ - otel-collector+ restart: always+ ```++ </Accordion>+ <Accordion title="Host IP: Use the Host IP to connect to OTel Collector">++ ```sh+ OTEL_EXPORTER_OTLP_ENDPOINT="http://192.168.10.15:4318"+ ```++ </Accordion>+ </AccordionGroup>+ ### Environment Variables++ OTel GPU Collector supports several environment variables for configuration. Below is a table that describes each variable:++ | Environment Variable | Description | Default Value |+ |---------------------------------|---------------------------------------------------------------|-------------------------|+ | `GPU_APPLICATION_NAME` | Name of the application running on the GPU | `default_app` |+ | `GPU_ENVIRONMENT` | Environment name (e.g., staging, production) | `production` |+ | `OTEL_EXPORTER_OTLP_ENDPOINT` | OpenTelemetry OTLP endpoint URL | (required) |+ | `OTEL_EXPORTER_OTLP_HEADERS` | Headers for authenticating with the OTLP endpoint | Ignore if using OpenLIT |</Step></Steps></Accordion>diff --git a/docs/latest/features/metrics.mdx b/docs/latest/features/metrics.mdxindex e1ba1805..212c37df 100644--- a/docs/latest/features/metrics.mdx+++ b/docs/latest/features/metrics.mdx@@ -1,15 +1,15 @@----title: 'LLM Application Metrics'-sidebarTitle: 'Metrics'+title: 'Metrics'+description: "Gather valuable metrics for your AI apps using OpenTelemetry auto instrumentation"----OpenLIT offers automatic instrumentation with OpenTelemetry for various LLM providers, frameworks, and VectorDBs, enabling you to gain valuable insights into the behavior and performance of your LLM applications through metrics.+OpenLIT offers automatic instrumentation with OpenTelemetry for various LLM providers, frameworks, and VectorDBs, enabling you to gain valuable insights into the behavior and performance of your AI applications through metrics.-This documentation covers metrics settings, understanding semantic conventions, and interpreting metric attributes, empowering you to enhance the monitoring and observability of your LLM applications.+This documentation covers metrics settings, understanding semantic conventions, and interpreting metric attributes, empowering you to enhance the monitoring and observability of your AI applications.<CardGroup cols={2}><Card title="Quickstart" href="/latest/quickstart" icon='bolt'>-Get Started with monitoring your LLM Applications in 2 simple steps+Get Started with monitoring your AI Applications in 2 simple steps</Card><Card title="Connections" href="/latest/connections/intro" icon='link'>Connect to your existing Observablity Stack@@ -97,7 +97,7 @@ This section outlines the OpenTelemetry metrics collected by **OpenLIT** from ap<CardGroup cols={2}><Card title="Integrations" href="/latest/integrations/introduction" icon='circle-nodes'>-Integrate your LLM Stack with OpenLIT+Integrate your AI Stack with OpenLIT</Card><Card title="Connections" href="/latest/connections/intro" icon='link'>Connect to your existing Observablity Stackdiff --git a/docs/latest/features/multiple-db.mdx b/docs/latest/features/multiple-db.mdxindex 29810092..6fde427d 100644--- a/docs/latest/features/multiple-db.mdx+++ b/docs/latest/features/multiple-db.mdx@@ -4,7 +4,18 @@ description: 'Connect and switch between multiple ClickHouse databases in OpenLIsidebarTitle: 'Connect Multiple Databases'----In the OpenLIT, you can add and switch between multiple ClickHouse databases. This feature enables users to maintain separate databases for different environments while using the same OpenLIT to analyze LLM Observability data.+OpenLIT allows you to add and switch between multiple ClickHouse databases, which helps manage observability data, prompts, and secrets separately for various environments. This feature enables users to maintain distinct databases tailored to specific environments or tenants, all while using a single OpenLIT deployment for efficient data analysis. The active database you select will power the data displayed across different sections of the application.+## When to Use This Feature++This feature is particularly useful in the following scenarios:++- **Different Environments**: If you're running multiple environments, such as development, testing, and production, you can keep their data separate and easily switch between them as needed.++- **Multi-Tenancy**: When supporting multiple tenants, this feature allows you to maintain isolated datasets for each tenant, ensuring data privacy and organization.++- **Centralized Management**: Manage all your ClickHouse databases from a single OpenLIT deployment, simplifying your AI Engineering data architecture and reducing overhead.++## Get started<Frame><img src="/images/multiple-db.png" />@@ -15,13 +26,14 @@ In the OpenLIT, you can add and switch between multiple ClickHouse databases. Th3. Click on the **Add new** option.4. Enter the ClickHouse database configuration details.5. Click **Save** to confirm.-6. Simply **enable** the desired database to view its data. This selected database will power the dashboards and request page.+6. Simply **enable** the desired database to view its data.+7. The selected active database will power data shown on the dashboards, requests page, Prompt Hub, and Vault secrets.---<CardGroup cols={2}><Card title="Integrations" href="/latest/integrations/introduction" icon='circle-nodes'>-Integrate your LLM Stack with OpenLIT+Integrate your AI Stack with OpenLIT</Card><Card title="Connections" href="/latest/connections/intro" icon='link'>Connect to your existing Observablity Stackdiff --git a/docs/latest/features/openground.mdx b/docs/latest/features/openground.mdxnew file mode 100644index 00000000..b273dd32--- /dev/null+++ b/docs/latest/features/openground.mdx@@ -0,0 +1,60 @@+---+title: 'OpenGround'+description: 'Test and compare different LLMs side-by-side based on performance, cost, and other key metrics'+sidebarTitle: 'OpenGround'+---++OpenGround enables developers to conduct side-by-side tests and comparisons of different Large Language Models (LLMs). By evaluating key factors such as performance and cost, OpenGround assists AI engineers and data scientists in selecting the most suitable LLM for their specific needs. Whether you're optimizing for efficiency, affordability, or other important metrics, OpenGround offers the tools you need to thoroughly assess various models efficiently.++## Features++- **Side-by-Side Comparison**: Simultaneously evaluate multiple LLMs to understand how they perform in real-time across various scenarios.+- **Performance Metrics**: Examine essential performance indicators like response time and token usage to gain deeper insights into each LLM's capabilities.+- **Response Comparison**: Compare the responses generated by different LLMs, assessing the quality, relevance, and appropriateness for specific tasks.+- **Cost Analysis**: Evaluate the cost implications of using different LLMs, helping you balance budget constraints with performance needs.+- **Intuitive Interface**: Use a user-friendly interface that simplifies the process of setting up tests, visualizing results, and making comparisons.+- **Comprehensive Reporting**: Generate detailed reports that compile and visualize comparison data, supporting informed decision-making.++## Get Started++<Steps>+ <Step title="List of existing experiments">+ Get a quick overview of all experiments created.+ 1. Navigate to the **OpenGround** in OpenLIT.+ 2. Explore the previosly created experiments.+ <Frame>+ <img src="/images/docs-openground-1.png" />+ </Frame>+ </Step>+ <Step title="Create a New Experiment">+ Set up new experiments to compare different LLMs side-by-side.+ 1. Click on **+ New** to start a new experiment.+ 2. In the editor, choose your first LLM provider. Configure the LLM by setting its parameters and enter your API key to enable requests.+ 3. Repeat the process for your second LLM provider. Choose the provider and configure its LLM parameters.+ 4. Once both LLMs are set up, enter your prompt and click **Compare Response**.+ 5. Review and analyze the responses from both LLMs to see how they compare.+ <Frame>+ <img src="/images/docs-openground-2.png" />+ <img src="/images/docs-openground-3.png" />+ </Frame>+ <Frame>+ <img src="/images/docs-openground-4.png" />+ <img src="/images/docs-openground-5.png" />+ </Frame>+ </Step>+ <Step title="View Experiment Details" stepNumber=" ">+ Once the experiment is creaetd, You can see information about the experiment along with details.+ <Frame>+ <img src="/images/docs-openground-5.png" />+ </Frame>+ </Step>+</Steps>++<CardGroup cols={2}>+<Card title="Integrations" href="/latest/integrations/introduction" icon='circle-nodes'>+Integrate your AI Stack with OpenLIT+</Card>+<Card title="Connections" href="/latest/connections/intro" icon='link'>+Connect to your existing Observablity Stack+</Card>+</CardGroup>diff --git a/docs/latest/features/overview.mdx b/docs/latest/features/overview.mdxnew file mode 100644index 00000000..d8b74011--- /dev/null+++ b/docs/latest/features/overview.mdx@@ -0,0 +1,43 @@+---+title: 'Overview'+---++<CardGroup cols={3}>++<Card title="Tracing" href="/latest/features/tracing" icon='bars-staggered'>+Automatically track your AI apps with OpenTelemetry to gain insights into performance and behavior.+</Card>++<Card title="Metrics" href="/latest/features/metrics" icon='magnifying-glass-chart'>+Gather valuable metrics for your AI apps using OpenTelemetry auto instrumentation.+</Card>++<Card title="GPU Monitoring" href="/latest/features/gpu" icon='server'>+Monitor NVIDIA GPUs with key metrics like usage, temperature, and power using OpenTelemetry.+</Card>++<Card title="Cost for Custom Models" href="/latest/features/pricing" icon='money-bill'>+Calculate costs for custom and fine-tuned LLMs using your own pricing data.+</Card>++<Card title="Prompt Hub" href="/latest/features/prompt-hub" icon='comments'>+Manage prompts centrally, fetch versions, and use variables for dynamic prompts.+</Card>++<Card title="Vault" href="/latest/features/vault" icon='vault'>+Store and access sensitive information like LLM API keys securely.+</Card>++<Card title="Exceptions Monitoring" href="/latest/features/exceptions" icon='bug'>+Automatically track and view detailed error stack traces in your app.+</Card>++<Card title="OpenGround" href="/latest/features/openground" icon='codepen'>+Compare different LLMs on performance, cost, and more side-by-side.+</Card>++<Card title="Multiple Databases" href="/latest/features/multiple-db" icon='database'>+Switch and manage data across environments and tenants easily.+</Card>++</CardGroup>\ No newline at end of filediff --git a/docs/latest/features/pricing.mdx b/docs/latest/features/pricing.mdxindex 6b097d0f..312177b6 100644--- a/docs/latest/features/pricing.mdx+++ b/docs/latest/features/pricing.mdx@@ -40,7 +40,7 @@ Users can provide their own `pricing_json` variable, which can either be a file<CardGroup cols={2}><Card title="Integrations" href="/latest/integrations/introduction" icon='circle-nodes'>-Integrate your LLM Stack with OpenLIT+Integrate your AI Stack with OpenLIT</Card><Card title="Connections" href="/latest/connections/intro" icon='link'>Connect to your existing Observablity Stackdiff --git a/docs/latest/features/prompt-hub.mdx b/docs/latest/features/prompt-hub.mdxindex f34823f2..391d20bb 100644--- a/docs/latest/features/prompt-hub.mdx+++ b/docs/latest/features/prompt-hub.mdx@@ -1,6 +1,6 @@---title: 'Prompt Hub'-sidebarTitle: 'Prompt Hub'+description: 'Manage prompts centrally, fetch versions, and use variables for dynamic prompts'---The Prompt-Hub allows you to manage prompts, fetch specific versions of the prompt, and compile prompts with variables.@@ -156,3 +156,12 @@ The Prompt-Hub allows you to manage prompts, fetch specific versions of the prom</Steps></Step></Steps>++<CardGroup cols={2}>+<Card title="Integrations" href="/latest/integrations/introduction" icon='circle-nodes'>+Integrate your AI Stack with OpenLIT+</Card>+<Card title="Connections" href="/latest/connections/intro" icon='link'>+Connect to your existing Observablity Stack+</Card>+</CardGroup>\ No newline at end of filediff --git a/docs/latest/features/tracing.mdx b/docs/latest/features/tracing.mdxindex 25f0046d..727d9c2b 100644--- a/docs/latest/features/tracing.mdx+++ b/docs/latest/features/tracing.mdx@@ -1,6 +1,6 @@----title: "LLM Application Traces"-sidebarTitle: "Tracing"+title: "Tracing"+description: "Automatically track your AI apps with OpenTelemetry to gain insights into performance and behavior"---OpenLIT provides OpenTelemetry Auto instrumentation for various LLM providers, frameworks, and VectorDBs, providing you with insights into the behavior and performance of your LLM applications.@@ -9,7 +9,7 @@ This documentation covers tracing settings, understanding semantic conventions,<CardGroup cols={2}><Card title="Quickstart" href="/latest/quickstart" icon="bolt">- Get Started with monitoring your LLM Applications in 2 simple steps+ Get Started with monitoring your AI Applications in 2 simple steps</Card><Card title="Connections" href="/latest/connections/intro" icon="link">Connect to your existing Observablity Stack@@ -153,7 +153,7 @@ This section outlines fundamental definitions related to span types and names us### General Span Attributes-General span attributes encapsulate the environmental and operational context of LLM applications, offering insights into where and how the spans are generated.+General span attributes encapsulate the environmental and operational context of AI applications, offering insights into where and how the spans are generated.| Attribute | Type | Description | Examples || ------------------------- | ------ | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |@@ -272,7 +272,7 @@ This section addresses attributes related to interactions with Vector Databaseshref="/latest/integrations/introduction"icon="circle-nodes">- Integrate your LLM Stack with OpenLIT+ Integrate your AI Stack with OpenLIT</Card><Card title="Connections" href="/latest/connections/intro" icon="link">Connect to your existing Observablity Stackdiff --git a/docs/latest/features/vault.mdx b/docs/latest/features/vault.mdxindex b53fcbb0..8a27d5d0 100644--- a/docs/latest/features/vault.mdx+++ b/docs/latest/features/vault.mdx@@ -1,6 +1,6 @@---title: 'Vault'-sidebarTitle: 'Vault'+description: 'Store and access sensitive information like LLM API keys securely'---Vault is designed to securely store and access sensitive information, such as API keys (e.g., `OPENAI_API_TOKEN`), eliminating the need to directly embed them in your code or environment variables. This centralized approach ensures that all developers can securely share and use the same API key without the necessity of distributing individual keys.@@ -116,3 +116,12 @@ Vault is designed to securely store and access sensitive information, such as AP</Steps></Step></Steps>++<CardGroup cols={2}>+<Card title="Integrations" href="/latest/integrations/introduction" icon='circle-nodes'>+Integrate your AI Stack with OpenLIT+</Card>+<Card title="Connections" href="/latest/connections/intro" icon='link'>+Connect to your existing Observablity Stack+</Card>+</CardGroup>\ No newline at end of filediff --git a/docs/latest/installation.mdx b/docs/latest/installation.mdxindex 2b73b2e4..c7dd8522 100644--- a/docs/latest/installation.mdx+++ b/docs/latest/installation.mdx@@ -14,14 +14,14 @@ OpenLIT flexibly adapts to your preferred environment. Select a deployment metho<path fill="#FFA500" d="M10.204 14.35l.007.01-.999 2.413a5.171 5.171 0 0 1-2.075-2.597l2.578-.437.004.005a.44.44 0 0 1 .484.606zm-.833-2.129a.44.44 0 0 0 .173-.756l.002-.011L7.585 9.7a5.143 5.143 0 0 0-.73 3.255l2.514-.725.002-.009zm1.145-1.98a.44.44 0 0 0 .699-.337l.01-.005.15-2.62a5.144 5.144 0 0 0-3.01 1.442l2.147 1.523.004-.002zm.76 2.75l.723.349.722-.347.18-.78-.5-.623h-.804l-.5.623.179.779zm1.5-3.095a.44.44 0 0 0 .7.336l.008.003 2.134-1.513a5.188 5.188 0 0 0-2.992-1.442l.148 2.615.002.001zm10.876 5.97l-5.773 7.181a1.6 1.6 0 0 1-1.248.594l-9.261.003a1.6 1.6 0 0 1-1.247-.596l-5.776-7.18a1.583 1.583 0 0 1-.307-1.34L2.1 5.573c.108-.47.425-.864.863-1.073L11.305.513a1.606 1.606 0 0 1 1.385 0l8.345 3.985c.438.209.755.604.863 1.073l2.062 8.955c.108.47-.005.963-.308 1.34zm-3.289-2.057c-.042-.01-.103-.026-.145-.034-.174-.033-.315-.025-.479-.038-.35-.037-.638-.067-.895-.148-.105-.04-.18-.165-.216-.216l-.201-.059a6.45 6.45 0 0 0-.105-2.332 6.465 6.465 0 0 0-.936-2.163c.052-.047.15-.133.177-.159.008-.09.001-.183.094-.282.197-.185.444-.338.743-.522.142-.084.273-.137.415-.242.032-.024.076-.062.11-.089.24-.191.295-.52.123-.736-.172-.216-.506-.236-.745-.045-.034.027-.08.062-.111.088-.134.116-.217.23-.33.35-.246.25-.45.458-.673.609-.097.056-.239.037-.303.033l-.19.135a6.545 6.545 0 0 0-4.146-2.003l-.012-.223c-.065-.062-.143-.115-.163-.25-.022-.268.015-.557.057-.905.023-.163.061-.298.068-.475.001-.04-.001-.099-.001-.142 0-.306-.224-.555-.5-.555-.275 0-.499.249-.499.555l.001.014c0 .041-.002.092 0 .128.006.177.044.312.067.475.042.348.078.637.056.906a.545.545 0 0 1-.162.258l-.012.211a6.424 6.424 0 0 0-4.166 2.003 8.373 8.373 0 0 1-.18-.128c-.09.012-.18.04-.297-.029-.223-.15-.427-.358-.673-.608-.113-.12-.195-.234-.329-.349-.03-.026-.077-.062-.111-.088a.594.594 0 0 0-.348-.132.481.481 0 0 0-.398.176c-.172.216-.117.546.123.737l.007.005.104.083c.142.105.272.159.414.242.299.185.546.338.743.522.076.082.09.226.1.288l.16.143a6.462 6.462 0 0 0-1.02 4.506l-.208.06c-.055.072-.133.184-.215.217-.257.081-.546.11-.895.147-.164.014-.305.006-.48.039-.037.007-.09.02-.133.03l-.004.002-.007.002c-.295.071-.484.342-.423.608.061.267.349.429.645.365l.007-.001.01-.003.129-.029c.17-.046.294-.113.448-.172.33-.118.604-.217.87-.256.112-.009.23.069.288.101l.217-.037a6.5 6.5 0 0 0 2.88 3.596l-.09.218c.033.084.069.199.044.282-.097.252-.263.517-.452.813-.091.136-.185.242-.268.399-.02.037-.045.095-.064.134-.128.275-.034.591.213.71.248.12.556-.007.69-.282v-.002c.02-.039.046-.09.062-.127.07-.162.094-.301.144-.458.132-.332.205-.68.387-.897.05-.06.13-.082.215-.105l.113-.205a6.453 6.453 0 0 0 4.609.012l.106.192c.086.028.18.042.256.155.136.232.229.507.342.84.05.156.074.295.145.457.016.037.043.09.062.129.133.276.442.402.69.282.247-.118.341-.435.213-.71-.02-.039-.045-.096-.065-.134-.083-.156-.177-.261-.268-.398-.19-.296-.346-.541-.443-.793-.04-.13.007-.21.038-.294-.018-.022-.059-.144-.083-.202a6.499 6.499 0 0 0 2.88-3.622c.064.01.176.03.213.038.075-.05.144-.114.28-.104.266.039.54.138.87.256.154.06.277.128.448.173.036.01.088.019.13.028l.009.003.007.001c.297.064.584-.098.645-.365.06-.266-.128-.537-.423-.608zM16.4 9.701l-1.95 1.746v.005a.44.44 0 0 0 .173.757l.003.01 2.526.728a5.199 5.199 0 0 0-.108-1.674A5.208 5.208 0 0 0 16.4 9.7zm-4.013 5.325a.437.437 0 0 0-.404-.232.44.44 0 0 0-.372.233h-.002l-1.268 2.292a5.164 5.164 0 0 0 3.326.003l-1.27-2.296h-.01zm1.888-1.293a.44.44 0 0 0-.27.036.44.44 0 0 0-.214.572l-.003.004 1.01 2.438a5.15 5.15 0 0 0 2.081-2.615l-2.6-.44-.004.005z"/></svg>}href="/latest/installation#kubernetes">- Deploy OpenLIT on a Kubernetes for scalable LLM Monitoring capabilities.+ Deploy OpenLIT on a Kubernetes for scalable AI Engineering capabilities.</Card><Cardtitle="Docker"icon="docker"href="/latest/installation#docker">- Deploy OpenLIT using Docker for an easy-to-manage LLM Monitoring setup.+ Deploy OpenLIT using Docker for an easy-to-manage AI Engineering setup.</Card></CardGroup>@@ -63,7 +63,7 @@ For a quick and straightforward setup, Docker Compose can be used to deploy Open<CardGroup cols={2}><Card title="Integrations" href="/latest/integrations/introduction" icon='circle-nodes'>-Integrate your LLM Stack with OpenLIT+Integrate your AI Stack with OpenLIT</Card><Card title="Connections" href="/latest/connections/intro" icon='link'>Connect to your existing Observablity Stackdiff --git a/docs/latest/introduction.mdx b/docs/latest/introduction.mdxindex ff0befa7..725a5600 100644--- a/docs/latest/introduction.mdx+++ b/docs/latest/introduction.mdx@@ -1,38 +1,48 @@---title: What is OpenLIT?-description: 'OpenTelemetry-native Observability and Evaluations for LLMs and GPUs'+description: 'Open Source Platform for AI Engineering'---<Frame>- <img src="/images/docs-main-banner.jpeg" />+ <img src="https://github.com/openlit/.github/raw/main/profile/assets/openlit-main-banner.png?raw=true" /></Frame>-**OpenLIT** is an **OpenTelemetry-native** tool designed to help developers gain insights into the performance of their LLM applications in production. It automatically collects LLM input and output metadata, and monitors GPU performance for self-hosted LLMs.-OpenLIT makes integrating observability into GenAI projects effortless with just **a single line of code**. Whether you're working with popular LLM providers such as OpenAI and HuggingFace, or leveraging vector databases like ChromaDB, OpenLIT ensures your applications are monitored seamlessly, providing critical insights including GPU performance stats for self-hosted LLMs to improve performance and reliability.+**OpenLIT** allows you to simplify your AI development workflow, especially for Generative AI and LLMs. It streamlines essential tasks like experimenting with LLMs, organizing and versioning prompts, and securely handling API keys.+With just one line of code, you can enable **OpenTelemetry-native** observability, offering full-stack monitoring that includes LLMs, vector databases, and GPUs. This enables developers to confidently build AI features and applications, transitioning smoothly from testing to production.-This project proudly follows the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) of the OpenTelemetry community, consistently updating to align with the latest standards in observability.+This project proudly follows and maintains the [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) with the OpenTelemetry community, consistently updating to align with the latest standards in Observability.+## โก Features-## What is LIT?-`LIT` stands for **Learning and Inference Tool**, which is a visual and interactive tool designed for understanding AI models and visualizing data.-The term `LIT` was introduced by [Google](https://developers.google.com/machine-learning/glossary#learning-interpretability-tool-lit).+- ๐ **Analytics Dashboard**: Monitor your AI application's health and performance with detailed dashboards that track metrics, costs, and user interactions, providing a clear view of overall efficiency.-## โก Features-+- ๐ **OpenTelemetry-native Observability SDKs**: Vendor-neutral SDKs to send traces and metrics to your existing observability tools.++- ๐ฒ **Cost Tracking for Custom and Fine-Tuned Models**: Tailor cost estimations for specific models using custom pricing files for precise budgeting.++- ๐ **Exceptions Monitoring Dashboard**: Quickly spot and resolve issues by tracking common exceptions and errors with a dedicated monitoring dashboard.++- ๐ญ **Prompt Management**: Manage and version prompts using Prompt Hub for consistent and easy access across applications.-- **Advanced Monitoring of LLM and VectorDB Performance**: OpenLIT offers automatic instrumentation that generates traces and metrics, providing insights into the performance and costs of your LLM and VectorDB usage. This helps you analyze how your applications perform in different environments, such as production, enabling you to optimize resource ussage and scale efficiently.-- **Cost Tracking for Custom and Fine-Tuned Models**: OpenLIT enables you to tailor cost tracking for specific models by using a custom JSON file. This feature allows for precise budgeting and ensures that cost estimations are perfectly aligned with your project needs.-- **OpenTelemetry-native & vendor-neutral SDKs**: OpenLIT is built with native support for OpenTelemetry, making it blend seamlessly with your projects. This vendor-neutral approach reduces barriers to integration, making OpenLIT an intuitive part of your software stack rather than an additional complexity.+- ๐ **API Keys and Secrets Management**: Securely handle your API keys and secrets centrally, avoiding insecure practices.++- ๐ฎ **Experiemnt with different LLMs**: Use OpenGround to explore, test and compare various LLMs side by side.++<CardGroup cols={1}>+<Card title="Take a Look at All Features in Detail" href="/latest/features/overview" icon='bolt'>+Explore powerful features that help you take your AI features from development to production with confidence.+</Card>+</CardGroup>## Getting StartedSelect from the following guides to learn more about how to use OpenLIT:<CardGroup cols={2}>-<Card title="Quickstart" href="/latest/quickstart" icon='bolt'>+<Card title="Quickstart: LLM Observability" href="/latest/quickstart" icon='bolt'>Get Started with monitoring your LLM Applications in 2 simple steps</Card><Card title="Integrations" href="/latest/integrations/introduction" icon='circle-nodes'>-Integrate your LLM Stack with OpenLIT+Integrate your AI Stack with OpenLIT</Card><Card title="Installation" href="/latest/installation" icon='circle-down'>Deploy OpenLIT in your preferred environmentdiff --git a/docs/latest/quickstart.mdx b/docs/latest/quickstart.mdxindex 091cbebc..ed583288 100644--- a/docs/latest/quickstart.mdx+++ b/docs/latest/quickstart.mdx@@ -1,11 +1,12 @@----title: "Quickstart"+title: "Get started with LLM Observability"+sidebarTitle: "Quickstart: Observability"description: "Quickly start monitoring your LLM Applications in just a single line of code"icon: "bolt"---This guide will walk you through setting up OpenTelemetry Auto Instrumentation for monitoring your LLM Application using OpenLIT. In just a few steps, you'll be able to track and analyze the performance and usage of your LLM Applications.-In this guide, we'll show how you can send OpenTelemetry traces and metrics from your LLM Applications to Grafana Cloud+In this guide, we'll show how you can send OpenTelemetry traces and metrics from your LLM Applications to OpenLIT.```mermaidflowchart TB;@@ -265,7 +266,7 @@ flowchart TB;</Steps>-You're all set! Following these steps should have you on your way to effectively monitoring your LLM applications with OpenTelemetry. If you wish to send telemetry to any other backend, Refer to our [Connections](./connections/intro).+You're all set! Following these steps should have you on your way to effectively monitoring your LLM applications with OpenTelemetry. If you wish to send telemetry to any other backend, Refer to our [Connections](./connections/intro) guide.If you have any questions or need support, reach out to our [community](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ).@@ -277,7 +278,7 @@ If you have any questions or need support, reach out to our [community](https://href="/latest/integrations/introduction"icon="circle-nodes">- Integrate your LLM Stack with OpenLIT+ Integrate your AI Stack with OpenLIT</Card><Card title="Connections" href="/latest/connections/intro" icon="link">Connect to your existing Observablity Stackdiff --git a/docs/latest/sdk-configuration.mdx b/docs/latest/sdk-configuration.mdxindex 4bcab3a8..29c6c5bd 100644--- a/docs/latest/sdk-configuration.mdx+++ b/docs/latest/sdk-configuration.mdx@@ -4,37 +4,104 @@ description: 'Configuring Options for OpenLIT SDK'icon: 'file-code'---+<Tabs>+<Tab title="Python">++## Observability - `openlit.init()`+Below is a detailed overview of the configuration options available, allowing you to adjust OpenLIT's behavior and functionality to align with your specific observability needs:-<Tabs>- <Tab title="Python">- | Argument | Description | Default Value |-|-------------------------|-----------------------------------------------------------------------------------------------|----------------|-| `environment` | The deployment environment of the application. | `"default"` |-| `application_name` | Identifies the name of your application. | `"default"` |-| `tracer` | An instance of OpenTelemetry Tracer for tracing operations. | `None` |-| `meter` | An OpenTelemetry Metrics instance for capturing metrics. | `None` |-| `otlp_endpoint` | Specifies the OTLP endpoint for transmitting telemetry data. | `None` |-| `otlp_headers` | Defines headers for the OTLP exporter, useful for backends requiring authentication. | `None` |-| `disable_batch` | A flag to disable batch span processing, favoring immediate dispatch. | `False` |-| `trace_content` | Enables tracing of content for deeper insights. | `True` |-| `disabled_instrumentors`| List of instrumentors to disable. | `None` |-| `disable_metrics` | If set, disables the collection of metrics. | `False` |-| `pricing_json` | URL or file path of the pricing JSON file. | `https://github.com/openlit/openlit/blob/main/assets/pricing.json` |-| `collect_gpu_stats` | Flag to enable or disable GPU metrics collection. | `False` |- </Tab>- <Tab title="Typescript">-| Argument | Description | Default Value |-|-------------------------|-------------------------------|----------------|-| `environment` | The deployment environment of the application. | `"default"` |-| `application_name` | Identifies the name of your application. | `"default"` |-| `tracer` | An instance of OpenTelemetry Tracer for tracing operations. | `undefined` |-| `otlpEndpoint` | Specifies the OTLP endpoint for transmitting telemetry data. | `undefined` |-| `otlp_headers` | Defines headers for the OTLP exporter, useful for backends requiring authentication. | {} |-| `disable_batch` | A flag to disable batch span processing, favoring immediate dispatch. | `false` |-| `trace_content` | Enables tracing of content for deeper insights. | `true` |-| `disabled_instrumentors`| List of instrumentors to disable. | `undefined` |-| `pricing_json` | URL or file path of the pricing JSON file. | `https://github.com/openlit/openlit/blob/main/assets/pricing.json` |- </Tab>- </Tabs>++| Argument | Description | Default Value | Required |+|-------------------------|-----------------------------------------------------------------------------------------------|----------------|----------|+| `environment` | The deployment environment of the application. | `"default"` | Yes |+| `application_name` | Identifies the name of your application. | `"default"` | Yes |+| `tracer` | An instance of OpenTelemetry Tracer for tracing operations. | `None` | No |+| `meter` | An OpenTelemetry Metrics instance for capturing metrics. | `None` | No |+| `otlp_endpoint` | Specifies the OTLP endpoint for transmitting telemetry data. | `None` | No |+| `otlp_headers` | Defines headers for the OTLP exporter, useful for backends requiring authentication. | `None` | No |+| `disable_batch` | A flag to disable batch span processing, favoring immediate dispatch. | `False` | No |+| `trace_content` | Enables tracing of content for deeper insights. | `True` | No |+| `disabled_instrumentors`| List of instrumentors to disable. | `None` | No |+| `disable_metrics` | If set, disables the collection of metrics. | `False` | No |+| `pricing_json` | URL or file path of the pricing JSON file. | `https://github.com/openlit/openlit/blob/main/assets/pricing.json` | No |+| `collect_gpu_stats` | Flag to enable or disable GPU metrics collection. | `False` | No |++## Prompt Hub - `openlit.get_prompt()`++Below are the parameters for use with the SDK for OpenLIT Prompt Hub for prompt management:++| Parameter | Description |+|-------------------|------------------------------------------------------------------------------------------------------------------------------------|+| `url` | Sets the OpenLIT URL. Defaults to the `OPENLIT_URL` environment variable. |+| `api_key` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |+| `name` | Sets the name to fetch a unique prompt. Use this or `prompt_id`. |+| `prompt_id` | Sets the ID to fetch a unique prompt. Use this or `name`. Optional |+| `version` | Set to `True` to get the prompt with variable substitution.. Optional |+| `shouldCompile` | Boolean value that compiles the prompt using the provided variables. Optional |+| `variables` | Sets the variables for prompt compilation. Optional |+| `meta_properties` | Sets the meta-properties for storing in the prompt's access history metadata. Optional |++## Vault - `openlit.get_secrets()`++Below are the parameters for use with the SDK for OpenLIT Vault for secret management:++| Parameter | Description |+|-------------------|------------------------------------------------------------------------------------------------------------------------------------|+| `url` | Sets the Openlit URL. Defaults to the `OPENLIT_URL` environment variable. |+| `api_key` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |+| `key` | Sets the key to fetch a specific secret. Optional |+| `should_set_env` | Boolean value that sets all the secrets as environment variables for the application. Optional |+| `tags` | Sets the tags for fetching only the secrets that have the mentioned tags assigned. Optional |++</Tab>+<Tab title="Typescript">++## Observability - `Openlit.init()`++Below is a detailed overview of the configuration options available, allowing you to adjust OpenLIT's behavior and functionality to align with your specific observability needs:++| Argument | Description | Default Value | Required |+| ------------------------ | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------ | -------- |+| `environment` | The deployment environment of the application. | `"default"` | No |+| `applicationName` | Identifies the name of your application. | `"default"` | No |+| `tracer` | An instance of OpenTelemetry Tracer for tracing operations. | `undefined` | No |+| `otlpEndpoint` | Specifies the OTLP endpoint for transmitting telemetry data. | `undefined` | No |+| `otlpHeaders` | Defines headers for the OTLP exporter, useful for backends requiring authentication. | `undefined` | No |+| `disableBatch` | A flag to disable batch span processing, favoring immediate dispatch. | `true` | No |+| `traceContent` | Enables tracing of content for deeper insights. | `true` | No |+| `disabledInstrumentations` | List of instrumentations to disable. | `undefined` | No |+| `instrumentations` | Object of instrumentation modules for manual patching | `undefined` | No |+| `pricing_json` | URL or file path of the pricing JSON file. | `https://github.com/openlit/openlit/blob/main/assets/pricing.json` | No |++## Prompt Hub - `Openlit.getPrompt()`++Below are the parameters for use with the SDK for OpenLIT Prompt Hub for prompt management:++| Parameter | Description |+|------------------|------------------------------------------------------------------------------------------------------------------------------------|+| `url` | Sets the OpenLIT URL. Defaults to the `OPENLIT_URL` environment variable or `http://127.0.0.1:3000` if not set. |+| `apiKey` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |+| `name` | Sets the name to fetch a unique prompt. Use this or `promptId`. |+| `promptId` | Sets the ID to fetch a unique prompt. Use this or `name`. Optional |+| `version` | Sets the version to retrieve a specific prompt. Optional |+| `shouldCompile` | Boolean value that compiles the prompt using the provided variables. Optional |+| `variables` | Sets the variables for prompt compilation. Optional |+| `metaProperties` | Sets the meta-properties for storing in the prompt's access history metadata. |+++## Vault - `Openlit.getSecrets()`++Below are the parameters for use with the SDK for OpenLIT Vault for secret management:++| Parameter | Description |+|------------------|------------------------------------------------------------------------------------------------------------------------------------|+| `url` | Sets the Openlit URL. Defaults to the `OPENLIT_URL` environment variable or `http://127.0.0.1:3000` if not set. |+| `apiKey` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |+| `key` | Sets the key to fetch a specific secret. Optional |+| `tags` | Sets the tags for fetching only the secrets that have the mentioned tags assigned. Optional |+| `shouldSetEnv` | Boolean value that sets all the secrets as environment variables for the application. Optional |++</Tab>+</Tabs>diff --git a/docs/mint.json b/docs/mint.jsonindex 47ae36b7..923cf5f6 100644--- a/docs/mint.json+++ b/docs/mint.json@@ -59,12 +59,15 @@{"group": "Features","pages": [+ "latest/features/overview","latest/features/tracing","latest/features/metrics","latest/features/gpu","latest/features/pricing","latest/features/prompt-hub","latest/features/vault",+ "latest/features/exceptions",+ "latest/features/openground","latest/features/multiple-db"],"version": "latest"diff --git a/sdk/typescript/README.md b/sdk/typescript/README.mdindex e16f5f85..d286d3ad 100644--- a/sdk/typescript/README.md+++ b/sdk/typescript/README.md@@ -166,7 +166,7 @@ Below is a detailed overview of the configuration options available, allowing yo| `instrumentations` | Object of instrumentation modules for manual patching | `undefined` | No || `pricing_json` | URL or file path of the pricing JSON file. | `https://github.com/openlit/openlit/blob/main/assets/pricing.json` | No |-### OpenLIT Prompt Hub - `openlit.get_prompt()`+### OpenLIT Prompt Hub - `Openlit.getPrompt()`Below are the parameters for use with the SDK for OpenLIT Prompt Hub for prompt management:@@ -182,7 +182,7 @@ Below are the parameters for use with the SDK for OpenLIT Prompt Hub for prompt| `metaProperties` | Sets the meta-properties for storing in the prompt's access history metadata. |-### OpenLIT Vault - `openlit.get_secrets()`+### OpenLIT Vault - `Openlit.getSecrets()`Below are the parameters for use with the SDK for OpenLIT Vault for secret management: